Automating Your Testing Pipeline with GitHub Actions

Created by: Amuthan Sakthivel

Last updated: May 5, 2022

Introduction

Test Automation adds tremendous value when integrated with CI/CD tools. Testing is a major pain point for many engineers, and as a result, numerous tools have arisen to make the process as painless as possible. Our company has always used Jenkins, one of the most popular automation frameworks out there.

The approach of using a dedicated testing platform is not without its trade-offs. It can be a hassle for the DevOps team who manages the platform to coordinate with the test engineers who use it, and testing pipelines are often under separate source control from the code they test (or in some cases, none at all). It is also inconvenient to maintain and pay for separate testing infrastructure in the form of Jenkins nodes.

Recently, we’ve moved to a new approach using GitHub Actions. On that platform, it is easy for our engineering teams to maintain their testing pipelines and source code in the same location with the same versioning, and to update those pipelines independently of the DevOps team when necessary. In this guide, I will share some of the most interesting ways we’ve been able to use GitHub Actions to automate our testing pipelines.

Prerequisites

Basic working knowledge of Github Actions and test automation

Topics

Use-case: Code quality with SonarQube
Use-case: Running automated tests within your framework
Use-case: Automated web testing with Selenoid
Use-case: Automated Android testing
Use-case: Automated iOS testing
Feature: Scheduling your automated tests
Feature: Publishing reports in GitHub Pages
Feature: Accepting input in your workflows

Code Quality with SonarQube

SonarQube is a popular tool used to catch possible vulnerabilities and other forms of technical debt, leading to more maintainable code. To get started:

Sign up for an account on SonarCloud using your GitHub account.
Create an organization - this will prompt you to authorize the application on GitHub
Select “Analyze new project” and pick a project - note that free accounts can only analyze the contents of public repositories.
Go to the Security tab under "My Account", enter a token name, and click “Generate”. Save this value to the clipboard for the next step.
Add the token to GitHub Secrets by going to Project settings → Secrets → Actions → New Repository Secret. In this example, we’ve name the secret SONAR_TOKEN
Under Actions tab in your project, select “New workflow”:

7. Select “set up a workflow yourself”:

8. Copy and paste the workflow below, replace the values of organization and projectKey with your own on the last line of YAML file.

name: sonarqube
on:
  push:
    branches:
      - main
  pull_request:
    types: [opened, synchronize, reopened]
  workflow_dispatch:

jobs:
  build:
    name: sonarqube
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0  # Shallow clones should be disabled for a better relevancy of analysis
      - name: Set up JDK 11
        uses: actions/setup-java@v1
        with:
          java-version: 11
      - name: Build and analyze
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}  # Needed to get PR information, if any
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
          SONAR_HOST_URL: https://sonarcloud.io
        run: mvn -B verify org.sonarsource.scanner.maven:sonar-maven-plugin:sonar -Dsonar.organization=YOUR_SONAR_ORG -Dsonar.projectKey=YOUR_PROKECT_KEY -DskipTests

9. This workflow will be triggered on every successful push to main and when certain actions are taken within PRs, and can also be manually triggered from the GitHub Actions tab:

Realistically, the pull_request types are most likely to fit your use-case - I’ve included the other triggers to showcase the flexibility of these workflows.

Running Automated Tests Within your Framework

Running unit tests on every pull request is an essential safeguard against unintentionally breaking your application. There are many reasons why this is the case, which I hope to cover in a future blog post.

For now, let’s dive into how we can accomplish this:

Write your unit tests.
Ensure it’s possible to run them from the commandline. In this example, I’ve configured them to run as a maven command using the surefire plugin, but this could just as easily be npm test on a Node.js project.
Setup a new workflow as in the previous example using the YAML file below:

name: unit-test
on:
  push:
    branches:
      - main
  pull_request:
    types: [opened, synchronize, reopened]
  workflow_dispatch:

jobs:
  build:
    name: automated-tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0  # Shallow clones should be disabled for a better relevancy of analysis
      - name: Set up JDK 11
        uses: actions/setup-java@v1
        with:
          java-version: 11
      - name: Run unit test
        run: mvn clean test -Punit-test

4. Like before, this unit test is triggered on every push to main or when a PR is opened, synchronized, or reopened. In case of failure, the automated tests simply need to be fixed and pushed, and the workflow will run again.

Automated Web Testing with Selenoid

Web testing is among the costliest to perform, both in terms of time and money. It’s typically necessary to connect to a dedicated server or spin up a container, both of which require significant effort to set up and cost money.

Selenoid is an excellent testing tool that serves as a complete replacement for the selenium grid. We can run a selenoid from within the GitHub runner and can use it to delegate our web tests and execute them.

Delegate all tests to http://localhost:4444/wd/hub

Sample Test Case:

 @Test
public void testGoogleSearchUsingSelenoid() {
        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability("browserName", "chrome");
        capabilities.setCapability("enableVNC", false);
        capabilities.setCapability("enableVideo", false);
        WebDriver driver = new RemoteWebDriver(new URL("http://localhost:4444/wd/hub"),capabilities);
        driver.get("https://google.co.in");
        Assert.assertEquals(driver.getTitle(), "Google");
}

Configure your web test as a maven command:

mvn clean test -Pweb

2. Create a new workflow using the YAML below:

name: Run web tests in Github runner

on:
  push:
    branches: [ main ]
  workflow_dispatch:
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Start Selenoid server
        uses: n-ton4/selenoid-github-action@master
        id: start-selenoid
        continue-on-error: false
        with:
            version: 1.10.1
            args: -limit 10
            browsers: chrome
            last-versions: 1

      - name: checkout
        uses: actions/checkout@v2

      - name: Run the tests
        run:
            mvn clean test -Pweb

This workflow uses the latest version of chrome and can rut up to 10 tests in parallel. More details can be found here.

Your runs might look something like this:

Automated Android Testing

If you have worked on mobile automation, you’re surely familiar with the pain of running your tests in emulators. These emulators are often quite expensive and can be frustrating to configure.

Fortunately, GitHub Actions runners are perfectly capable of running these tests, and the results will be identical to testing with emulators or physical hardware.

Yes, that’s right! You don’t have to spend to run mobile tests.

Sample Test Case :

 @Test
 public void testAppLaunchAndroid() {
        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability(MobileCapabilityType.PLATFORM_NAME, Platform.ANDROID);
        capabilities.setCapability(MobileCapabilityType.AUTOMATION_NAME, "UIAutomator2");
        capabilities.setCapability("uiautomator2ServerInstallTimeout", 60000);
        capabilities.setCapability(MobileCapabilityType.APP,System.getProperty("user.dir")+"/ApiDemos-debug.apk");
        AndroidDriver<AndroidElement> driver = new AndroidDriver<>(new URL("http://127.0.0.1:4723/wd/hub"),
                capabilities);
        driver.findElementByAccessibilityId("Animation").click();
        //Add your assertions
    }

Configure a maven command to run your Android tests:

mvn clean test -Pandroid

2. Create a workflow based on the YAML below:

name: Run android tests in github runner

on:
  push:
    branches: [ master ]

  workflow_dispatch:

jobs:
  test:
    runs-on: macos-latest
    strategy:
      matrix:
        api-level: [25]
    steps:
      - name: checkout
        uses: actions/checkout@v2

      - name: Set up JDK 1.11
        uses: actions/setup-java@v1
        with:
          java-version: 11

      - uses: actions/setup-node@v2
        with:
          node-version: '12'
      - run: |
            npm install -g appium@v1.22
            appium -v
            appium &>/dev/null &

      - name: AVD cache
        uses: actions/cache@v2
        id: avd-cache
        with:
          path: |
            ~/.android/avd/*
            ~/.android/adb*
          key: avd-${{ matrix.api-level }}

      - name: create AVD and generate snapshot for caching
        if: steps.avd-cache.outputs.cache-hit != 'true'
        uses: reactivecircus/android-emulator-runner@v2
        with:
          api-level: ${{ matrix.api-level }}
          force-avd-creation: false
          emulator-options: -no-window -gpu swiftshader_indirect -noaudio -no-boot-anim -camera-back none
          disable-animations: false
          script: echo "Generated AVD snapshot for caching."

      - name: run android tests
        uses: reactivecircus/android-emulator-runner@v2
        with:
          api-level: ${{ matrix.api-level }}
          force-avd-creation: false
          emulator-options: -no-snapshot-save -no-window -gpu swiftshader_indirect -noaudio -no-boot-anim -camera-back none
          disable-animations: true
          script: mvn clean test -Pandroid

These tests are run in the latest version of macOS and an API level of 25. Multiple levels can be tested simultaneously by specifying an array, e.g. [25, 27]. We also leverage an AVD cache here to reduce the startup time of the Android emulator

More information about how the emulator action works can be found Emulator.

Your runs might look something like this:

Automated iOS Testing

Automated testing on iOS can be challenging not only due to the complexity of writing Appium tests, but also because they must be run on a Mac. As a result, most iOS automated tests require a cloud server. However, we can run these tests in the iOS simulator provided by GitHub Actions without provisioning any additional architecture.

Sample Test Case :

 @Test
 public void testAppLaunchIOS() {
        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability(MobileCapabilityType.PLATFORM_NAME, Platform.IOS);
        capabilities.setCapability(MobileCapabilityType.AUTOMATION_NAME, "XCUITest");
        capabilities.setCapability("isHeadless",true);
        capabilities.setCapability(MobileCapabilityType.DEVICE_NAME, "iPod touch (7th generation)");
        capabilities.setCapability(MobileCapabilityType.APP,
                System.getProperty("user.dir")+"/DailyCheck.zip");
        AndroidDriver<AndroidElement> driver = new AndroidDriver<>
                (new URL("http://127.0.0.1:4723/wd/hub"),capabilities);
    }

Configure your iOS tests to be run as a maven command:

mvn clean test -Pios

2. Create a workflow based on the YAML below:

name: Run appium iOS test in Github Runner

on:
  push:
    branches: [ master ]
  workflow_dispatch:
jobs:
  build:
    runs-on: macos-latest
    steps:
      - uses: actions/checkout@v2

      - name: Set up JDK 1.11
        uses: actions/setup-java@v1
        with:
          java-version: 11

      - uses: actions/setup-node@v2
        with:
          node-version: '12'
      - run: |
          npm install -g appium@v1.22
          appium -v
          appium &>/dev/null &
          mvn clean test -Pios

You will notice that this workflow requires significantly less configuration than the Android tests!

Scheduling Your Automated Tests

It’s always good practice to run automated tests when code changes, but it can also be useful to run them on a schedule to ensure consistency. The workflow provides an example of how to schedule a job to run at 10PM every day using standard cron syntax:

name: Schedule web tests in Github runner

on:
  schedule:
    - cron: '0 22 * * *'
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Start Selenoid server
        uses: n-ton4/selenoid-github-action@master
        id: start-selenoid
        continue-on-error: false
        with:
            version: 1.10.1
            args: -limit 10
            browsers: chrome
            last-versions: 1

      - name: checkout
        uses: actions/checkout@v2

      - name: Run the tests
        run:
            mvn clean test -Pweb

This workflow will be triggered every day at 10 PM and execute (in this example) web tests.

Publishing Reports in Github Pages

One of the chief concerns when automating test suites is how to retrieve the reports. While it is possible to send alerts via email or Slack, when tests are run often these notifications can become overwhelming, and it’s easy to ignore them if they just become “noise”. An alternative approach would be to expose your test results on a static website, making it easy to see the latest results in an intelligible format at your convenience. We can leverage an action called actions-gh-pages to publish test results to a static site on GitHub Pages:

Create a new branch called gh-pages in your repository:

2. Navigate to the project settings in Github.

3. Navigate to Project Settings → Code and automation → Pages, and select gh-pages as the branch:

5. Create a new workflow such as in the example below:

name: Publish report in Github Pages

on:
  push:
    branches: [ master ]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Start Selenoid server
        uses: n-ton4/selenoid-github-action@master
        id: start-selenoid
        continue-on-error: false
        with:
          version: 1.10.1
          args: -limit 10
          browsers: chrome
          last-versions: 1

      - name: checkout project
        uses: actions/checkout@v2

      - name: execute tests
        run: mvn clean test -Pweb

      - name: Deploy report to Github Pages
        if: always()
        uses: peaceiris/actions-gh-pages@v2
        env:
          PERSONAL_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          PUBLISH_BRANCH: gh-pages
          PUBLISH_DIR: ./test-output

In this example, I created a static index.html file under the test-output directory, hence why that is my PUBLISH_DIR.

6. When this workflow runs, it will also run a second workflow called “pages build and deployment”:

7. To see the resulting automation report, simply click on the link from the results page:

Accepting Input in your Workflows

Sometimes it’s useful to be able to pass certain inputs to our automatic suite, such as a grid URL, username, password, or maven profile. The example workflow below takes two inputs:

Maven Profile
The URL of a remote server running chrome (this could point to any cloud infra service)

name: Provide inputs to workflow

on:
  workflow_dispatch:
    inputs:
      mavenProfile:
        description: 'web or android or ios or unit-test'
        required: true
        default: 'web'
      remoteURL:
        description: 'selenoid url if hosted outside'
        required: true
        default: 'http://localhost:4444/wd/hub'

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Start Selenoid server
        uses: n-ton4/selenoid-github-action@master
        id: start-selenoid
        continue-on-error: false
        with:
            version: 1.10.1
            args: -limit 10
            browsers: chrome
            last-versions: 1

      - name: checkout
        uses: actions/checkout@v2

      - name: Run the tests
        run:
            mvn clean test -P${{ github.event.inputs.mavenProfile }} -DremoteURL=${{ github.event.inputs.remoteURL }}

Note that these input values can be referenced anywhere with this syntax: ${{ github.event.inputs.INPUT_NAME }}.

To execute this workflow:

Navigate to the Github Actions tab.
Supply the required input - fields with required: true must be provided by the user unless a default is specified.
Trigger the workflow by selecting “Run Workflow”.

That’s a Wrap!

Though it is impossible to cover all the use cases, I hope you’ve learned a bit about how to leverage GitHub Actions to automate your testing pipeline. As we’ve seen, it can be used to perform any type of testing. And it’s easy to run those tests on a trigger or a schedule, with our without input, and even publish the results with minimal configuration. If you’d like to dig deeper into these examples, here's a link to the project. Until next time!