knowledge-kitchen

Continuous Integration - Automated Software Testing

Automatically run unit tests.

1. Overview
2. Typical Workflow
3. Best Practices
4. Software as a Service
5. Circle CI
6. GitHub Actions
7. Conclusions

Overview

Concept

As developers work on a project, they make changes.

• Every time a change is made, there is a risk of bugs being introduced.

• In order to discover bugs, the software must be periodically built and tested.

• Continous Integration (CI) is the automated process of building and testing software everytime a component of the system changes.

Benefits

Automated Continuous Integration has several benefits:

• uncover bugs more quickly through frequent builds, tests, and notifications

• improve quality

• reduce time to validate builds

• reduce human labor costs

Typical Workflow

Overview

Version Control

When a developer makes changes to a system, they typically check those changes into a version control system.

• A continuous integration server will be notified whenever a new change has been made.

• This notification can arrive either by continually polling the version control logs looking for changes, or by receiving an API call (i.e. a hook) from the version control system when a change is made.

Build System

Once a change to the version control repository has been detected, the Continous Integration server will automatically trigger a new build to begin in order to be able to test the system once the build is done.

• Build Systems typically remove any artefacts from previous builds, and assemble the system from scratch.

• Builds can occur on a developer’s local machine, on a build server, or even on the target production server.

• In a Continuous Integration pipeline, once the build is done, the testing framework begins running automated tests.

Testing Framework

Once a fresh build is complete, the testing framework will run automated tests on the system.

• Unit Tests, Integration Tests, System Tests, and any other tests that support automation can all be executed.

• Once testing is complete, developers are notified of the success or failure of all tests.

Notifications

Developers are notified, immediately upon completion of the build and test phases, whether each of the steps succeeded or failed.

With most Continuous Integration tools, notifications can be sent via

• email

• SMS

• Slack, Discord, or MS Teams messages

• notification apps

• calls to any arbitrary API endpoint for custom notifications or follow-up automated steps

Badges

In popular project management tools such as GitHub, badges automatically posted to a repository’s README.md can indicate the last known success or failure of the build and tests for all to see.

Best Practices

Commit small changes

Committing small changes to version control limits the scope of what has changed so failures in building or testing are more quickly identified and remedied.

You break it, you fix it!

If a given developer has broken the build, it is their responsibility to fix it.

• Developers are often expected to be always on call to fix broken builds…

• … or whatever else a manager thinks is important.

Test the right stuff

It’s common to miss obvious bugs because automated unit and integration tests have poor code coverage or test the wrong things.

• This is a problem with all unit testing and is not unique to Continuous Integration automation of those tests.

What to produce

Plan on producing a status, a log, and an artefact.

• the status is the result, whether the build succeeded or failed

• the log contains a list of the steps that were completed in the build, and the results of all tests

• the artefact can be anything that is intended to be produced by the CI server, such as the built software in a bundled and ready-to-deply form, or a generated report on the code coverage of the unit tests, or an auto-generated web page documenting the system, etc.

Software as a Service

Continuous Deployment

An extension of Continuous Integration is Continuous Deployment (CD)…

• Assuming Continuous Integration build/test passes, the CI/CD automation server takes the additional step of…

• Deploying (cloning, installing and running) the built system on a production server - the server where customers can immediately access the latest version.

CI/CD Automation pipelines

Continuous Integration (CI), followed by Continuous Deployment (CD) goes hand-in-hand with Software as a Service (SaaS).

• SaaS systems are deployed not to the customers’ local machines, but to the software distributor’s own servers.

• Doing so allows the software distributor to maintain control and ownership of software at all times while granting access to the software to customers / end-users, rather than actually deploying software on customers’ own machines.

• Since changes to the source code immediately trigger new builds and deployment of the latest code, customers will always have immediate access to the latest stable version of the software.

Circle CI

Hosted continuous integration service

Circle CI is a proprietary hosted Continuous Integration web service with a free tier of service that is suitable for small projects.

Virtual build servers

When a change is detected in version control, Circle CI boots up a new virtual server.

• Circle CI clones the version control repository into the virtual machine, and performs the build and test.

Key features

Circle CI comes with the common features of any continuous integration server.

• hosted continuous integration solution

• free tier

• at its simplest, requires the addition of a configuration file to a repository

• supports building and testing of many programming languages

• integrates with other popular DevOps tools and practices

Basic configuration

Circle CI’s build and test configuration is written in YAML (YAML Ain’t Markup Language) and saved into a file named config.yml in a .circleci directory of your project.

• See an example config for a React.js/Express.js-based app

GitHub integration

To link Circle CI to a GitHub repository, such that it detects changes to the repository and triggers continuous integration builds and testing, go to circleci.com.

• Follow the instructions there to link to a GitHub repository. You will be asked to authorize the app on GitHub, and then can modify settings within the travis web app. The setup is relatively straightforward.

• As a best practice, have Circle CI perform automated builds on pull requests. This allows the pull request reviewer to be sure the code builds and passes automated tests prior to merging the code.

Highly configurable

Circle CI supports detailed instructions and can run arbitrary code at various lifecycle points in the build.

Messenger integration

Circle CI can send notifications to Discord, Slack, MS Teams or other messenger apps indicating the results of the latest build.

• To do so, the Circle CI app must be authorized for the given messenger app workspace.

• So your instructor may, unfortunately, be reluctant to allow this.

Badges

You can embed build status badges that Circle CI will use to automatically show the latest status of your build into your project’s README.md.

• The Markdown code to use for status badge images are shown on your Circle CI project settings page, and are something like:

[![CircleCI](https://circleci.com/gh/software-assignments-spring2022/your-repo-name-here/tree/master.svg?style=shield)](https://circleci.com/gh/software-assignments-spring2022/your-repo-name-here/tree/master)

Warnings Cause Build Failure

By default, both errors and warnings produced by a project will cause the continuous integration server to fail the build.

• React.js projects are particularly prone to issuing many warnings that cause the build to fail.

• if an environment variable CI is set to false, most continuous integration servers will ignore warnings.

• For a React.js project, set this environment variable by modifying the build script in the package.json file to: ` “build”: “CI=false && react-scripts build”`

• For other types of projects, it is possible to set this environment variable directly within the workflow file.

GitHub Actions

Overview

Like other continuous integration services, GitHub Actions is a hosted service that runs automated builds, tests, and other processes on a virtual machine.

Virtual build servers

When a change is detected in version control (whether a push, pull request, merge, or other), GitHub Actions boots up a new virtual server.

• GitHub Actions clones the version control repository into the virtual machine, and performs the build and test.

Key features

GitHub Actions comes with the common features of any continuous integration server.

• hosted continuous integration solution

• free tier

• at its simplest, requires the addition of a configuration file to a repository

• supports building and testing of many programming languages

• integrates with other popular DevOps tools and practices

GitHub integration

As a part of a GitHub repository, GitHub Actions does not need any special setup in order to detects change to the repository and trigger continuous integration builds and testing.

• As a best practice, have GitHub Actions perform automated builds on pull requests. This allows the pull request reviewer to be sure the code builds and passes automated tests prior to merging the code.

• This also better conserves the free executable minutes given to the repository owner’s account.

Basic configuration

Configuration files, known as “workflows”, are written in YAML (YAML Ain’t Markup Language).

• configuration files are stored in a .github/workflows directory in the project repository.

• workflows in GitHub Actions, as in other automation tools, can run aribtrary shell commands to perform any task.

• GitHub maintains a marketplace of pre-built actions that can be used in your own custom workflows.

• GitHub Actions currently offers a limited number of execution minutes for free, and charges for additional usage.

Highly configurable

GitHub Actions supports detailed instructions and can run arbitrary code at various lifecycle points in the build.

Example

A Python build/test workflow that runs on every pull request. Linting using pylint or other code linter could easily be added.

on: [pull_request]
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.6", "3.8", "3.10"]
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python $
        uses: actions/setup-python@v4
        with:
          python-version: $
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install pytest
          if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
      - name: Test with pytest
        run: |
          pytest

Badges

You can embed build/test status badges that GitHub Actions will use to automatically show the latest status of your build into your project’s README.md.

• The Markdown code to use for status badge images are shown on your GitHub Actions project settings page, and are something like:

![Your custom badge name](https://github.com/<OWNER>/<REPOSITORY>/actions/workflows/<WORKFLOW_FILE>/badge.svg)

Messenger integration

GitHub Actions can send notifications to Discord, Slack, MS Teams or other messenger apps indicating the results of the latest build.

• To do so, GitHub Actions must be authorized for the given messenger app workspace.

• So your instructor may, unfortunately, be reluctant to allow this.

Warnings Cause Build Failure

By default, both errors and warnings produced by a project will cause the continuous integration server to fail the build.

• React.js projects are particularly prone to issuing many warnings that cause the build to fail.

• if an environment variable CI is set to false, most continuous integration servers will ignore warnings.

• For a React.js project, set this environment variable by modifying the build script in the package.json file to: ` “build”: “CI=false && react-scripts build”`

• For other types of projects, it is possible to set this environment variable directly within the workflow file.

Conclusions

Thank you. Bye.