knowledge-kitchen

Software Testing - Ensuring Software Works as Expected

Validating the functionality of a codebase.

1. Overview
2. Automation
3. Static Analysis
4. Code Linting
5. Unit Testing
6. Code Coverage
7. Integration Testing
8. User Acceptance Testing
9. Regression Testing
10. System Testing
11. Test-Driven Development
12. Behavior-Driven Development
13. Conclusions

Overview

Substance

Testing helps verify that the software performs as expected.

• Functionality

• Speed

• Load

• Security

• etc.

Style

Testing can also help verify that the software has been written well.

• Syntax

• Style

• Maintainability

Sanity

Testing can also help prove that the software is usable and that you haven’t built it all for nil.

Automation

Rise of the Robots

While much of running software tests can now be automated, a few tasks still require humans.

• Writing the tests… although this too may become fully automated eventually

• Testing whether humans are able and happy to use the software… although some seem to think they can automate users away.

Static Analysis

Concept

Static analysis tools perform a static analysis of the codebase, meaning the code need not be executed for it to be analyzed and for problems to be reported.

Examples of static analysis include:

• code linting - checking code for syntax and style errors

• type checking - checking that data such as function arguments and values assigned to object properties are of the correct type.

Advantage

Static analysis linting and type checking tools are especially useful when a programming language is interpreted, since interpreters do not report compilation errors, and errors are often only discovered at runtime.

Code Linting

Concept

A code linter is a tool that automatically checks both syntax and style of code.

Features

A code linter is a software tool that can…

• be set to follow a set of code rules and standards

• flag suspicious usage in code that does not meet the defined rules and standards

• auto-fix some syntax and style problems

• integrate with most popular code editors

Advantages

Code linters offer some advantages over human code reviews, such as:

• fast

• accurate

• consistent

• impersonal

Recommendations (Javascript)

Use ESLint linter with the Prettier formatter for Javascript.

• An extension exists to integrate ESLint into the popular Visual Studio Code IDE, and another for Prettier.

• Can be configured to test against any Javascript coding conventions - there are two popular competing Javascript style guides: Google’s and AirBnb’s.

• There are also people and teams who decide on their own code styles.

• Follow these instructions - or these if you don’t have a LinkedIn Learning account - to set it up ESLint with the AirBnB style guide and the Prettier formatter.

Recommendations (Javascript) continued

Once you have installed the Prettier or other similar formatter extension to Visual Studio Code, you can have it auto-format your code every time you save a file. To turn this on, go to Visual Studio Code’s settings and search for “format on save”. Activate the checkbox.

Recommendations (Python)

Use pylint for Python.

• A Pylint extension for Visual Studio Code exists, and similar extensions for other popular IDEs do too.

• To modify pylint’s default coding standards (which follow PEP 8), output them to a file with pylint --generate-rcfile > .pylintrc. If using Visual Studio Code, point the Pylint extension to that file by adding the setting, "pylint.args": ["--rcfile=./.pylintrc"], to the project’s workspace settings.json file.

• Black is a Python code formatter - it automatically formats the code to adhere to the PEP 8 coding standard, and extensions for this also exist for popular IDEs, including one for Visual Studio Code. Configure it to automatically format the code when you save by adding the following setting to your workspace’s settings.json file:

  "[python]": {
    "editor.defaultFormatter": "ms-python.black-formatter",
    "editor.formatOnSave": true
  }
  

• See the Beginner’s Guide to Coding Standards in Python

Type Checking

Concept

Type checking is a type of static analysis that verifies the data types of variables, function arguments, and object properties. Setting these values to a type that contradicts the type annotations added to the code results in an error.

• type checkers can allow a dynamically-typed languages, such as Javascript and Python, to be used in a way similar to statically-typed languages, such as Java and C, should one desire such a thing.

Advantages

Type checking offers a few specific advantages that make it attractive, especially for large teams and projects, where the quality of people and processes may vary:

• documentation - type checking can serve as a form of documentation of the expected behavior of the code

• debugging - errors from incorrect types can help identify bugs or usages that run contrary to the intention of the code

• speed - if the code is compiled, type checking can help the compiler optimize the code for the expected data types

Disadvantages

Type checking also has some notable disadvantages, especially for small teams and projects attempting to move quickly:

• complexity - type checking adds complexity to the code, which can make it harder to read and understand, especially for new developers

• maintenance - type checking must be maintained the same as all code, which creates additional burden

• false sense of security - type checking does not guarantee that the code will run correctly, since it does not check the logic of the code, only the types of the data

Recommendations (Javascript)

For small projects and teams, it is recommended not to use a type checker. However, if you insist on using a type checker, TypeScript is the most popular type checker for Javascript at the time of this writing.

An example of function with Typescript annotations for the parameter variable and return value:

function printToConsole(s: string): void {
  console.log(s)
}

printToConsole(`Hello world`)

Custom object and array structures can be defined as well.

Recommendations (Python)

Should one desire type checking, Python allows (but does not enforce) type hints as part of the language. The mypy type checking tool can be used to enforce typing.

# an example function with a type hint for the parameter variable and return value
def print_to_console(s: str) -> None:
    print(s)

Custom object and array structures can be defined as well.

Unit Testing

Concept

A unit is the smallest testable unit of code - typically a function.

• Unit testing verifies that each unit behaves as expected, given certain inputs.

Javascript Example

A unit test example in Javascript using the mocha unit testing framework and Node.js’s built-in assertion library, assert:

// use node.js's built-in assert assertion library
const assert = require("assert")

// a set of tests of array functions
describe("Array", function () {
  // one particular unit test
  describe("#indexOf()", function () {
    // assert what should be returned
    it("should return -1 when the value is not present", function () {
      // test that assertion
      assert.equal(-1, [1, 2, 3].indexOf(4))
    })
  })
})

Python Example

A unit test example in Python using pytest:

import pytest
from some_package import some_module

class Tests:

  def test_some_function(self):
    '''
    Verify some_function() returns a non-empty string.
    '''
    actual = some_module.some_function() # get the actual return value of the function
    assert isinstance(actual, str), f"Expected some_function() to return a string. Instead, it returned {actual}"
    assert len(actual) > 0, f"Expected some_function() not to be empty. Instead, it returned a string with {len(actual)} characters"

Features

Unit tests are performed on each unit in isolation, in the sense that they…

• run in isolation and do not depend upon one-another.

• are not concerned with user interactions or user interfaces.

• do not require any changes to the production code.

• only test code belonging to the system, not platform code, 3rd party code or external systems such as databases or APIs.

Advantages

Unit testing has discrete advantages that become increasingly important as a project grows in complexity and age.

• Failed unit tests indicate bugs that must be fixed

• Unit tests are fast to write and faster run

• Running unit tests can be automated, with a human notified of any failure

Recommendation

Use mocha and chai for Javascript unit testing.

• mocha is a Javascript unit test framework

• by default it uses Node.js’s built-in assert assertion library.

• but chai is a more elaborate assertion library that is ironically often paired with mocha

Integration Testing

Concept

Whereas unit testing tests units in-situ in isolation, integration testing tests units in vivo.

• Unit tests are meant to validate each unit in isolation from any external influences.

• Integration tests are purposefully designed to test units as they interact with those outside influences.

• For example, whereas a a unit test of the front-end of a web app might make an HTTP request to a mock server or API and receive a mock response , a unit test would make an HTTP request to a real server or API, receivee the real response, to ensuree it is handled correctly.

Features

Integration tests…

• test that all components of the system interact as expected, sending and receiving messages from one-another correctly in all expected circumstances

• test that the system interacts with any external dependencies, such as libraries, databases, and APIs correctly

• can be of partial or full environments, including external bits like databases and services, or not

• can include user interfaces and results of particular system interactions, such as changes to content in databases and logs that result from certain actions

• can be run on one system, or across several systems

Code Coverage

Concept

The term code coverage refers to the percent of all code which is executed when unit and integration tests are run.

• Code coverage tools automatically calculate this as tests are run.

• While 100% code coverage is the ideal, anywhere above 80% is pretty well-covered.

Limitations

Code coverage does not indicate that the code has been tested in all possible scenarios

• High code coverage does not indicate that the code is good.

• 100% code coverage does not indicate that the code is well-tested or that it will run well.

Recommendation (Javascript)

Use c8 or istanbul for Javascript code coverage analysis.

• integrates well with the mocha unit testing framework

• An example of using istanbul in a Node.js project’s package.json script:

{
    "scripts": {
      "test": "nyc mocha --timeout=3000"
    }
}

• Code coverage analysis could then be triggered from the command line.

npm test

Recommendation (Python)

Use [coverage.py](https://coverage.readthedocs.io/ for Python code coverage analysis.

User Acceptance Testing

Concept

User Acceptance Testing (UAT) …

• tests whether users will accept the software.

• i.e., verifies that the software works as expected from an end-user’s point of view

• is a form of Integration Testing, since all the parts must inter-operate in order for the system to be used.

Scripts

User Acceptance Tests test users in specific use cases or user stories.

• Users are asked to run through scripts of common scenarios and interactions they might encounter using the software

• Each script includes a set of steps for the human tester to go through to perform the test, e.g. ```
  1. Go to the login page
  2. Enter your username
  3. Enter your password
  4. Click the login button
  5. Verify that you are logged in ```
• The tester then records whether the user successfully completed the task, how long it took them, and notes any problems the user encountered.

Regression Testing

Concept

The term regression testing refers to the re-running of old tests when new tests are run.

• This ensures that the entire codebase works well, even when new features are developed, bugs are fixed, etc.

System Testing

Concept

Testing of the non-functional requirements, such as …

• load handling - handling the load under expected conditions

• stress testing - handling the load at higher-than expected conditions

• security testing - making sure the system and its users are safe

Test-Driven Development

Concept

Test-Driven Development (TDD) is the practice of writing tests, particularly unit tests, before production code is written.

Advantages

Test-Driven Development has several touted benefits:

• code coverage - every bit of code has a test developed for it before the code to be tested has even been written

• debugging - since tests are run with every code change, it is easy to identify the new code that created a bug

• documentation - the tests themselves become a specification of the system

• planning - it forces developers to think about what they want their code to do before writig it

Criticisms

Test-Driven Development has been attacked on many fronts:

• It is often seen as a bit too extreme -in fact, TDD originated from what is called the eXtreme Programming (XP) methodology.

• In most cases (for better or worse), the requirements of systems are not fully known until development starts

• It is not unusually for new requirements to surface late in the development cycle.

• Many developers consider it helpful to have an ongoing feedback loop between specifying, developing and testing.

Behavior-Driven Development

Concept

An alternative preferred by some to Test-Driven Development is Behavior-Driven Development (BDD).

• BDD performs all the same duties as TDD, at the same times, for the same reasons.

• The difference arises from the language used in writing the code - BDD attempts to offer testing in a more human-centric language.

• Javascript’s chai assertion library, for example, supports either TDD or BDD style assertions, e.g.

• A TDD-style assertion stating the expected value in a variable: assert.equal(foo, 'bar')

• The same assertion in BDD-style: expect(foo).to.equal('bar') - notice the more human-friendly phrasing of the code.

Conclusions

You now have experienced a nice-and-easy overview of the different categories of software testing in common practice. The next step would be to try them out!

• Thank you. Bye.