Software Testing - Ensuring Software Works as Expected

# Software Testing

Validating the functionality of a codebase.

---

# Agenda

1. [Overview](#overview)
1. [Automation](#automation)
1. [Static Analysis](#static-analysis)
1. [Code Linting](#code-linting)
1. [Unit Testing](#unit-testing)
1. [Code Coverage](#code-coverage)
1. [Integration Testing](#integration-testing)
1. [User Acceptance Testing](#user-acceptance-testing)
1. [Regression Testing](#regression-testing)
1. [System Testing](#system-testing)
1. [Test-Driven Development](#test-driven-development)
1. [Behavior-Driven Development](#behavior-driven-development)
1. [Conclusions](#conclusions)

---

# Overview

## Substance

Testing helps verify that the software performs as expected.

- Functionality

- Speed

- Load

- Security

- etc.

---

## Style

Testing can also help verify that the software has been written well.

- Syntax

- Style

- Maintainability

---

## Sanity

Testing can also help prove that the **software is usable** and that you haven't built it all for nil.

---

# Automation

## Rise of the Robots

While much of running software tests can now be automated, a few tasks still require humans.

- Writing the tests... although this too [may become fully automated eventually](https://duckduckgo.com/?q=autogenerate+unit+tests)

- Testing whether humans are able and happy to use the software... although [some seem to think they can automate users away](https://duckduckgo.com/?q=automate+user+acceptance+testing).

---

# Static Analysis

## Concept

Static analysis tools perform a **static analysis** of the codebase, meaning the code need not be executed for it to be analyzed and for problems to be reported.

Examples of `static analysis` include:

- **code linting** - checking code for syntax and style errors

- **type checking** - checking that data such as function arguments and values assigned to object properties are of the correct type.

---

## Advantage

Static analysis linting and type checking tools are _especially useful when a programming language is interpreted_, since interpreters do not report compilation errors, and errors are often only discovered at runtime.

---

# Code Linting

## Concept

A code linter is a tool that automatically checks both **syntax** and **style** of code.

---

## Features

A code linter is a software tool that can...

- be set to follow a set of code rules and standards

- flag suspicious usage in code that does not meet the defined rules and standards

- auto-fix some syntax and style problems

- integrate with most popular code editors

---

## Advantages

Code linters offer some advantages over human code reviews, such as:

- fast

- accurate

- consistent

- impersonal

---

## Recommendations (Javascript)

Use [ESLint](https://github.com/eslint/eslint) linter with the [Prettier](https://prettier.io/) formatter for Javascript.

- An [extension](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint) exists to integrate ESLint into the popular Visual Studio Code IDE, and another for [Prettier](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode).

- Can be configured to test against any Javascript **coding conventions** - there are three popular competing Javascript style guides: [Google's](https://google.github.io/styleguide/jsguide.html), [AirBnb's](https://github.com/airbnb/javascript), and Feross Aboukhadijeh's [Standard Style](https://github.com/standard/standard)

- There are also people and teams who decide on their own code styles.

- Follow [these instructions](https://www.linkedin.com/learning/building-a-website-with-node-js-and-express-js-3/setting-up-eslint-and-prettier?u=2131553) - or [these](https://www.youtube.com/watch?v=SydnKbGc7W8) if you don't have a LinkedIn Learning account - to set it up ESLint with the AirBnB style guide and the Prettier formatter.

---

## Recommendations (Javascript) continued

Once you have installed the `Prettier` or other similar formatter extension to Visual Studio Code, you can have it auto-format your code every time you save a file. To turn this on, go to Visual Studio Code's settings and search for "`format on save`". Activate the checkbox.

![Setting to format on save in VSCode](../assets/software-testing/vscode-format-on-save.png)

---

## Recommendations (Python)

Use [pylint](https://github.com/pylint-dev/pylint) for Python.

- A [Pylint extension](https://marketplace.visualstudio.com/items?itemName=ms-python.pylint) for Visual Studio Code exists, and similar extensions for other popular IDEs do too.

- To veer from `pylint`'s default coding standards (which follow [PEP 8](https://peps.python.org/pep-0008/)), output them to a file with `pylint --generate-rcfile > .pylintrc`. If using Visual Studio Code, point the Pylint extension to that file by adding the setting, `"pylint.args": ["--rcfile=./.pylintrc"]`, to the project's [workspace settings.json file](https://code.visualstudio.com/docs/getstarted/settings#_workspace-settingsjson-location).

---

## Recommendations (Python, continued)

[Black](https://pypi.org/project/black/) is a Python code formatter - it automatically formats the code to adhere to the `PEP 8` coding standard, and extensions for this also exist for popular IDEs, including [one for Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=ms-python.black-formatter).

Configure it to automatically format the code when you save by adding the following setting to your [workspace's `settings.json` file](https://code.visualstudio.com/docs/getstarted/settings#_workspace-settings):

```javascript
"[python]": {
  "editor.defaultFormatter": "ms-python.black-formatter",
  "editor.formatOnSave": true
}
```

- See the [Beginner's Guide to Coding Standards in Python](https://docs.pylint.org/tutorial.html)

---

# Type Checking

## Concept

Type checking is a type of **static analysis** that verifies the data types of variables, function arguments, and object properties. Setting these values to a type that contradicts the _type annotations_ added to the code results in an error.

- type checkers can allow a dynamically-typed languages, such as Javascript and Python, to be used in a way similar to statically-typed languages, such as Java and C, should one desire such a thing.

---

## Advantages

Type checking offers a few specific advantages that make it attractive, especially for large teams and projects, where the quality of people and processes may vary:

- **documentation** - type checking can serve as a form of documentation of the expected behavior of the code

- **debugging** - errors from incorrect types can help identify bugs or usages that run contrary to the intention of the code

- **speed** - if the code is compiled, type checking can help the compiler optimize the code for the expected data types

---

## Disadvantages

Type checking also has some notable disadvantages, especially for small teams and projects attempting to move quickly:

- **complexity** - type checking adds complexity to the code, which can make it harder to read and understand, especially for new developers

- **maintenance** - type checking must be maintained the same as all code, which creates additional burden

- **false sense of security** - type checking does not guarantee that the code will run correctly, since it does not check the logic of the code, only the types of the data

---

## Recommendations (Javascript)

For small projects and teams, it is recommended not to use a type checker. However, if you insist on using a type checker, [TypeScript](https://www.typescriptlang.org/) is the most popular type checker for Javascript at the time of this writing.

An example of function with Typescript annotations for the parameter variable and return value:

```js
function printToConsole(s: string): void {
  console.log(s)
}

printToConsole(`Hello world`)
```

Custom object and array structures can be defined as well.

---

## Recommendations (Python)

Should one desire type checking, Python allows (but does not enforce) [type hints](https://docs.python.org/3/library/typing.html) as part of the language. The [mypy](https://mypy.readthedocs.io/en/stable/getting_started.html) type checking tool can be used to enforce typing.

```python
# an example function with a type hint for the parameter variable and return value
def print_to_console(s: str) -> None:
    print(s)
```

Custom object and array structures can be defined as well.

---

# Unit Testing

## Concept

A unit is the smallest testable unit of code - typically a function.

- [Unit testing](/content/courses/software-engineering/slides/unit-testing) verifies that each unit behaves as expected, given certain inputs.

---

## Javascript Example

A unit test example in Javascript using the `mocha` unit testing framework and Node.js's built-in assertion library, `assert`:

```js
// use node.js's built-in assert assertion library
const assert = require("assert")

// a set of tests of array functions
describe("Array", function () {
  // one particular unit test
  describe("#indexOf()", function () {
    // assert what should be returned
    it("should return -1 when the value is not present", function () {
      // test that assertion
      assert.equal(-1, [1, 2, 3].indexOf(4))
    })
  })
})
```

---

## Python Example

A unit test example in Python using `pytest`:

```python
import pytest
from some_package import some_module

class Tests:

def test_some_function(self):
    '''
    Verify some_function() returns a non-empty string.
    '''
    actual = some_module.some_function() # get the actual return value of the function
    assert isinstance(actual, str), f"Expected some_function() to return a string. Instead, it returned {actual}"
    assert len(actual) > 0, f"Expected some_function() not to be empty. Instead, it returned a string with {len(actual)} characters"

```

---

## Features

Unit tests are performed on each unit in isolation, in the sense that they...

- run in isolation and do not depend upon one-another.

- are not concerned with user interactions or user interfaces.

- do not require any changes to the production code.

- only test code belonging to the system, not platform code, 3rd party code or external systems such as databases or APIs.

---

## Advantages

Unit testing has discrete advantages that become increasingly important as a project grows in complexity and age.

- Failed unit tests indicate bugs that **must** be fixed

- Unit tests are fast to write and faster run

- Running unit tests can be automated, with a human notified of any failure

---

## Recommendation (Javascript)

Use [mocha](https://mochajs.org/) and [chai](https://www.chaijs.com/) for Javascript unit testing.

- `mocha` is a Javascript unit test framework

- by default it uses Node.js's built-in [`assert`](https://nodejs.org/api/assert.html) assertion library.

- but `chai` is a more elaborate [BDD-style](#behavior-driven-development) assertion library that is ironically often paired with `mocha`

---

## Recommendation (Python)

Use [pytest](https://docs.pytest.org/en/stable/) for Python unit testing.

- `pytest` is a Python unit test framework

- By default, it uses Python's built-in [`assert`](https://docs.python.org/3/reference/simple_stmts.html#assert) statement.

- but it can be configured to use other assertion libraries, including [BDD-style](#behavior-driven-development) assertion libraries such as [`grappa`](https://pypi.org/project/grappa/) and [`assertpy`](https://pypi.org/project/assertpy/)

---

# Integration Testing

## Concept

Whereas unit testing tests units _in-situ_ in isolation, integration testing tests units _in vivo_.

- **Unit tests** are meant to validate each unit in isolation from any external influences.

- **Integration tests** are purposefully designed to test units as they interact with those outside influences.

The tools used for integration testing are usually the same as those used for unit testing.

---

## Comparison

For example, consider testing the front-end code of a web application that makes an `HTTP` request to a back-end server or API for some data.

- a _unit test_ of the front-end of a web app might make an `HTTP` request to a **mock server or API** receive a **mock response** that is controlled by the tester, and evaluate it to ensure it is handled correctly by the front-end code.

- an _integration test_ would make an `HTTP` request to the **real server or API**, receive the **real response**, and evaluate it to ensure it is handled correctly by the front-end code.

In other words, in unit tests, systems or parts of code that are external to the unit of code being tested are mocked or faked. In integration tests, they are not. Otherwise, they are both tests of the same code done in the same manner.

---

## Features

Integration tests...

- test that all components of the system interact as expected, sending and receiving messages from one-another correctly in all expected circumstances

- test that the system interacts with any external dependencies, such as libraries, databases, and APIs correctly

- can be of partial or full environments, including external bits like databases and services, or not

- can include user interfaces and results of particular system interactions, such as changes to content in databases and logs that result from certain actions

- can be run on one system, or across several systems

---

# Code Coverage

## Concept

The term `code coverage` refers to the percent of all code which is executed when unit or integration tests are run.

- Code coverage tools automatically calculate this as tests are run.

- While `100%` code coverage is the ideal, anywhere above `80%` is pretty well-covered.

---

## Limitations

Code coverage does not indicate that the code has been tested in all possible scenarios

- High code coverage **does not** indicate that the code is good.

- 100% code coverage **does not** indicate that the code is well-tested or that it will run well.

---

## Recommendation (Javascript)

Use [c8](https://www.npmjs.com/package/c8) for Javascript code coverage analysis.

- integrates well with the mocha unit testing framework

- An example of using `c8` in a Node.js project's `package.json` script:

```js
{
    "scripts": {
      "test": "c8 mocha --timeout=3000"
    }
}
```

- Code coverage analysis could then be triggered from the command line.

```bash
npm test
```

---

## Recommendation (Python)

Use [coverage.py](https://coverage.readthedocs.io/)

- install it into your [Python virtual environment](../python-virtual-environments), e.g. `pipenv install coverage`

- use it to run your tests, e.g. `coverage run -m pytest`

- generate a code coverage report, e.g. `coverage report -m`

---

## Recommendation (Python, continued)

If using `pytest` as the testing framework, you can alternatively integrate `coverage.py` directly into `pytest` commands by installing the [pytest-cov](https://pytest-cov.readthedocs.io/en/latest/readme.html) package.

- install it into your [Python virtual environment](../python-virtual-environments), e.g. `pipenv install pytest-cov`

- include coverage reporting in your `pytest` results with the `--cov` flag, e.g. `pytest --cov=.` or `python -m pytest --cov=.`, depending on how you usually run your tests.

---

# User Acceptance Testing

## Concept

User Acceptance Testing (UAT) ...

- tests whether users will accept the software.

- i.e., verifies that the software works as expected from an end-user's point of view

- is a form of Integration Testing, since all the parts must inter-operate in order for the system to be used.

---

## Scripts

User Acceptance Tests test users in specific [use cases](../uml-diagrams#use-cases) or [user stories](../requirements-engineering#user-stories).

- Users are asked to run through scripts of common scenarios and interactions they might encounter using the software

- Each script includes a set of steps for the human tester to go through to perform the test, e.g.
  ```
  1. Go to the login page
  2. Enter your username
  3. Enter your password
  4. Click the login button
  5. Verify that you are logged in
  ```

- The tester then records whether the user successfully completed the task, how long it took them, and notes any problems the user encountered.

---

# Regression Testing

## Concept

The term `regression testing` refers to the re-running of old tests when new tests are run.

- This ensures that the entire codebase works well, even when new features are developed, bugs are fixed, etc.

---

# System Testing

## Concept

Testing of the non-functional requirements, such as ...

- **load handling** - handling the load under expected conditions

- **stress testing** - handling the load at higher-than expected conditions

- **security testing** - making sure the system and its users are safe

---

# Test-Driven Development

## Concept

Test-Driven Development (TDD) is the practice of writing tests, particularly unit tests, before production code is written.

---

## Advantages

Test-Driven Development has several touted benefits:

- **code coverage** - every bit of code has a test developed for it before the code to be tested has even been written

- **debugging** - since tests are run with every code change, it is easy to identify the new code that created a bug

- **documentation** - the tests themselves become a specification of the system

- **planning** - it forces developers to think about what they want their code to do before writig it

---

## Criticisms

Test-Driven Development has been attacked on many fronts:

- It is often seen as a bit too extreme -in fact, TDD originated from what is called the **eXtreme Programming** (XP) methodology.

- In most cases (for better or worse), the requirements of systems are not fully known until development starts

- It is not unusually for new requirements to surface late in the development cycle.

- Many developers consider it helpful to have an ongoing feedback loop between specifying, developing and testing.

---

# Behavior-Driven Development

## Concept

An alternative style of writing assertions, known as **Behavior-Driven Development** (BDD), is preferred by some developers.

- BDD performs all the same duties as regular assertions, at the same times, and for the same reasons.

- Like Test-Driven Development, BDD attempts to use tests as a form of documentation.

- The difference between BDD and other approaches to testing arises from the style of the language used in writing the code - BDD attempts to offer testing in a more human-centric language, at least according to its proponents.

---

## Examples

For example, Javascript's `chai` and Python's `assertpy` libraries both support BDD-style assertions.

- A "regular" assertion stating the expected value in a variable would be `assert.equal(foo, 'bar')` (Javascript) or `assert foo == 'bar'` (Python).

- The same assertion in BDD-style could be `expect(foo).to.equal('bar')` (Javascript) or `assert_that(foo).is_equal_to('bar')` (Python).

Notice the more "human-friendly" phrasing of the code in the BDD-style assertions.

---

## Considerations

Proponents of BDD-style assertions claim that they are easier to read and understand by "regular" people. But is this true?

- Critics claim that BDD is overly flowery and verbose and makes it more _difficult to understand_ what is truly going on in each statement.

- Critics claim that BDD-style assertions focus more on "_magic_" (i.e. "wow, it just works...") than on logic (i.e. "here's how it works...")

Your author sides with the critics. [Cui bono?](https://en.wikipedia.org/wiki/Cui_bono%3F)

---

# Conclusions

You now have experienced a nice-and-easy overview of the different categories of software testing in common practice. The next step would be to try them out!

- Thank you. Bye.