Software Testing - Ensuring Software Works as Expected

# Software Testing

Validating the functionality of a codebase.

---

# Agenda

1. [Overview](#overview)
1. [Automation](#automation)
1. [Static Analysis](#static-analysis)
1. [Code Linting](#code-linting)
1. [Unit Testing](#unit-testing)
1. [Code Coverage](#code-coverage)
1. [Integration Testing](#integration-testing)
1. [User Acceptance Testing](#user-acceptance-testing)
1. [Regression Testing](#regression-testing)
1. [System Testing](#system-testing)
1. [Test-Driven Development](#test-driven-development)
1. [Behavior-Driven Development](#behavior-driven-development)
1. [Conclusions](#conclusions)

---

# Overview

## Substance

Testing helps verify that the software performs as expected.

- Functionality

- Speed

- Load

- Security

- etc.

---

## Style

Testing can also help verify that the software has been written well.

- Syntax

- Style

- Maintainability

---

## Sanity

Testing can also help prove that the **software is usable** and that you haven't built it all for nil.

---

# Automation

## Rise of the Robots

While much of running software tests can now be automated, a few tasks still require humans.

- Writing the tests... although this too [may become fully automated eventually](https://duckduckgo.com/?q=autogenerate+unit+tests)

- Testing whether humans are able and happy to use the software... although [some seem to think they can automate users away](https://duckduckgo.com/?q=automate+user+acceptance+testing).

---

# Static Analysis

## Concept

Static analysis tools perform a **static analysis** of the codebase, meaning the code need not be executed for it to be analyzed and for problems to be reported.

Examples of `static analysis` include:

- **code linting** - checking code for syntax and style errors

- **type checking** - checking that data such as function arguments and values assigned to object properties are of the correct type.

---

## Advantage

Static analysis linting and type checking tools are _especially useful when a programming language is interpreted_, since interpreters do not report compilation errors, and errors are often only discovered at runtime.

---

# Code Linting

## Concept

A code linter is a tool that automatically checks both **syntax** and **style** of code.

---

## Features

A code linter is a software tool that can...

- be set to follow a set of code rules and standards

- flag suspicious usage in code that does not meet the defined rules and standards

- auto-fix some syntax and style problems

- integrate with most popular code editors

---

## Advantages

Code linters offer some advantages over human code reviews, such as:

- fast

- accurate

- consistent

- impersonal

---

## Recommendations (Javascript)

Use [ESLint](https://github.com/eslint/eslint) linter with the [Prettier](https://prettier.io/) formatter for Javascript.

- An [extension](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint) exists to integrate ESLint into the popular Visual Studio Code IDE, and another for [Prettier](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode).

- Can be configured to test against any Javascript **coding conventions** - there are two popular competing Javascript style guides: [Google's](https://google.github.io/styleguide/jsguide.html) and [AirBnb's](https://github.com/airbnb/javascript).

- There are also people and teams who decide on their own code styles.

- Follow [these instructions](https://www.linkedin.com/learning/building-a-website-with-node-js-and-express-js-3/setting-up-eslint-and-prettier?u=2131553) - or [these](https://www.youtube.com/watch?v=SydnKbGc7W8) if you don't have a LinkedIn Learning account - to set it up ESLint with the AirBnB style guide and the Prettier formatter.

---

## Recommendations (Javascript) continued

Once you have installed the `Prettier` or other similar formatter extension to Visual Studio Code, you can have it auto-format your code every time you save a file. To turn this on, go to Visual Studio Code's settings and search for "`format on save`". Activate the checkbox.

![Setting to format on save in VSCode](../assets/software-testing/vscode-format-on-save.png)

---

## Recommendations (Python)

Use [pylint](https://github.com/pylint-dev/pylint) for Python.

- A [Pylint extension](https://marketplace.visualstudio.com/items?itemName=ms-python.pylint) for Visual Studio Code exists, and similar extensions for other popular IDEs do too.

- To modify `pylint`'s default coding standards (which follow [PEP 8](https://peps.python.org/pep-0008/)), output them to a file with `pylint --generate-rcfile > .pylintrc`. If using Visual Studio Code, point the Pylint extension to that file by adding the setting, `"pylint.args": ["--rcfile=./.pylintrc"]`, to the project's [workspace settings.json file](https://code.visualstudio.com/docs/getstarted/settings#_workspace-settingsjson-location).

- [Black](https://pypi.org/project/black/) is a Python code formatter - it automatically formats the code to adhere to the `PEP 8` coding standard, and extensions for this also exist for popular IDEs, including [one for Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=ms-python.black-formatter). Configure it to automatically format the code when you save by adding the following setting to your workspace's `settings.json` file:
  ```javascript
  "[python]": {
    "editor.defaultFormatter": "ms-python.black-formatter",
    "editor.formatOnSave": true
  }
  ```

- See the [Beginner's Guide to Coding Standards in Python](https://docs.pylint.org/tutorial.html)

---

# Type Checking

## Concept

Type checking is a type of **static analysis** that verifies the data types of variables, function arguments, and object properties. Setting these values to a type that contradicts the _type annotations_ added to the code results in an error.

- type checkers can allow a dynamically-typed languages, such as Javascript and Python, to be used in a way similar to statically-typed languages, such as Java and C, should one desire such a thing.

---

## Advantages

Type checking offers a few specific advantages that make it attractive, especially for large teams and projects, where the quality of people and processes may vary:

- **documentation** - type checking can serve as a form of documentation of the expected behavior of the code

- **debugging** - errors from incorrect types can help identify bugs or usages that run contrary to the intention of the code

- **speed** - if the code is compiled, type checking can help the compiler optimize the code for the expected data types

---

## Disadvantages

Type checking also has some notable disadvantages, especially for small teams and projects attempting to move quickly:

- **complexity** - type checking adds complexity to the code, which can make it harder to read and understand, especially for new developers

- **maintenance** - type checking must be maintained the same as all code, which creates additional burden

- **false sense of security** - type checking does not guarantee that the code will run correctly, since it does not check the logic of the code, only the types of the data

---

## Recommendations (Javascript)

For small projects and teams, it is recommended not to use a type checker. However, if you insist on using a type checker, [TypeScript](https://www.typescriptlang.org/) is the most popular type checker for Javascript at the time of this writing.

An example of function with Typescript annotations for the parameter variable and return value:

```js
function printToConsole(s: string): void {
  console.log(s)
}

printToConsole(`Hello world`)
```

Custom object and array structures can be defined as well.

---

## Recommendations (Python)

Should one desire type checking, Python allows (but does not enforce) [type hints](https://docs.python.org/3/library/typing.html) as part of the language. The [mypy](https://mypy.readthedocs.io/en/stable/getting_started.html) type checking tool can be used to enforce typing.

```python
# an example function with a type hint for the parameter variable and return value
def print_to_console(s: str) -> None:
    print(s)
```

Custom object and array structures can be defined as well.

---

# Unit Testing

## Concept

A unit is the smallest testable unit of code - typically a function.

- [Unit testing](/content/courses/software-engineering/slides/unit-testing) verifies that each unit behaves as expected, given certain inputs.

---

## Javascript Example

A unit test example in Javascript using the `mocha` unit testing framework and Node.js's built-in assertion library, `assert`:

```js
// use node.js's built-in assert assertion library
const assert = require("assert")

// a set of tests of array functions
describe("Array", function () {
  // one particular unit test
  describe("#indexOf()", function () {
    // assert what should be returned
    it("should return -1 when the value is not present", function () {
      // test that assertion
      assert.equal(-1, [1, 2, 3].indexOf(4))
    })
  })
})
```

---

## Python Example

A unit test example in Python using `pytest`:

```python
import pytest
from some_package import some_module

class Tests:

def test_some_function(self):
    '''
    Verify some_function() returns a non-empty string.
    '''
    actual = some_module.some_function() # get the actual return value of the function
    assert isinstance(actual, str), f"Expected some_function() to return a string. Instead, it returned {actual}"
    assert len(actual) > 0, f"Expected some_function() not to be empty. Instead, it returned a string with {len(actual)} characters"

```

---

## Features

Unit tests are performed on each unit in isolation, in the sense that they...

- run in isolation and do not depend upon one-another.

- are not concerned with user interactions or user interfaces.

- do not require any changes to the production code.

- only test code belonging to the system, not platform code, 3rd party code or external systems such as databases or APIs.

---

## Advantages

Unit testing has discrete advantages that become increasingly important as a project grows in complexity and age.

- Failed unit tests indicate bugs that **must** be fixed

- Unit tests are fast to write and faster run

- Running unit tests can be automated, with a human notified of any failure

---

## Recommendation

Use [mocha](https://mochajs.org/) and [chai](https://www.chaijs.com/) for Javascript unit testing.

- `mocha` is a Javascript unit test framework

- by default it uses Node.js's built-in `assert` assertion library.

- but `chai` is a more elaborate assertion library that is ironically often paired with mocha

---

# Integration Testing

## Concept

Whereas unit testing tests units in-situ in isolation, integration testing tests units in vivo.

- Unit tests are meant to validate each unit in isolation from any external influences.

- Integration tests are purposefully designed to test units as they interact with those outside influences.

- For example, whereas a a unit test of the front-end of a web app might make an HTTP request to a **mock server or API** and receive a **mock response** , a unit test would make an HTTP request to a **real server or API**, receivee the **real response**, to ensuree it is handled correctly.

---

## Features

Integration tests...

- test that all components of the system interact as expected, sending and receiving messages from one-another correctly in all expected circumstances

- test that the system interacts with any external dependencies, such as libraries, databases, and APIs correctly

- can be of partial or full environments, including external bits like databases and services, or not

- can include user interfaces and results of particular system interactions, such as changes to content in databases and logs that result from certain actions

- can be run on one system, or across several systems

---

# Code Coverage

## Concept

The term `code coverage` refers to the percent of all code which is executed when unit and integration tests are run.

- Code coverage tools automatically calculate this as tests are run.

- While 100% code coverage is the ideal, anywhere above 80% is pretty well-covered.

---

## Limitations

Code coverage does not indicate that the code has been tested in all possible scenarios

- High code coverage **does not** indicate that the code is good.

- 100% code coverage **does not** indicate that the code is well-tested or that it will run well.

---

## Recommendation (Javascript)

Use [c8](https://www.npmjs.com/package/c8) or [istanbul](https://istanbul.js.org/) for Javascript code coverage analysis.

- integrates well with the mocha unit testing framework

- An example of using istanbul in a Node.js project's `package.json` script:

```js
{
    "scripts": {
      "test": "nyc mocha --timeout=3000"
    }
}
```

- Code coverage analysis could then be triggered from the command line.

```bash
npm test
```

---

## Recommendation (Python)

Use [coverage.py](https://coverage.readthedocs.io/ for Python code coverage analysis.

---

# User Acceptance Testing

## Concept

User Acceptance Testing (UAT) ...

- tests whether users will accept the software.

- i.e., verifies that the software works as expected from an end-user's point of view

- is a form of Integration Testing, since all the parts must inter-operate in order for the system to be used.

---

## Scripts

User Acceptance Tests test users in specific [use cases](../uml-diagrams#use-cases) or [user stories](../requirements-engineering#user-stories).

- Users are asked to run through scripts of common scenarios and interactions they might encounter using the software

- Each script includes a set of steps for the human tester to go through to perform the test, e.g.
  ```
  1. Go to the login page
  2. Enter your username
  3. Enter your password
  4. Click the login button
  5. Verify that you are logged in
  ```

- The tester then records whether the user successfully completed the task, how long it took them, and notes any problems the user encountered.

---

# Regression Testing

## Concept

The term `regression testing` refers to the re-running of old tests when new tests are run.

- This ensures that the entire codebase works well, even when new features are developed, bugs are fixed, etc.

---

# System Testing

## Concept

Testing of the non-functional requirements, such as ...

- **load handling** - handling the load under expected conditions

- **stress testing** - handling the load at higher-than expected conditions

- **security testing** - making sure the system and its users are safe

---

# Test-Driven Development

## Concept

Test-Driven Development (TDD) is the practice of writing tests, particularly unit tests, before production code is written.

---

## Advantages

Test-Driven Development has several touted benefits:

- **code coverage** - every bit of code has a test developed for it before the code to be tested has even been written

- **debugging** - since tests are run with every code change, it is easy to identify the new code that created a bug

- **documentation** - the tests themselves become a specification of the system

- **planning** - it forces developers to think about what they want their code to do before writig it

---

## Criticisms

Test-Driven Development has been attacked on many fronts:

- It is often seen as a bit too extreme -in fact, TDD originated from what is called the **eXtreme Programming** (XP) methodology.

- In most cases (for better or worse), the requirements of systems are not fully known until development starts

- It is not unusually for new requirements to surface late in the development cycle.

- Many developers consider it helpful to have an ongoing feedback loop between specifying, developing and testing.

---

# Behavior-Driven Development

## Concept

An alternative preferred by some to Test-Driven Development is Behavior-Driven Development (BDD).

- BDD performs all the same duties as TDD, at the same times, for the same reasons.

- The difference arises from the language used in writing the code - BDD attempts to offer testing in a more human-centric language.

- Javascript's `chai` assertion library, for example, supports either TDD or BDD style assertions, e.g.

- A TDD-style assertion stating the expected value in a variable: `assert.equal(foo, 'bar')`

- The same assertion in BDD-style: `expect(foo).to.equal('bar')` - notice the more human-friendly phrasing of the code.

---

# Conclusions

You now have experienced a nice-and-easy overview of the different categories of software testing in common practice. The next step would be to try them out!

- Thank you. Bye.