Testing

Testing#

Education objectives

importance of testing
test-driven development (TDD)
simple pytest usage
coverage

Software testing#

From https://en.wikipedia.org/wiki/Software_testing:

Software testing is the act of checking whether software satisfies expectations.

Why testing?#

Coding without testing is dangerous.

Fig. 1 quand-je-déploie-en-prod-sans-tester#
To make sure I conform with the specs, and/or define correct specs.

Fig. 2 solid-code-wrong-specs#
To avoid regression:
- when there is a refactor
- when there is a critical code evolution
- when it crashes, to select where to look for the pb

What do we test?#

Unit tests
Functional tests

Unit tests#

Test that functions conform with the specifications.

def add(a, b):
    """ add a and b and return a+b"""
    return a + b


def test_add():
    assert add(1, 2) == 3
    ...

Functional tests#

Check that the code works when assembling different functions, i.e. the functions can work together!

Test-driven development#

From https://en.wikipedia.org/wiki/Test-driven_development:

Test-driven development (TDD) is a way of writing code that involves writing an automated unit-level test case that fails, then writing just enough code to make the test pass, then refactoring both the test code and the production code, then repeating with another new test case.

Pytest and coverage#

Pytest#

Pytest is a software testing framework that helps you write and run readable and scalable tests. It is not part of the standard library so it needs to be installed

Tests are functions whose names starts with test_ written in files starting test_*.py. They check for behaviors of code. They check that code continues to work after modifications.

They can be executed by calling pytest. Alternatively, one can run pytest test_xxx.py to only run this test file.

There are several useful options (see pytest -h) but here is a selection of the most useful ones:

  -v, --verbose         Increase verbosity
  -s                    Shortcut for --capture=no

  -x, --exitfirst       Exit instantly on first error or failed test

  --lf, --last-failed   Rerun only the tests that failed at the last run (or all
                        if none failed)
  --ff, --failed-first  Run all tests, but run the last failures first.

Tip

pytest --pdb --pdbcls=IPython.terminal.debugger:TerminalPdb starts a debug session where an error was raised (pdb is the builtin Python debugger).

The related help says:

  --pdb                 Start the interactive Python debugger on errors or
                        KeyboardInterrupt
  --pdbcls=modulename:classname
                        Specify a custom interactive Python debugger for use
                        with --pdb.For example:
                        --pdbcls=IPython.terminal.debugger:TerminalPdb

One can remember about the command pytest -h | grep pdb, to find again this useful command.

Test coverage and the Coverage package#

The notion of test coverage is useful. It is important to know which code is at least executed during testing. The coverage is the percentage of lines run by the test suite.

One can measure the coverage with the package coverage and the Pytest plugin pytest-cov (pip install pytest coverage pytest-cov).

If your code is in a directory src and your test files in a directory tests, pytest --cov=src tests will run your tests, measure the coverage and produce a short report. You can then produce a html visualisation of these results by running coverage html.

Exercise#

Exercise 18

The goal is to write a function that returns the sum of the first argument with twice the second argument. First write a test for this function. Try to use pytest!

Solution to Exercise 18

First write the tests in a file named test_*.py

from my_mod import add_second_twice

def test_add_second_twice():
    """ test add second twice"""
    print("testing add second twice with int ", end="")
    assert add_second_twice(3, 5) == 13
    print("...OK")
    print("testing add second twice with strings ", end="")
    assert add_second_twice("aa", "bb") == "aabbbb"
    print("...OK")
    print("testing add second twice with list ", end="")
    assert add_second_twice([1,2], [3,4]) == [1, 2, 3, 4, 3, 4]
    print("...OK")
    print("test add second twice OK with int, string and list")

and empty functions

def add_second_twice(arg0, arg1):
    """Return the sum of the first argument with twice the second one.
        Arguments should be of type that support
        sum and product by an integer
        (e.g. numerical, string, list, ...)
        :param arg0: first argument
        :param arg1: second argument
        :return: arg0 + 2 * arg1
    """
    pass

Then implement the function and test:

def add_second_twice(arg0, arg1):
    """Return the sum of the first argument with twice the second one.
    Arguments should be of type that support sum and product by
    an integer (e.g. numerical, string, list, ...)
    :param arg0: first argument
    :param arg1: second argument
    :return: arg0 + 2 * arg1
    """
    result = arg0 + 2 * arg1
    print(f"arg0 + 2*arg1 = {arg0} + 2*{arg1} = {result}")
    return result


def test_add_second_twice():
    """test add second twice"""
    print("testing add second twice with int ", end="")
    assert add_second_twice(3, 5) == 13
    print("...OK")
    print("testing add second twice with strings ", end="")
    assert add_second_twice("aa", "bb") == "aabbbb"
    print("...OK")
    print("testing add second twice with list ", end="")
    assert add_second_twice([1, 2], [3, 4]) == [1, 2, 3, 4, 3, 4]
    print("...OK")
    print("test add second twice OK with int, string and list")


test_add_second_twice()

testing add second twice with int arg0 + 2*arg1 = 3 + 2*5 = 13
...OK
testing add second twice with strings arg0 + 2*arg1 = aa + 2*bb = aabbbb
...OK
testing add second twice with list arg0 + 2*arg1 = [1, 2] + 2*[3, 4] = [1, 2, 3, 4, 3, 4]
...OK
test add second twice OK with int, string and list