Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Testing Python a micro tutorial

Testing Demo

Testing is probably one of the most overlooked technique in the programming world.

Traditionally in the software development industry we had a clear separation between "programmers" or "coders" or "software engineers" on one hand and "qa people" who would do the Quality Assurance of the product.

Traditionally the company and thus the programmers relied on the QA people to find the bugs and to ensure high quality.

This approach breaks down for a variety of reasons in a number of cases.

Fast-paced development with frequent releases - CI/CD Continuous Integration / Continuous Delivery

In the age of fast-paced development when the release cycles aren't measured in months and not even in days, but in hours or minutes, there is no time for a separate, manual QA process. You must automate the testing and verification process.

Academia

If you are a student you don't have the luxury of having a people do QA for you. In fact you might be the person doing manual QA for the projects of your professor. I teach Python programming to biology and in general life-sciences students. Some of them will end up writing lots of software to support the research and thesis. The don't have a separate QA department that would check the applications they wrote. They have to do it themselves.

Open Source

Most of the Open Source developers work on their project in their spare time. They don't have any means to pay someone to do quality assurance for them. They are also not very interested in doing QA themselves nor they want to fix the same bug twice. So Open Source developers tend to write a lot of automated tests. Certainly a lot more than in a corporation. This is true even for the same person in the two different situations. I know a number of people who write lots of tests for their Open Source projects, but almost none at work.

Testing Flask

Before getting to our own simple example, let's see the tests of one of the most popular Python libraries. Flask is a minimalistic framework to write web applications in Python. It is an open source project that has been around for many year.

We can, of course, install it using the regular tools, e.g. pip install flask, but what we are really interested in is to see how a developer of Flask can check that changes made to the project or a newer version of Python does not break anything that was working before. For this we need to get a local copy of the developmen version of the source code.

We can do this by cloning the GitHub repository of Flask

git clone https://github.com/pallets/flask.git
cd flask
pip install -r requirements/dev.txt
pip install -e .
pytest

How do you test your code?

This mini-series is for people who don't have the time to delve into the way you'd write tests for your Python code, but would like to get a quick overview of the possibilities.

However before we can get into actually testing things, it is worth to think about and even to discuss the following questions.

  • What kind of things do you test?

  • Web application?

  • Command line application?

  • Databases?

  • ...

What is testing?

So what do we really mean when we mean testing?

For every piece of code wether its is a small module or a huge application you can have the following equasion.

There some environment the code works in. It might be just the interpreter/compiler in case of a single stand-alone function, or it might include multiple networking elements, servers, databases, ioT deviecs etc. No matter what, the environment is called by the testing people the "Fixture".

Then execute the code - the Application Under Test - and give it some input.

The result should be some "Expected Output".

So this is our equasion.

  • Fixture + Input = Expected Output

What is testing really?

In reality, however, many times we don't get exactly the expected output. Instead there is a small (or big) difference. That's the bug.

The goal of (automated) testing is to make it easy and cheap to notice when these bugs creep in.

To put it in other words, when you write your code you can check if the result is as expected either manually or by writing some automated tests. The question, how will you know your piece of code still works half a year from now when someone made some changes to some other part of the code?

Will you repeate all the manual tests you did earlier? You won't have time for that.

On the other hand if you automated your tests in the first place, then you can easily, quickly and cheaply run them again and you can verify if everything still works as earlier or if a bug appeared.

  • Fixture + Input = Expected Output + Bugs

Testing demo tools

In these examples we are going to see 3 Python modules that can be used for testing.

  • doctest
  • unittest
  • pytest

Testing demo methodology

We won't delve deep into the capabilities of these testing libraries. We will only us a very simple example to show how to write a passing and a failing test.

  • Have a simple AUT - Application Under Test with an obvious bug
  • Write a passing test
  • Write a failing test

Testing demo - AUT - Application Under Test

Given the following module with a single function, how can we use this function and how can we test it?

def add(x, y):
    return x * y

def multiply(x, y):
    return x + y

# Yes, I know there are bugs in this code!

You probably noticed that our function was called add and so the expectation is that it will be able to add two numbers. However the implementation has a bug. It actually multiplies the two numbers. I know it is a very obvious issue, but it is great as it allows us to see the mechanics of testing without getting distracted by a complex implementation and a complex problem.

Rest assured, the mechanism of the testing would be the same even if our function was calculating the moon-landing trajectory.

Testing demo - use the module

Before we start writing an "automated test", let's see how one could test this code "manually". In reality I see this many times, that people write short snippets of code to check if their real code works properly, but they don't turn these small snippets into real tests.

Basically the question is "How can we use the add function of the mymath module?"

The code is straight forward. We import the module. We import the "sys" module to be able to access the command line arguments. We take two arguments from the command line, call the function, and print the result.

Then, if we would like to make sure our code works well, we can compare that result to some expected result.

Based on this everything works fine.

import mymath
import sys

if len(sys.argv) != 3:
    exit(f"Usage {sys.argv[0]} NUMBER NUMBER")

a = int(sys.argv[1])
b = int(sys.argv[2])

result = mymath.add(a, b)

print(result)

python use_mymath.py 2 2
4

Testing demo: doctest

  • doctest
  • $?
  • %ERRORLEVEL%

The first way we are going to look at is using the "doctest" module. It is a very nice tool that allows us to test our code and to also verify that our documentation is aligned with the code. In addition to that, doctest is a standard module. It comes with every installation of Python so you don't need to worry about installation.

The big drawback is that it is not really useful for anything complex.

So how does it work?

In Python if you add a string immediately after the declaration of the function - meaning the line immediately after the "def" statement - that string becomes the documentation of the function. It can be a one-line string or a multi-line string using triple-quotes.

In the documentation you can write free text and you can also write examples as if one was using the interactive shell of Python. For these examples we have code snippets preceded with 3 greater-than signs, the prompt of the in Python interactive shell. The line immediately after that contains the result that you'd see if you actually typed in the expression into the interactive shell.

Doctest will read your source code, look at all the functions you have and for each function it will look at the documentation of the function. If in the documentation it sees 3 greater-than signs then it will take the content of that line as code to be executed and the next line will be the expected result. Doctest will execute each code snippet and compare it with the expected results. Effectively checking if the examples in your documentation and the implementation are aligned.

We can run doctest in the following way: python -m doctest mymath.py. If all the tests pass, then this execution will print nothing. This lack of positive feedback is a bit strange so you might want to check the so-called "exit code" of the execution. On Unix systems such as Linux and OSX, you'd inspect the $? environment variable while on MS Windows you need to inspect the %ERRORLEVEL% variable. On all of these systems you can use the echo command to inspect the variables. In either case 0 indicates success.

def add(x, y):
    """
    This function will add two numbers together
    >>> add(2, 2)
    4
    >>>
    And here we can have more documentation.
    """
    return x * y

def multiply(x, y):
    return x + y

# Yes, I know there are bugs in this code!



$ python -m doctest mymath.py
$ echo $?
0

> python -m doctest mymath.py
> echo %ERRORLEVEL%
0

Testing demo: doctest with failure

Of course we know that our code is not perfect (to say the least) so at one point someone will complain about the incorrect results received, for example in case they try to add 3 and 3. Before running and fixing the code however it is better to write a test case with the expected correct result that will fail.

So we added another example to the documentation.

If we run the same command as we did earlier we'll get an extensive output on the screen and the exit code with have some value different from 0.

At this point you'd probably also go and fix the code, but you have also increased the number of tests and eliminated the possibility of this failure to return unnoticed.

def add(x, y):
    """
    This function will add two numbers together
    >>> add(2, 2)
    4
    >>> add(3, 3)
    6
    >>>
    And here we can have more documentation.
    """
    return x * y

def multiply(x, y):
    return x + y

# Yes, I know there are bugs in this code!

{% embed include file="src/examples/testing-demo/doctest_fail/mymath.out)

$ python -m doctest mymath.py
$ echo $?
1

> python -m doctest mymath.py
> echo %ERRORLEVEL%
1

Testing demo: Unittest success

  • unittest
  • TestCase
  • assertEqual

Python comes with a built-in module for writing tests. Its name is unittest which might be a bit confusing as this module can be used to any kind of more complex feature-tests and other modules can be also used to write so called unit-tests.

Unlike the doctests that were part of the actual code, the unittest library calls for separate test files. It is recommended that the names of files start with the test_ prefix as that will make it easy for the various testing tools to locate them.

Inside the file you'd need to import both the unittest module and the module that we are testing. mystest in this case.

We need a class with a name that starts with Test and inherits from unittest.TestCase. In the class we can have one or more testing functions. Each one starts with a test_ prefix. Inside the function we can call the function that we are testing and we can compare the result returned by it to some expected value. We can compare them in various ways using the various assert-methods of the unittest.TestCase. In this example we used the assertEqual method as we wanted to make sure the actual return value equals the expected value.

We can run the tests using python -m unittest test_one_with_unittest.py. It will have some output on the screen indicating all the tests passed. The exit-code will be 0 as expected.

import unittest
import mymath

class TestMath(unittest.TestCase):
    def test_math(self):
        self.assertEqual(mymath.add(2, 2), 4)

{% embed include file="src/examples/testing-demo/test_one_with_unittest.out)

$ python -m unittest test_one_with_unittest.py
$ echo $?
0

> python -m unittest test_one_with_unittest.py
> echo %ERRORLEVEL%
0

Testing demo: Unittest failure

When we get the report on the incorrect results when adding 3 and 3, we can added another test-case. We could have added another assertion to the test_math function or we could have created a separare class with its own function, but in this case we opted creating a separate test-function.

We won't go into the pros and contras of each strategy now as we are only interested in the basic technique.

If we run the tests now the output will indicate that it ran 2 test-cases and one of them failed. It even shows use some details about the expected value and the actual value that can be really useful understanding the source of the problem.

Note there is also .F in the output. The dot indicates the test-function that passed, the F indicates the test-function that failed.

The exit code is again different from 0.

BTW this exit-code is used by the various CI systems to understand the results of the tests.

import unittest
import mymath

class TestMath(unittest.TestCase):
    def test_math(self):
        self.assertEqual(mymath.add(2, 2), 4)

    def test_more_math(self):
        self.assertEqual(mymath.add(3, 3), 6)

{% embed include file="src/examples/testing-demo/test_with_unittest.out)

$ python -m unittest test_with_unittest.py
$ echo $?
1

> python -m unittest test_with_unittest.py
> echo %ERRORLEVEL%
1

Testing demo: pytest using classes

  • pytest
  • assert

In our third example we are going to use the pytest module. The only drawback of the pytest module is that it does not come with the installation of Python itself. It is not a huge issue though as you probably install hundreds of other modules as well.

These days Pytest seems like the most popular testing library for Python.

We'll have several examples using Pytest.

In order to use it you'd create a file with a name that starts with test_ prefix. We need to import the module we are testing but we don't need to import pytest. Actually we don't even use pytest inside the code. (At least not in the simple use-cases.) In the file you need to create a class starting with Test, but this class does not need to inherit from any special class. In the class we can have one or more test-functions starting with the prefix test_. In the function we call the function we are testing and we compare the results to the expected results.

We use the built-in assert function of Python to check if the results were true.

No need to learn various specialized assert-statements as we had in the unittest module.

We run the test using the pytest command.

We'll get some output. Here too the single dot after the name of the test file indicates that there was one successful test function.

The exit-code of this execution in 0 as was the case with unittest.

pip install pytest

import mymath

class TestMath():
    def test_math(self):
        assert mymath.add(2, 2) == 4

{% embed include file="src/examples/testing-demo/test_with_pytest_class.out)

$ pytest test_with_pytest_class.py
$ echo $?
0

> pytest test_with_pytest_class.py
> echo %ERRORLEVEL%
0

Testing demo: pytest using classes - failure

Here too we can add additional test-functions to the same test-class. Executing pytest will print .F indicating one passing test-function and one failing test function. We'll get detailed explanation where the failure happened.

The exit-code will be different from 0 helping the CI systems and any other external system to know that the tests have failed.

import mymath

class TestMath():
    def test_math(self):
        assert mymath.add(2, 2) == 4

    def test_more_math(self):
        assert mymath.add(3, 3) == 6

{% embed include file="src/examples/testing-demo/test_with_pytest_class_failure.out)

$ pytest test_with_pytest_class_failure.py
$ echo $?
1

> pytest test_with_pytest_class_failure.py
> echo %ERRORLEVEL%
1

Testing demo: pytest without classes

In the previous example we used a test-class to write our tests, but in reality in many cases we don't need the classes. We could just as well write plain test-functions as in this example.

Test-functions without a class around them are easier to write and understand and they are a lot simplert to graps. So unless you really need the features a class can provide I'd recommend you use functions only. After all our test code should be a lot more simple than our application code.

pip install pytest

import mymath

def test_math():
    assert mymath.add(2, 2) == 4

{% embed include file="src/examples/testing-demo/test_with_pytest.out)

$ pytest test_with_pytest.py
$ echo $?
0

> pytest test_with_pytest.py
> echo %ERRORLEVEL%
0

Testing demo: pytest without classes failure

import mymath

def test_math():
    assert mymath.add(2, 2) == 4

def test_more_math():
    assert mymath.add(3, 3) == 6

{% embed include file="src/examples/testing-demo/test_with_pytest_failure.out)

$ pytest test_with_pytest.py
$ echo $?
1

> pytest test_with_pytest.py
> echo %ERRORLEVEL%
1

Testing demo: Failure in one sub

import mymath

def test_math():
    assert mymath.add(3, 3) == 6
    assert mymath.add(2, 2) == 4

Testing demo: pytest run doctests

The nice thing about pytest that it can also run all the doctests in your module. So you can start your testing journey with doctest and later switch to pytest.

You can easily test your examples in your documentation.

$ pytest --doctest-modules mymath.py

Testing demo: pytest run unittest

Pytest can also run the unit-test. You don't even need to tell it anything special. It will introspect the test code and if it notices tests-classes that are based on unittest it will execute them using the unittest module.

$ pytest test_one_with_unittest.py
$ pytest test_with_unittest.py

Test demo: test coverage

pip install pytest-cover

$ pytest test_with_pytest.py --cov mymath --cov-report html --cov-report term

{% embed include file="src/examples/testing-demo/test_with_pytest_cover.out)

Open htmlcov/index.html

Exercise: Testing demo - anagrams

  • An anagram is a pair of words that are created from exactly the same set of characters, but of different order.
  • For example listen and silent
  • Or bad credit and debit card
  • Given the following module with the is_anagram function write tests for it. (in a file called test_anagram.py" %}
  • Write a failing test as well.
  • Try doctest, unittest, and pytest as well.
def is_anagram(a_word, b_word):
    return sorted(a_word) == sorted(b_word)

Sample code to use the Anagram module.

from anagram import is_anagram
import sys

if len(sys.argv) != 3:
    exit(f"Usage {sys.argv[0]} WORD WORD")

if is_anagram(sys.argv[1], sys.argv[2]):
    print("Anagram")
else:
    print("NOT")

Exercise: Test previous solutions

  • Go back to your solutions to the previous exercises
  • Write tests
  • If you feel it is hard, maybe you need to change the code to make it more testable.

Solution: Testing demo

from anagram import is_anagram

def test_anagram():
    assert is_anagram("silent", "listen")
    assert is_anagram("bad credit", "debit card")

def test_not_anagram():
    assert not is_anagram("abc", "def")

def test_should_be_anagram_spaces():
    assert is_anagram("anagram", "nag a ram")


def test_should_be_anagram_case():
    assert is_anagram("Silent", "Listen")