Python Library 'Unittest': Generate Multiple Tests Programmatically

Python library 'unittest': Generate multiple tests programmatically

Not tested:

class TestPreReqs(unittest.TestCase):
    ...

def create_test (pair):
    def do_test_expected(self):
        self.assertEqual(under_test(pair[0]), pair[1])
    return do_test_expected

for k, pair in enumerate ([(23, 55), (4, 32)]):
    test_method = create_test (pair)
    test_method.__name__ = 'test_expected_%d' % k
    setattr (TestPreReqs, test_method.__name__, test_method)

If you use this often, you could prettify this by using utility functions and/or decorators, I guess. Note that pairs are not an attribute of TestPreReqs object in this example (and so setUp is gone). Rather, they are "hardwired" in a sense to the TestPreReqs class.

How do you generate dynamic (parameterized) unit tests in Python?

This is called "parametrization".

There are several tools that support this approach. E.g.:

pytest's decorator
parameterized

The resulting code looks like this:

from parameterized import parameterized

class TestSequence(unittest.TestCase):
    @parameterized.expand([
        ["foo", "a", "a",],
        ["bar", "a", "b"],
        ["lee", "b", "b"],
    ])
    def test_sequence(self, name, a, b):
        self.assertEqual(a,b)

Which will generate the tests:

test_sequence_0_foo (__main__.TestSequence) ... ok
test_sequence_1_bar (__main__.TestSequence) ... FAIL
test_sequence_2_lee (__main__.TestSequence) ... ok

======================================================================
FAIL: test_sequence_1_bar (__main__.TestSequence)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/parameterized/parameterized.py", line 233, in <lambda>
    standalone_func = lambda *a: func(*(a + p.args), **p.kwargs)
  File "x.py", line 12, in test_sequence
    self.assertEqual(a,b)
AssertionError: 'a' != 'b'

For historical reasons I'll leave the original answer circa 2008):

I use something like this:

import unittest

l = [["foo", "a", "a",], ["bar", "a", "b"], ["lee", "b", "b"]]

class TestSequense(unittest.TestCase):
    pass

def test_generator(a, b):
    def test(self):
        self.assertEqual(a,b)
    return test

if __name__ == '__main__':
    for t in l:
        test_name = 'test_%s' % t[0]
        test = test_generator(t[1], t[2])
        setattr(TestSequense, test_name, test)
    unittest.main()

Python's unittest and dynamic creation of test cases

For this you should use test generators in nose. All you need to do is yield a tuple, with the first being a function and the rest being the args. From the docs here is the example.

def test_evens():
    for i in range(0, 5):
        yield check_even, i, i*3

def check_even(n, nn):
    assert n % 2 == 0 or nn % 2 == 0

How do I run multiple Python test cases in a loop?

Using unittest you can show the difference between two sequences all in one test case.

seq1 = range(1, 11)
seq2 = (fn(j) for j in seq1)
assertSequenceEqual(seq1, seq2)

If that's not flexible enough, using unittest, it is possible to generate multiple tests, but it's a bit tricky.

def fn(i): ...
output = ...

class TestSequence(unittest.TestCase):
    pass

for i in range(1,11):
    testmethodname = 'test_fn_{0}'.format(i)
    testmethod = lambda self: self.assertEqual(fn(i), output[i])
    setattr(TestSequence, testmethodname, testmethod)

Nose makes the above easier through test generators.

import nose.tools

def test_fn():
    for i in range(1, 11):
        yield nose.tools.assert_equals, output[i], fn(i)

How do I run all Python unit tests in a directory?

With Python 2.7 and higher you don't have to write new code or use third-party tools to do this; recursive test execution via the command line is built-in. Put an __init__.py in your test directory and:

python -m unittest discover <test_directory>
# or
python -m unittest discover -s <directory> -p '*_test.py'

You can read more in the python 2.7
or python 3.x unittest documentation.

Update for 2021:

Lots of modern python projects use more advanced tools like pytest. For example, pull down matplotlib or scikit-learn and you will see they both use it.

It is important to know about these newer tools because when you have more than 7000 tests you need:

more advanced ways to summarize what passes, skipped, warnings, errors
easy ways to see how they failed
percent complete as it is running
total run time
ways to generate a test report
etc etc

Python Run Unittest as Package Import Error

It is worthwhile mentioning up front that the Flask documents say to run the application as a package, and set the environment variable: FLASK_APP. The application then runs from the project root: $ python -m flask run. Now the imports will include the application root, such as app.models.transactions. Since the unittest is being run in the same way, as a package from the project root, all imports are resolved there as well.

The crux of the problem can be described in the following way. The test_app.py needs access to sideways imports, but if it runs as a script like:

/sandbox/test$ python test_app.py

it has __name__==__main__. This means that imports, such as from models.transactions import TransactionsModel, will not be resolved because they are sideways and not lower in the hierarchy. To work around this the test_app.py can be run as a package:

/sandbox$ python unittest -m test.test_app

The -m switch is what tells Python to do this. Now the package has access to app.model because it is running in /sandbox. The imports in test_app.py must reflect this change and become something like:

from app.models.transactions import TransactionsModel

To make the test run, the imports in the application must now be relative. For example, in app.resources:

from ..models.transactions import TransactionsModel

So the tests run successfully, but if the application is run it fails! This is the crux to the problem. When the application is run as a script from /sandbox/app$ python app.py it hits this relative import ..models.transactions and returns an error that the program is trying to import above the top level. Fix one, and break the other.

How can one work around this without having to set the PYTHONPATH? A possible solution is to use a conditional in the package __init__.py to do conditional imports. An example of what this looks like for the resources package is:

if __name__ == 'resources':
    from models.transactions import TransactionsModel
    from controllers.transactions import get_transactions
elif __name__ == 'app.resources':
    from ..models.transactions import TransactionsModel
    from ..controllers.transactions import get_transactions

The last obstacle to overcome is how do we get this pulled into the resources.py. The imports done in the __init__.py are bound to that file and are not available to resources.py. Normally one would include the following import in resources.py:

import resources

But, again, is it resources or app.resources? It would seem like the difficulty has just moved further down the road for us. The tools offered by importlib can help here, for example the following will make the correct import:

from importlib import import_module
import_module(__name__)

There are other methods that can be used. For example,

TransactionsModel = getattr(import_module(__name__), 'TransactionsModel')

This fixed the error in the present context.

Another, more direct solution is using absolute imports in the modules themselves. For example, in resources:

models_root = os.path.join(os.path.dirname(__file__), '..', 'models')
fp, file_path, desc = imp.find_module(module_name, [models_root])
TransactionsModel = imp.load_module(module_name, fp, file_path, 
    desc).TransactionsModel
TransactionType = imp.load_module(module_name, fp, file_path, 
    desc).TransactionType

Just a note concerning changing the PYTHONPATH with sys.path.append(app_root)
in resources.py. This works well and is a few lines of code located where it needs to be. Additionally, it only changes the path for the executing file and reverts back when finished. Seems like a good use case for unittest. One concern might be when the application is moved to different environments.

unittest - run the same test for a list of inputs and outputs

Check out DDT (Data-Driven/Decorated Tests).

DDT allows you to multiply a test case by running it with different test data, making it appear as multiple test cases.

consider this example, using DDT:

import unittest

from ddt import ddt, data, unpack

@ddt
class TestName(unittest.TestCase):

        # simple decorator usage:
        @data(1, 2)
        def test_greater_than_zero(self, value):
            self.assertGreater(value, 0)

        # passing data in tuples to achieve the 
        # scenarios from your given example:
        @data(('Bob', 'Bob'), ('Alice', 'Alice'))
        @unpack
        def test_name(self, first_value, second_value):
            name, expected_name = first_value, second_value
            self.assertEquals(name, expected_name)

if __name__ == '__main__':
        unittest.main(verbosity=2)

I defined 2 test methods in the above code, but 4 test cases will be run, using the data I supplied in the decorator.

Output:

test_greater_than_zero_1 (__main__.TestName) ... ok
test_greater_than_zero_2 (__main__.TestName) ... ok
test_name_('Alice', 'Alice') (__main__.TestName) ... ok
test_name_('Bob', 'Bob') (__main__.TestName) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK

Paramaterize unit tests in python

Note that this is precisely one of the most common uses of the recent addition of funcargs in py.test.

In your case you'd get:

def pytest_generate_tests(metafunc):
    if 'data' in metafunc.funcargnames:
        metafunc.parametrize('data', [1,2,3,4])

def test_data(data):
    assert data > 0

[EDIT] I should probably add that you can also do that as simply as

@pytest.mark.parametrize('data', [1,2,3,4])
def test_data(data):
    assert data > 0

So I'd say that py.test is a great framework for parameterized unit testing...

How do you generate dynamic (parameterized) unit tests in Python?

This is called "parametrization".

There are several tools that support this approach. E.g.:

pytest's decorator
parameterized

The resulting code looks like this:

from parameterized import parameterized

class TestSequence(unittest.TestCase):
    @parameterized.expand([
        ["foo", "a", "a",],
        ["bar", "a", "b"],
        ["lee", "b", "b"],
    ])
    def test_sequence(self, name, a, b):
        self.assertEqual(a,b)

Which will generate the tests:

test_sequence_0_foo (__main__.TestSequence) ... ok
test_sequence_1_bar (__main__.TestSequence) ... FAIL
test_sequence_2_lee (__main__.TestSequence) ... ok

======================================================================
FAIL: test_sequence_1_bar (__main__.TestSequence)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/parameterized/parameterized.py", line 233, in <lambda>
    standalone_func = lambda *a: func(*(a + p.args), **p.kwargs)
  File "x.py", line 12, in test_sequence
    self.assertEqual(a,b)
AssertionError: 'a' != 'b'

For historical reasons I'll leave the original answer circa 2008):

I use something like this:

import unittest

l = [["foo", "a", "a",], ["bar", "a", "b"], ["lee", "b", "b"]]

class TestSequense(unittest.TestCase):
    pass

def test_generator(a, b):
    def test(self):
        self.assertEqual(a,b)
    return test

if __name__ == '__main__':
    for t in l:
        test_name = 'test_%s' % t[0]
        test = test_generator(t[1], t[2])
        setattr(TestSequense, test_name, test)
    unittest.main()

Why PyTest is not collecting tests (collected 0 items)?

pytest gathers tests according to a naming convention. By default any file that is to contain tests must be named starting with test_, classes that hold tests must be named starting with Test, and any function in a file that should be treated as a test must also start with test_.

If you rename your test file to test_sorts.py and rename the example function you provide above as test_integer_sort, then you will find it is automatically collected and executed.

This test collecting behavior can be changed to suit your desires. Changing it will require learning about configuration in pytest.

Python Library 'Unittest': Generate Multiple Tests Programmatically