Test Case Execution Order in Pytest

Test case execution order in pytest

In general you can configure the behavior of basically any part of pytest using its well-specified hooks.

In your case, you want the "pytest_collection_modifyitems" hook, which lets you re-order collected tests in place.

That said, it does seem like ordering your tests should be easier -- this is Python after all! So I wrote a plugin for ordering tests: "pytest-ordering". Check out the docs or install it from pypi. Right now I recommend using @pytest.mark.first and @pytest.mark.second, or one of the @pytest.mark.order# markers, but I have some ideas about more useful APIs. Suggestions welcome :)

Edit: pytest-ordering seems abandoned at the moment, you can also check out pytest-order (a fork of the original project by the author).

Edit2: In pytest-order, only one marker (order) is supported, and the mentioned examples would read @pytest.mark.order("first"), @pytest.mark.order("second"), or @pytest.mark.order(#) (with # being any number).

pytest execution order of tests in a class

By default the tests will be executed in the order they are defined in the class/module. In your case:

test_class.py::TestClass::test_one PASSED
test_class.py::TestClass::test_two PASSED

Consider that in general it's a bad practise writing tests that are dependent on each other. If later tests are run in parallel, you will have flakiness, or if you install a plugin for random test execution e.g. https://pypi.python.org/pypi/pytest-randomly, or if you leave the project and someone else will have to debug tests that would start failing out of the blue.

I'd recommend combining two tests into one. All that matters is you have some test scenario. Does it matter if you have 2 tests or 1 if you still have same confidence in your code?

Run Pytest Classes in Custom Order

Approach

You can use the pytest_collection_modifyitems hook to modify the order of collected tests (items) in place. This has the additional benefit of not having to install any third party libraries.

With some custom logic, this allows to sort by class.

Full example

Say we have three test classes:

1. TestExtract
2. TestTransform
3. TestLoad

Say also that, by default, the test order of execution would be alphabetical, i.e.:

TestExtract -> TestLoad -> TestTransform

which does not work for us due to test class interdependencies.

We can add pytest_collection_modifyitems to conftest.py as follows to enforce our desired execution order:

# conftest.py
def pytest_collection_modifyitems(items):
    """Modifies test items in place to ensure test classes run in a given order."""
    CLASS_ORDER = ["TestExtract", "TestTransform", "TestLoad"]
    class_mapping = {item: item.cls.__name__ for item in items}

    sorted_items = items.copy()
    # Iteratively move tests of each class to the end of the test queue
    for class_ in CLASS_ORDER:
        sorted_items = [it for it in sorted_items if class_mapping[it] != class_] + [
            it for it in sorted_items if class_mapping[it] == class_
        ]
    items[:] = sorted_items

Some comments on the implementation details:

• Test classes can live in different modules
• CLASS_ORDER does not have to be exhaustive. You can reorder just those classes on which to want to enforce an order (but note: if reordered, any non-reordered class will execute before any reordered class)
• The test order within the classes is kept unchanged
• It is assumed that test classes have unique names
• items must be modified in place, hence the final items[:] assignment

Execute pytest in order

I think the project maintainer does not have much time to support the project anymore: last update was 5 months ago and the issue tracker is filled with related opened issues. One of them is closely related to your problem:

• pytest.mark.order1 doesn't work after updating from 0.3 to 0.4

---

I've actually found that the run(order=N) works better but don't use negative indexing - they are not going to work. The following works for me:

import pytest

@pytest.mark.run(order=3)
def test_three():
    assert True

@pytest.mark.run(order=4)
def test_four():
    assert True

@pytest.mark.run(order=2)
def test_two():
    assert True

@pytest.mark.run(order=1)
def test_one():
    assert True

Produces:

test.py::test_one PASSED
test.py::test_two PASSED
test.py::test_three PASSED
test.py::test_four PASSED

---

Related Topics