Changing Order of Unit Tests in Python

Unittest tests order

You can disable it by setting sortTestMethodsUsing to None:

import unittest
unittest.TestLoader.sortTestMethodsUsing = None

For pure unit tests, you folks are right; but for component tests and integration tests...
I do not agree that you shall assume nothing about the state.
What if you are testing the state?

For example, your test validates that a service is auto-started upon installation. If in your setup, you start the service, then do the assertion, and then you are no longer testing the state, but you are testing the "service start" functionality.

Another example is when your setup takes a long time or requires a lot of space and it just becomes impractical to run the setup frequently.

Many developers tend to use "unit test" frameworks for component testing...so stop and ask yourself, am I doing unit testing or component testing?

Python unittest.TestCase execution order

Don't make them independent tests - if you want a monolithic test, write a monolithic test.

class Monolithic(TestCase):
  def step1(self):
      ...

  def step2(self):
      ...

  def _steps(self):
    for name in dir(self): # dir() result is implicitly sorted
      if name.startswith("step"):
        yield name, getattr(self, name) 

  def test_steps(self):
    for name, step in self._steps():
      try:
        step()
      except Exception as e:
        self.fail("{} failed ({}: {})".format(step, type(e), e))

If the test later starts failing and you want information on all failing steps instead of halting the test case at the first failed step, you can use the subtests feature: https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests

(The subtest feature is available via unittest2 for versions prior to Python 3.4: https://pypi.python.org/pypi/unittest2 )

changing order of unit tests in Python

You can change the default sorting behavior by setting a custom comparison function. In unittest.py you can find the class variable unittest.TestLoader.sortTestMethodsUsing which is set to the builtin function cmp by default.

For example you can revert the execution order of your tests with doing this:

import unittest
unittest.TestLoader.sortTestMethodsUsing = lambda _, x, y: cmp(y, x)

Order of tests in python unittest

Option 1.

One solution to this (as a workaround) was given here - which suggests writing the tests in numbered methods step1, step2, etc., then collecting and storing them via dir(self) and yielding them to one test_ method which trys each.

Not ideal but does what you expect. Each test sequence has to be a single TestClass (or adapt the method given there to have more than one sequence generating method).

Option 2.

Another solution, also in the linked question, is you name your tests alphabetically+numerically sorted so that they will execute in that order.

But in both cases, write monolithic tests, each in their own Test Class.

P.S. I agree with all the comments that say unit testing shouldn't be done this way; but there are situations where unit test frameworks (like unittest and pytest) get used to do integration tests which need modular independent steps to be useful. Also, if QA can't influence Dev to write modular code, these kinds of things have to be done.

Execution order of python unitests by their declaration

To start with unit tests are supposed to be independent. So must be python-unittest. Tests executed through python-unittest should be designed in such a way that they should be able to be run independently. Pure unit tests offer a benefit that when they fail, they often depicts what exactly went wrong. Still we tend to write functional tests, integration tests, and system tests with the unittest framework and these tests won't be feasible to run without ordering them since Selenium automates the Browsing Context. To achieve the ordering, you at-least need to use a better naming convention for the testnames, as an example: test_1, test_2, test_3, etc and this works because the tests are sorted respect to the built-in ordering for strings.

However, as per your observation the problem appears with test_10 and so on where sorting order seems to break. As an example, among 3 tests with name as test_1, test_2 and test_10, it seems unittest executes test_10 before test_2:

Code:

import unittest

class Test(unittest.TestCase):

    @classmethod
    def setUp(self):
        print("I'm in setUp")

    def test_1(self):
        print("I'm in test 1")

    def test_2(self):
        print("I'm in test 2")

    def test_10(self):
        print("I'm in test 10")

    @classmethod
    def tearDown(self):
        print("I'm in tearDown")

if __name__ == "__main__":
    unittest.main()

Console Output:

Finding files... done.
Importing test modules ... done.

I'm in setUp
I'm in test 1
I'm in tearDown
I'm in setUp
I'm in test 10
I'm in tearDown
I'm in setUp
I'm in test 2
I'm in tearDown
----------------------------------------------------------------------
Ran 3 tests in 0.001s

OK

Solution

Different solutions were offered in different discussions and some of them are as follows:

@max in the discussion Unittest tests order suggested to set the sortTestMethodsUsing to None as follows:
```
import unittest
unittest.TestLoader.sortTestMethodsUsing = None
```
@atomocopter in the discussion changing order of unit tests in Python suggested to set the sortTestMethodsUsing to some value as follows:
```
import unittest
unittest.TestLoader.sortTestMethodsUsing = lambda _, x, y: cmp(y, x)
```
@ElmarZander in the discussion Unittest tests order suggested to use nose and write your testcases as functions (and not as methods of some TestCase derived class) nose doesn't fiddle with the order, but uses the order of the functions as defined in the file.
@Keiji in the discussion Controlling the order of unittest.TestCases mentions:

sortTestMethodsUsing expects a function like Python 2's cmp, which
has no equivalent in Python 3 (I went to check if Python 3 had a <=>
spaceship operator yet, but apparently not; they expect you to rely on
separate comparisons for < and ==, which seems much a backwards
step...). The function takes two arguments to compare, and must return
a negative number if the first is smaller. Notably in this particular
case, the function may assume that the arguments are never equal, as
unittest will not put duplicates in its list of test names.

With this in mind, here's the simplest way I found to do it, assuming
you only use one TestCase class:

def make_orderer():
    order = {}

    def ordered(f):
        order[f.__name__] = len(order)
        return f

    def compare(a, b):
        return [1, -1][order[a] < order[b]]

    return ordered, compare

ordered, compare = make_orderer()
unittest.defaultTestLoader.sortTestMethodsUsing = compare

Then, annotate each test method with @ordered:

class TestMyClass(unittest.TestCase):
    @ordered
    def test_run_me_first(self):
        pass

    @ordered
    def test_do_this_second(self):
        pass

    @ordered
    def test_the_final_bits(self):
        pass

if __name__ == '__main__':
    unittest.main()

This relies on Python calling annotations in the order the annotated
functions appear in the file. As far as I know, this is intended, and
I'd be surprised if it changed, but I don't actually know if it's
guaranteed behavior. I think this solution will even work in Python 2
as well, for those who are unfortunately stuck with it, though I
haven't had a chance to test this.

If you have multiple TestCase classes, you'll need to run ordered, compare = make_orderer() once per class before the class
definition, though how this can be used with sortTestMethodsUsing
will be more tricky and I haven't yet been able to test this either.

For the record, the code I am testing does not rely on the test
order being fixed - and I fully understand that you shouldn't rely on
test order, and this is the reason people use to avoid answering this
question. The order of my tests could be randomised and it'd work just
as well. However, there is one very good reason I'd like the order to
be fixed to the order they're defined in the file: it makes it so much
easier to see at a glance which tests failed.

How to run unittest test cases in the order they are declared

The solution is to create a TestSuite explicitly, instead of letting unittest.main() follow all its default test discovery and ordering behavior. Here's how I got it to work:

import unittest

class TestCaseB(unittest.TestCase):
    def runTest(self):
        print("running test case B")

class TestCaseA(unittest.TestCase):
    def runTest(self):
        print("running test case A")

import inspect
def get_decl_line_no(cls):
    return inspect.getsourcelines(cls)[1]

# get all test cases defined in this module
test_case_classes = list(filter(lambda c: c.__name__ in globals(), 
                                unittest.TestCase.__subclasses__()))

# sort them by decl line no
test_case_classes.sort(key=get_decl_line_no)

# make into a suite and run it
suite = unittest.TestSuite(cls() for cls in test_case_classes)
unittest.TextTestRunner().run(suite)

This gives the desired output:

running test case B
.running test case A
.
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

It is important to note that the test method in each class must be named runTest.

Execution order on Python unittest

Better do not do it.

Tests should be independent.

To do what you want best would be to put the code into functions that are called by the test.

Like that:

def assert_can_log_in(self):
    ...

def test_1(self):
    self.assert_can_log_in()
    ...

def test_2(self):
    self.assert_can_log_in()
    ...

Or even to split the test class and put the assertions into the setUp function.

class LoggedInTests(unittest.TestCase):
    def setUp(self):
        # test for login or not - your decision

    def test_1(self):
        ...

When I split the class I often write more and better tests because the tests are split up and I can see better through all the cases that should be tested.

How to sort unittest TestCases properly?

^{After doing a lot of research, aided greatly by SO and Python's help and not at all by unittest's documentation, I got the answer I initially wanted so I figured I'd write this up to help others, because this is a fair (and apparently common) request.}

To run a specific test case, we need to make a new TestSuite for that TestCase. Let's do that for any number of TestCases:

def suiteFactory(*testcases):

    ln    = lambda f: getattr(tc, f).__code__.co_firstlineno
    lncmp = lambda a, b: ln(a) - ln(b)

    test_suite = unittest.TestSuite()
    for tc in testcases:
        test_suite.addTest(unittest.makeSuite(tc, sortUsing=lncmp))

    return test_suite

It's pleasingly simple:

Define a function to get a function's line number.
In Python 3, the attribute we're after changed from func.im_func.func_code.co_firstlineno to func.__code__.co_firstlineno, which you can see using dir(anyFunction).
Define a function to sort two arguments based on cmping their line numbers. cmp isn't in Python 3 on account of People Can Do Math, so I've just done exactly what it did in the interest of readability.
Make a new blank TestSuite(), and give it a TestCase or ten, then tell it to sort that TestCase's methods using point #2: cmping their line numbers.

Now we need to sort the file's TestCase subclasses.

To do this, we can look at the globals() and their attributes.

def caseFactory():

    from inspect import findsource

    g = globals().copy()

    cases = [
        g[obj] for obj in g
            if obj.startswith("Test")
            and issubclass(g[obj], unittest.TestCase)
    ]

    ordered_cases = sorted(cases, key=lambda f: findsource(f)[1])

    return ordered_cases

This will just get all the subclasses of unittest.TestCase that begin with Test, or any naming convention you prefer, and then sort them by their line number: findsource(object) returns source code, and line number as index 1, which is what we care about.

To wrap it into something we can use:

if __name__ == "__main__":
    cases = suiteFactory(*caseFactory())
    runner = unittest.TextTestRunner(verbosity=2)
    runner.run(cases)

~~This does compartmentalise the output, but probably a good thing if the lowest-level tests are at the top (or bottom, or somewhere) in the file, and should run before the higher-level tests.~~

So the full output is then:

test_run_me_first (__main__.Test_MyTests) ... ok
test_2nd_run_me (__main__.Test_MyTests) ... ok
test_and_me_last (__main__.Test_MyTests) ... ok
test_first (__main__.Test_AnotherClass) ... ok
test_after_first (__main__.Test_AnotherClass) ... ok
test_de_last_ding (__main__.Test_AnotherClass) ... ok

----------------------------------------------------------------------
Ran 6 tests in 0.000s

OK

Success!

You can find a more interesting version of this on Github gist.

Asserting execution order in python unittest

One workaround would to be to create a separate mock object, attach methods to it and use assert_has_calls() to check the call order:

converter = Converter(encoded_file_path)
converter.change_to_temp_dir = Mock()
converter.do_stuff = Mock()
converter.change_to_original_dir = Mock()

m = Mock()
m.configure_mock(first=converter.change_to_temp_dir,
                 second=converter.do_stuff,
                 third=converter.change_to_original_dir)

converter.run()
m.assert_has_calls([call.first(), call.second(), call.third()])