Python Unittest.Testcase Execution Order

Python unittest.TestCase execution order

Don't make them independent tests - if you want a monolithic test, write a monolithic test.

class Monolithic(TestCase):
  def step1(self):
      ...

  def step2(self):
      ...

  def _steps(self):
    for name in dir(self): # dir() result is implicitly sorted
      if name.startswith("step"):
        yield name, getattr(self, name) 

  def test_steps(self):
    for name, step in self._steps():
      try:
        step()
      except Exception as e:
        self.fail("{} failed ({}: {})".format(step, type(e), e))

If the test later starts failing and you want information on all failing steps instead of halting the test case at the first failed step, you can use the subtests feature: https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests

(The subtest feature is available via unittest2 for versions prior to Python 3.4: https://pypi.python.org/pypi/unittest2 )

Execution order on Python unittest

Better do not do it.

Tests should be independent.

To do what you want best would be to put the code into functions that are called by the test.

Like that:

def assert_can_log_in(self):
    ...

def test_1(self):
    self.assert_can_log_in()
    ...

def test_2(self):
    self.assert_can_log_in()
    ...

Or even to split the test class and put the assertions into the setUp function.

class LoggedInTests(unittest.TestCase):
    def setUp(self):
        # test for login or not - your decision

    def test_1(self):
        ...

When I split the class I often write more and better tests because the tests are split up and I can see better through all the cases that should be tested.

How to run unittest test cases in the order they are declared

The solution is to create a TestSuite explicitly, instead of letting unittest.main() follow all its default test discovery and ordering behavior. Here's how I got it to work:

import unittest

class TestCaseB(unittest.TestCase):
    def runTest(self):
        print("running test case B")

class TestCaseA(unittest.TestCase):
    def runTest(self):
        print("running test case A")

import inspect
def get_decl_line_no(cls):
    return inspect.getsourcelines(cls)[1]

# get all test cases defined in this module
test_case_classes = list(filter(lambda c: c.__name__ in globals(), 
                                unittest.TestCase.__subclasses__()))

# sort them by decl line no
test_case_classes.sort(key=get_decl_line_no)

# make into a suite and run it
suite = unittest.TestSuite(cls() for cls in test_case_classes)
unittest.TextTestRunner().run(suite)

This gives the desired output:

running test case B
.running test case A
.
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK

It is important to note that the test method in each class must be named runTest.

Execution order of python unitests by their declaration

To start with unit tests are supposed to be independent. So must be python-unittest. Tests executed through python-unittest should be designed in such a way that they should be able to be run independently. Pure unit tests offer a benefit that when they fail, they often depicts what exactly went wrong. Still we tend to write functional tests, integration tests, and system tests with the unittest framework and these tests won't be feasible to run without ordering them since Selenium automates the Browsing Context. To achieve the ordering, you at-least need to use a better naming convention for the testnames, as an example: test_1, test_2, test_3, etc and this works because the tests are sorted respect to the built-in ordering for strings.

However, as per your observation the problem appears with test_10 and so on where sorting order seems to break. As an example, among 3 tests with name as test_1, test_2 and test_10, it seems unittest executes test_10 before test_2:

• Code:

  import unittest
  
  class Test(unittest.TestCase):
  
      @classmethod
      def setUp(self):
          print("I'm in setUp")
  
      def test_1(self):
          print("I'm in test 1")
  
      def test_2(self):
          print("I'm in test 2")
  
      def test_10(self):
          print("I'm in test 10")
  
      @classmethod
      def tearDown(self):
          print("I'm in tearDown")
  
  if __name__ == "__main__":
      unittest.main()
  

• Console Output:

  Finding files... done.
  Importing test modules ... done.
  
  I'm in setUp
  I'm in test 1
  I'm in tearDown
  I'm in setUp
  I'm in test 10
  I'm in tearDown
  I'm in setUp
  I'm in test 2
  I'm in tearDown
  ----------------------------------------------------------------------
  Ran 3 tests in 0.001s
  
  OK
  

---

Solution

Different solutions were offered in different discussions and some of them are as follows:

• @max in the discussion Unittest tests order suggested to set the sortTestMethodsUsing to None as follows:

  import unittest
  unittest.TestLoader.sortTestMethodsUsing = None
  

• @atomocopter in the discussion changing order of unit tests in Python suggested to set the sortTestMethodsUsing to some value as follows:

  import unittest
  unittest.TestLoader.sortTestMethodsUsing = lambda _, x, y: cmp(y, x)
  

• @ElmarZander in the discussion Unittest tests order suggested to use nose and write your testcases as functions (and not as methods of some TestCase derived class) nose doesn't fiddle with the order, but uses the order of the functions as defined in the file.

• @Keiji in the discussion Controlling the order of unittest.TestCases mentions:

sortTestMethodsUsing expects a function like Python 2's cmp, which
has no equivalent in Python 3 (I went to check if Python 3 had a <=>
spaceship operator yet, but apparently not; they expect you to rely on
separate comparisons for < and ==, which seems much a backwards
step...). The function takes two arguments to compare, and must return
a negative number if the first is smaller. Notably in this particular
case, the function may assume that the arguments are never equal, as
unittest will not put duplicates in its list of test names.

With this in mind, here's the simplest way I found to do it, assuming
you only use one TestCase class:

def make_orderer():
    order = {}

    def ordered(f):
        order[f.__name__] = len(order)
        return f

    def compare(a, b):
        return [1, -1][order[a] < order[b]]

    return ordered, compare

ordered, compare = make_orderer()
unittest.defaultTestLoader.sortTestMethodsUsing = compare

Then, annotate each test method with @ordered:

class TestMyClass(unittest.TestCase):
    @ordered
    def test_run_me_first(self):
        pass

    @ordered
    def test_do_this_second(self):
        pass

    @ordered
    def test_the_final_bits(self):
        pass

if __name__ == '__main__':
    unittest.main()

This relies on Python calling annotations in the order the annotated
functions appear in the file. As far as I know, this is intended, and
I'd be surprised if it changed, but I don't actually know if it's
guaranteed behavior. I think this solution will even work in Python 2
as well, for those who are unfortunately stuck with it, though I
haven't had a chance to test this.

If you have multiple TestCase classes, you'll need to run ordered,
compare = make_orderer() once per class before the class
definition, though how this can be used with sortTestMethodsUsing
will be more tricky and I haven't yet been able to test this either.

For the record, the code I am testing does not rely on the test
order being fixed - and I fully understand that you shouldn't rely on
test order, and this is the reason people use to avoid answering this
question. The order of my tests could be randomised and it'd work just
as well. However, there is one very good reason I'd like the order to
be fixed to the order they're defined in the file: it makes it so much
easier to see at a glance which tests failed.

Unittest tests order

You can disable it by setting sortTestMethodsUsing to None:

import unittest
unittest.TestLoader.sortTestMethodsUsing = None

For pure unit tests, you folks are right; but for component tests and integration tests...
I do not agree that you shall assume nothing about the state.
What if you are testing the state?

For example, your test validates that a service is auto-started upon installation. If in your setup, you start the service, then do the assertion, and then you are no longer testing the state, but you are testing the "service start" functionality.

Another example is when your setup takes a long time or requires a lot of space and it just becomes impractical to run the setup frequently.

Many developers tend to use "unit test" frameworks for component testing...so stop and ask yourself, am I doing unit testing or component testing?

Order of tests in python unittest

Option 1.

One solution to this (as a workaround) was given here - which suggests writing the tests in numbered methods step1, step2, etc., then collecting and storing them via dir(self) and yielding them to one test_ method which trys each.

Not ideal but does what you expect. Each test sequence has to be a single TestClass (or adapt the method given there to have more than one sequence generating method).

Option 2.

Another solution, also in the linked question, is you name your tests alphabetically+numerically sorted so that they will execute in that order.

But in both cases, write monolithic tests, each in their own Test Class.

P.S. I agree with all the comments that say unit testing shouldn't be done this way; but there are situations where unit test frameworks (like unittest and pytest) get used to do integration tests which need modular independent steps to be useful. Also, if QA can't influence Dev to write modular code, these kinds of things have to be done.

Python unittest run TestCases in a specific order

The answer to this question is to use a unittest.TestSuite which preserves the order in which tests are added. You can do the following:

loader = unittest.TestLoader()
suite = unittest.TestSuite()
tests_to_run = [TestCaseA, TestCaseB, TestCaseC]
for test in tests_to_run:
    suite.addTests(loader.loadTestsFromTestCase(test)
runner = unittest.TextTestRunner()
runner.run(suite)

How to test the execution order of several functions in python?

Ask yourself the question, "why is the order important?" If you can't tell from outside the difference of calls, you don't have to test them.
If these are for example database updates you have to write a database mockup which logs the order of updates, or make a select statement, that can check, if the updates ocured in the correct order.

Asserting execution order in python unittest

One workaround would to be to create a separate mock object, attach methods to it and use assert_has_calls() to check the call order:

converter = Converter(encoded_file_path)
converter.change_to_temp_dir = Mock()
converter.do_stuff = Mock()
converter.change_to_original_dir = Mock()

m = Mock()
m.configure_mock(first=converter.change_to_temp_dir,
                 second=converter.do_stuff,
                 third=converter.change_to_original_dir)

converter.run()
m.assert_has_calls([call.first(), call.second(), call.third()])

---

Related Topics