Unit Testing for C++ Code - Tools and Methodology

Unit testing for C++ code - Tools and methodology

Applying unit tests to legacy code was the very reason Working Effectively with Legacy Code was written. Michael Feathers is the author - as mentioned in other answers, he was involved in the creation of both CppUnit and CppUnitLite.

Unit Testing C Code

One unit testing framework in C is Check; a list of unit testing frameworks in C can be found here and is reproduced below. Depending on how many standard library functions your runtime has, you may or not be able to use one of those.

AceUnit

AceUnit (Advanced C and Embedded Unit) bills itself as a comfortable C code unit test framework. It tries to mimick JUnit 4.x and includes reflection-like capabilities. AceUnit can be used in resource constraint environments, e.g. embedded software development, and importantly it runs fine in environments where you cannot include a single standard header file and cannot invoke a single standard C function from the ANSI / ISO C libraries. It also has a Windows port. It does not use forks to trap signals, although the authors have expressed interest in adding such a feature. See the AceUnit homepage.

GNU Autounit

Much along the same lines as Check, including forking to run unit tests in a separate address space (in fact, the original author of Check borrowed the idea from GNU Autounit). GNU Autounit uses GLib extensively, which means that linking and such need special options, but this may not be a big problem to you, especially if you are already using GTK or GLib. See the GNU Autounit homepage.

cUnit

Also uses GLib, but does not fork to protect the address space of unit tests.

CUnit

Standard C, with plans for a Win32 GUI implementation. Does not currently fork or otherwise protect the address space of unit tests. In early development. See the CUnit homepage.

CuTest

A simple framework with just one .c and one .h file that you drop into your source tree. See the CuTest homepage.

CppUnit

The premier unit testing framework for C++; you can also use it to test C code. It is stable, actively developed, and has a GUI interface. The primary reasons not to use CppUnit for C are first that it is quite big, and second you have to write your tests in C++, which means you need a C++ compiler. If these don’t sound like concerns, it is definitely worth considering, along with other C++ unit testing frameworks. See the CppUnit homepage.

embUnit

embUnit (Embedded Unit) is another unit test framework for embedded systems. This one appears to be superseded by AceUnit. Embedded Unit homepage.

MinUnit

A minimal set of macros and that’s it! The point is to show how easy it is to unit test your code. See the MinUnit homepage.

CUnit for Mr. Ando

A CUnit implementation that is fairly new, and apparently still in early development. See the CUnit for Mr. Ando homepage.

This list was last updated in March 2008.

More frameworks:

CMocka

CMocka is a test framework for C with support for mock objects. It's easy to use and setup.

See the CMocka homepage.

Criterion

Criterion is a cross-platform C unit testing framework supporting automatic test registration, parameterized tests, theories, and that can output to multiple formats, including TAP and JUnit XML. Each test is run in its own process, so signals and crashes can be reported or tested if needed.

See the Criterion homepage for more information.

HWUT

HWUT is a general Unit Test tool with great support for C. It can help to create Makefiles, generate massive test cases coded in minimal 'iteration tables', walk along state machines, generate C-stubs and more. The general approach is pretty unique: Verdicts are based on 'good stdout/bad stdout'. The comparison function, though, is flexible. Thus, any type of script may be used for checking. It may be applied to any language that can produce standard output.

See the HWUT homepage.

CGreen

A modern, portable, cross-language unit testing and mocking framework for C and C++. It offers an optional BDD notation, a mocking library, the ability to run it in a single process (to make debugging easier). A test runner which discover automatically the test functions is available. But you can create your own programmatically.

All those features (and more) are explained in the CGreen manual.

Wikipedia gives a detailed list of C unit testing frameworks under List of unit testing frameworks: C

Check Unit testing in C: Best way to test static methods

It's debatable whether or not static functions should be tested, as they aren't part of the public API.

I test static functions by including the unit-under-test, rather than linking against it:

foo.c (Unit under test)

static int foo(int x)
{
    return x;
}

/* ... */

test_foo.c

#include "foo.c"

void test_foo(void)
{
    assert( foo(42) == 42 );
}

int main(void)
{
    test_foo();
}

Compile just that test:

$ gcc -Wall -Werror -o test_foo test_foo.c
$ ./test_foo

Writing unit tests for C code

In general, I agree with Wes's answer - it is going to be much harder to add tests to code that isn't written with tests in mind. There's nothing inherent in C that makes it impossible to test - but, because C doesn't force you to write in a particular style, it's also very easy to write C code that is difficult to test.

In my opinion, writing code with tests in mind will encourage shorter functions, with few arguments, which helps alleviate some of the pain in your examples.

First, you'll need to pick a unit testing framework. There are a lot of examples in this question (though sadly a lot of the answers are C++ frameworks - I would advise against using C++ to test C).

I personally use TestDept, because it is simple to use, lightweight, and allows stubbing. However, I don't think it is very widely used yet. If you're looking for a more popular framework, many people recommend Check - which is great if you use automake.

Here are some specific answers for your use cases:

Passing around a large 'context' struct pointer

For this case, you can build an instance of the struct with the pre conditions manually set, then check the status of the struct after the function has run. With short functions, each test will be fairly straightforward.

No easy way to test failure on dependent functions

I think this is one of the biggest hurdles with unit testing C.
I've had success using TestDept, which allows run time stubbing of dependent functions. This is great for breaking up tightly coupled code. Here's an example from their documentation:

void test_stringify_cannot_malloc_returns_sane_result() {
  replace_function(&malloc, &always_failing_malloc);
  char *h = stringify('h');
  assert_string_equals("cannot_stringify", h);
}

Depending on your target environment, this may or may not work for you. See their documentation for more details.

Functions with lots of parameters

This probably isn't the answer you're looking for, but I would just break these up into smaller functions with fewer parameters. Much much easier to test.

Static or hidden functions

It's not super clean, but I have tested static functions by including the source file directly, enabling calls of static functions. Combined with TestDept for stubbing out anything not under test, this works fairly well.

 #include "implementation.c"

 /* Now I can call foo(), defined static in implementation.c */

---

A lot of C code is legacy code with few tests - and in those cases, it is generally easier to add integration tests that test large parts of the code first, rather than finely grained unit tests. This allows you to start refactoring the code underneath the integration test to a unit-testable state - though it may or may not be worth the investment, depending on your situation. Of course, you'll want to be able to add unit tests to any new code written during this period, so having a solid framework up and running early is a good idea.

If you are working with legacy code, this book
(Working effectively with legacy code by Michael Feathers) is great further reading.

Unit Testing Code Coverage

Code coverage refers to the execution of the code from within the test execution. It doesn't actually have anything to do with what is tested, just that it is executed at some point while your tests are running.

Code coverage will return 100% if every command in that method is executed at any point during your testing process. If you want to review the code coverage (using the VS 2013 Ultimate code coverage tool) for an individual test, selected it in the Test Manager window and click Test/Analyze Code Coverage/Selected Tests to see what that particular test covers.

Unit testing in C++

Here are similar questions that you may want to look at:

• Unit testing for C++ code - Tools and methodology

• C++ unit testing framework

I recommend you check out Google's unit testing framework in addition to CppUnit.

How to write good Unit Tests?

An important point (that I didn't realise in the beginning) is that Unit Testing is a testing technique that can be used by itself, without the need to apply the full Test Driven methodology.

For example, you have a legacy application that you want to improve by adding unit tests to problem areas, or you want to find bugs in an existing app. Now you write a unit test to expose the problem code and then fix it. These are semi test-driven, but can completely fit in with your current (non-TDD) development process.

Two books I've found useful are:

Test Driven Development in Microsoft .NET

A very hands on look at Test Driven development, following on from Kent Becks' original TDD book.

Pragmatic Unit Testing with C# and nUnit

It comes straight to the point what unit testing is, and how to apply it.

In response to your points:

1. A Unit test, in practical terms is a single method in a class that contains just enough code to test one aspect / behaviour of your application. Therefore you will often have many very simple unit tests, each testing a small part of your application code. In nUnit for example, you create a TestFixture class that contains any number of test methods. The key point is that the tests "test a unit" of your code, ie a smallest (sensible) unit as possible. You don't test the underlying API's you use, just the code you have written.

2. There are frameworks that can take some of the grunt work out of creating test classes, however I don't recommmend them. To create useful unit tests that actually provide a safety net for refactoring, there is no alternative but for a developer to put thought into what and how to test their code. If you start becoming dependent on generating unit tests, it is all too easy to see them as just another task that has to be done. If you find yourself in this situation you're doing it completely wrong.

3. There are no simple rules as to how many unit tests per class, per method etc. You need to look at your application code and make an educated assessment of where the complexity exists and write more tests for these areas. Most people start by testing public methods only because these in turn usually exercise the remainder of the private methods. However this is not always the case and sometimes it is necessary to test private methods.

In short, even experienced unit testers start by writing obvious unit tests, then look for more subtle tests that become clearer once they have written the obvious tests. They don't expect to get every test up-front, but instead add them as they come to their mind.

How force a class to get a UnitTest for every public method?

a) see which methods are untested?

This is fairly easy with a Code Coverage tool. There's one built into Visual Studio, but there are other options as well.

b) force a developer who adds a method to a class to write a unit test for it (e.g. I achieve to test all 50 methods and tomorrow someone adds a 51. method)?

Don't make hard rules based on coverage!

All experience (including mine) shows that if you set hard code coverage targets, the only result you'll get is that developers will begin to game the system. The result will be worse code.

Here's an example; consider this Reverse method:

public static string Reverse(string s)
{
    char[] charArray = s.ToCharArray();
    Array.Reverse(charArray);
    return new string(charArray);
}

You can write a single test to get 100 % coverage of this method. However, as it's given here, the implementation is brittle: what happens if s is null? The method is going to throw a NullReferenceException.

A better implementation would be to check s for null, and add a branch that handles that case, but unless you write a test for it, code coverage will drop.

If you have developers who don't want to write tests, but you demand a hard code coverage target, such developers will leave the Reverse function as is, instead of improving it.

I've seen such gaming of the system in the wild. It will happen if you institute a rule without buy-in from developers.

You'll need to teach developers how their jobs could become easier if they add good unit tests.

Once developers understand this, they'll add tests themselves, and you don't need any rules.

As long as developers don't understand this, the rule will only hurt you even more, because the 'tests' you'll get out of it will be written without understanding.

c) raise a warning or a error when a method is untested?

As explained above, code coverage is a poor tool if used for an absolute metric. On the other hand, it can be useful if you start monitoring trends.

In general, code coverage should increase until it reaches some plateau where it's impractical to increase it more.

Sometimes, it may even drop, which can be okay, as long as there's a good reason for it. Monitoring coverage trends will tell you when the coverage drops, and then you can investigate. Sometimes, there's a good reason (such as when someone adds a Humble Object), but sometimes, there isn't, and you'll need to talk to the developer(s) responsible for the drop.

---

Related Topics