(Image retrieved from
http://blog.hinshelwood.com/. According to google image search this image is labeled for reuse. Please let me know if you disagree.)
Unit testing in C++
Perhaps you recognize the following story. You're finished writing C++ code, and now you want to add some unit tests, because it makes the metrics look good. You're using a well-established C++ unit testing framework, like CppUnit. Adding a new test requires manually adding files to the solution, adding seemingly duplicated information in .cpp and .h files (i.e. implementations and declarations) and invoking obscure macros. Or perhaps you copy an existing test file and start editing it to test new stuff. Or worse: you extend an existing test with some extra lines to avoid creating all the boilerplate code over and over again.
However, being a programmer and not a file copier, I would really appreciate if we could use our computer a bit more efficiently to specify tests.
YouNeedTests: Python3's doctest meets fitnesse for C++ code
doctest? fitnesse?
The brilliant thing about python's doctest module is how it extracts tests from comments embedded in the source code. This leads to executable documentation which by its very nature
never needs to become outdated. Doctest is typically used to write unit tests: tests written by programmers for programmers, trying to test little bits of code in isolation.
The brilliant thing about fitnesse is how it specifies different test scenarios in table form.
Fitnesse is typically used to define acceptance tests: tests written by business and QA people and intended for business and QA people. Acceptance tests usually involve more objects than unit tests.
requirements
I wanted a tool that could be used with C++, and that transforms comments embedded in the C++ code into unit tests. I also wanted a way to specify different test scenarios in table form. Because I prefer python over C++ in matters of text processing, I decided this was an ideal opportunity to wet my feet in several "hot" and "established" technologies:
- git as source code management system. The brilliant thing about git is its distributed model, its insane flexibility, speed and compactness, and its superb support for branching and merging.
- python3, the somewhat controversial, non-backward compatible successor to the very popular python2 language. I still have to find out what's so brilliant about python3 - but so far python3 hasn't been a negative experience.
- yaml to format the comments from which the code will be generated. The brilliant thing about yaml is how it combines expressiveness of XML with a much more human friendly syntax. Human friendly enough that I didn't create a domain specific language to specify tests in.
- googletest as backend unit testing framework. The brilliant thing about the googletest framework is how it requires fairly little boilerplate code to write simple tests. The second brilliant thing about googletest is that it supports death-tests, i.e. you can check if a piece of code crashes as expected.
- CMake for the build system. The brilliant thing about CMake is how cross-platform, easy to use and versatile it is as a build system. It also comes with built-in provision for testing (and loads of stuff I haven't discovered yet).
- mako to generate boiler plate code and build scripts. The brilliant thing about mako templates is their ease of use and the fast speeds with which they are rendered.
Example
Here's a simple comment that will 6 independent tests from a table of scenarios.
Not all tests need to have a table, one can also embed raw code if desired.
A comment like the above consists of several parts:
- a testsuite name
- a testsuite type: for now 3 types are supported:
- COLUMNBASED: CODE section contains a list of tables. Each line in each table is an independent test.
- ROWBASED: CODE section contains a list of tables. Each table becomes a test. Each line in a table is one step in the test.
- RAW: CODE section contains normal C++ code.
- a LINK section: this specifies which .cpp files should be linked in with the test
- an INCLUDE section: this specifies which .h files to include in with the test
- a STUBS section: this can contain arbitrary C++ code to resolve linker errors. Only stub those parts of the code which are not testing. There's no point in testing stubs :)
- a PRE section: code in the PRE section is executed before anything from the table is execute
- a POST section: code in the POST section is executed after executing statements from the table, but before executing assertions from the table
- a CODE section. In case of a COLUMNBASED or ROWBASED test suite, the CODE section contains a list of tables. Each table has a table header that specifies code templates with placeholders $1, $2, .... The placeholders will be filled in with values from the table cells in the same column. Each table has a NULL column (the ~ symbol) which separates the statements on a line from the assertions.
- The line containing the string 'twovalues' in table 'TABLE1', e.g. will result in the following lines of code being generated:
where the double comma (",,") causes the code template to be unrolled. (Of course the system will also generate all the rest of the required boilerplate code and buildscripts, finally leading to this .cpp file (buildscript not shown here):
Despite being used with a toy example, the table format already results in quite some space and boiler-plate reduction :)
All the generated tests can be run at once by issuing "make test" in the output folder:
More detailed test results are available by individually running tests:
Proof of concept code!
Got curious? Hop over to
Gitorious!
LGPL'd proof of concept code - for now I only tried to use it on debian sid amd64.
