Guide to Testing in Empirical¶
This document details how testing works in Empirical, both for writing and understanding tests. Empirical makes use of the Catch testing framework, the documentation of which is available here.
Running Tests¶
- In the root directory of Empirical, use the maketarget
test
, like so:: - make test
The tests will compile and execute automatically, and you should see output that looks something like this:
cd tests && make test
make[1]: Entering directory '/home/jgf/git/Empirical/tests'
g++ -std=c++11 test_driver.cc -o test.o
# Execute tests
./test.o
===============================================================================
All tests passed (562 assertions in 27 test cases)
If you wish to see detailed coverage data you can use the maketarget coverage
:
make coverage
Again, the tests will compile (this time with coverage flags) and execute, generating coverage data. This data will be analyzed and stuffed into helpful HTML files. You’ll see output that initially looks like the normal tests, followed by a lot of output, and then:
Overall coverage rate:
lines......: 81.7% (946 of 1158 lines)
functions..: 87.0% (463 of 532 functions)
The HTML info will give breakdowns on line-by-line coverage on each file. It is highly recommended that you consult these to verify that code is well covered. To view these files, open tests/html/index.html in your favorite browser.
Writing Tests¶
It is required that contributions to the Empirical library have test coverage. Though writing tests can be a complex task in some cases the Catch testing framework is extremely easy to use.
In general the best way to understand how to write tests is to look at the existing tests. I
recommend skimming through test_tools.cc
for an overview.
If you are creating a new test file you will need to include the file you’ve made in the
test_driver.cc
file. That is, suppose you create a file test_potatoes.cc
. You will then need
to edit test_driver.cc
so that it looks something like this:
#define CATCH_CONFIG_MAIN
#include "../third-party/catch/single_include/catch.hpp"
#include "test_tools.cc"
#include "test_geometry.cc"
#include "test_scholar.cc"
#include "test_potatoes.cc"
To write a test case you simply use the TEST_CASE
macro provided by Catch:
TEST_CASE("Test name goes here", "[test classification here]")
{
// body of test
}
Within a test case you can use the REQUIRE
macro like an assert, to require certain conditions
within the test:
REQUIRE(1==1); // will pass, obviously
REQUIRE(1==0); // will fail, and Catch will complain
If a REQUIRE
fails, Catch will expand it for you to show you what was compared. Supposing we
have a test case like the following:
TEST_CASE("testing tests", "[demo]")
{
bool a = false, b = true;
REQUIRE(a == b);
}
It would execute like so:
demo.cc:4: FAILED:
REQUIRE( a == b )
with expansion:
false == true
===============================================================================
test cases: 1 | 1 failed
assertions: 1 | 1 failed
This allows for easier debugging of failed tests.
Catch provides several different frameworks for constructing test cases which are detailed within their documentation.