Testing, Testing, 123

Read time: 26 minutes (6600 words)

../_images/updating-itself.jpg

There is an old saying in software development that goes like this:

If you did not test it, it does not work!

We are about to set off on an adventure to build a simulator for a complete computer system. Our model is the attinh85 chip, but we will only build a part of this machine.

We will be building a bunch of parts, each one representing a digital component inside a real machine. We will wire those components up in the same way real digital components are interconnected in a chip.

So, how will we know if anything in this setup works, without waiting for the complete application to come together?

The answer is simple: we will test each part in isolation!

Huh?

Most of your work as a developer will involve only part of some bigger project. The entire project may be building a huge application and the team may involve hundreds of developers, eah building only one part of the overall system.

Since you will not have access to the entire code base, you need a way to test just your part, so you can gain confidence that your part is “production ready”.

But you have bit been trained to do testing. You are still learning how to write simple programs. We need a simple test scheme.

Testing a Function

Suppose you need to write a magical square root routine that is much faster than the canned on available on every system. You need to use the same interface as the “real” square root routine, so you build a program that looks like this as a start:

main.cpp
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
#include <iostream>
#include <math.h>

float sqrt2(float x) {
    return sqrt(x);
}

int main(void) {
    float x = 2.0;
    std::cout 
        << "Testing sqrt2" 
        << std::endl;
    std::cout 
        << "\tthe square root of " 
        << x 
        << " is " 
        << sqrt2(x) 
        << std::endl;
}

Note

Believe it or not, I have given an assignment in a beginning programming class to calculate the square root using an iteration scheme, and had students submit exactly this code. It ain’t about the answer, it is about the method!

$ g++ -o test main.cpp
$ ./test
Testing sqrt2
	the square root of 2 is 1.41421

Does this constitute a good test? Not really! All it proves is that the code runs, and produces some answer. You still need to decide if this answer if correct, and that is left to you to figure out.

Even worse, this is just one test, and that is definitely not enough to prove that the code is working.

Let’s see is we can do better.

Breaking out a Unit of code

We are going to test a chunk of this code, not the complete application. The only part we are really concerned with is the sqrt function. Let’s start off by breaking that part out into its own file:

main.cpp
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
#include <iostream>

float sqrt2(float x);

int main(void) {
    float x = 2.0;
    std::cout 
        << "Testing sqrt2" 
        << std::endl;
    std::cout 
        << "\tthe square root of " 
        << x 
        << " is " 
        << sqrt2(x) 
        << std::endl;
}

sqrt2.cpp
1
2
3
4
5
#include <math.h>

float sqrt2(float x) {
    return sqrt(x);
}

We can still run this by doing this:

$ g++ -o test main.cpp sqrt2.cpp
$ ./test
Testing sqrt2
	the square root of 2 is 1.41421

Now the program is configured for testing. We can compile the function alone and build a test application that exercises it. We will do that using the catch.hpp framework.

Catch Testing

Basically, all we need to do to test this function separately from any application that will use it (like our main.cpp here) is to set up a separate directory where all test files will live. I usually set up a tests directory in the project directory for this purpose.

All we need to do is download the simple header file that makes up the catch.hpp test framework, and either land it in the project include directory, or place it in the tests folder directly.

Next, we need to build a single file in the tests directory named test-main.cpp. This file will cause catch.hpp to build a main program that will be used in the test application we will build.

This file is very simple:

tests/test_main.cpp
1
2
3
#define CATCH_CONFIG_MAIN
#include "catch.hpp"

Not much there. The single define in this file is what triggers the C++ preprocessor to add code to this file that is your main function. We do not need to worry about the magic happening here.

To get started with this testing framework, let;s make sure the test system works. Here is a sanity check that proves things are working properly:

tests/test_sanity.cpp
1
2
3
4
5
#include "catch.hpp"

TEST_CASE( "sanity test", "sanity" ){
    REQUIRE(true);
}

Look at the code here. The include brings in the catch.hpp magic. The TEST_CASE definition is in the catch.hpp header, and it creates a function that the main test logic will call. The two parameters in this function give a name to the test, and a group name you can use to only run tests associated with the named “group” We will ignore that for now.

Inside this magic test function, you see a single line that constitutes the “test we want to run. In this example, all we want to do is test that the test passes. The stuff inside the parentheses is a logical expression (in this case, just the value true). If that expression evaluates to true, catch.hpp will assume that the test passed. If it produced false, the test failed.

I am betting this test will pass. Let’s see:

$ make clean
rm -f   testapp tests/test_main.o tests/test_sanity.o
$ make
g++ -c -std=c++11 -Iinclude tests/test_main.cpp -o tests/test_main.o
g++ -c -std=c++11 -Iinclude tests/test_sanity.cpp -o tests/test_sanity.o
g++ -o testapp tests/test_main.o tests/test_sanity.o
$ make test
./testapp
===============================================================================
All tests passed (1 assertion in 1 test case)

That says we are running properly

Adding a Real Test

Now that we know that the testing system works, we are ready to really test our code.

I leave the sanity test in place, as it does not hurt anything. Create a new test file that looks like this:

tests/test_sqrt2.cpp
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
#include "catch.hpp"
#include "sqrt2.h"
#include <math.h>

TEST_CASE( "basic sqrt test", "sqrt2" ){
    float x = 2.0;
    float result = sqrt2(x);
    float checkval = sqrt(x);
    REQUIRE(result == checkval);
}

This time, we set up the test code to access the function we want to test. I added a header file for this, since I should have set that up in splitting the code into two files. That header just has the prototype for the sqrt2 function.

include/sqrt2.h
1
2
float sqrt2(float x);

Now, we need to compile the new test and regenerate the testapp program.

$ make clean
rm -f   testapp tests/test_main.o tests/test_sanity.o tests/test_sqrt2.o
$ make
g++ -c -std=c++11 -Iinclude tests/test_main.cpp -o tests/test_main.o
g++ -c -std=c++11 -Iinclude tests/test_sanity.cpp -o tests/test_sanity.o
g++ -c -std=c++11 -Iinclude tests/test_sqrt2.cpp -o tests/test_sqrt2.o
g++ -o testapp tests/test_main.o tests/test_sanity.o tests/test_sqrt2.o lib/sqrt2.o

The project Makefile we are using in this class compiles any code in the lib directory. Since our test function is in that directory, this compiles sqrt2.cpp and creates the object file we need to link with our test code. Notice that we add this object file to the link step when we built the testapp program.

Run the new test:

$ make test
./testapp
===============================================================================
All tests passed (2 assertions in 2 test cases)

It looks like this function is working. Well it should work, all we are doing it verifying that the real sqrt method returns the same value as our new sqtr2 function.

Obviously, in a real development project, we would write our own code to calculate the square root, and then this test is going to tell you something useful!

Improving the test

Just running one test is not enough to gain confidence in your code. You should run several tests to see how it performs.

Here is a better version of the test code for our function:

tests/test_sqrt2.cpp
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
#include "catch.hpp"
#include "sqrt2.h"
#include <math.h>
 
float test_set[] = {2.0,3.0,4.0,5.0,6.0};

TEST_CASE( "basic sqrt test", "sqrt2" ){
    for(int i=0;i<5;i++) {
        float x = test_set[i];
        float result = sqrt2(x);
        float checkval = sqrt(x);
        REQUIRE(result == checkval);
    }
}

Now we have

$ make clean
rm -f   testapp tests/test_main.o tests/test_sanity.o tests/test_sqrt2.o
$ make
g++ -c -std=c++11 -Iinclude tests/test_main.cpp -o tests/test_main.o
g++ -c -std=c++11 -Iinclude tests/test_sanity.cpp -o tests/test_sanity.o
g++ -c -std=c++11 -Iinclude tests/test_sqrt2.cpp -o tests/test_sqrt2.o
g++ -o testapp tests/test_main.o tests/test_sanity.o tests/test_sqrt2.o lib/sqrt2.o
$ make test
./testapp
===============================================================================
All tests passed (6 assertions in 2 test cases)

The number of passing tests is going up. In a real project, you may have hundreds of tests cases you run. We never throw away tests. As the project evolves, new code could break old code that used to work. Running all the tests over and over is called regression testing. (We want to make sure we are moving forward, mot backward in our work!)

This is still now a really good set of tests, but we are at least exercising the function more now. You should test “edge cases” and even test with bad data, and see what happens.

Your goal is to write a set of test cases that make you confident in this “unit” of code, and be willing to turn it loose on real users!

Testing Classes

The strategy for testing a C++ class is identical to what we just did. You include the class header file in your test file, then create an object from the class. You should exercise that object, and use any accessor methods you have to make sure you are getting the results you expect.

Once you get the hang of this, adding testing to your development workflow is pretty easy to do, and then you will begin really testing your code.