Testing, Testing, 123

Read time: 32 minutes (8159 words)

There is an old saying in software development that goes like this:

If you did not test it, it does not work!

We are about to set off on an adventure to build a simulator for something from our real world, a simple pool table with a bunch of balls bouncing around.

We will be building a bunch of parts, each one representing some aspect of our simulated world. How will we know if anything in this setup works, without waiting for the complete application to come together?

The answer is simple: we will test each part in isolation!

Huh?

Most of your work as a developer will involve only part of some bigger project. The entire project may be building a huge application and the team may involve hundreds of developers, each building only one part of the overall system.

Since you may not have access to the entire code base, you need a way to test just your part, so you can gain confidence that your part is “production ready”.

But you have not been trained to do testing. You are still learning how to write simple programs. We need a simple test scheme.

Setting up Your Project

Any project of reasonable complexity deserves to be set up properly. In this class, we will use this standard structure for all future projects:

Project-Name/
    |
    +- .git/ Git management folder (you get this when cloning a repo from GH)
    |
    +- README.rst - tell the world what this project is about
    |
    +- .gitignore - to keep files out of your repo on GH
    |
    +- Makefile - manages building your project
    |
    +- src/ - where main.cpp lives
    |
    +- lib/ where all class .cpp files live
    |
    +- include/ where all header files live
    |
    +- tests/ where all test code lives

Warning

The “/” indicates a folder name. On a PC, remember to use the back slash.

You can build these folders and files uging any scheme you like. I usually use mkdir` to build folders, and gvim to create any text files for my project, code or things like Makefile. We will add a few folders to this basic layout as needed.

Once this setup is in place, we are ready to begin programming.

Testing a Function

Note

In this section, we will just create code in the root level of our project. All files live there until we move then later.

Suppose you need to write a magical square root routine that is much faster than the default one available on every system. You need to use the same interface as the “real” square root routine, so you build a program that looks like this as a start:

main.cpp

#include <iostream>
#include <math.h>

float sqrt2(float x) {
    return sqrt(x);
}

int main(void) {
    float x = 2.0;
    std::cout 
        << "Testing sqrt2" 
        << std::endl;
    std::cout 
        << "\tthe square root of " 
        << x 
        << " is " 
        << sqrt2(x) 
        << std::endl;
}

Note

Believe it or not, I have given an assignment in a beginning programming class to calculate the square root using an iteration scheme, and had students submit exactly this code. It ain’t about the answer, it is about the method!

$ g++ -o test main.cpp

$ ./test
Testing sqrt2
	the square root of 2 is 1.41421

Does this constitute a good test? Not really! All it proves is that the code runs, and produces some answer. You still need to decide if this answer is correct, and that is left to you to figure out.

Even worse, this is just one test, and that is definitely not enough to prove that the code is working.

Let’s see is we can do better.

Breaking out a Unit of code

We are going to test a chunk of this code, not the complete application. The only part we are really concerned with is the sqrt function. Let’s start off by breaking that part out into its own file:

main.cpp

#include <iostream>

float sqrt2(float x);

int main(void) {
    float x = 2.0;
    std::cout 
        << "Testing sqrt2" 
        << std::endl;
    std::cout 
        << "\tthe square root of " 
        << x 
        << " is " 
        << sqrt2(x) 
        << std::endl;
}

sqrt2.cpp

#include <math.h>

float sqrt2(float x) {
    return sqrt(x);
}

We can still run this by doing this:

$ g++ -o test main.cpp sqrt2.cpp

$ ./test
Testing sqrt2
	the square root of 2 is 1.41421

Now the program is configured for testing. We can compile the function alone and build a test application that exercises it. We will do that using a neat C++ tool called the Catch.

Setting up Catch

Catch is a single header file system that sets up unit testing in a C++ project. All you need to do is add the header file to your project, then write a few short files.

Clone Catch

The Catch project code is on GitHub (naturally!) In your course folder (not in another repo), do this:

$ cd cosc1337
$ git clone https://github.com/catchorg/Catch2.git

Inside of this project you will find a single file that you need to copy into your project folder:

$ cp Catch2/single_include/catch2/catch.hpp Pool-Simulation/include

Catch Testing

Basically, all we need to do to test this function separately from any application that will use it (like our main.cpp here) is to set up a separate directory where all test files will live. I create a tests directory in the project directory for this purpose.

Then, we need to install the simple header file that makes up the catch.hpp test framework in either land it in the project include directory, or place it in the tests folder directly.

Next, we need to build a single file in the tests directory named test-main.cpp. This file will cause catch.hpp to build a main program that will be used in the test application we will build.

This file is very simple:

tests/test_main.cpp

#define CATCH_CONFIG_MAIN
#include "catch.hpp"

Not much there. The single define in this file is what triggers the C++ preprocessor to add code to this file that is your main function. We do not need to worry about the magic happening here.

To get started with this testing framework, let;s make sure the test system works properly, and get a look at how hard writing tests will be. Here is a sanity check that proves things are working properly:

tests/test_sanity.cpp

#include "catch.hpp"

TEST_CASE( "sanity test", "sanity" ){
    REQUIRE(true);
}

Look at the code here. The include brings in the catch.hpp magic. The TEST_CASE definition is in the catch.hpp header, and it creates a function that the main test logic will call. The two parameters in this function give a name to the test, and a group name you can use to only run tests associated with the named “group” We will ignore that for now.

Inside this magic test function, you see a single line that constitutes the “test we want to run. In this example, all we want to do is test that the test passes. The stuff inside the parentheses is a logical expression (in this case, just the value true). If that expression evaluates to true, catch.hpp will assume that the test passed. If it produced false, the test failed.

Getting Ready to Test

It is time to put files where they belong!

Move the main.cpp file into the src folder, and the sqrt2,cpp file into the lib folder.

We will use Make to manage building out application and tests. Here is a basic Makefile that you can use for all future lab projects in this class:

Makefile

# Makefile for building Graphics demo on command line
TARGET 	= demo
TSTTGT	= testapp

SSRCS	= $(wildcard src/*.cpp)
LSRCS	= $(wildcard lib/*.cpp)
TSRCS	= $(wildcard tests/*.cpp)

SOBJS	= $(SSRCS:.cpp=.o)
LOBJS	= $(LSRCS:.cpp=.o)
TOBJS	= $(TSRCS:.cpp=.o)
AOBJS	= $(SOBJS) $(LOBJS) $(TOBJS)

CFLAGS	= -std=c++11 -Iinclude
CXX = g++
ifeq ($(OS), Windows_NT)
	EXT = .exe
	RM	= del
	PRE =
	LDFLAGS = -glut32 -lglu32 0lopengl32 -Wl,--subsystem,windows
else
	EXT =
	RM 	= rm -f
	PRE = ./
	UNAME_S := $(shell uname -s)
	ifeq ($(UNAME_S), Linux)
		LFLAGS = -lGL -lGLU, -lglut
	endif
	ifeq ($(UNAME_S), Darwin)
		CFLAGS += -Wno-deprecated-declarations
		LFLAGS = -framework OpenGL -framework GLUT
	endif
endif

.PHONY:	all
all:	$(TARGET)$(EXT)
	$(PRE)$<

.PHONY: test
test:	$(TSTTGT)$(EXT)
	$(PRE)$<

$(TARGET)$(EXT):	$(SOBJS) $(LOBJS)
	$(CXX) -o $@ $(LFLAGS) $^

$(TSTTGT)$(EXT):	$(TOBJS) $(LOBJS)
	$(CXX) -o $@ $(LFLAGS) $^

%.o:	%.cpp
	$(CXX) -c $< -o $@ $(CFLAGS)

clean:
	$(RM) $(TARGET)$(EXT) $(TSTTGT)$(EXT) $(iAOBJS)

This Makefile is pretty sophisticated, and it should handle most moderate C++ projects, as long as you land files in the right folders.

Note

I will show a modular Makefile system in our next lcture. YOu can use that one if you like, or stay with this one for the projects in this class.

Running the Tests

I am betting this test will pass. Let’s see:

$ make clean
rm -f demo testapp

$ make
g++ -o demo -framework OpenGL -framework GLUT src/main.o lib/sqrt2.o
./demo
Testing sqrt2
	the square root of 2 is 1.41421

$ make test
g++ -o testapp -framework OpenGL -framework GLUT tests/test_main.o tests/test_sanity.o lib/sqrt2.o
./testapp
===============================================================================
All tests passed (1 assertion in 1 test case)

That says we are running properly

Adding a Real Test

Now that we know that the testing system works, we are ready to really test our code.

I leave the sanity test in place, as it does not hurt anything. Create a new test file that looks like this:

tests/test_sqrt2.cpp

#include "catch.hpp"
#include "sqrt2.h"
#include <math.h>
 
float test_set[] = {2.0,3.0,4.0,5.0,6.0};

TEST_CASE( "basic sqrt test", "sqrt2" ){
    for(int i=0;i<5;i++) {
        float x = test_set[i];
        float result = sqrt2(x);
        float checkval = sqrt(x);
        REQUIRE(result == checkval);
    }
}

This time, we set up the test code to access the function we want to test. I added a header file for this, since I should have set that up in splitting the code into two files. That header just has the prototype for the sqrt2 function.

include/sqrt2.h

#pragme once

float sqrt2(float x);

Now, we need to compile the new test and regenerate the testapp program.

$ make clean
rm -f demo testapp

$ make
g++ -o demo -framework OpenGL -framework GLUT src/main.o lib/sqrt2.o
./demo
Testing sqrt2
	the square root of 2 is 1.41421

The project Makefile we are using in this class compiles any code in the lib directory. Since our test function is in that directory, this compiles sqrt2.cpp and creates the object file we need to link with our test code. Notice that we add this object file to the link step when we built the testapp program.

Run the new test:

$ make test
g++ -o testapp -framework OpenGL -framework GLUT tests/test_main.o tests/test_sanity.o tests/test_sqrt2.o lib/sqrt2.o
./testapp
===============================================================================
All tests passed (2 assertions in 2 test cases)

It looks like this function is working. Well it should work, all we are doing it verifying that the real sqrt method returns the same value as our new sqtr2 function.

Obviously, in a real development project, we would write our own code to calculate the square root, and then this test is going to tell you something useful!

Improving the test

Just running one test is not enough to gain confidence in your code. You should run several tests to see how it performs.

Here is a better version of the test code for our function:

tests/test_sqrt2.cpp

#include "catch.hpp"
#include "sqrt2.h"
#include <math.h>
 
float test_set[] = {2.0,3.0,4.0,5.0,6.0};

TEST_CASE( "basic sqrt test", "sqrt2" ){
    for(int i=0;i<5;i++) {
        float x = test_set[i];
        float result = sqrt2(x);
        float checkval = sqrt(x);
        REQUIRE(result == checkval);
    }
}

Now we have

$ make clean
rm -f demo testapp

$ make
g++ -o demo -framework OpenGL -framework GLUT src/main.o lib/sqrt2.o
./demo
Testing sqrt2
	the square root of 2 is 1.41421

$ make test
g++ -o testapp -framework OpenGL -framework GLUT tests/test_main.o tests/test_sanity.o tests/test_sqrt2.o lib/sqrt2.o
./testapp
===============================================================================
All tests passed (2 assertions in 2 test cases)

The number of passing tests is going up. In a real project, you may have hundreds of tests cases you run. We never throw away tests. As the project evolves, new code could break old code that used to work. Running all the tests over and over is called regression testing. (We want to make sure we are moving forward, mot backward in our work!)

This is still now a really good set of tests, but we are at least exercising the function more now. You should test “edge cases” and even test with bad data, and see what happens.

Your goal is to write a set of test cases that make you confident in this “unit” of code, and be willing to turn it loose on real users!

Testing Classes

The strategy for testing a C++ class is identical to what we just did. You include the class header file in your test file, then create an object from the class. You should exercise that object, and use any accessor methods you have to make sure you are getting the results you expect.

Once you get the hang of this, adding testing to your development workflow is pretty easy to do, and then you will begin really testing your code.

Let’s take our new testing skills and try a Poolball simulation.