Testing, Testing, 123 ##################### .. include:: /references.inc .. vim:ft=rst spell: .. wordcount:: .. image:: images/updating-itself.jpg :align: center There is an old saying in software development that goes like this:: If you did not test it, it does not work! We are about to set off on an adventure to build a simulator for something from our real world, a simple pool table with a bunch of balls bouncing around. We will be building a bunch of parts, each one representing some aspect of our simulated world. How will we know if anything in this setup works, without waiting for the complete application to come together? The answer is simple: we will test each part in isolation! Huh? Most of your work as a developer will involve only part of some bigger project. The entire project may be building a huge application and the team may involve hundreds of developers, each building only one part of the overall system. Since you may not have access to the entire code base, you need a way to test just your part, so you can gain confidence that your part is "production ready". But you have not been trained to do testing. You are still learning how to write simple programs. We need a simple test scheme. Setting up Your Project *********************** Any project of reasonable complexity deserves to be set up properly. In this class, we will use this standard structure for all future projects: .. code-block:: text Project-Name/ | +- .git/ Git management folder (you get this when cloning a repo from GH) | +- README.rst - tell the world what this project is about | +- .gitignore - to keep files out of your repo on GH | +- Makefile - manages building your project | +- src/ - where main.cpp lives | +- lib/ where all class .cpp files live | +- include/ where all header files live | +- tests/ where all test code lives .. warning:: The "/" indicates a folder name. On a PC, remember to use the back slash. You can build these folders and files uging any scheme you like. I usually use `mkdir`` to build folders, and ``gvim`` to create any text files for my project, code or things like ``Makefile``. We will add a few folders to this basic layout as needed. Once this setup is in place, we are ready to begin programming. Testing a Function ****************** .. note:: In this section, we will just create code in the root level of our project. All files live there until we move then later. Suppose you need to write a magical square root routine that is much faster than the default one available on every system. You need to use the same interface as the "real" square root routine, so you build a program that looks like this as a start: .. literalinclude:: code/ex01/main.cpp :linenos: :caption: main.cpp .. note:: Believe it or not, I have given an assignment in a beginning programming class to calculate the square root using an iteration scheme, and had students submit exactly this code. It ain't about the answer, it is about the method! .. command-output:: g++ -o test main.cpp :cwd: code/ex01 .. command-output:: ./test :cwd: code/ex01 Does this constitute a good test? Not really! All it proves is that the code runs, and produces some answer. You still need to decide if this answer is correct, and that is left to you to figure out. Even worse, this is just one test, and that is definitely not enough to prove that the code is working. Let's see is we can do better. Breaking out a Unit of code *************************** We are going to test a chunk of this code, not the complete application. The only part we are really concerned with is the ``sqrt`` function. Let's start off by breaking that part out into its own file: .. literalinclude:: code/ex02/main.cpp :linenos: :caption: main.cpp .. literalinclude:: code/ex02/sqrt2.cpp :linenos: :caption: sqrt2.cpp We can still run this by doing this: .. command-output:: g++ -o test main.cpp sqrt2.cpp :cwd: code/ex02 .. command-output:: ./test :cwd: code/ex02 Now the program is configured for testing. We can compile the function alone and build a test application that exercises it. We will do that using a neat C++ tool called the Catch_. Setting up Catch **************** Catch_ is a single header file system that sets up unit testing in a C++ project. All you need to do is add the header file to your project, then write a few short files. Clone Catch =========== The Catch_ project code is on GitHub_ (naturally!) In your course folder (not in another repo), do this: .. code-block:: bash $ cd cosc1337 $ git clone https://github.com/catchorg/Catch2.git Inside of this project you will find a single file that you need to copy into your project folder: .. code-block:: bash $ cp Catch2/single_include/catch2/catch.hpp Pool-Simulation/include Catch Testing ************* Basically, all we need to do to test this function separately from any application that will use it (like our ``main.cpp`` here) is to set up a separate directory where all test files will live. I create a ``tests`` directory in the project directory for this purpose. Then, we need to install the simple header file that makes up the `catch.hpp`_ test framework in either land it in the project ``include`` directory, or place it in the ``tests`` folder directly. Next, we need to build a single file in the ``tests`` directory named ``test-main.cpp``. This file will cause `catch.hpp`_ to build a main program that will be used in the test application we will build. This file is very simple: .. literalinclude:: code/ex03/tests/test_main.cpp :linenos: :caption: tests/test_main.cpp Not much there. The single ``define`` in this file is what triggers the C++ preprocessor to add code to this file that is your main function. We do not need to worry about the magic happening here. To get started with this testing framework, let;s make sure the test system works properly, and get a look at how hard writing tests will be. Here is a sanity check that proves things are working properly: .. literalinclude:: code/ex03/tests/test_sanity.cpp :linenos: :caption: tests/test_sanity.cpp Look at the code here. The ``include`` brings in the `catch.hpp`_ magic. The ``TEST_CASE`` definition is in the catch.hpp_ header, and it creates a function that the main test logic will call. The two parameters in this function give a name to the test, and a group name you can use to only run tests associated with the named "group" We will ignore that for now. Inside this magic test function, you see a single line that constitutes the "test we want to run. In this example, all we want to do is test that the test passes. The stuff inside the parentheses is a logical expression (in this case, just the value ``true``). If that expression evaluates to ``true``, `catch.hpp`_ will assume that the test passed. If it produced ``false``, the test failed. Getting Ready to Test ********************* It is time to put files where they belong! Move the ``main.cpp`` file into the ``src`` folder, and the ``sqrt2,cpp`` file into the ``lib`` folder. We will use Make_ to manage building out application and tests. Here is a basic ``Makefile`` that you can use for all future lab projects in this class: .. literalinclude:: code/ex04/Makefile :linenos: :caption: Makefile This ``Makefile`` is pretty sophisticated, and it should handle most moderate C++ projects, as long as you land files in the right folders. .. note:: I will show a modular Makefile system in our next lcture. YOu can use that one if you like, or stay with this one for the projects in this class. Running the Tests ****************** I am betting this test will pass. Let's see: .. command-output:: make clean :cwd: code/ex04 .. command-output:: make :cwd: code/ex04 .. command-output:: make test :cwd: code/ex04 That says we are running properly Adding a Real Test ****************** Now that we know that the testing system works, we are ready to really test our code. I leave the ``sanity`` test in place, as it does not hurt anything. Create a new test file that looks like this: .. literalinclude:: code/ex05/tests/test_sqrt2.cpp :linenos: :caption: tests/test_sqrt2.cpp This time, we set up the test code to access the function we want to test. I added a header file for this, since I should have set that up in splitting the code into two files. That header just has the prototype for the ``sqrt2`` function. .. literalinclude:: code/ex05/include/sqrt2.h :linenos: :caption: include/sqrt2.h Now, we need to compile the new test and regenerate the ``testapp`` program. .. command-output:: make clean :cwd: code/ex05 .. command-output:: make :cwd: code/ex05 The project ``Makefile`` we are using in this class compiles any code in the ``lib`` directory. Since our test function is in that directory, this compiles ``sqrt2.cpp`` and creates the object file we need to link with our test code. Notice that we add this object file to the link step when we built the ``testapp`` program. Run the new test: .. command-output:: make test :cwd: code/ex05 It looks like this function is working. Well it should work, all we are doing it verifying that the real ``sqrt`` method returns the same value as our new ``sqtr2`` function. Obviously, in a real development project, we would write our own code to calculate the square root, and then this test is going to tell you something useful! Improving the test ****************** Just running one test is not enough to gain confidence in your code. You should run several tests to see how it performs. Here is a better version of the test code for our function: .. literalinclude:: code/ex05/tests/test_sqrt2.cpp :linenos: :caption: tests/test_sqrt2.cpp Now we have .. command-output:: make clean :cwd: code/ex05 .. command-output:: make :cwd: code/ex05 .. command-output:: make test :cwd: code/ex05 The number of passing tests is going up. In a real project, you may have hundreds of tests cases you run. We never throw away tests. As the project evolves, new code could break old code that used to work. Running all the tests over and over is called ``regression testing``. (We want to make sure we are moving forward, mot backward in our work!) This is still now a really good set of tests, but we are at least exercising the function more now. You should test "edge cases" and even test with bad data, and see what happens. Your goal is to write a set of test cases that make you confident in this "unit" of code, and be willing to turn it loose on real users! Testing Classes *************** The strategy for testing a C++ class is identical to what we just did. You include the class header file in your test file, then create an object from the class. You should exercise that object, and use any accessor methods you have to make sure you are getting the results you expect. Once you get the hang of this, adding testing to your development workflow is pretty easy to do, and then you will begin really testing your code. Let's take our new testing skills and try a Poolball simulation.