Modular Makefiles ################# .. include:: /header.inc .. vim:filetype=rst spell: I hope you have figured out how useful Make_ can be in helping you manage your project. Unfortunately ``Makefiles`` tend to overwhelm beginners as they try to get Make_ to take on more management tasks. I have seen single ``Makefiles`` that print out over 15 pages long! As programmers, we know how to tame something like that! We decompose it into modules! Let's try that with our Project ``Makefile``. Before we get started, it is worthwhile to have a copy of the documentation for Make_. * :download:`make.pdf` We will be focusing on several aspects of how Make_ works in this lecture, and this reference helps explain thing in more detail. Make Includes ************* Make lets you "include" another file containing instructions to be followed. Those instructions can include definition lines, and target blocks. We can use this feature to gather together all Make lines associated with some part of our project build process. Let's start off by setting up a top-level ``Makefile`` we can use for our development project. .. literalinclude:: code/ex1/Makefile :linenos: :caption: Makefile Believe it or not, this file is the complete ``Makefile`` you will need for all projects you set up (plus one directory full of ``Makefile`` modules - we will get to that!) This file shows a bit more of the power of Make_. The first few lines figure out where this project lives on your system, then extracts the folder name holding your project. If you name that folder correctly, it should be your project name. Make_ has many built-in functions that can process either strings of text, get information from your operating system, or launch external programs that can return results into your ``Makefile``. These features will be handy in our work here. The ``PWD`` variable is set to the full path to the directory where your ``Makefile`` lives. The ``notdir`` function processes that path, stripping off all of the leading directory names, leaving just the name of the final folder. Here, we are using that name as our project name. If you do not like that name, a simple edit here can set the project name to anything you like. .. note:: Remember that you can rename any Git_ repository folder to something you like, setting up a nice name for your project. GitHub Classroom forces those names to be something pretty ugly, IMHO! Notice that :include" line. Here I am asking Make to include every file in a subdirectory named "mk" with name ending in ".mk" into this top-level ``Makefile``. Each of those files will contain our "modules" for this project. Project Structure ***************** We want Make_ to do several things for us. Certainly, we want it to build our project application. But, we also want other things done as well. Let's start off by examining our project in a bit more detail. We set up a structure for our project earlier. That structure is not complete, and we will fix that here. Source Code =========== I proposed placing all human interfacing parts of our project (including that traditional ``main.cpp`` file) in the ``src`` folder. When we build things, this folder will be the place where the primary project application code lives. The source files we find here must be compiled to build that application, but the linking step will need to find all the other application components, which will not live in this folder. Obviously, we will be compiling all source files in this folder, and we will be linking these files with other components (if they exist) to build the final project application. We will need to build a list of all source files in this folder for processing, a task we will delegate to one of our ``Makefile`` modules. Library Code ============ All major application code will be stored in the ``lib`` folder. Actually, we might like to subdivide all of this code into subdirectories focusing on some major aspect of our project. That would make finding code when needed much easier. (I once worked on a project that had over 150 files, all lumped into one folder, with names that helped no one figure out what each file was about!) While we could just compile each of the source files in this folder into object files, it is common to build special files called "archives" which are just a fancy package containing a set of object files. The linker can look into these archive files to located missing parts of the application it builds. .. note:: If the project gets more complex, placing all library object files in one giant library might not make sense. In that case, the project would logically be split up into smaller component parts, each of which should generate a separate archive file. Test Code ========= Since we are using Catch_ as our testing framework, the ``tests`` folder is where all test code will be located. We will compile all files in this folder into object files, then link all of them with the needed code from the ``lib`` folder to build a test application. Include Headers =============== All header files required by any code in this project (except system headers, of course) will be located in a single, easy to find, place: the ``include folder``. We will not be processing files in this folder. Instead, these files will be included in other program source files, and we will tell the compiler how to find them. .. note:: If you start using C++ templates, that may need to change. Project Documentation ===================== We need to write documentation for our project as we develop the code. Doing this after the project is well underway is too late. The documentation we need is not just comments in your code. It is additional information explaining *why* the code was set up this way. Those details are being set up while you write your code, and that is the best time to write things down. We will use the Python Sphinx_ project to create nice documentation for our project. Although Sphinx_ was originally developed to support documenting Python code, it is in widespread use to document all kinds of projects. Since Sphinx_ is a Python tool, we will need to have Python installed on our development system. Python is very handy for general use in managing your system, so this is not a bad thing! Sphinx_ can generate all kinds of documentation products. The most common output is HTML pages suitable for publishing on a web server. Less common, since it requires installing another tool, one that produces very nice publications, is LaTeX_. I have used LaTeX_ every since it first appeared on the scene in the 1980s, for all kinds of publications, including a Master's Thesis at Texas State, so I have it installed on all my development machines. Using that, I can generate PDF files from the documentation. Since we are sure to be hosting our project on GitHub_, you should know that you get a website for free with every project repository you set up. All we need to do is make sure our documentation files end up in a folder named ``docs``, something we can set up easily in our build system. I will keep the original documentation source files, written in |RST|_ in another folder named ``rst``. When you step back and look at the project folder, there are a lot of directories in there, but each has a specific purpose, and locating things you need is pretty easy. Help System ************ When I started out building this modular ``Makefile`` system I stumbled onto an idea that is going to be very useful for beginning developers (or even seasoned ones as well). With a little help from an external Python program, we can get Make_ to display a help message detailing the targets supported by this ``Makefile``. We can also set up a debugging system to check all of the Make_ variable settings, which is useful while building this system. Here is a "module" we will place in the ``mk`` folder that adds the help system: .. literalinclude:: code/ex1/mk/help.mk :linenos: :caption: mk/help.mk This module simply sets up a "help" target. We add the ``PHONY`` marker to tell Make_ that this is not going to be a file in the project folder. The rule lines simply call a Python helper script, passing to that script a list of all files that make up part of the ``Makefile``. The list includes all modules we "include" into our top-level ``Makefile``. The line that runs Python starts with an "@". This stops the system from printing out the command line. All we want to see is what the Python code displays. The Python helper code is a bit scary, unless you learned about "Regular Expressions" in your Python class (If not, this kind of thing is very handy!). .. literalinclude:: code/ex1/mk/pyhelp.py :linenos: :caption: mk/pyhelp.py Basically, we are looking at all the lines in each ``Makefile`` component file. On every target line we see, we look for two sharp characters, and the text following those. The target name and that final text are printed out in a form suitable for display. Let's try it out with just this one file in our ``mk`` folder: .. program-output:: make help :cwd: code/ex1 Not very exciting, but as we add new targets, this will become more helpful. Debugging Targets ================= A lot of what Make_ does depends on building lists of things to process. These lists are just strings, usually containing the names of files. While building this ``Makefile`` system, it was very helpful to be able to see all of the manes I defined before Make_ starts doing the real work. Here is a module that will do the job: .. literalinclude:: code/ex2/mk/debug.mk :linenos: :caption: mk/debug.mk This module provides two new targets, one that displays variables defined in this project, and one showing all variable names, including names Make_ sets up internally. Let's see the help output: .. program-output:: make help :cwd: code/ex2 And the debug output for this project so far: .. program-output:: make debug :cwd: code/ex2 There is definitely some Make_ trickery going on here. The variable list is fairly easy to figure out, but the ``filter_out`` line deletes names not set up in this ``Makefile``. Also, the ``origin`` function tells us where the name was defined. Detecting the OS **************** There is one serious issue I want this Make_ system to address. I have students in my classes who own all kinds of systems. I want this ``Makefile`` to handle any of the "big three" systems I see: Mac OS-X, Linux, and Windows! Here is a module that can detect the platform we are running on: .. literalinclude:: code/ex3/mk/os-detect.mk :linenos: :caption: mk/os-detect.mk Notice that this module has no new targets, all it does is set up a ``PLATFORM`` variable telling us what system we are using. The scheme for figuring this out depends on system environment variable set on all Windows platforms, or on the output of a system standard tool, ``uname`` which is present on Mac and Linux systems. Here is out new output: .. program-output:: make debug :cwd: code/ex3 The important new variables created by this module are these: * ``EXT`` - the file extension for executable programs * ``PREFIX`` - Linux and Mac require "./" in front of a program name to run a program in the current directory. * PLATFORM - Set to "Windows", Linux", or "Mac" depending on the system. We will use these variables later. Notice how Make_ provides conditional code that lets you decide what variables to define in this system. Building a C++ Application ************************** We are ready to set up modules that will build C++ projects, organized according to the structure we are using. The key to doing this is building a list of files to compile, then link to build the applications we need. First, let's set up a module that figures out the names of all source files in each of the three directories we are using for such files. The one wrinkle is that we are now going to allow subdirectories in the ``lib`` folder. We need a way for Make_ to dig out those file names, including the path to each file. For example, if we put a source file named component.cpp in the ``lib/part`` folder, I want to tell make to compile ``lib/part/component.cpp`` and build ``lib/part/component.o``. Make Makefiles you see require that you list the subdirectory names, then tell Make to search in each one. I wanted something simpler, and do not want to list directory names at all. The simple solution to this is to let Python dig out the file names with a simple script: .. literalinclude:: code/ex4/mk/pyfind.py :linenos: :caption: mk/pyfind.py This script recursively searches all folders and subfolders beginning in the named location for files ending with the specified extension. It returns a list of those names in a form ready for Make_ to save in a new variable. With this tool in hand, we can create a list of all source files in each of the two main places application files will be found. We can also create a list of the object files we need from these lists. .. literalinclude:: code/ex4/mk/cpp-project.mk :linenos: :caption: mk/cpp-project.mk .. note:: To get this to run, I had to work through how Make_ expands variables. The ":=" operator causes Make_ to expand a variable that depends on other variables right away. The standard "=" operator waits until the variable is needed to expand things. THis causes all kinds of confusion until you get used to the idea! There are several new targets defined here: * ``all`` the default rule usually builds the project application * ``clean`` - removes all generated files, leaving only essential files behind * ``run`` - this will run the application, in case you forget its name. Let's see if this runs for a simple "Hello World" kind of application. Here is our first source file: .. literalinclude:: code/ex4/src/main.cpp :linenos: :caption: src/main.cpp Now, we can build the application using these commands: .. program-output:: make clean :cwd: code/ex4 .. program-output:: make :cwd: code/ex4 .. program-output:: make run :cwd: code/ex4 .. note:: I included a "clean" command to make sure you see all files being compiled: Let's see what our help system has to say now: .. program-output:: make help :cwd: code/ex4 .. note:: I probably should sort these. Maybe later! Adding a Library File ********************* We need to test building a program with a library file sitting in a subdirectory of ``lib``. For this test, I will simply set up a message function and place it in a file in a subfolder named ``utilities`` under ``lib``. Here is the new main.cpp .. literalinclude:: code/ex5/src/main.cpp :linenos: :caption: src/main.cpp And the utility function: .. literalinclude:: code/ex5/lib/utilities/message.cpp :linenos: :caption: lib/utilities/message.cpp Again, we can build the application using these commands: .. program-output:: make clean :cwd: code/ex5 .. program-output:: make :cwd: code/ex5 .. program-output:: make run :cwd: code/ex5 Do you see that the ``Makefile`` successfully found the files inside subfolders, a nd the compile ran correctly. Build Issues ************ The ``Makefile`` system we have at this point is about the same as the simple one we have been using for our projects up to this point. It does have new features that make using and extending it easier, though. There is one serious deficiency here, though! The ``main.cpp`` file includes a header file, and so does the ``message.cpp`` file. What happens if we modify that header file? The build system should recognize that and recompile both files to make sure they still work with the new header file. While we could add this information ourselves, doing so defeats my goal of not needing to modify this ``Makefile`` system at all to use it in other projects. Fortunately, the compiler can figure out these dependencies for us. All we need to do is add another rule in our ``cpp-project.mk`` file: .. literalinclude:: code/ex6/mk/cpp-project.mk :linenos: :caption: mk/cpp-project.mk The new lines will create special "dependency" files with a ".d" extension. The compiler can build these files for us by adding the ``-MMD`` option to the ``CFLAGS`` variable. Look at the bottom of this file. There we include all these dependency files in our ``Makefile`` so Make_ will know about these new dependencies when building files for us. Here is the file generated by the compiler: .. literalinclude:: code/ex6/src/main.d :linenos: :caption: src/main.d Now, if we modify the header file, Make_ will figure out that ``main.cpp`` needs to be recompiled. C++ Lint Checks *************** Next, we need to add lint checks to see if our code is "good" (according to Google!) Here is the new module: .. literalinclude:: code/ex7/mk/cpp-lint.mk :linenos: :caption: mk/cpp-lint.mk The new target in this file depends on Python, but we have that covered. All we need to do to use this is install ``cpplint`` in our Python installation. Let's see how bad our demo code is now: .. program-output:: make lint :cwd: code/ex7 Looks like we have some work to do! Testing with Catch ****************** We will use the Catch_ system for unit testing our C++ code. Here is the module that sets that up: .. literalinclude:: code/ex7/mk/cpp-test.mk :linenos: :caption: mk/cpp-test.mk You will need to create the ``tests`` folder and place the standard ``test_main.cpp`` file there. After than add tests as you wish to this folder. The new ``test`` target will build and run your tests. Project Documentation ********************* :Reference: https://jamwheeler.com/college-productivity/how-to-write-beautiful-code-documentation/ Last, but certainly not least, we need ot add project documentation to our setup. In this section, we will be using Python Sphinx_, so you need to make sure to add these tools to your Python installation: .. code-block:: bash $ pip install sphinx $ pip install cloud-sptheme .. note:: There are other themes available for generating nice web pages from your documentation. This theme is pretty popular. We need to create two folders for this setup: * ``docs`` - where final web pages will be stored * ``rst`` - where we will write our documentation We are going to set our projects up so the web pages for our documentation can be published using something called "GitHub Pages", a free service you get for every repository you set up on GithUb_. Do this: .. code-block:: bash $ mkdir docs $ mkdir rst In the ``rst`` folder place these two files: .. literalinclude:: code/ex8/rst/conf.py :linenos: :caption: rst/conf.py This is a configuration file used by Sphinx. You should modify this file to suit your project. Only a few lines need modification for most projects. .. literalinclude:: code/ex8/rst/index.rst :linenos: :caption: rst/index.rst This is the top level document. This file will automatically include all files in the ``introduction`` subdirectory in the ``rst`` folder. These files must be named with a ".rst" extension. To control the order of these files, I usually name files with numbers at the front. Something like this: .. code-block:: text introduction/01-introduction.rst introduction/02-building-notes.rst And so on. Here is the new module for our ``Makefile`` system: .. literalinclude:: code/ex8/mk/sphinx.mk :linenos: :caption: mk/sphinx.mk With this new module, you can run this command to build your documentation files: .. code-block:: bash $ make html When the smoke clears, you will find new HTML pages in the ``docs`` folder. You can see your pages locally by opening up the ``index.html`` file using any web browser. (Or, double-click on that file name in your file explorer tool.) When you push your project to Github_, these new web pages will be included on GitHub_ as well as your project code files. Once you have the ``docs`` folder on GitHub_ open up the ``settings`` menu option on your project page. Scroll down to the section labeled ``GitHub Pages``. Click on the box that is labeled "None" and select the option ``master branch/docs folder``. I add a line to my project README file that directs visitors to this project to the correct URL, provided by GitHub: * https://ghusername.github.io/reponame There, now you have no excuse! Start writing documentation that explains what you are thinking as you create this cool new project! Wrapping Up *********** Phew, that is a lot to digest in one sitting. This attempt at modularizing a fairly complex ``Makefile`` is working fairly well now, but is still has a few issues that need attention. The order in which Make_ includes files matters more than I would like. For that reason, I had to name the files so they showed up in the correct order. I could number the files, using the same scheme I use for SPhinx_, and I may do that later. I also need to add one thing to this project setup: testing on Travis-CI. I will cover that in a separate note. You could copy the contents of the ``mk`` folder to a place on your system and remove that folder from projects. IN that case, you would need to alter the single "include" line in the main ``Makefile``. That would greatly simplify building multiple projects using this system. I have done that for my own projects, but not for studnt projects. Until you lear what is going on, I feel it is best to keep the entire project self contained.