Modular Makefiles

Read time: 71 minutes (17927 words)

I hope you have figured out how useful Make can be in helping you manage your project. Unfortunately Makefiles tend to overwhelm beginners as they try to get Make to take on more management tasks. I have seen single Makefiles that print out over 15 pages long! As programmers, we know how to tame something like that! We decompose it into modules! Let’s try that with our Project Makefile.

Before we get started, it is worthwhile to have a copy of the documentation for Make.

We will be focusing on several aspects of how Make works in this lecture, and this reference helps explain thing in more detail.

Make Includes

Make lets you “include” another file containing instructions to be followed. Those instructions can include definition lines, and target blocks. We can use this feature to gather together all Make lines associated with some part of our project build process. Let’s start off by setting up a top-level Makefile we can use for our development project.

Makefile
1
2
3
4
5
6
7
8
# Modular Make - top level makefile
PROJPATH = $(PWD)
PROJNAME = $(notdir $(PROJPATH))

include $(wildcard mk/*.mk)
TARGET := $(PROJNAME)$(EXT)
TSTTGT := testapp$(EXT)

Believe it or not, this file is the complete Makefile you will need for all projects you set up (plus one directory full of Makefile modules - we will get to that!)

This file shows a bit more of the power of Make. The first few lines figure out where this project lives on your system, then extracts the folder name holding your project. If you name that folder correctly, it should be your project name.

Make has many built-in functions that can process either strings of text, get information from your operating system, or launch external programs that can return results into your Makefile. These features will be handy in our work here.

The PWD variable is set to the full path to the directory where your Makefile lives. The notdir function processes that path, stripping off all of the leading directory names, leaving just the name of the final folder. Here, we are using that name as our project name. If you do not like that name, a simple edit here can set the project name to anything you like.

Note

Remember that you can rename any Git repository folder to something you like, setting up a nice name for your project. GitHub Classroom forces those names to be something pretty ugly, IMHO!

Notice that :include” line. Here I am asking Make to include every file in a subdirectory named “mk” with name ending in “.mk” into this top-level Makefile. Each of those files will contain our “modules” for this project.

Project Structure

We want Make to do several things for us. Certainly, we want it to build our project application. But, we also want other things done as well. Let’s start off by examining our project in a bit more detail.

We set up a structure for our project earlier. That structure is not complete, and we will fix that here.

Source Code

I proposed placing all human interfacing parts of our project (including that traditional main.cpp file) in the src folder. When we build things, this folder will be the place where the primary project application code lives. The source files we find here must be compiled to build that application, but the linking step will need to find all the other application components, which will not live in this folder.

Obviously, we will be compiling all source files in this folder, and we will be linking these files with other components (if they exist) to build the final project application. We will need to build a list of all source files in this folder for processing, a task we will delegate to one of our Makefile modules.

Library Code

All major application code will be stored in the lib folder. Actually, we might like to subdivide all of this code into subdirectories focusing on some major aspect of our project. That would make finding code when needed much easier. (I once worked on a project that had over 150 files, all lumped into one folder, with names that helped no one figure out what each file was about!)

While we could just compile each of the source files in this folder into object files, it is common to build special files called “archives” which are just a fancy package containing a set of object files. The linker can look into these archive files to located missing parts of the application it builds.

Note

If the project gets more complex, placing all library object files in one giant library might not make sense. In that case, the project would logically be split up into smaller component parts, each of which should generate a separate archive file.

Test Code

Since we are using Catch as our testing framework, the tests folder is where all test code will be located. We will compile all files in this folder into object files, then link all of them with the needed code from the lib folder to build a test application.

Include Headers

All header files required by any code in this project (except system headers, of course) will be located in a single, easy to find, place: the include folder. We will not be processing files in this folder. Instead, these files will be included in other program source files, and we will tell the compiler how to find them.

Note

If you start using C++ templates, that may need to change.

Project Documentation

We need to write documentation for our project as we develop the code. Doing this after the project is well underway is too late. The documentation we need is not just comments in your code. It is additional information explaining why the code was set up this way. Those details are being set up while you write your code, and that is the best time to write things down.

We will use the Python Sphinx project to create nice documentation for our project. Although Sphinx was originally developed to support documenting Python code, it is in widespread use to document all kinds of projects. Since Sphinx is a Python tool, we will need to have Python installed on our development system. Python is very handy for general use in managing your system, so this is not a bad thing!

Sphinx can generate all kinds of documentation products. The most common output is HTML pages suitable for publishing on a web server. Less common, since it requires installing another tool, one that produces very nice publications, is LaTeX. I have used LaTeX every since it first appeared on the scene in the 1980s, for all kinds of publications, including a Master’s Thesis at Texas State, so I have it installed on all my development machines. Using that, I can generate PDF files from the documentation.

Since we are sure to be hosting our project on GitHub, you should know that you get a website for free with every project repository you set up. All we need to do is make sure our documentation files end up in a folder named docs, something we can set up easily in our build system. I will keep the original documentation source files, written in reStructuredText in another folder named rst.

When you step back and look at the project folder, there are a lot of directories in there, but each has a specific purpose, and locating things you need is pretty easy.

Help System

When I started out building this modular Makefile system I stumbled onto an idea that is going to be very useful for beginning developers (or even seasoned ones as well). With a little help from an external Python program, we can get Make to display a help message detailing the targets supported by this Makefile. We can also set up a debugging system to check all of the Make variable settings, which is useful while building this system.

Here is a “module” we will place in the mk folder that adds the help system:

mk/help.mk
1
2
3
4
5
# Makefile help system

.PHONY: help
help:	## display help messages
	@python mk/pyhelp.py$(MAKEFILE_LIST)

This module simply sets up a “help” target. We add the PHONY marker to tell Make that this is not going to be a file in the project folder. The rule lines simply call a Python helper script, passing to that script a list of all files that make up part of the Makefile. The list includes all modules we “include” into our top-level Makefile.

The line that runs Python starts with an “@”. This stops the system from printing out the command line. All we want to see is what the Python code displays.

The Python helper code is a bit scary, unless you learned about “Regular Expressions” in your Python class (If not, this kind of thing is very handy!).

mk/pyhelp.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
import sys
import re

def main():
    help_re = re.compile(r"^([a-zA-Zi_-]*:).*?##(.*)$")

    modules = sys.argv
    del modules[0]
    for m in modules:
        fin = open(m,'r')
        lines = fin.readlines()
        for l in lines:
            m = help_re.match(l)
            if m:
                item = m.group(1).strip()
                defn = m.group(2).strip()
                print("%-20s %s" %(item,defn))

main()

Basically, we are looking at all the lines in each Makefile component file. On every target line we see, we look for two sharp characters, and the text following those. The target name and that final text are printed out in a form suitable for display.

Let’s try it out with just this one file in our mk folder:

help:                display help messages

Not very exciting, but as we add new targets, this will become more helpful.

Debugging Targets

A lot of what Make does depends on building lists of things to process. These lists are just strings, usually containing the names of files. While building this Makefile system, it was very helpful to be able to see all of the manes I defined before Make starts doing the real work.

Here is a module that will do the job:

mk/debug.mk
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
.PHONY:	debug
debug: ## display local make variables defined
	@$(foreach V, $(sort $(.VARIABLES)), \
		$(if $(filter-out environment% default automatic,\
			$(origin $V)), \
			$(warning $V = $($V) )) \
	)

.PHONY:	debug-all
debug-all: ## display all make variables defined
	@$(foreach V, $(sort $(.VARIABLES)), \
		$(warning $V = $($V) ) \
	)

This module provides two new targets, one that displays variables defined in this project, and one showing all variable names, including names Make sets up internally.

Let’s see the help output:

debug:               display local make variables defined
debug-all:           display all make variables defined
help:                display help messages

And the debug output for this project so far:

mk/debug.mk:3: .DEFAULT_GOAL = debug 
mk/debug.mk:3: CURDIR = /Users/rblack/_projects/lectures/cosc1337/lectures/11-Modular-Make/code/ex2 
mk/debug.mk:3: MAKEFILE_LIST =  Makefile mk/debug.mk mk/help.mk 
mk/debug.mk:3: MAKEFLAGS =  
mk/debug.mk:3: PROJNAME = cosc1337 
mk/debug.mk:3: PROJPATH = /Users/rblack/_projects/lectures/cosc1337 
mk/debug.mk:3: SHELL = /bin/sh 
mk/debug.mk:3: TARGET = cosc1337 
mk/debug.mk:3: TSTTGT = testapp 
make[1]: Nothing to be done for `debug'.

There is definitely some Make trickery going on here. The variable list is fairly easy to figure out, but the filter_out line deletes names not set up in this Makefile. Also, the origin function tells us where the name was defined.

Detecting the OS

There is one serious issue I want this Make system to address. I have students in my classes who own all kinds of systems. I want this Makefile to handle any of the “big three” systems I see: Mac OS-X, Linux, and Windows!

Here is a module that can detect the platform we are running on:

mk/os-detect.mk
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
ifeq ($(OS), Windows_NT)
	EXT = .exe
	PREFIX =
	RM	= del
	WHICH := where
	PLATFORM := Windows
else
	EXT =
	PREFIX := ./
	RM 	= rm -f
	WHICH := which
	UNAME_S := $(shell uname -s)
	ifeq ($(UNAME_S), Linux)
		PLATFORM := Linux
	endif
	ifeq ($(UNAME_S), Darwin)
		PLATFORM := Mac
	endif
endif

Notice that this module has no new targets, all it does is set up a PLATFORM variable telling us what system we are using. The scheme for figuring this out depends on system environment variable set on all Windows platforms, or on the output of a system standard tool, uname which is present on Mac and Linux systems.

Here is out new output:

mk/debug.mk:3: .DEFAULT_GOAL = debug 
mk/debug.mk:3: CURDIR = /Users/rblack/_projects/lectures/cosc1337/lectures/11-Modular-Make/code/ex3 
mk/debug.mk:3: EXT =  
mk/debug.mk:3: MAKEFILE_LIST =  Makefile mk/debug.mk mk/os-detect.mk 
mk/debug.mk:3: MAKEFLAGS =  
mk/debug.mk:3: PLATFORM = Mac 
mk/debug.mk:3: PREFIX = ./ 
mk/debug.mk:3: PROJNAME = cosc1337 
mk/debug.mk:3: PROJPATH = /Users/rblack/_projects/lectures/cosc1337 
mk/debug.mk:3: RM = rm -f 
mk/debug.mk:3: SHELL = /bin/sh 
mk/debug.mk:3: TARGET = cosc1337 
mk/debug.mk:3: TSTTGT = testapp 
mk/debug.mk:3: UNAME_S = Darwin 
mk/debug.mk:3: WHICH = which 
make[1]: Nothing to be done for `debug'.

The important new variables created by this module are these:

  • EXT - the file extension for executable programs

  • PREFIX - Linux and Mac require “./” in front of a program name to run a program in the current directory.

  • PLATFORM - Set to “Windows”, Linux”, or “Mac” depending on the system.

We will use these variables later.

Notice how Make provides conditional code that lets you decide what variables to define in this system.

Building a C++ Application

We are ready to set up modules that will build C++ projects, organized according to the structure we are using. The key to doing this is building a list of files to compile, then link to build the applications we need.

First, let’s set up a module that figures out the names of all source files in each of the three directories we are using for such files. The one wrinkle is that we are now going to allow subdirectories in the lib folder. We need a way for Make to dig out those file names, including the path to each file.

For example, if we put a source file named component.cpp in the lib/part folder, I want to tell make to compile lib/part/component.cpp and build lib/part/component.o. Make Makefiles you see require that you list the subdirectory names, then tell Make to search in each one. I wanted something simpler, and do not want to list directory names at all. The simple solution to this is to let Python dig out the file names with a simple script:

mk/pyfind.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
import os
import sys

dir = sys.argv[1]
ext = sys.argv[2]
result = []
for root, dirs, files in os.walk(dir):
    for name in files:
        if name.endswith(ext):
            result.append(os.path.join(root,name))

print(" ".join(result));


This script recursively searches all folders and subfolders beginning in the named location for files ending with the specified extension. It returns a list of those names in a form ready for Make to save in a new variable.

With this tool in hand, we can create a list of all source files in each of the two main places application files will be found. We can also create a list of the object files we need from these lists.

mk/cpp-project.mk
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
CXX		:= g++
TARGET	:= $(PROJNAME)$(EXT)

HDRS	:= $(shell python mk/pyfind.py include .h)

USRCS	:= $(wildcard src/*.cpp)
LSRCS	:= $(shell python mk/pyfind.py lib .cpp)
ALLSRCS := $(USRCS) $(LSRCS)

UOBJS   := $(USRCS:.cpp=.o)
LOBJS   := $(LSRCS:.cpp=.o)

ifeq ($(PLATFORM), Windows)
	DOBJS := $(subst /,\,$(UOBJS)) $(subst /.\,$(LOBJS))
else
	DOBJS := $(UOBJS) $(LOBJS)
endif

CFLAGS	:= -std=c++11 -Iinclude

.PHONY: all
all: $(TARGET) ## build application (default)

$(TARGET):	$(UOBJS) $(LOBJS)
	g++ -o $@ $^ $(LFLAGS)

%.o:	%.cpp
	$(CXX) -c -o $@ $< $(CFLAGS)

.PHONY:	run
run:	$(TARGET)	## launch primary build application
	$(PREFIX)$(TARGET)

.PHONY: clean
clean:	## remove all build artifacts
	$(RM) $(TARGET) $(DOBJS)

Note

To get this to run, I had to work through how Make expands variables. The “:=” operator causes Make to expand a variable that depends on other variables right away. The standard “=” operator waits until the variable is needed to expand things. THis causes all kinds of confusion until you get used to the idea!

There are several new targets defined here:

  • all the default rule usually builds the project application

  • clean - removes all generated files, leaving only essential files behind

  • run - this will run the application, in case you forget its name.

Let’s see if this runs for a simple “Hello World” kind of application.

Here is our first source file:

src/main.cpp
1
2
3
4
5
6
#include <iostream>

int main(void) {
    std::cout << "Pool Ball Simulator" << std::endl;
}

Now, we can build the application using these commands:

rm -f cosc1337 src/main.o
g++ -c -o src/main.o src/main.cpp -std=c++11 -Iinclude
g++ -o cosc1337 src/main.o
./cosc1337
Pool Ball Simulator

Note

I included a “clean” command to make sure you see all files being compiled:

Let’s see what our help system has to say now:

all:                 build application (default)
run:                 launch primary build application
clean:               remove all build artifacts
debug:               display local make variables defined
debug-all:           display all make variables defined
help:                display help messages

Note

I probably should sort these. Maybe later!

Adding a Library File

We need to test building a program with a library file sitting in a subdirectory of lib. For this test, I will simply set up a message function and place it in a file in a subfolder named utilities under lib.

Here is the new main.cpp

src/main.cpp
1
2
3
4
5
6
7
8
#include <iostream>
#include "message.h"

int main(void) {
    std::cout << "Pool Ball Simulator" << std::endl;
    message();
}

And the utility function:

lib/utilities/message.cpp
1
2
3
4
5
6
#include <iostream>

void message(void) {
    std::cout << "\tmessage from subfolder" << std::endl;
}

Again, we can build the application using these commands:

rm -f cosc1337 src/main.o lib/utilities/message.o
g++ -c -o src/main.o src/main.cpp -std=c++11 -Iinclude
g++ -c -o lib/utilities/message.o lib/utilities/message.cpp -std=c++11 -Iinclude
g++ -o cosc1337 src/main.o lib/utilities/message.o
./cosc1337
Pool Ball Simulator
	message from subfolder

Do you see that the Makefile successfully found the files inside subfolders, a nd the compile ran correctly.

Build Issues

The Makefile system we have at this point is about the same as the simple one we have been using for our projects up to this point. It does have new features that make using and extending it easier, though. There is one serious deficiency here, though!

The main.cpp file includes a header file, and so does the message.cpp file. What happens if we modify that header file? The build system should recognize that and recompile both files to make sure they still work with the new header file. While we could add this information ourselves, doing so defeats my goal of not needing to modify this Makefile system at all to use it in other projects.

Fortunately, the compiler can figure out these dependencies for us. All we need to do is add another rule in our cpp-project.mk file:

mk/cpp-project.mk
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
CXX		:= g++
TARGET	:= $(PROJNAME)$(EXT)

HDRS	:= $(shell python mk/pyfind.py include .h)

USRCS	:= $(wildcard src/*.cpp)
LSRCS	:= $(shell python mk/pyfind.py lib .cpp)
ALLSRCS := $(USRCS) $(LSRCS)

UOBJS   := $(USRCS:.cpp=.o)
LOBJS   := $(LSRCS:.cpp=.o)

DEPS	:= $(USRCS:.cpp=.d) $(LSRCS:.cpp=.d)

ifeq ($(PLATFORM), Windows)
	DOBJS := $(subst /,\,$(UOBJS)) $(subst /.\,$(LOBJS))
else
	DOBJS := $(UOBJS) $(LOBJS)
endif

CFLAGS	:= -std=c++11 -Iinclude -MMD

.PHONY: all
all: $(TARGET) ## build application (default)

$(TARGET):	$(UOBJS) $(LOBJS)
	g++ -o $@ $^ $(LFLAGS)

%.o:	%.cpp
	$(CXX) -c -o $@ $< $(CFLAGS)

.PHONY:	run
run:	$(TARGET)	## launch primary build application
	$(PREFIX)$(TARGET)

.PHONY: clean
clean:	## remove all build artifacts
	$(RM) $(TARGET) $(DOBJS) $(DEPS)

-include $(DEPS)

The new lines will create special “dependency” files with a “.d” extension. The compiler can build these files for us by adding the -MMD option to the CFLAGS variable.

Look at the bottom of this file. There we include all these dependency files in our Makefile so Make will know about these new dependencies when building files for us.

Here is the file generated by the compiler:

src/main.d
1
src/main.o: src/main.cpp include/message.h

Now, if we modify the header file, Make will figure out that main.cpp needs to be recompiled.

C++ Lint Checks

Next, we need to add lint checks to see if our code is “good” (according to Google!)

Here is the new module:

mk/cpp-lint.mk
1
2
3
4
5
6
7
8
LINT		:= cpplint
LINTFLAGS	:= --repository=. --recursive --extensions=cpp,h --quiet
LINTFLAGS	+= --filter=-build/include_subdir

.PHONY: lint 
lint:	$(SRCS) $(HDRS) ## Run cpp lint over project files
	@$(LINT) $(LINTFLAGS) include lib src

The new target in this file depends on Python, but we have that covered. All we need to do to use this is install cpplint in our Python installation.

Let’s see how bad our demo code is now:

make[1]: cpplint: No such file or directory
make[1]: *** [lint] Error 1

Looks like we have some work to do!

Testing with Catch

We will use the Catch system for unit testing our C++ code. Here is the module that sets that up:

mk/cpp-test.mk
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
TSTSRCS	:= $(wildcard tests/*.cpp)
TSTOBJS	:= $(TSTSRCS:.cpp=.o)
DOBJS	+= $(TSTOBJS)

TSTTGT	:= testapp$(EXT)

.PHONY:	test
test:	$(TSTTGT)	## Run project unit tests using catch.hpp
	$(PREFIX)$(TSTTGT)

$(TSTTGT):  $(TSTOBJS) $(LOBJS)
	$(CXX) $(LFLAGS) -o $@ $^

You will need to create the tests folder and place the standard test_main.cpp file there. After than add tests as you wish to this folder. The new test target will build and run your tests.

Project Documentation

Reference

https://jamwheeler.com/college-productivity/how-to-write-beautiful-code-documentation/

Last, but certainly not least, we need ot add project documentation to our setup. In this section, we will be using Python Sphinx, so you need to make sure to add these tools to your Python installation:

$ pip install sphinx
$ pip install cloud-sptheme

Note

There are other themes available for generating nice web pages from your documentation. This theme is pretty popular.

We need to create two folders for this setup:

  • docs - where final web pages will be stored

  • rst - where we will write our documentation

We are going to set our projects up so the web pages for our documentation can be published using something called “GitHub Pages”, a free service you get for every repository you set up on GithUb.

Do this:

$ mkdir docs
$ mkdir rst

In the rst folder place these two files:

rst/conf.py
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# -*- coding: utf-8 -*-

import os
import sys
import cloud_sptheme as csp

sys.path.insert(0, os.path.abspath('.'))

# -- Project information -----------------------------------------------------

project = 'PoolBalls'
copyright = '2019, Roie R. Black'
author = 'Roie R. Black'

version = '0.0.1'
release = '0.0.1'

# -- General configuration ---------------------------------------------------

extensions = [
    'sphinx.ext.githubpages',
]
templates_path = ['_templates']
source_suffix = '.rst'
master_doc = 'index'
language = None
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
pygments_style = None

# -- Options for HTML output -------------------------------------------------
html_theme = 'cloud'
html_theme_path = [csp.get_theme_dir()]
html_theme_options = { "roottarget": "index" }
html_static_path = ['_static']

This is a configuration file used by Sphinx. You should modify this file to suit your project. Only a few lines need modification for most projects.

rst/index.rst
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
Pool Ball Simulator
###################
:Author: Roie R. Black
:Date: Feb 22, 2019
:Email: rblack@austincc.edu

.. toctree::
   :maxdepth: 1
   :glob:

   introduction/*

Indices and tables
******************

* :ref:`genindex`
* :ref:`search`

This is the top level document. This file will automatically include all files in the introduction subdirectory in the rst folder. These files must be named with a “.rst” extension. To control the order of these files, I usually name files with numbers at the front. Something like this:

introduction/01-introduction.rst
introduction/02-building-notes.rst

And so on.

Here is the new module for our Makefile system:

mk/sphinx.mk
1
2
3
4
5
6
.PHONY: html
html:	## run Sphinx to generate html pages
	cd rst && \
	sphinx-build -b html -d _build/doctrees . ../docs


With this new module, you can run this command to build your documentation files:

$ make html

When the smoke clears, you will find new HTML pages in the docs folder. You can see your pages locally by opening up the index.html file using any web browser. (Or, double-click on that file name in your file explorer tool.)

When you push your project to Github, these new web pages will be included on GitHub as well as your project code files. Once you have the docs folder on GitHub open up the settings menu option on your project page. Scroll down to the section labeled GitHub Pages. Click on the box that is labeled “None” and select the option master branch/docs folder.

I add a line to my project README file that directs visitors to this project to the correct URL, provided by GitHub:

There, now you have no excuse! Start writing documentation that explains what you are thinking as you create this cool new project!

Wrapping Up

Phew, that is a lot to digest in one sitting. This attempt at modularizing a fairly complex Makefile is working fairly well now, but is still has a few issues that need attention. The order in which Make includes files matters more than I would like. For that reason, I had to name the files so they showed up in the correct order. I could number the files, using the same scheme I use for SPhinx, and I may do that later.

I also need to add one thing to this project setup: testing on Travis-CI. I will cover that in a separate note.

You could copy the contents of the mk folder to a place on your system and remove that folder from projects. IN that case, you would need to alter the single “include” line in the main Makefile. That would greatly simplify building multiple projects using this system. I have done that for my own projects, but not for studnt projects. Until you lear what is going on, I feel it is best to keep the entire project self contained.