Modular Makefiles¶
Read time: 71 minutes (17927 words)
I hope you have figured out how useful Make can be in helping you manage your
project. Unfortunately Makefiles
tend to overwhelm beginners as they try to
get Make to take on more management tasks. I have seen single Makefiles
that print out over 15 pages long! As programmers, we know how to tame
something like that! We decompose it into modules! Let’s try that with our
Project Makefile
.
Before we get started, it is worthwhile to have a copy of the documentation for Make.
We will be focusing on several aspects of how Make works in this lecture, and this reference helps explain thing in more detail.
Make Includes¶
Make lets you “include” another file containing instructions to be followed.
Those instructions can include definition lines, and target blocks. We can use
this feature to gather together all Make lines associated with some part of our
project build process. Let’s start off by setting up a top-level Makefile
we can use for our development project.
1 2 3 4 5 6 7 8 | # Modular Make - top level makefile
PROJPATH = $(PWD)
PROJNAME = $(notdir $(PROJPATH))
include $(wildcard mk/*.mk)
TARGET := $(PROJNAME)$(EXT)
TSTTGT := testapp$(EXT)
|
Believe it or not, this file is the complete Makefile
you will need for all
projects you set up (plus one directory full of Makefile
modules - we will
get to that!)
This file shows a bit more of the power of Make. The first few lines figure out where this project lives on your system, then extracts the folder name holding your project. If you name that folder correctly, it should be your project name.
Make has many built-in functions that can process either strings of text, get
information from your operating system, or launch external programs that can
return results into your Makefile
. These features will be handy in our work
here.
The PWD
variable is set to the full path to the directory where your
Makefile
lives. The notdir
function processes that path, stripping off
all of the leading directory names, leaving just the name of the final folder.
Here, we are using that name as our project name. If you do not like that name,
a simple edit here can set the project name to anything you like.
Note
Remember that you can rename any Git repository folder to something you like, setting up a nice name for your project. GitHub Classroom forces those names to be something pretty ugly, IMHO!
Notice that :include” line. Here I am asking Make to include every file in a
subdirectory named “mk” with name ending in “.mk” into this top-level
Makefile
. Each of those files will contain our “modules” for this project.
Project Structure¶
We want Make to do several things for us. Certainly, we want it to build our project application. But, we also want other things done as well. Let’s start off by examining our project in a bit more detail.
We set up a structure for our project earlier. That structure is not complete, and we will fix that here.
Source Code¶
I proposed placing all human interfacing parts of our project (including that
traditional main.cpp
file) in the src
folder. When we build things,
this folder will be the place where the primary project application code lives.
The source files we find here must be compiled to build that application, but
the linking step will need to find all the other application components, which
will not live in this folder.
Obviously, we will be compiling all source files in this folder, and we will be
linking these files with other components (if they exist) to build the final
project application. We will need to build a list of all source files in this
folder for processing, a task we will delegate to one of our Makefile
modules.
Library Code¶
All major application code will be stored in the lib
folder. Actually, we
might like to subdivide all of this code into subdirectories focusing on some
major aspect of our project. That would make finding code when needed much
easier. (I once worked on a project that had over 150 files, all lumped into one
folder, with names that helped no one figure out what each file was about!)
While we could just compile each of the source files in this folder into object files, it is common to build special files called “archives” which are just a fancy package containing a set of object files. The linker can look into these archive files to located missing parts of the application it builds.
Note
If the project gets more complex, placing all library object files in one giant library might not make sense. In that case, the project would logically be split up into smaller component parts, each of which should generate a separate archive file.
Test Code¶
Since we are using Catch as our testing framework, the tests
folder is
where all test code will be located. We will compile all files in this folder
into object files, then link all of them with the needed code from the lib
folder to build a test application.
Include Headers¶
All header files required by any code in this project (except system headers,
of course) will be located in a single, easy to find, place: the include
folder
. We will not be processing files in this folder. Instead, these files
will be included in other program source files, and we will tell the compiler
how to find them.
Note
If you start using C++ templates, that may need to change.
Project Documentation¶
We need to write documentation for our project as we develop the code. Doing this after the project is well underway is too late. The documentation we need is not just comments in your code. It is additional information explaining why the code was set up this way. Those details are being set up while you write your code, and that is the best time to write things down.
We will use the Python Sphinx project to create nice documentation for our project. Although Sphinx was originally developed to support documenting Python code, it is in widespread use to document all kinds of projects. Since Sphinx is a Python tool, we will need to have Python installed on our development system. Python is very handy for general use in managing your system, so this is not a bad thing!
Sphinx can generate all kinds of documentation products. The most common output is HTML pages suitable for publishing on a web server. Less common, since it requires installing another tool, one that produces very nice publications, is LaTeX. I have used LaTeX every since it first appeared on the scene in the 1980s, for all kinds of publications, including a Master’s Thesis at Texas State, so I have it installed on all my development machines. Using that, I can generate PDF files from the documentation.
Since we are sure to be hosting our project on GitHub, you should know that
you get a website for free with every project repository you set up. All we
need to do is make sure our documentation files end up in a folder named
docs
, something we can set up easily in our build system. I will keep the
original documentation source files, written in reStructuredText in another folder named
rst
.
When you step back and look at the project folder, there are a lot of directories in there, but each has a specific purpose, and locating things you need is pretty easy.
Help System¶
When I started out building this modular Makefile
system I stumbled onto an
idea that is going to be very useful for beginning developers (or even seasoned
ones as well). With a little help from an external Python program, we can get
Make to display a help message detailing the targets supported by this
Makefile
. We can also set up a debugging system to check all of the Make
variable settings, which is useful while building this system.
Here is a “module” we will place in the mk
folder that adds the help system:
1 2 3 4 5 | # Makefile help system
.PHONY: help
help: ## display help messages
@python mk/pyhelp.py$(MAKEFILE_LIST)
|
This module simply sets up a “help” target. We add the PHONY
marker to tell
Make that this is not going to be a file in the project folder. The rule lines
simply call a Python helper script, passing to that script a list of all files
that make up part of the Makefile
. The list includes all modules we
“include” into our top-level Makefile
.
The line that runs Python starts with an “@”. This stops the system from printing out the command line. All we want to see is what the Python code displays.
The Python helper code is a bit scary, unless you learned about “Regular Expressions” in your Python class (If not, this kind of thing is very handy!).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | import sys
import re
def main():
help_re = re.compile(r"^([a-zA-Zi_-]*:).*?##(.*)$")
modules = sys.argv
del modules[0]
for m in modules:
fin = open(m,'r')
lines = fin.readlines()
for l in lines:
m = help_re.match(l)
if m:
item = m.group(1).strip()
defn = m.group(2).strip()
print("%-20s %s" %(item,defn))
main()
|
Basically, we are looking at all the lines in each Makefile
component file.
On every target line we see, we look for two sharp characters, and the text
following those. The target name and that final text are printed out in a form
suitable for display.
Let’s try it out with just this one file in our mk
folder:
help: display help messages
Not very exciting, but as we add new targets, this will become more helpful.
Debugging Targets¶
A lot of what Make does depends on building lists of things to process. These
lists are just strings, usually containing the names of files. While building
this Makefile
system, it was very helpful to be able to see all of the
manes I defined before Make starts doing the real work.
Here is a module that will do the job:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | .PHONY: debug
debug: ## display local make variables defined
@$(foreach V, $(sort $(.VARIABLES)), \
$(if $(filter-out environment% default automatic,\
$(origin $V)), \
$(warning $V = $($V) )) \
)
.PHONY: debug-all
debug-all: ## display all make variables defined
@$(foreach V, $(sort $(.VARIABLES)), \
$(warning $V = $($V) ) \
)
|
This module provides two new targets, one that displays variables defined in this project, and one showing all variable names, including names Make sets up internally.
Let’s see the help output:
debug: display local make variables defined
debug-all: display all make variables defined
help: display help messages
And the debug output for this project so far:
mk/debug.mk:3: .DEFAULT_GOAL = debug
mk/debug.mk:3: CURDIR = /Users/rblack/_projects/lectures/cosc1337/lectures/11-Modular-Make/code/ex2
mk/debug.mk:3: MAKEFILE_LIST = Makefile mk/debug.mk mk/help.mk
mk/debug.mk:3: MAKEFLAGS =
mk/debug.mk:3: PROJNAME = cosc1337
mk/debug.mk:3: PROJPATH = /Users/rblack/_projects/lectures/cosc1337
mk/debug.mk:3: SHELL = /bin/sh
mk/debug.mk:3: TARGET = cosc1337
mk/debug.mk:3: TSTTGT = testapp
make[1]: Nothing to be done for `debug'.
There is definitely some Make trickery going on here. The variable list is
fairly easy to figure out, but the filter_out
line deletes names not set up
in this Makefile
. Also, the origin
function tells us where the name was
defined.
Detecting the OS¶
There is one serious issue I want this Make system to address. I have students
in my classes who own all kinds of systems. I want this Makefile
to handle
any of the “big three” systems I see: Mac OS-X, Linux, and Windows!
Here is a module that can detect the platform we are running on:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | ifeq ($(OS), Windows_NT)
EXT = .exe
PREFIX =
RM = del
WHICH := where
PLATFORM := Windows
else
EXT =
PREFIX := ./
RM = rm -f
WHICH := which
UNAME_S := $(shell uname -s)
ifeq ($(UNAME_S), Linux)
PLATFORM := Linux
endif
ifeq ($(UNAME_S), Darwin)
PLATFORM := Mac
endif
endif
|
Notice that this module has no new targets, all it does is set up a
PLATFORM
variable telling us what system we are using. The scheme for
figuring this out depends on system environment variable set on all Windows
platforms, or on the output of a system standard tool, uname
which is
present on Mac and Linux systems.
Here is out new output:
mk/debug.mk:3: .DEFAULT_GOAL = debug
mk/debug.mk:3: CURDIR = /Users/rblack/_projects/lectures/cosc1337/lectures/11-Modular-Make/code/ex3
mk/debug.mk:3: EXT =
mk/debug.mk:3: MAKEFILE_LIST = Makefile mk/debug.mk mk/os-detect.mk
mk/debug.mk:3: MAKEFLAGS =
mk/debug.mk:3: PLATFORM = Mac
mk/debug.mk:3: PREFIX = ./
mk/debug.mk:3: PROJNAME = cosc1337
mk/debug.mk:3: PROJPATH = /Users/rblack/_projects/lectures/cosc1337
mk/debug.mk:3: RM = rm -f
mk/debug.mk:3: SHELL = /bin/sh
mk/debug.mk:3: TARGET = cosc1337
mk/debug.mk:3: TSTTGT = testapp
mk/debug.mk:3: UNAME_S = Darwin
mk/debug.mk:3: WHICH = which
make[1]: Nothing to be done for `debug'.
The important new variables created by this module are these:
EXT
- the file extension for executable programs
PREFIX
- Linux and Mac require “./” in front of a program name to run a program in the current directory.PLATFORM - Set to “Windows”, Linux”, or “Mac” depending on the system.
We will use these variables later.
Notice how Make provides conditional code that lets you decide what variables to define in this system.
Building a C++ Application¶
We are ready to set up modules that will build C++ projects, organized according to the structure we are using. The key to doing this is building a list of files to compile, then link to build the applications we need.
First, let’s set up a module that figures out the names of all source files in
each of the three directories we are using for such files. The one wrinkle is
that we are now going to allow subdirectories in the lib
folder. We need a
way for Make to dig out those file names, including the path to each file.
For example, if we put a source file named component.cpp in the lib/part
folder, I want to tell make to compile lib/part/component.cpp
and build
lib/part/component.o
. Make Makefiles you see require that you list the
subdirectory names, then tell Make to search in each one. I wanted something
simpler, and do not want to list directory names at all. The simple solution to
this is to let Python dig out the file names with a simple script:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | import os
import sys
dir = sys.argv[1]
ext = sys.argv[2]
result = []
for root, dirs, files in os.walk(dir):
for name in files:
if name.endswith(ext):
result.append(os.path.join(root,name))
print(" ".join(result));
|
This script recursively searches all folders and subfolders beginning in the named location for files ending with the specified extension. It returns a list of those names in a form ready for Make to save in a new variable.
With this tool in hand, we can create a list of all source files in each of the two main places application files will be found. We can also create a list of the object files we need from these lists.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | CXX := g++
TARGET := $(PROJNAME)$(EXT)
HDRS := $(shell python mk/pyfind.py include .h)
USRCS := $(wildcard src/*.cpp)
LSRCS := $(shell python mk/pyfind.py lib .cpp)
ALLSRCS := $(USRCS) $(LSRCS)
UOBJS := $(USRCS:.cpp=.o)
LOBJS := $(LSRCS:.cpp=.o)
ifeq ($(PLATFORM), Windows)
DOBJS := $(subst /,\,$(UOBJS)) $(subst /.\,$(LOBJS))
else
DOBJS := $(UOBJS) $(LOBJS)
endif
CFLAGS := -std=c++11 -Iinclude
.PHONY: all
all: $(TARGET) ## build application (default)
$(TARGET): $(UOBJS) $(LOBJS)
g++ -o $@ $^ $(LFLAGS)
%.o: %.cpp
$(CXX) -c -o $@ $< $(CFLAGS)
.PHONY: run
run: $(TARGET) ## launch primary build application
$(PREFIX)$(TARGET)
.PHONY: clean
clean: ## remove all build artifacts
$(RM) $(TARGET) $(DOBJS)
|
Note
To get this to run, I had to work through how Make expands variables. The “:=” operator causes Make to expand a variable that depends on other variables right away. The standard “=” operator waits until the variable is needed to expand things. THis causes all kinds of confusion until you get used to the idea!
There are several new targets defined here:
all
the default rule usually builds the project application
clean
- removes all generated files, leaving only essential files behind
run
- this will run the application, in case you forget its name.
Let’s see if this runs for a simple “Hello World” kind of application.
Here is our first source file:
1 2 3 4 5 6 | #include <iostream>
int main(void) {
std::cout << "Pool Ball Simulator" << std::endl;
}
|
Now, we can build the application using these commands:
rm -f cosc1337 src/main.o
g++ -c -o src/main.o src/main.cpp -std=c++11 -Iinclude
g++ -o cosc1337 src/main.o
./cosc1337
Pool Ball Simulator
Note
I included a “clean” command to make sure you see all files being compiled:
Let’s see what our help system has to say now:
all: build application (default)
run: launch primary build application
clean: remove all build artifacts
debug: display local make variables defined
debug-all: display all make variables defined
help: display help messages
Note
I probably should sort these. Maybe later!
Adding a Library File¶
We need to test building a program with a library file sitting in a
subdirectory of lib
. For this test, I will simply set up a message
function and place it in a file in a subfolder named utilities
under
lib
.
Here is the new main.cpp
1 2 3 4 5 6 7 8 | #include <iostream>
#include "message.h"
int main(void) {
std::cout << "Pool Ball Simulator" << std::endl;
message();
}
|
And the utility function:
1 2 3 4 5 6 | #include <iostream>
void message(void) {
std::cout << "\tmessage from subfolder" << std::endl;
}
|
Again, we can build the application using these commands:
rm -f cosc1337 src/main.o lib/utilities/message.o
g++ -c -o src/main.o src/main.cpp -std=c++11 -Iinclude
g++ -c -o lib/utilities/message.o lib/utilities/message.cpp -std=c++11 -Iinclude
g++ -o cosc1337 src/main.o lib/utilities/message.o
./cosc1337
Pool Ball Simulator
message from subfolder
Do you see that the Makefile
successfully found the files inside subfolders, a
nd the compile ran correctly.
Build Issues¶
The Makefile
system we have at this point is about the same as the simple
one we have been using for our projects up to this point. It does have new
features that make using and extending it easier, though. There is one serious
deficiency here, though!
The main.cpp
file includes a header file, and so does the message.cpp
file. What happens if we modify that header file? The build system should
recognize that and recompile both files to make sure they still work with the
new header file. While we could add this information ourselves, doing so
defeats my goal of not needing to modify this Makefile
system at all to
use it in other projects.
Fortunately, the compiler can figure out these dependencies for us. All we need
to do is add another rule in our cpp-project.mk
file:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | CXX := g++
TARGET := $(PROJNAME)$(EXT)
HDRS := $(shell python mk/pyfind.py include .h)
USRCS := $(wildcard src/*.cpp)
LSRCS := $(shell python mk/pyfind.py lib .cpp)
ALLSRCS := $(USRCS) $(LSRCS)
UOBJS := $(USRCS:.cpp=.o)
LOBJS := $(LSRCS:.cpp=.o)
DEPS := $(USRCS:.cpp=.d) $(LSRCS:.cpp=.d)
ifeq ($(PLATFORM), Windows)
DOBJS := $(subst /,\,$(UOBJS)) $(subst /.\,$(LOBJS))
else
DOBJS := $(UOBJS) $(LOBJS)
endif
CFLAGS := -std=c++11 -Iinclude -MMD
.PHONY: all
all: $(TARGET) ## build application (default)
$(TARGET): $(UOBJS) $(LOBJS)
g++ -o $@ $^ $(LFLAGS)
%.o: %.cpp
$(CXX) -c -o $@ $< $(CFLAGS)
.PHONY: run
run: $(TARGET) ## launch primary build application
$(PREFIX)$(TARGET)
.PHONY: clean
clean: ## remove all build artifacts
$(RM) $(TARGET) $(DOBJS) $(DEPS)
-include $(DEPS)
|
The new lines will create special “dependency” files with a “.d” extension. The compiler can
build these files for us by adding the -MMD
option to the CFLAGS
variable.
Look at the bottom of this file. There we include all these dependency files in
our Makefile
so Make will know about these new dependencies when building files
for us.
Here is the file generated by the compiler:
1 | src/main.o: src/main.cpp include/message.h
|
Now, if we modify the header file, Make will figure out that main.cpp
needs to be recompiled.
C++ Lint Checks¶
Next, we need to add lint checks to see if our code is “good” (according to Google!)
Here is the new module:
1 2 3 4 5 6 7 8 | LINT := cpplint
LINTFLAGS := --repository=. --recursive --extensions=cpp,h --quiet
LINTFLAGS += --filter=-build/include_subdir
.PHONY: lint
lint: $(SRCS) $(HDRS) ## Run cpp lint over project files
@$(LINT) $(LINTFLAGS) include lib src
|
The new target in this file depends on Python, but we have that covered. All we
need to do to use this is install cpplint
in our Python installation.
Let’s see how bad our demo code is now:
make[1]: cpplint: No such file or directory
make[1]: *** [lint] Error 1
Looks like we have some work to do!
Testing with Catch¶
We will use the Catch system for unit testing our C++ code. Here is the module that sets that up:
1 2 3 4 5 6 7 8 9 10 11 12 | TSTSRCS := $(wildcard tests/*.cpp)
TSTOBJS := $(TSTSRCS:.cpp=.o)
DOBJS += $(TSTOBJS)
TSTTGT := testapp$(EXT)
.PHONY: test
test: $(TSTTGT) ## Run project unit tests using catch.hpp
$(PREFIX)$(TSTTGT)
$(TSTTGT): $(TSTOBJS) $(LOBJS)
$(CXX) $(LFLAGS) -o $@ $^
|
You will need to create the tests
folder and place the standard
test_main.cpp
file there. After than add tests as you wish to this folder.
The new test
target will build and run your tests.
Project Documentation¶
Last, but certainly not least, we need ot add project documentation to our setup. In this section, we will be using Python Sphinx, so you need to make sure to add these tools to your Python installation:
$ pip install sphinx
$ pip install cloud-sptheme
Note
There are other themes available for generating nice web pages from your documentation. This theme is pretty popular.
We need to create two folders for this setup:
docs
- where final web pages will be stored
rst
- where we will write our documentation
We are going to set our projects up so the web pages for our documentation can be published using something called “GitHub Pages”, a free service you get for every repository you set up on GithUb.
Do this:
$ mkdir docs
$ mkdir rst
In the rst
folder place these two files:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | # -*- coding: utf-8 -*-
import os
import sys
import cloud_sptheme as csp
sys.path.insert(0, os.path.abspath('.'))
# -- Project information -----------------------------------------------------
project = 'PoolBalls'
copyright = '2019, Roie R. Black'
author = 'Roie R. Black'
version = '0.0.1'
release = '0.0.1'
# -- General configuration ---------------------------------------------------
extensions = [
'sphinx.ext.githubpages',
]
templates_path = ['_templates']
source_suffix = '.rst'
master_doc = 'index'
language = None
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
pygments_style = None
# -- Options for HTML output -------------------------------------------------
html_theme = 'cloud'
html_theme_path = [csp.get_theme_dir()]
html_theme_options = { "roottarget": "index" }
html_static_path = ['_static']
|
This is a configuration file used by Sphinx. You should modify this file to suit your project. Only a few lines need modification for most projects.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | Pool Ball Simulator
###################
:Author: Roie R. Black
:Date: Feb 22, 2019
:Email: rblack@austincc.edu
.. toctree::
:maxdepth: 1
:glob:
introduction/*
Indices and tables
******************
* :ref:`genindex`
* :ref:`search`
|
This is the top level document. This file will automatically include all files
in the introduction
subdirectory in the rst
folder. These files must be
named with a “.rst” extension. To control the order of these files, I usually
name files with numbers at the front. Something like this:
introduction/01-introduction.rst
introduction/02-building-notes.rst
And so on.
Here is the new module for our Makefile
system:
1 2 3 4 5 6 | .PHONY: html
html: ## run Sphinx to generate html pages
cd rst && \
sphinx-build -b html -d _build/doctrees . ../docs
|
With this new module, you can run this command to build your documentation files:
$ make html
When the smoke clears, you will find new HTML pages in the docs
folder. You
can see your pages locally by opening up the index.html
file using any web
browser. (Or, double-click on that file name in your file explorer tool.)
When you push your project to Github, these new web pages will be included on
GitHub as well as your project code files. Once you have the docs
folder
on GitHub open up the settings
menu option on your project page. Scroll
down to the section labeled GitHub Pages
. Click on the box that is labeled
“None” and select the option master branch/docs folder
.
I add a line to my project README file that directs visitors to this project to the correct URL, provided by GitHub:
There, now you have no excuse! Start writing documentation that explains what you are thinking as you create this cool new project!
Wrapping Up¶
Phew, that is a lot to digest in one sitting. This attempt at modularizing a
fairly complex Makefile
is working fairly well now, but is still has a few
issues that need attention. The order in which Make includes files matters
more than I would like. For that reason, I had to name the files so they showed
up in the correct order. I could number the files, using the same scheme I use
for SPhinx, and I may do that later.
I also need to add one thing to this project setup: testing on Travis-CI. I will cover that in a separate note.
You could copy the contents of the mk
folder to a place on your system and
remove that folder from projects. IN that case, you would need to alter the
single “include” line in the main Makefile
. That would greatly simplify
building multiple projects using this system. I have done that for my own
projects, but not for studnt projects. Until you lear what is going on, I feel
it is best to keep the entire project self contained.