Recently, I’ve had to rediscover some of my knowledge on makefiles for a work project. As these things go, when I had knowledge on makefiles it had all seemed obvious; so, no need to write it down, right?
Anyway, this is my attempt at properly archiving this knowledge for when I need it again five to ten years from now.
In case you don’t know, make is a build tool originally
created by Stuart Feldman in 1976 at Bell Labs. While make
is most commonly associated with C and C++ projects, it can be used in
any context where you want to rebuild only the relevant parts of
something after a dependency has changed.
Nowadays, many programming languages ship with their own build
system, like Rust/Cargo, Zig, or Go; but make is not
limited to just pure programming languages or build processes that yield
a library or executable.
Let’s imagine, for example, that we wanted to write posts for our blog in Markdown. In order for people on the internet to read those posts, the Markdown files will need to be converted to HTML. Usually, this part is done by a static site generator, such as Hakyll, or Pelican. This blog also used to be built using Hakyll, as you can read here.
These tools work great and they make it easy to get started! But let’s assume for a moment, that we had a lazy afternoon, out wife is out of town, we’re sitting on the couch with our cat, and we have all this newly rediscovered knowledge on makefiles. Could we build our blog with just that? How hard could it be, right?
pandoc
- Turning Markdown into HTML Since 2006
The tool that many static site generators use internally is called pandoc, a universal document
converter.
pandoc really deserves its own blog post, so for the
purposes of our little project here, it’s enough to know that it
essentially converts any markup format - such as Markdown - into any
other markup format - such as HTML. It can also work with templates
which describe how a standalone document should look like.
If you’ve looked at the templating languages used by various static
site generators, you may have noticed that they are all very similar
(not to say identical). The reason is that they use pandoc
internally, and the templating language is the one pandoc
uses. Case in point, we will not have to adjust our existing templates
for this project.
The command for turning one of our Markdown posts
(foo.md) into the appropriate HTML (foo.html)
looks like this:
pandoc --standalone --template=templates/post.html -o foo.html foo.md
The Structure of a Blog
The source code of our blog has a pretty simple structure; you have some posts written in Markdown, some templates that describe how to turn this Markdown into HTML, and some CSS to make it look a little more fancy:
posts/
2022-05-10-foo.md
...
templates/
post.html
...
css/
style.css
The index.html will have a list of recent blog posts and
some general information about the blog, like an Atom feed, a git forge,
and a contact email address.
From a programmer’s point of view, we have a compilation step which
turns each Markdown file into an HTML file, followed by the generation
of the index.html and atom.xml, and finally
bundling these outputs with the CSS into a single directory with the
following structure:
out/
atom.xml
index.html
css/
style.css
posts/
2022-05-10-foo.html
...
Up to this point, we could theoretically perform these steps manually each time we write a blog post, especially if we don’t publish a new one very frequently. However, that’s not really satisfying, is it?
Let’s imagine we had many blog posts and compilation of each
individual blog post took a long time. We would only ever want to
recompile what’s necessary leaving everything else untouched, right?
That’s where make comes in.
make -
Building Software Since 1976
make works based on three things: targets,
their prerequisites, and the recipes of how to build
the targets. In a makefile, this looks like this:
[target]: [prerequisites]
[recipe]For example, we can use our pandoc call as a recipe to
build our foo.html whenever the prerequisite
foo.md or the post.html template changes.
Let’s put that in a file called Makefile.
foo.html: foo.md templates/post.html
pandoc --standalone --template=templates/post.html -o foo.html foo.mdWe can then build this by running this on the command line:
make foo.html
To determine whether there is anything that needs to be done,
make checks whether the last-modified time of the target is
older than the last-modified time of any of its prerequisites. If so, it
rebuilds the target, otherwise it does not do anything.
Running make without specifying a target will build the
first target in the list, so we can specify a top level target that
builds our whole blog by specifying all relevant targets as a
prerequisite:
blog: out/foo.html out/bar.html
out/foo.html: posts/foo.md templates/post.html
pandoc --standalone --template=templates/post.html -o out/foo.html posts/foo.md
out/bar.html: posts/bar.md templates/post.html
pandoc --standalone --template=templates/post.html -o out/bar.html posts/bar.mdSo far, we’re only compiling the Markdown files into HTML files and
putting them in the out/posts directory. We’re still
missing the atom.xml feed and the index.html,
so let’s define two additional targets and add them to the prerequisites
for our main target.
blog: out/foo.html out/bar.html out/index.html out/atom.xml
out/index.html: out/foo.html out/bar.html build_index.sh templates/index_top.html templates/index_bottom.html
sh build_index.sh
out/atom.xml: out/foo.html out/bar.html build_feed.sh
sh build_feed.shFor the sake of brevity, we will skip over how we actually generate these two files. We encapsulate the logic in two small bash scripts. You can find them in the repo for this blog if you are interested.
Now, this is already a workable setup, but we still have to change
our makefile every time we create a new blog post and add the new
dependency. This would become rather tedious. Ideally, we want to just
type make after adding a post and see that all the files
are created/updated correctly without ever touching our makefile. To
achieve this, we will use some built-in functions of
make.
OUT_DIR := out
sources := $(wildcard posts/*.md)
posts := $(subst .md,.html,$(sources))
posts := $(subst posts/,$(OUT_DIR)/posts/,$(posts))First we define a variable OUT_DIR to hold the name of
the directory where we want to create all the files, so that it’s easy
to change if we ever feel like it. Then we define another variable
sources which holds the names of all Markdown files in
posts/. Then we replace the .md with
.html, and prefix the names with the value in
OUT_DIR.
For our example above, the posts variable looks like
this: out/posts/foo.html out/posts/bar.html.
This allows us to make the prerequisites for our various targets static:
blog: $(posts) $(OUT_DIR)/index.html $(OUT_DIR)/atom.xml
$(OUT_DIR)/index.html: $(posts) build_index.sh templates/index_top.html templates/index_bottom.html
sh build_index.sh
$(OUT_DIR)/atom.xml: $(posts) build_feed.sh
sh build_feed.shBut we still need to create a new target for each blog post. To get around this we can use pattern rules:
$(posts): $(OUT_DIR)/posts/%.html: posts/%.md templates/post.html
pandoc --standalone --template=templates/post.html -o $@ $<Don’t worry, this looks much worse than it is. This essentially goes
through the individual targets in posts and tries to match
the pattern $(OUT_DIR)/posts/%.html where % is
a wildcard. If the pattern matches, it defines a corresponding target
with the prerequisites posts/%.md templates/post.html where
% contains the value that was matched in the wildcard
before.
Finally, $@ contains the current target, and
$< contains the first prerequisite. These are built-in
automatic variables in make. There are a few more of these,
but we won’t need them.
As for the CSS, it doesn’t have any prerequisites (other than itself)
or need for code generation, so we simply copy it into the appropriate
directory and add it as a prerequisite for the blog
target.
blog: $(posts) $(OUT_DIR)/index.html $(OUT_DIR)/atom.xml $(OUT_DIR)/css/style.css
$(OUT_DIR)/css/style.css: css/style.css
cp $< $@And that’s it, now we can just type make on the command
line, and all changes will be updated on our blog.
The Extra Mile and Phoniness
So, what’s left? Well, it’s common practice in makefiles to define a
clean target which removes some or all build results, for
example, intermediate files that are no longer needed after compilation
has finished. It would also be nice to have a shortcut to actually
publishing the updated version of the blog, e.g., a publish
target.
Both of these targets, as well as our top-level blog
target have one thing in common: they are not actual files on the file
system. While we do not plan to have any files named like this in our
blog, it is a good opportunity to introduce one final concept of
make: phoniness.
A target marked as .PHONY will always be built when
called. Even if there is a file that matches the target name and none of
its prerequisites are newer than it. So, let’s define our final two
targets and add some phoniness to the relevant targets.
.PHONY: blog clean publish
blog: $(posts) $(OUT_DIR)/index.html $(OUT_DIR)/css/style.css $(OUT_DIR)/atom.xml
publish: blog
tar -C $(OUT_DIR) -cvz . > $(OUT_DIR).tar.gz
hut pages publish -d blog.erikwastaken.dev $(OUT_DIR).tar.gz
clean:
rm -rf $(OUT_DIR)Further Reading
If this has made you interested in make and makefiles,
there’s more information at Learn
Makefiles. I also recommend Clay Dowling’s blog series on makefiles.
They’re what got me started on makefiles years ago and are quite
enjoyable reads!