Makefiles

An automation tool that organizes commands for our application's processes.

Goku Mohandas

📬 Receive new lessons straight to your inbox (once a month) and join 30K+ developers in learning how to responsibly develop, deploy & maintain ML.

Intuition

Even though we're only about halfway through the course, there are already so many different commands to keep track of. To help organize everything, we're going to use a Makefile which is a automation tool that organizes our commands. We'll start by create this file in our project's root directory.

touch Makefile

At the top of our Makefile we need to specify the shell environment we want all of our commands to execute in:

# Makefile
SHELL = /bin/bash

Components

Inside our Makefile, we'll be creating a list of rules. These rules have a target which can sometimes have prerequisites that need to be met (can be other targets) and on the next line a Tab followed by a recipe which specifies how to create the target.

# Makefile
target: prerequisites
<TAB> recipe

For example, if we wanted to create a rule for styling our files, we would add the following to our Makefile:

# Styling
style:
    black .
    flake8
    python3 -m isort .

Tabs vs. spaces

Makefiles require that indention be done with a , instead of spaces where we'll receive an error:

Makefile:: *** missing separator.  Stop.

Luckily, editors like VSCode automatically change indentation to tabs even if other files use spaces.

Targets

We can execute any of the rules by typing make <target> in the terminal:

# Make a target
$ make style

black .
All done! ✨ 🍰 ✨
8 files left unchanged.
flake8
python3 -m isort .
Skipped 1 files

Similarly, we can set up our Makefile for creating a virtual environment:

# Environment
venv:
    python3 -m venv venv
    source venv/bin/activate && \
    python3 -m pip install pip setuptools wheel && \
    python3 -m pip install -e .

&& signifies that we want these commands to execute in one shell (more on this below).

PHONY

A Makefile is called as such because traditionally the targets are supposed to be files we can make. However, Makefiles are also commonly used as command shortcuts, which can lead to confusion when a Makefile target and a file share the same name! For example if we have a file called venv (which we do) and a target in your Makefile called venv, when you run make venv we'll get this message:

$ make venv

make: `venv' is up to date.

In this situation, this is the intended behavior because if a virtual environment already exists, then we don't want ot make that target again. However, sometimes, we'll name our targets and want them to execute whether it exists as an actual file or not. In these scenarios, we want to define a PHONY target in our makefile by adding this line above the target:

.PHONY: <target_name>

Most of the rules in our Makefile will require the PHONY target because we want them to execute even if there is a file sharing the target's name.

# Styling
.PHONY: style
style:
    black .
    flake8
    isort .

Prerequisites

Before we make a target, we can attach prerequisites to them. These can either be file targets that must exist or PHONY target commands that need to be executed prior to making this target. For example, we'll set the style target as a prerequisite for the clean target so that all files are formatted appropriately prior to cleaning them.

# Cleaning
.PHONY: clean
clean: style
    find . -type f -name "*.DS_Store" -ls -delete
    find . | grep -E "(__pycache__|\.pyc|\.pyo)" | xargs rm -rf
    find . | grep -E ".pytest_cache" | xargs rm -rf
    find . | grep -E ".ipynb_checkpoints" | xargs rm -rf
    find . | grep -E ".trash" | xargs rm -rf
    rm -f .coverage

Variables

We can also set and use variables inside our Makefile to organize all of our rules.

We can set the variables directly inside the Makefile. If the variable isn't defined in the Makefile, then it would default to any environment variable with the same name.
```
# Set variable
MESSAGE := "hello world"

# Use variable
greeting:
    @echo ${MESSAGE}
```
We can also use variables passed in when executing the rule like so (ensure that the variable is not overridden inside the Makefile):
```
make greeting MESSAGE="hi"
```

Shells

Each line in a recipe for a rule will execute in a separate sub-shell. However for certain recipes such as activating a virtual environment and loading packages, we want to execute all steps in one shell. To do this, we can add the .ONESHELL special target above any target.

# Environment
.ONESHELL:
venv:
    python3 -m venv venv
    source venv/bin/activate
    python3 -m pip install pip setuptools wheel
    python3 -m pip install -e .

However this is only available in Make version 3.82 and above and most Macs currently use version 3.81. You can either update to the current version or chain your commands with && as we did previously:

# Environment
venv:
    python3 -m venv venv
    source venv/bin/activate && \
    python3 -m pip install pip setuptools wheel && \
    python3 -m pip install -e .

Help

The last thing we'll add to our Makefile (for now at least) is a help target to the very top. This rule will provide an informative message for this Makefile's capabilities:

.PHONY: help
help:
    @echo "Commands:"
    @echo "venv    : creates a virtual environment."
    @echo "style   : executes style formatting."
    @echo "clean   : cleans all unnecessary files."

make help

Commands:
venv    : creates a virtual environment.
style   : executes style formatting.
clean   : cleans all unnecessary files.

There's a whole lot more to Makefiles but this is plenty for most applied ML projects.

To cite this content, please use:

@article{madewithml,
    author       = {Goku Mohandas},
    title        = { Makefile - Made With ML },
    howpublished = {\url{https://madewithml.com/}},
    year         = {2022}
}