Skip to content

Makefiles

View all lessons
An automation tool that organizes commands for our application's processes.
Goku Mohandas
· ·
Repository

📬  Receive new lessons straight to your inbox (once a month) and join 20K+ developers in learning how to responsibly deliver value with ML.

Intuition

We have just started and there are already so many different commands to keep track of. To help with this, we're going to use a Makefile which is a automation tool that organizes our commands. This makes it very easy for us to organize relevant commands as well as organize it for others who may be new to our application.

Components

Inside our Makefile, we can see a list of rules (help, venv, clean, etc.). These rules have a target which can sometimes have prerequisites that need to be met (can be other targets) and on the next line a Tab followed by a recipe which specifies how to create the target.

1
2
3
# Makefile
target: prerequisites
<TAB> recipe

Targets

We can execute any of the rules by typing make <target>:

# View all rules
$ make help
👉  Commands:
    venv   : creates development environment.
    style  : runs style formatting.
    clean  : cleans all unecessary files.
    dvc    : pushes versioned artifacts to blob storage.
    test   : run non-training tests.

# Make a target
$ make venv
python3 -m venv venv
...

PHONY

A Makefile is called as such because traditionally the targets are supposed to be files we can make. However, Makefiles are also commonly used as command shortcuts which can lead to confusion when a file with a certain name exists and a Makefile rule with the same name exists! For example if you a directory called docs and a target in your Makefile called docs, when you run make docs you'll get this message:

$ make docs
make: `docs` is up to date.

We can fix this by defining a PHONY target in our makefile by adding this line above the target: 

1
.PHONY: <target_name>

Most of the rules in our Makefile will require the PHONY target because we want them to execute even if there is a file sharing the target's name. An exception to this is the venv target because we don't want to create a venv directory if it already exists.

Prerequisites

Before we make a target, we can attach prerequisites to them. These can either be file targets that must exist or PHONY target commands that need to be executed prior to making this target. We use the style target as a prerequisite for the clean target so that all files are formatted appropriately prior to cleaning them.

1
2
3
4
5
6
7
8
# Cleaning
.PHONY: clean
clean: style
    find . -type f -name "*.DS_Store" -ls -delete
    find . | grep -E "(__pycache__|\.pyc|\.pyo)" | xargs rm -rf
    find . | grep -E ".pytest_cache" | xargs rm -rf
    find . | grep -E ".ipynb_checkpoints" | xargs rm -rf
    rm -f .coverage

Variables

We can also set and use variables inside our Makefile to organize all of our rules.

  • We can set the variables directly inside the Makefile. If the variable isn't defined in the Makefile, then it would default to any environment variable with the same name. 

    1
2
3
4
5
6
    # Set variable
MESSAGE := "hello world"

# Use variable
greeting:
    @echo ${MESSAGE}

  • We can also use variables passed in when executing the rule like so (ensure that the variable is not overriden inside the Makefile): 

    1
    make greeting MESSAGE="hi"

Shells

Each line in a recipe for a rule will execute in a separate sub-shell. However for certain recipes such as activating a virtual environment and loading packages, we want to execute all steps in one shell. To do this, we can add the .ONESHELL special target above any target.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# Environment
.ONESHELL:
venv:
    python3 -m venv ${name}
    source ${name}/bin/activate
    python -m pip install --upgrade pip setuptools wheel
    python -m pip install -e ".[dev]" --no-cache-dir
    pre-commit install
    pre-commit autoupdate
    pip uninstall dataclasses -y

However this is only available in Make version 3.82 and above and most Macs currently user version 3.81. You can either update to the current version or chain your commands with &&.

1
2
3
4
5
6
7
8
9
# Environment
venv:
    python3 -m venv ${name}
    source ${name}/bin/activate && \
    python -m pip install --upgrade pip setuptools wheel && \
    python -m pip install -e ".[dev]" --no-cache-dir && \
    pre-commit install && \
    pre-commit autoupdate && \
    pip uninstall dataclasses -y

Note

There's a whole lot more to Makefiles but this is plenty for most applied ML projects.

To cite this lesson, please use:

1
2
3
4
5
6
@article{madewithml,
    title  = "Makefiles - Made With ML",
    author = "Goku Mohandas",
    url    = "https://madewithml.com/courses/mlops/makefile/"
    year   = "2021",
}