Iteratively Improving ML Systems

Improving on our solution iteratively over time.

Goku Mohandas

📬 Receive new lessons straight to your inbox (once a month) and join 30K+ developers in learning how to responsibly deliver value with ML.

Intuition

We don't want to spend months of time developing a complicated solution only to learn that the entire problem has changed. The main idea here is to close the loop quickly, which involves:

Create a minimum viable product (MVP) that satisfies a baseline performance.
Iterate on your solution by using the feedback.
Constantly reassess to ensure your objective hasn't changed.

Product development cycle

Creating the MVP for solutions that requires machine learning often involves going manual before ML.

deterministic, high interpretability, low-complexity MVP (ex. rule based)
establish baselines for objective comparisons
allows you to ship quickly and get feedback from users

Manual solutions

What are examples of manual solutions?

Show answer

visualizations
aggregate analytics
rule-based (feature based heuristics, regex, etc.)

Challenges

Deploying solutions is actually quite easy (from an engineering standpoint) but maintaining and iterating is the challenge. This is why we need to close the loop quickly first, so we can properly experiment with a process. This allows to change one thing as a time (improve data, models, etc.), evaluate it and adapt as necessary.

collect signals from UI/UX to best approximate performance
determine window / rolling performances on overall and key slices of data
monitor (performance, concept drift, etc.) to know when to update
identify subsets of data that are worth labeling / oversampling
address iteration bottlenecks (ex. data quality checks)
constantly reassess your objective

Read more about the maintenance and iteration challenges in our data-driven development lesson.

Real-world

While it may be easy (and justified) to say "don't use machine learning" as your first attempt at a solution, your leadership may not like that. Most likely, you're being paid to work on problems where traditional software isn't able to deliver.

Justifying simple solutions

How can we justify the time for implementing simple, rule-based, solutions as the first solution to leadership?

Show answer

helps us understand the problem and discover what you don't know
helps us quickly identify shortcomings in the data, system, etc.
ease of experimentation: once we set up our system, we can easily evaluate future solutions
allows for conversations with domain experts for feature engineering
solution allows for v1 which can start the data collection process required for more sophisticated solutions

Application

For our solution, we'll have an initial set of baselines where we'll start with a rule-based approach and then slowly add complexity (rule-based → regression → CNN). Though you are welcome to use any stochastic model you like because all the subsequent lessons are model agnostic.

For the purpose of this course, even our MVP will be an ML model, however we would normally deploy the rule-based approach first as long as it satisfies a performance threshold so we can quickly close the loop.

As for monitoring and iterating on our solution, we'll be looking at things like overall performance, class specific performances, # of relevant tags, etc. We'll also create workflows to look at new data for anomalies, apply active learning, ease the annotation process, etc. More on this in our testing and monitoring lessons.

Resources

To cite this lesson, please use:

@article{madewithml,
    author       = {Goku Mohandas},
    title        = { Iteration - Made With ML },
    howpublished = {\url{https://madewithml.com/}},
    year         = {2021}
}