Iteratively Improving ML Systems
Repository · Video
We don't want to spend months of time developing a complicated solution only to learn that the entire problem has changed. The main idea here is to close the loop quickly, which involves:
- Create a minimum viable product (MVP) that satisfies a baseline performance.
- Iterate on your solution by using the feedback.
- Constantly reassess to ensure your objective hasn't changed.
Creating the MVP for solutions that requires machine learning often involves going manual before ML.
- deterministic, high interpretability, low-complexity MVP (ex. rule based)
- establish baselines for objective comparisons
- allows you to ship quickly and get feedback from users
What are examples of manual solutions?
- aggregate analytics
- rule-based (feature based heuristics, regex, etc.)
Deploying solutions is actually quite easy (from an engineering standpoint) but maintaining and iterating is the challenge. This is why we need to close the loop quickly first, so we can properly experiment with a process. This allows to change one thing as a time (improve data, models, etc.), evaluate it and adapt as necessary.
- collect signals from UI/UX to best approximate performance
- determine window / rolling performances on overall and key slices of data
- monitor (performance, concept drift, etc.) to know when to update
- identify subsets of data that are worth labeling / oversampling
- address iteration bottlenecks (ex. data quality checks)
- constantly reassess your objective
Read more about the maintenance and iteration challenges in our data-driven development lesson.
While it may be easy (and justified) to say "don't use machine learning" as your first attempt at a solution, your leadership may not like that. Most likely, you're being paid to work on problems where traditional software isn't able to deliver.
Justifying simple solutions
How can we justify the time for implementing simple, rule-based, solutions as the first solution to leadership?
- helps us understand the problem and discover what you don't know
- helps us quickly identify shortcomings in the data, system, etc.
- ease of experimentation: once we set up our system, we can easily evaluate future solutions
- allows for conversations with domain experts for feature engineering
- solution allows for v1 which can start the data collection process required for more sophisticated solutions
For our solution, we'll have an initial set of baselines where we'll start with a rule-based approach and then slowly add complexity (rule-based → regression → CNN). Though you are welcome to use any stochastic model you like because all the subsequent lessons are model agnostic.
For the purpose of this course, even our MVP will be an ML model, however we would normally deploy the rule-based approach first as long as it satisfies a performance threshold so we can quickly close the loop.
As for monitoring and iterating on our solution, we'll be looking at things like overall performance, class specific performances, # of relevant tags, etc. We'll also create workflows to look at new data for anomalies, apply active learning, ease the annotation process, etc. More on this in our testing and monitoring lessons.
To cite this lesson, please use:
1 2 3 4 5 6