Iteration
Improving on our solution iteratively over time.
Intuition
We don't want to spend months of time developing a complicated solution only to learn that the entire problem has changed. The main idea here is to close the loop, which involves: 1. Create a minimum viable product (MVP) that satisfies a baseline performance. 2. Iterate on your solution by using the feedback. 3. Constantly reassess to ensure your objective hasn't changed.
Creating the MVP for solutions that requires machine learning often involves going manual before ML.
- deterministic, high interpretability, low-complexity MVP (ex. rule based)
- establish baselines for objective comparisons
- allows you to ship quickly and get feedback from users
Deploying solutions is actually quite easy (from an engineering POV) but maintaining and iterating upon it is quite the challenge.
- collect signals from UX
- monitor (performance, concept drift, etc.) to know when to update
- constantly reassess your objective
- iteration bottlenecks (ex. data quality checks)
Application
For our solution, we'll have an initial set of baselines where we'll start with a rule-based approach and then slowly add complexity (regression → CNN → Transformers).
Note
For the purpose of this course, even our MVP will be an ML model, however we would normally deploy the rule-based approach first as long as it satisfies a performance threshold.
As for monitoring and iterating on our solution, we'll be looking at things like overall performance, class specific performances, # of relevant tags, etc. We'll also create workflows to look at new data for anomalies, apply active learning, ease the annotation process, etc.