Retraining, Essential.

Examined in practice. 

Published in  Towards Data Science,      By Claire Longo

Embracing Automated Retraining

How to move away from retraining at a set cadence (or not at all) in favor of a dynamic approach

This piece was co-authored by Trevor LaViale

While the industry has invested a lot in processes and techniques for knowing when to deploy a model into production, there is arguably less collective knowledge on the equally important task of knowing when to retrain a model. In truth, knowing when to retrain a model is hard due to factors like delays in feedback or labels for live predictions. In practice, many models are in production with no retraining at all, use manual retraining methods, or are retraining without optimizing or studying the cadence.

This post is written to help data scientists and machine learning engineering teams embrace automated retraining.

Approaches for Retraining

There are two core approaches to automated model retraining:

Fixed: Retraining a set cadence (e.g., daily, weekly, monthly)

Dynamic: Ad-hoc triggered retraining based on model performance metrics.

While the fixed approach is straightforward to implement, there are some drawbacks. Compute costs can be higher than necessary, and the frequent retraining can lead to inconsistencies from one model to another, while infrequent retraining schedules can lead to a stale model.

The dynamic approach can prevent models from going stale, and optimize the compute cost. While there are numerous approaches to retraining, here are some recommended best practices for dynamic model retraining that will keep models healthier and performant.

Generalized Retraining Architecture