Retraining, Essential.

Examined in practice. 

Published in  Towards Data Science,      By Claire Longo

Embracing Automated Retraining

How to move away from retraining at a set cadence (or not at all) in favor of a dynamic approach

This piece was co-authored by Trevor LaViale

While the industry has invested a lot in processes and techniques for knowing when to deploy a model into production, there is arguably less collective knowledge on the equally important task of knowing when to retrain a model. In truth, knowing when to retrain a model is hard due to factors like delays in feedback or labels for live predictions. In practice, many models are in production with no retraining at all, use manual retraining methods, or are retraining without optimizing or studying the cadence.

This post is written to help data scientists and machine learning engineering teams embrace automated retraining.

Approaches for Retraining

There are two core approaches to automated model retraining:

Fixed: Retraining a set cadence (e.g., daily, weekly, monthly)

Dynamic: Ad-hoc triggered retraining based on model performance metrics.

While the fixed approach is straightforward to implement, there are some drawbacks. Compute costs can be higher than necessary, and the frequent retraining can lead to inconsistencies from one model to another, while infrequent retraining schedules can lead to a stale model.

The dynamic approach can prevent models from going stale, and optimize the compute cost. While there are numerous approaches to retraining, here are some recommended best practices for dynamic model retraining that will keep models healthier and performant.

Generalized Retraining Architecture

Machines Unlearning

Perhaps yet more important, unlearn in new contexts.  Say for maintenance. With new or changed metadata.  And help us understand the difference.

Now that machines can learn, can they unlearn?

Researchers see if they can remove sensitive data without retraining AI from scratch.  By Tom Simonite, Wired.com

Companies of all kinds use machine learning to analyze people’s desires, dislikes, or faces. Some researchers are now asking a different question: How can we make machines forget?

A nascent area of computer science dubbed machine unlearning seeks ways to induce selective amnesia in artificial intelligence software. The goal is to remove all trace of a particular person or data point from a machine-learning system, without affecting its performance.

If made practical, the concept could give people more control over their data and the value derived from it. Although users can already ask some companies to delete personal data, they are generally in the dark about what algorithms their information helped tune or train. Machine unlearning could make it possible for a person to withdraw both their data and a company’s ability to profit from it.

Although intuitive to anyone who has rued what they shared online, that notion of artificial amnesia requires some new ideas in computer science. Companies spend millions of dollars training machine-learning algorithms to recognize faces or rank social posts, because the algorithms often can solve a problem more quickly than human coders alone. But once trained, a machine-learning system is not easily altered, or even understood. The conventional way to remove the influence of a particular data point is to rebuild a system from the beginning, a potentially costly exercise. “This research aims to find some middle ground,” says Aaron Roth, a professor at the University of Pennsylvania who is working on machine unlearning. “Can we remove all influence of someone’s data when they ask to delete it, but avoid the full cost of retraining from scratch?”