Building Reliable AI Models

 Below quite interesting,  here just the intro, full article linked to. Reviewing.  I note the point about changing data in context, which I always emphasize in models. 

7 Techniques for Building Reliable AI Models  By Beena Ammanath in future.16z.com

This is an edited excerpt from Trustworthy AI: A Business Guide for Navigating Trust and Ethics in AI by Beena Ammanath (Wiley, March 2022). Ammanath is executive director of the Global Deloitte AI Institute and leads Trustworthy & Ethical Technology at Deloitte. She has held leadership positions in artificial intelligence and data science at multiple companies, and is the founder of Humans For AI, an organization dedicated to increasing diversity in AI.

With AI model training, datasets are a proxy for the real world. Models are trained on one dataset and tested against another, and if the results are similar, there is an expectation that the model functions can translate to the operational environment. What works in the lab should work consistently in the real world, but for how long? Perfect operating scenarios are rare in AI, and real-world data is messy and complex. This has led to what leading AI researcher Andrew Ng called a “proof-of-concept-to-production gap,” where models train as desired but fail once they are deployed. It is partly a problem of robustness and reliability.

When outputs are inconsistently accurate and become worse over time, the result is uncertainty. Data scientists are challenged to build provably robust, consistently accurate AI models in the face of changing real-world data. In the information flux, the algorithm can meander away, with small changes in input cascading into large shifts in function.

To be sure, not all tools operate in environments prone to dramatic change, and not all AI models present the same levels of risk and consequence if they become inaccurate or undependable. The task for enterprises as they grow their AI footprint is to weigh robustness and reliability as a component of their AI strategy and align the processes, people, and technologies that can manage and correct for errors in a dynamic environment. .... '