What are Our Models Learning?

In some of my earliest work in statistics, we were informed of the "Clever Hans Effect" Which basically means your model is learning a pattern in the data which is unrelated to an the answer you seek.  It may work on the training and test sets, but the prediction evidence comes from some other  evidence hidden in the system.  This can happen when you have too much data,  or you are carrying along variables that are irrelevant.  The model might now work appropriately, but later drift to other results. Dangerous in autonomous systems.  In social data it can occur when humans hint at results  they want, which is why double-blind tests are constructed.   Overall rather important as we depend more on ML. ....

Unmasking Clever Hans predictors and assessing what machines really learn
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek & Klaus-Robert Müller 
Nature Communications volume 10, Article number: 1096 (2019)  

Abstract
Current learning machines have successfully solved hard application problems, reaching high accuracy and displaying seemingly intelligent behavior. Here we apply recent techniques for explaining decisions of state-of-the-art learning machines and analyze various tasks from computer vision and arcade games. This showcases a spectrum of problem-solving behaviors ranging from naive and short-sighted, to well-informed and strategic. We observe that standard performance evaluation metrics can be oblivious to distinguishing these diverse problem solving behaviors. 

Furthermore, we propose our semi-automated Spectral Relevance Analysis that provides a practically effective way of characterizing and validating the behavior of nonlinear learning machines. This helps to assess whether a learned model indeed delivers reliably for the problem that it was conceived for. Furthermore, our work intends to add a voice of caution to the ongoing excitement about machine intelligence and pledges to evaluate and judge some of these recent successes in a more nuanced manner.

Introduction
Artificial intelligence systems, based on machine learning (ML), are increasingly assisting our daily life. They enable industry and the sciences to convert a never ending stream of data—which per se is not informative—into information that may be helpful and actionable. ML has become a basis of many services and products that we use.

While it is broadly accepted that the nonlinear ML methods being used as predictors to maximize some prediction accuracy, are effectively (with few exceptions, such as shallow decision trees) black boxes; this intransparency regarding explanation and reasoning is preventing a wider usage of nonlinear prediction methods in the sciences (see Fig. 1a why understanding nonlinear learning machines is difficult). Due to this black-box character, a scientist may not be able to extract deep insights about what the nonlinear system has learned, despite the urge to unveil the underlying natural structures. In particular, the conclusion in many scientific fields has so far been to prefer linear models1,2,3,4 in order to rather gain insight (e.g. regression coefficients and correlations) even if this comes at the expense of predictivity.  ... " 

The article is also discussed here:
https://towardsdatascience.com/deep-learning-meet-clever-hans-3576144dc5a9