NASA Webb Space Telescope Data Could be Misinterpreted

Data May be Misinterpreted?   Is always an issue.  Models that use the data also must be good.  CNN should look to themselves as a glaring example of bad interpretation.. 

ACM TECHNEWS

NASA Webb Space Telescope Data Could Be Misinterpreted   By CNN

Massachusetts Institute of Technology (MIT) scientists warn of a possible disconnect between the power of the U.S. National Aeronautics and Space Administration's James Webb Space Telescope (JWST) and current datasets.

"The data we will be getting from the JWST will be incredible, but ... our insights will be limited if our models don't match it in quality," said MIT's Clara Sousa-Silva.

New research suggests some of the light-decoding tools researchers employ to understand exoplanets cannot fully accommodate the JWST's spectral data, which could undermine the search for extraterrestrial life.

The researchers are calling for improvements to opacity models to enable them to handle the JWST's subtle measurements.

From CNN

View Full Article  

Interpretable Machine Learning and Diagnosis

Well covered in ACM pieces of late, useful takes. 

ACM PRACTICE

Interpretable Machine Learning: Moving from Mythos to Diagnostics

By Valerie Chen, Jeffrey Li, Joon Sik Kim, Gregory Plumb, Ameet Talwalkar

Communications of the ACM, August 2022, Vol. 65 No. 8, Pages 43-50  10.1145/3546036

The emergence of machine learning as a society-changing technology in the past decade has triggered concerns about people's inability to understand the reasoning of increasingly complex models. The field of interpretable machine learning (IML) grew out of these concerns, with the goal of empowering various stakeholders to tackle use cases, such as building trust in models, performing model debugging, and generally informing real human decision-making.7,10,17

Yet despite the flurry of IML methodological development over the past several years, a stark disconnect characterizes the current overall approach. As shown in Figure 1, IML researchers develop methods that typically optimize for diverse but narrow technical objectives, yet their claimed use cases for consumers remain broad and often underspecified. Echoing similar critiques about the field,17 it has thus remained difficult to evaluate these claims sufficiently and to translate methodological advances into widespread practical impact. .... '

Making Decision Trees Accurate and Explainable

Explaining AI, Decision Trees

Berkeley Artificial Intelligence Research
Making Decision Trees Accurate Again: Explaining What Explainable AI Did Not   By Alvin Wan  

The interpretability of neural networks is becoming increasingly necessary, as deep learning is being adopted in settings where accurate and justifiable predictions are required. These applications range from finance to medical imaging. However, deep neural networks are notorious for a lack of justification. Explainable AI (XAI) attempts to bridge this divide between accuracy and interpretability, but as we explain below, XAI justifies decisions without interpreting the model directly.

What is “Interpretable”?
Defining explainability or interpretability for computer vision is challenging: What does it even mean to explain a classification for high-dimensional inputs like images? As we discuss below, two popular definitions involve saliency maps and decision trees, but both approaches have their weaknesses. .... " 

Complexity of Showing How AI Decides

Thoughtful piece on the problem of how to reasonably provide explanation.  Have now looked at several means of addressing the problem.   Consider carefully the decisions being driven and risks involved.

Grilling the answers: How businesses need to show how AI decides

As artificial intelligence becomes more widespread, so the need to render it explainable increases. How can companies navigate the technical and ethical challenges?
By   Lindsay Clark in Computerweek

Show your working: generations of mathematics students have grown up with this mantra. Getting the right answer is not enough. To get top marks, students must demonstrate how they got there. Now, machines need to do the same.

As artificial intelligence (AI) is used to make decisions affecting employment, finance or justice, as opposed to which film a consumer might want to watch next, the public will insist it explains its working.

Sheffield University professor of AI and robotics Noel Sharkey drove home the point when he told The Guardian that decisions based on machine learning could not be trusted because they were so “infected with biases”.

Sharkey called for an end to the application of machine learning to life-changing decisions until they could be proven safe in the same way that drugs are introduced into healthcare.  ... " 

Video: Interpretable Machine Learning

I mentioned this article in an earlier post, where I discuss in more detail, here here a short video introduction.

Techniques for Interpretable Machine Learning from CACM on Vimeo.

Mengnan Du and Xia Hu discuss "Techniques for Interpretable Machine Learning," a Review Article in the January 2020 CACM. ... 

Techniques for Interpretable Machine Learning

Very good piece I am reading in the January CACM.  The most important aspect of considering AI-ML type models in the real world.  Good introduction, useful key insights, but ultimately quite technical.  Bottom line is that research is still needed and 'Model explanation and surprising artifacts are often two sides of the same coin'.  Complex models may extract and codify biases and other 'artifacts' of metadata from training data.   Test and re-test under varying context.  Maintenance is more that just tracking performance over time.  Consider embedded models of risk.

I highly recommend subscribing to CACM if you are technically involved.

Techniques for Interpretable Machine Learning
By Mengnan Du, Ninghao Liu, Xia Hu

Communications of the ACM, January 2020, Vol. 63 No. 1, Pages 68-77
10.1145/3359786

Machine learning is progressing at an astounding rate, powered by complex models such as ensemble models and deep neural networks (DNNs). These models have a wide range of real-world applications, such as movie recommendations of Netflix, neural machine translation of Google, and speech recognition of Amazon Alexa. Despite the successes, machine learning has its own limitations and drawbacks. The most significant one is the lack of transparency behind their behaviors, which leaves users with little understanding of how particular decisions are made by these models. Consider, for instance, an advanced self-driving car equipped with various machine learning algorithms does not brake or decelerate when confronting a stopped firetruck. This unexpected behavior may frustrate and confuse users, making them wonder why. Even worse, the wrong decisions could cause severe consequences if the car is driving at highway speeds and might ultimately crash into the firetruck. The concerns about the black-box nature of complex models have hampered their further applications in our society, especially in those critical decision-making domains like self-driving cars.

Interpretable machine learning would be an effective tool to mitigate these problems. It gives machine learning models the ability to explain or to present their behaviors in understandable terms to humans,10 which is called interpretability or explainability and we use them interchangeably in this article. Interpretability would be an indispensable part for machine learning models in order to better serve human beings and bring benefits to society. For end users, explanation will increase their trust and encourage them to adopt machine learning systems. From the perspective of machine learning system developers and researchers, the provided explanation can help them better understand the problem, the data and why a model might fail, and eventually increase the system safety. Thus, there is a growing interest among the academic and industrial community in interpreting machine learning models and gaining insights into their working mechanisms.

Interpretable machine learning techniques can generally be grouped into two categories: intrinsic interpretability and post-hoc interpretability, depending on the time when the interpretability is obtained.23 Intrinsic interpretability is achieved by constructing self-explanatory models which incorporate interpretability directly to their structures. The family of this category includes decision tree, rule-based model, linear model, attention model, and so on. In contrast, the post-hoc one requires creating a second model to provide explanations for an existing model. The main difference between these two groups lies in the trade-off between model accuracy and explanation fidelity. Inherently interpretable models could provide accurate and undistorted explanation but may sacrifice prediction performance to some extent. The post-hoc ones are limited in their approximate nature while keeping the underlying model accuracy intact.  ... "

More Aims at Explainable AI

More interesting moves in providing explanation to AI.  Here from Ga Tech.   As in the previous recent  post, this asks the question:   What does explainable mean?   And to whom in what context?  Simplicity is good, if everyone agrees to its truth and value.

A Breakthrough in Explainable AI   By Joe Dysart in CACM
A new artificial intelligence (AI) agent offers easy-to-understand English explanations of the AI's analysis and decisions.

Such tools of explanation are considered critical by those working in AI who fear users may be reluctant to embrace AI applications that make recommendations whose rationales are shrouded in mystery.

"As AI pervades all aspects of our lives, there is a distinct need for human-centered AI design that makes black-boxed AI systems explainable to everyday users," says Upol Ehsan, a doctoral student in the School of Interactive Computing at the Georgia Institute of Technology (Georgia Tech) and lead author of the study.  "Our work takes a formative step toward understanding the role of language-based explanations and how humans perceive them."

Adds Devi Parikh, an assistant professor in Georgia Tech's School of Interactive Computing, "Just like human teams are more effective when team members are transparent and can understand the rationale behind each others decisions or opinions, human-AI teams will be more effective if AI systems explain themselves and are somewhat interpretable to a human."

Eshan and his research team set out to solve the explainable AI problem by developing an AI agent that could offer easy-to-understand explanations to humans in certain settings.

For their research, the team—including researchers from Georgia Tech, Cornell University, and the University of Kentucky—decided to create an AI agent that would analyze and explain moves in the video game "Frogger." The game is an ideal choice for developing an AI agent, given its simplicity: the entire goal of the game is to move an animated frog across a screen, enabling it to dodge oncoming vehicles and other animated hazards.

The researchers trained their AI agent by first asking gamers to play Frogger as they explained the rationale behind each action they took, move by move."  ... ' 

Machine Learning Interpretability: Free eBook

Its the ultimate need for any problem solving method, interpreting the results you get from complex methods.

Just scanned this new O'Reilly eBook.  30+ pages, nicely done, includes links and technical details.  But could still be utilized by engineering oriented management.   Good simple graphics, and emphasis on readily understood methods like decision trees.   Could have used some more industry specific examples, to indicate the breadth of needs.    Link to a download below. 

The latest data science superpower—interpreting ML
Amp up your superpowers with this free ebook  ... 

Machine learning algorithms are incredibly useful and increasingly complex. But when the complexity outpaces interpretability, human trust suffers, leading to stalled adoption, regulation, and difficulties with model documentation.

That's why the latest data science superpower is the ability to interpret machine learning—amp up your powers with this free ebook.

An Introduction to Machine Learning Interpretability takes you through the basics and provides a set of machine learning techniques, algorithms, and models that will help you improve the accuracy of your predictive models while maintaining interpretability.Machine learning algorithms are incredibly useful and increasingly complex. But when the complexity outpaces interpretability, human trust suffers, leading to stalled adoption, regulation, and difficulties with model documentation.

That's why the latest data science superpower is the ability to interpret machine learning—amp up your powers with this free ebook.

An Introduction to Machine Learning Interpretability takes you through the basics and provides a set of machine learning techniques, algorithms, and models that will help you improve the accuracy of your predictive models while maintaining interpretability.  ..... " 

Interpreting and Securely Using Machine Learning

 Good piece here, which discusses the nature of Trust, Causality, Transferability, Informativeness,

....Yes to that, but ultimately is how you can link, and use the results as part of a current or proposed business process.   Try that first ...

Interpreting Machine Leaning Models: A Myth or Reality?

 Despite the predictive capabilities of supervised machine learning, can we trust the machines? As much as we want the models to be good, we also want them to be interpretable. Yet, the task of interpretation often remains vague.

Despite the proliferation of machine learning into our daily lives ranging from finance to justice, a majority of the users find their models difficult to understand. This lack of a commonly agreed upon definition or the ill-definition of the interpretability means that rather than being a monolithic concept, interpretability embeds various related concepts.

Interpretability is mostly used in the field of supervised learning in comparison to other fields of machine learning such as reinforcement or interactive learning. Existing research studies approach interpretability as a means to establish trust. Yet, it needs to be clarified whether trust refers to the robustness of a model’s performance or to some other properties.

Viewing interpretability simply as a low-level mechanistic understanding of models might be problematic. Despite the capability of machines of discovering causal structure in data, they still are far from being perfect for offering relevant matches for the tasks they are supposed to solve in the real life. One reason for this failure might be the oversimplification of optimization goals so that they fail to fulfill more complicates real-life goals. Another reason might be the unrepresentativeness of the training data of the related deployment ecosystem. Besides, given a model’s complexity, all of parameters, algorithms, factors of human agency need to be taken into account.

Whenever there is a gap between the goals of supervised learning and the costs of a real world deployment setting, demand for interpretability would emerge. Not every real life goal can be coded as simple functions. To give a specific example, an algorithm designed to make hiring decisions would not be able to optimize all of productivity and ethics. So, a formal model that would work within the context of a real-life environment would be a struggle. In order to overcome this struggle, here are some aspects of interpretability to be taken into account: ...  " 

Considering Interpretability

Good look at what interpretability is.  My view is that to interpret means to understand what it means in developed context, and in future.    The context I usually have to most closely deal with is how is it interpreted by decision makers.  Thoughtful and considerable article here,  I am reviewing,

In Machine Learning, the concept of Interpretability is both important and slippery
Via ACMQueue
https://arxiv.org/abs/1606.03490

Computer Science > Machine Learning
The Mythos of Model Interpretability
By Zachary C. Lipton

(Submitted on 10 Jun 2016 (v1), last revised 6 Mar 2017 (this version, v3))
Supervised machine learning models boast remarkable predictive capabilities. But can you trust your model? Will it work in deployment? What else can it tell you about the world? We want models to be not only good, but interpretable. And yet the task of interpretation appears underspecified. Papers provide diverse and sometimes non-overlapping motivations for interpretability, and offer myriad notions of what attributes render models interpretable. Despite this ambiguity, many papers proclaim interpretability axiomatically, absent further explanation. In this paper, we seek to refine the discourse on interpretability. First, we examine the motivations underlying interest in interpretability, finding them to be diverse and occasionally discordant. Then, we address model properties and techniques thought to confer interpretability, identifying transparency to humans and post-hoc explanations as competing notions. Throughout, we discuss the feasibility and desirability of different notions, and question the oft-made assertions that linear models are interpretable and that deep neural networks are not.  ... " 

Interpretability Testing Examined

Testing machine learning interpretability techniques  In O'Reilly

By Patrick Hall,Navdeep Gill,Lingyao Meng 

The importance of testing your tools, using multiple tools, and seeking consistency across various interpretability techniques.

This post contains excerpts from the report “An Introduction to Machine Learning Interpretability,”    ... Read the full report on O'Reilly's learning platform.

Interpreting machine learning models is a pretty hot topic in data science circles right now. Machine learning models need to be interpretable to enable wider adoption of advanced predictive modeling techniques, to prevent socially discriminatory predictions, to protect against malicious hacking of decisioning systems, and simply because machine learning models affect our work and our lives. Like others in the applied machine learning field, my colleagues and I at H2O.ai have been developing machine learning interpretability software for the past 18 months or so.

We were able to give a summary of applied concerns in the interpretability field in an O’Reilly report earlier this year. What follows here are excerpts of that report, plus some new, bonus material. This post will focus on a few important, but seemingly less often discussed, interpretability issues: the approximate nature of machine learning interpretability techniques, and how to test model explanations. ... "

Seeking Interpret-ability and Explanation as Components of our Brain

Modeling the apparent structure and resulting operation of the brain is difficult ....   We have lots of neurons gathering data,  and interacting in ways we do not fully understand.   And that results in high level cognitive concepts, like language or consciousness.   This piece looks at these interactions and seeks to produce some interpretive models based on things we know how to model well enough today, interpret and tag visual scenes.  But that is yet a small and simplistic portion of what the Brain does.  What does it mean for deep intelligence?  Like being given a mass of Lego blocks and being asked to model a city without a map.  You can see the engineers starting to sweat.

The Building Blocks of Interpretability

Interpretability techniques are normally studied in isolation.
We explore the powerful interfaces that arise when you combine them — and the rich structure of this combinatorial space. .... 

Researchers from Google and CMU explore ... 

Chris Olah, Google Brain
Arvind Satyanarayan. Google Brain  .... 

With the growing success of neural networks, there is a corresponding need to be able to explain their decisions — including building confidence about how they will behave in the real-world, detecting model bias, and for scientific curiosity. In order to do so, we need to both construct deep abstractions and reify (or instantiate) them in rich interfaces [1] . With a few exceptions [2, 3, 4] , existing work on interpretability fails to do these in concert.

The machine learning community has primarily focused on developing powerful methods, such as feature visualization [5, 6, 7, 8, 9, 10] , attribution [7, 11, 12, 13, 14, 15, 16, 17] , and dimensionality reduction [18] , for reasoning about neural networks. However, these techniques have been studied as isolated threads of research, and the corresponding work of reifying them has been neglected. On the other hand, the human-computer interaction community has begun to explore rich user interfaces for neural networks [19, 20, 21] , but they have not yet engaged deeply with these abstractions. To the extent these abstractions have been used, it has been in fairly standard ways. As a result, we have been left with impoverished interfaces (e.g., saliency maps or correlating abstract neurons) that leave a lot of value on the table. Worse, many interpretability techniques have not been fully actualized into abstractions because there has not been pressure to make them generalizable or composable. .... 

Interpreting Predictive Models with Skater

One of the most important issues in data science,  interpreting models,  especially for particular contextual use.  And part of the context is always metadata driven. ...

In O'Reilly: 

Interpreting predictive models with Skater: Unboxing model opacity

A deep dive into model interpretation as a theoretical concept and a high-level overview of Skater.
By Pramit Choudhary  ....

Particularly like these general overview statements of interpretability:

" ... Ideally, you should be able to query the model to understand the what, why, and how of its algorithmic decisions: 

What information can the model provide to avoid prediction errors? You should be able to query and understand latent variable interactions in order to evaluate and understand, in a timely manner, what features are driving predictions. This will ensure the fairness of the model.

Why did the model behave in a certain way? You should be able to identify and validate the relevant variables driving the model’s outputs. Doing so will allow you to trust in the reliability of the predictive system, even in unforeseen circumstances. This diagnosis will ensure accountability and safety of the model.


How can we trust the predictions made by the model? You should be able to validate any given data point to demonstrate to business stakeholders and peers that the model works as expected. This will ensure transparency of the model.   ...  " 

Fixing Statistics in Algorithms, Process

Thoughtful piece.  And increasingly important as we decide to algorithmically, and sometimes non transparently  implement methods deeper in the process.   Will that make cognitive bias from humans less likely, or just embed them more deeply?  A challenge for big data and AI.

Five Ways to Fix Statistics in Nature

By Jeff Leek, Blakeley B. McShane, Andrew Gelman, David Colquhoun, Michèle B. Nuijten & Steven N. Goodman

As debate rumbles on about how and how much poor statistics is to blame for poor reproducibility, Nature asked influential statisticians to recommend one change to improve science. The common theme? The problem is not our maths, but ourselves.

To use statistics well, researchers must study how scientists analyse and interpret data and then apply that information to prevent cognitive mistakes.

In the past couple of decades, many fields have shifted from data sets with a dozen measurements to data sets with millions. Methods that were developed for a world with sparse and hard-to-collect information have been jury-rigged to handle bigger, more-diverse and more-complex data sets. No wonder the literature is now full of papers that use outdated statistics, misapply statistical tests and misinterpret results. The application of P values to determine whether an analysis is interesting is just one of the most visible of many shortcomings.   .... "

Feature Visualization

In the Google Blog, they provide an overview and pointer to deeper information.  An image rich and informative article.

Feature Visualization
Posted by Christopher Olah, Research Scientist, Google Brain Team and Alex Mordvintsev, Research Scientist, Google Research

Have you ever wondered what goes on inside neural networks? Feature visualization is a powerful tool for digging into neural networks and seeing how they work.

Our new article, published in Distill, does a deep exploration of feature visualization, introducing a few new tricks along the way!

Building on our work in DeepDream, and lots of work by others since, we are able to visualize what every neuron a strong vision model (GoogLeNet [1]) detects. Over the course of multiple layers, it gradually builds up abstractions: first it detects edges, then it uses those edges to detect textures, the textures to detect patterns, and the patterns to detect parts of objects….   " 

" .... There is a growing sense that neural networks need to be interpretable to humans. The field of neural network interpretability has formed in response to these concerns. As it matures, two major threads of research have begun to coalesce: feature visualization and attribution. ... " 

Interpreting Machine Learning

Interesting piece,  somewhat wordy with embedded technical aspect.  But worth the read.

Ideas on interpreting machine learning
Mix-and-match approaches for visualizing data and interpreting machine learning models and results.    By Patrick Hall, Wen Phan, SriSatish Ambati     from O'Reilly

How to Mislead with Data

 I have seen it often in the enterprise and elsewhere.  Good points made in article below, and always worth repeating.    But missing is the current tendency to dress the results. Aka 'Tell a Story', or use some sort of pre-established narrative.  Or use an infographic to simplify into an instantly understandable point to be made.    Yes, that's confirmation bias, or model-confirmation bias.

Also very common and hard to detect unless you are shown the raw data and can manipulate it, which is rare.  Helps to do a risk analysis to understand the cost of a wrong model. Or involve the context owner early and often.

How to Lie with Data  Posted by Karolis Urbonas    In DSC

We expect that data scientists and analysts should be objective and base their conclusions on data. Now while the name of the job implies that “data” is the fundamental material that is used to do their jobs, it is not impossible to lie with it. Quite the opposite – the data scientist is affected by unconscious biases, peer pressure, urgency, and if that’s not enough – there are inherent risks in the process of data analysis and interpretation that lead to lying. It happens all the time while the intentions might be truly honest – though we all know the saying “The road to Hell is paved with good intentions”.

As every industry in every country is affected by data revolution we need to make sure we are aware of the dangerous mechanisms that can affect the output of any data project..... " 

Spotting Fake Info in Data

Would add to beyond 'fake', tags like,  'expired',  'contradictory', 'signals of bias', ' unstable metadata', 'evidence of value', 'risk of use (or non-use)' ... just a few I have had experience with .... there are more.   The thought is good though. Every piece of data should have metadata that flags its interpretation.   In this case its use for strategic espionage.   But for other contexts as well.

Can an app help spies spot phony info? Syracuse researchers, others get $11M to study
By Rick Moriarty | rmoriarty@syracuse.com

on January 26, 2017 at 11:49 AM, updated January 26, 2017 at 12:17 PM
Syracuse, N.Y. -- A team that includes researchers from Syracuse University and SRC Inc. is developing digital tools that will use crowdsourcing and other techniques to help the nation's intelligence analysts do their jobs better.

Among the things the software would do is help intelligence analysts determine the credibility of the information they are receiving, researchers said. Another goal is to make the applications more like computer games, researchers said. ... " 

ACM Statement on Algorithmic Transparency and Accountability

The well known ACM professional society issued a statement about algorithmic transparency and accountability today.  I have been a member and participant for many years.   Algorithms have been part of computing forever, but only recently have they been closely examined regarding their implications, especially as they interact with the public.  And we interact with them every day.

 Algorithms are models, and inherently have bias.  They make and position decisions for and with us. This will further expand with the use of AI.   The ACM statement, the first few paragraphs below, the complete document is at the link, does an excellent job of laying out the problem, and their professional position.  Well stated:

Statement on Algorithmic Transparency and Accountability
Computer algorithms are widely employed throughout our economy and society to make decisions that have far-reaching impacts, including their applications for education, access to credit, healthcare, and employment.    The ubiquity of algorithms in our everyday lives is an important reason to focus on addressing challenges associated with the design and technical aspects of algorithms and preventing bias from the onset.

An algorithm is a self-contained step-by-step set of operations that computers and other 'smart' devices carry out to perform calculation, data processing, and automated reasoning tasks. 

Increasingly, algorithms implement institutional decision-making based on analytics, which involves the discovery, interpretation, and communication of meaningful patterns in data. Especially valuable in areas rich with recorded information, analytics relies on the simultaneous application of statistics, computer programming, and operations research to quantify performance.

There is also growing evidence that some algorithms and analytics can be opaque, making it impossible to determine when their outputs may be biased or erroneous. Computational models can be distorted as a result of biases contained   ..... " 

Medicine and Chatbots

Been using email with my doctors for some time.    Even the point of sharing data and its interpretation, at least at a high level.  Seems they still run nervous on the provision of recorded 'advice' and I wonder how that will follow to 'bots' as part of the conversation.   Early AI tried this, and Watson as a physicians assistant is claiming success, at least in interpreting research.

 Example here is interesting:

 In Digital Trends
The Chatbot will see your now: AI may play doctor in the future of Healthcare ... By Dyllan Furness