Unpacking Black Box Models

Quite interesting development.  I had to talk the nature and implications of What a 'black box' (BB) . was for management many times.  A BB is simply a method that is not precisely know in its operation.  Usually such an algorithm's method IS known to the person or AI that developed it, but is obscure to the people who need its operation in context. .  It may be 'explainable',  but it has never been sufficiently explained to the user who wants or needs it. .  It may be the explanation is too difficult.  It may require considerable math, statistics or AI.   Or, and not uncommon, the user may not have asked for an explanation.  Perhaps because they liked the outcome of the BB.   Lots of practical subtilties here. Measuring understanding is a useful step.

Unpacking Black-Box Models

By MIT News, May 11, 2022

A mathematical framework developed by researchers at the Massachusetts Institute of Technology and Microsoft Research aims to quantify and evaluate the understandability of a machine learning model's explanations for its predictions.

The framework, called ExSum (explanation summary), can evaluate a rule on an entire dataset. ExSum enables the user to see if a rule holds up based on three metrics: coverage, or how broadly applicable the rule is across the entire dataset; validity, or the percentage of individual examples that agree with the rule; and sharpness, or how precise the rule is.

Said MIT's Yilun Zhou, "Before this work, if you have a correct local explanation, you are done. You have achieved the holy grail of explaining your model. We are proposing this additional dimension of making sure these explanations are understandable." ... 

Researchers have created a mathematical framework to evaluate explanations of machine-learning models and quantify how well people understand them.... 

MIT News, full article.

Chatbot explains Apps

Intriguing idea.  Could a similar means explain potential security issues with an App or system?  Or its uses in context.  I like the idea of explanatory analyses available for complex systems.  Could have used them to deliver alongside systems we delivered to decision makers.

Chatbot Can Explain Apps, Show How They Access Hardware or Data  By Aalto University (Finland)

Researchers at Finland's Aalto University, the University of Luxembourg, and Germany's University of Bayreuth developed a chatbot that can help designers and developers create new apps and explain apps to end-users.

Hey GUI (Graphical User Interface) answers questions with images, screenshots, and simple text phrases.

Aalto's Kashyap Todi said, "Hey GUI eliminates the need for coding skills or technical expertise. Anyone can use it to search for information on, for example, which one of my apps is asking for permission to use the camera, or where is the search button on this screen."

The researchers surveyed more than 100 people to determine the most desirable chatbot features, and how users preferred to interact with chatbots.

Said Todi, "This is an important first step towards developing chatbots that can help users find information about apps using

From Aalto University (Finland)

View Full Article  

Apologetic AI is a Trend?

Wonder how well that works.    Having worked with consumer-computers since the very early days, never recall an attempt at apologies in this way. Even when we built avatars to represent products, it never had 'sorry' in its vocabulary.  We had to deal with errors.    What are our expectations here?   At what level do we expect apologies, or just dismiss it and lose our respect for the interaction? 

Apologetic AI Is A Somewhat Sorry Trend, Especially For Autonomous Cars   By Lance Eliot, the AI Trends Insider   

AI is starting to apologize.   

That’s the latest trend for AI that directly interacts with people. The notion seems to be that if the AI has to deliver unfavorable news or appears to have made a potential mistake, it ought to be civil about the matter and emit an apology. AI developers are either opting to include the apology-generating capacity or they are being pressed by system designers and managers to infuse the “sorry about that” capability.  

This might seem at first glance as a marvelous addition to an AI system and would presumably be valuable to construct. Sorry to say that the AI being apologetic has both upsides and downsides. 

Let’s begin by considering a context that will help to reveal the pros and cons of AI-powered apologies. Imagine that you apply online for a car loan and the AI system determines that you are not worthy, as it were, and promptly turns you down. The belief is that this would be an ideal moment for the AI to offer you a “heartfelt” apology.   

It might go something like this: Dear loan applicant, it is with great sorrow that I must inform you of the unfortunate news that your request to borrow funds to buy a car is hereby denied. Please know that you are not alone in having been spurned and accept this apology for any discomfort that might arise from this outcome. Sincerely, the AI system that reluctantly rebuffed your request.   

Do you think this apology will make the person feel any better about the AI-powered decision?   

Well, I doubt it would for most people, nonetheless, there is an emerging trend of having AI systems produce these kinds of messages.   .... ' 

Explaining AI in Context

Not to say I am using the connection with autonomous cars, but am very much into how we explain with AI in all sorts of contexts.    We built some AI systems in our early days that could have used very precise explanatory capabilities, in order to keep its credibility over many maintenance cycles,  but it could only be crudely done at the time.   Here a nice case study in the here and now.  

The Rocky Road Toward Explainable AI (XAI) For AI Autonomous Cars 

The AI systems doing the piloting of autonomous cars will need to provide explanations to curious passengers about the route being undertaken    By Lance Eliot, the AI Trends Insider  

Our lives are filled with explanations. You go to see your primary physician due to a sore shoulder. The doctor tells you to rest your arm and avoid any heavy lifting. In addition, a prescription is given. You immediately wonder why you would need to take medication and also are undoubtedly interested in knowing what the medical diagnosis and overall prognosis are. 

So, you ask for an explanation. 

In a sense, you have just opened a bit of Pandora’s box, at least in regard to the nature of the explanation that you might get. For example, the medical doctor could rattle a lengthy and jargon-filled indication of shoulder anatomy and dive deeply into the chemical properties of the medication that has been prescribed. That’s probably not the explanation you were seeking.   

It used to be that physicians did not expect patients to ask for explanations. Whatever was said by the doctor was considered sacrosanct. The very nerve of asking for an explanation was tantamount to questioning the veracity of a revered medical opinion. Some doctors would gruffly tell you to simply do as they have instructed (no questions permitted) or might utter something rather insipid like your shoulder needs help and this is the best course of action. Period, end of story.   

Nowadays, medical doctors are aware of the need for viable explanations. There is specialized “bedside” training that takes place in medical schools. Hospitals have their own in-house courses. upcoming medical doctors are graded on how they interact with patients. And so on.   

Though that certainly has opened the door toward improved interaction with patients, it does not necessarily completely solve the explanations issue. 

Knowing how to best provide an explanation is both art and science. You need to consider that there is the explainer that will be providing the explanation, and there is a person that will be the recipient of the explanation.    ... '

Explanations and Contexts

An example of the need for explain-ability. Like in a conversation with a human, we could want the option of getting an explanation of a solution.  But the nature of an explanation does often depend on context. Is is for management or an engineer?  Is it for a current set of data or a generalization?    Does it depend on some regulation or special constraints?  Context is often key.  Often occurred in our work. 

Researchers Develop 'Explainable' Algorithm

University of Toronto (Canada), Matthew Tierney, March 31, 2021

An "explainable" artificial intelligence (XAI) algorithm developed by researchers at Canada's University of Toronto (U of T) and LG AI Research was designed to find and fix defects in display screens. XAI addresses issues with the "black box" approach of machine learning strategies, in which the artificial intelligence makes decisions entirely on its own. With XAI's "glass box" approach, XAI algorithms are run simultaneously with traditional algorithms to audit the validity and level of their learning performance, perform debugging, and identify training efficiencies. U of T's Mahesh Sudhakar said LG "had an existing [machine learning] model that identified defective parts in LG products with displays, and our task was to improve the accuracy of the high-resolution heat maps of possible defects while maintaining an acceptable run time." The new XAI algorithm, Semantic Input Sampling for Explanation (SISE), outperformed comparable approaches on industry benchmarks.... ' 

Explaining Reinforcement Learning

 Been recently involved with reinforcement learning, and found this interesting.   I see some advances in this space worth following.  Especially in RL, you need explanation regarding decisions, in order to support reasonable tuning.  

Dear Reinforcement Learning Agent, please explain your actions.

Explainable Reinforcement Learning for Longitudinal Control

Roman Liessner   in TowardDataScience

Here I present research with Jan Dohmen and Marco Wiering.

TL;DR: Reinforcement learning is promising for achieving new best performances in a variety of applications. However, as long as the learned actions remain intransparent, their use in security-relevant applications is unlikely. The new RL-SHAP Diagram presented here opens the black box and gives a new perspective to the Reinforcement Learning decision-making. ..."

Tool Turns Math into Pictures

Lovely thought, you can browse the images, and edit or choose the best.  To produce a best explanation.   A means to communicate math concepts with management, decision makers?   Good examples at the link.

Carnegie Mellon Tool Automatically Turns Math Into Pictures
By Byron Spice

A tool created by Carnegie Mellon University (CMU) researchers allows anyone to render mathematical abstractions as illustrations. The Penrose tool enables diagram-drawing experts to encode their math-into-diagram methods; users simply type in an ordinary mathematical expression, and Penrose produces the drawing. Once the computer learns how the user wants to visualize mathematical objects, it uses the encoded rules to draw several candidate diagrams, which the user can from choose and edit. The researchers created a special programming language for this purpose, which CMU’s Keenan Crane said mathematicians should have no trouble learning. Said Crane, "Our vision is to be able to dust off an old math textbook from the library, drop it into the computer, and get a beautifully illustrated book—that way, more people understand."  .. '

Making Decision Trees Accurate and Explainable

Explaining AI, Decision Trees

Berkeley Artificial Intelligence Research
Making Decision Trees Accurate Again: Explaining What Explainable AI Did Not   By Alvin Wan  

The interpretability of neural networks is becoming increasingly necessary, as deep learning is being adopted in settings where accurate and justifiable predictions are required. These applications range from finance to medical imaging. However, deep neural networks are notorious for a lack of justification. Explainable AI (XAI) attempts to bridge this divide between accuracy and interpretability, but as we explain below, XAI justifies decisions without interpreting the model directly.

What is “Interpretable”?
Defining explainability or interpretability for computer vision is challenging: What does it even mean to explain a classification for high-dimensional inputs like images? As we discuss below, two popular definitions involve saliency maps and decision trees, but both approaches have their weaknesses. .... " 

Complexity of Showing How AI Decides

Thoughtful piece on the problem of how to reasonably provide explanation.  Have now looked at several means of addressing the problem.   Consider carefully the decisions being driven and risks involved.

Grilling the answers: How businesses need to show how AI decides

As artificial intelligence becomes more widespread, so the need to render it explainable increases. How can companies navigate the technical and ethical challenges?
By   Lindsay Clark in Computerweek

Show your working: generations of mathematics students have grown up with this mantra. Getting the right answer is not enough. To get top marks, students must demonstrate how they got there. Now, machines need to do the same.

As artificial intelligence (AI) is used to make decisions affecting employment, finance or justice, as opposed to which film a consumer might want to watch next, the public will insist it explains its working.

Sheffield University professor of AI and robotics Noel Sharkey drove home the point when he told The Guardian that decisions based on machine learning could not be trusted because they were so “infected with biases”.

Sharkey called for an end to the application of machine learning to life-changing decisions until they could be proven safe in the same way that drugs are introduced into healthcare.  ... " 

Video: Interpretable Machine Learning

I mentioned this article in an earlier post, where I discuss in more detail, here here a short video introduction.

Techniques for Interpretable Machine Learning from CACM on Vimeo.

Mengnan Du and Xia Hu discuss "Techniques for Interpretable Machine Learning," a Review Article in the January 2020 CACM. ... 

Techniques for Interpretable Machine Learning

Very good piece I am reading in the January CACM.  The most important aspect of considering AI-ML type models in the real world.  Good introduction, useful key insights, but ultimately quite technical.  Bottom line is that research is still needed and 'Model explanation and surprising artifacts are often two sides of the same coin'.  Complex models may extract and codify biases and other 'artifacts' of metadata from training data.   Test and re-test under varying context.  Maintenance is more that just tracking performance over time.  Consider embedded models of risk.

I highly recommend subscribing to CACM if you are technically involved.

Techniques for Interpretable Machine Learning
By Mengnan Du, Ninghao Liu, Xia Hu

Communications of the ACM, January 2020, Vol. 63 No. 1, Pages 68-77
10.1145/3359786

Machine learning is progressing at an astounding rate, powered by complex models such as ensemble models and deep neural networks (DNNs). These models have a wide range of real-world applications, such as movie recommendations of Netflix, neural machine translation of Google, and speech recognition of Amazon Alexa. Despite the successes, machine learning has its own limitations and drawbacks. The most significant one is the lack of transparency behind their behaviors, which leaves users with little understanding of how particular decisions are made by these models. Consider, for instance, an advanced self-driving car equipped with various machine learning algorithms does not brake or decelerate when confronting a stopped firetruck. This unexpected behavior may frustrate and confuse users, making them wonder why. Even worse, the wrong decisions could cause severe consequences if the car is driving at highway speeds and might ultimately crash into the firetruck. The concerns about the black-box nature of complex models have hampered their further applications in our society, especially in those critical decision-making domains like self-driving cars.

Interpretable machine learning would be an effective tool to mitigate these problems. It gives machine learning models the ability to explain or to present their behaviors in understandable terms to humans,10 which is called interpretability or explainability and we use them interchangeably in this article. Interpretability would be an indispensable part for machine learning models in order to better serve human beings and bring benefits to society. For end users, explanation will increase their trust and encourage them to adopt machine learning systems. From the perspective of machine learning system developers and researchers, the provided explanation can help them better understand the problem, the data and why a model might fail, and eventually increase the system safety. Thus, there is a growing interest among the academic and industrial community in interpreting machine learning models and gaining insights into their working mechanisms.

Interpretable machine learning techniques can generally be grouped into two categories: intrinsic interpretability and post-hoc interpretability, depending on the time when the interpretability is obtained.23 Intrinsic interpretability is achieved by constructing self-explanatory models which incorporate interpretability directly to their structures. The family of this category includes decision tree, rule-based model, linear model, attention model, and so on. In contrast, the post-hoc one requires creating a second model to provide explanations for an existing model. The main difference between these two groups lies in the trade-off between model accuracy and explanation fidelity. Inherently interpretable models could provide accurate and undistorted explanation but may sacrifice prediction performance to some extent. The post-hoc ones are limited in their approximate nature while keeping the underlying model accuracy intact.  ... "

XAI: Google Explainable AI as a Service

Google takes on Explainable AI, try it free at the link ... Note in contrast IBM's Explainability Toolkit.  Now ask explainability of what, to whom?  Further KDNuggets has a good view of this as 'explainable AI as a service'.

Understand AI output and build trust

Explainable AI is a set of tools and frameworks to help you develop interpretable and inclusive machine learning models and deploy them with confidence. With it, you can understand feature attributions in AutoML Tables and AI Platform and visually investigate model behavior using the What-If Tool. It also further simplifies model governance through continuous evaluation of models managed using AI Platform.

Design interpretable and inclusive AI

Build interpretable and inclusive AI systems from the ground up with tools designed to help detect and resolve bias, drift, and other gaps in data and models. AI Explanations in AutoML Tables and AI Platform provide data scientists with the insight needed to improve data sets or model architecture and debug model performance. The What-If Tool lets you investigate model behavior at a glance.

Simple and fully managed

Deploy AI with confidence

Grow end-user trust and improve transparency with human-interpretable explanations of machine learning models. When deploying a model on AutoML Tables or AI Platform, you get a prediction and a score in real time indicating how much a factor affected the final result. While explanations don’t reveal any fundamental relationships in your data sample or population, they do reflect the patterns the model found in the data. .... "

Talk on Explainable AI

All analytics should be explainable in business context.  And the person responsible for that part of the business should also be able to explain how and why it works.

Voices in AI – Episode 101: A Conversation with Cindi Howsen  By Byron  Reese 

On Episode 101 of Voices in AI, Byron speaks with Cindi Howsen of Thoughtspot about the direction of explainable AI and where we are going as an industry.

Listen to this episode or read the full transcript at www.VoicesinAI.com

Transcript Excerpt

 This is Voices in AI brought to you by GigaOm and I’m Byron Reese. Today my guest is Cindi Howson. She is the Chief Data Strategy Officer at ThoughtSpot. She holds a degree in English from the University of Maryland and an MBA from my alma mater, Rice University. Welcome to the show, Cindy..... "

(Update) AI Explainability Toolkit Talk and Technology

From last weeks talk on the just released open source explainabilty toolkit.   This can be seen as a fundamental part of most conversations.   When we interact with colleagues or with professionals, and get recommendations, we often have to ask the question 'Why?'.  This is an attempt at preloading AI originating answers to that question, based on a number of common templates.

http://cognitive-science.info/wp-content/uploads/2019/09/AIX360-CSIG-V1-2019-09-05.pdf  (Slides)

http://cognitive-science.info/community/weekly-update/  Update: Recording: https://www.youtube.com/watch?v=Yn4yduyoQh4

http://aix360.mybluemix.net/   (Technical link, demos)

What does it take to trust AI decisions ? 
AI is now used in many high-stakes decision making applications.

Addressing:
Is it fair?  Is it easy to understand?  Did anyone tamper with it?  Is it accountable?  

Very good talk, lots of great progress shown here,  but still lots more to do.   Everyone doing serious work with AI systems should examine this work and see how their system could link to this capability.  And extend it.   More to follow.

IBM Research AI Explainability 360 Toolkit

By Vijay Arya, Rachel Bellamy, Pin-Yu Chen,Payel Das, Amit Dhurandhar, MaryJo Fitzgerald,Michael Hind, Samuel Hoffman,Stephanie Houde, Vera Liao, Ronny Luss,Sameep Mehta, Saska Mojsilovic, Sami Mourad,Pablo Pedemonte, John Richards,Prasanna Sattigeri, Moninder Singh,Karthikeyan Shanmugam, Kush Varshney,Dennis Wei, Yunfeng Zhang, Ramya Raghavendra .... 

AI Explainability 360 Toolkit

From today's talk:

http://cognitive-science.info/wp-content/uploads/2019/09/AIX360-CSIG-V1-2019-09-05.pdf (Slides)

http://cognitive-science.info/community/weekly-update/  Update: Recording: https://www.youtube.com/watch?v=Yn4yduyoQh4

http://aix360.mybluemix.net/   (Technical link)

What does it take to trust AI Decisions ? 

AI IS NOW USED IN MANY HIGH-STAKES DECISION MAKING APPLICATIONS

Addressing:
Is it fair?  Is it easy to understand?  Did anyone tamper with it?  Is it accountable?  

Very good talk, lots of great progress shown here,  but still lots more to do.   Everyone doing serious work with AI systems should examine this work and see how their system could link to this capability.  And extend it.   More to follow.

IBM Research AI Explainability 360 Toolkit

By Vijay Arya, Rachel Bellamy, Pin-Yu Chen,Payel Das, Amit Dhurandhar, MaryJo Fitzgerald,Michael Hind, Samuel Hoffman,Stephanie Houde, Vera Liao, Ronny Luss,Sameep Mehta, Saska Mojsilovic, Sami Mourad,Pablo Pedemonte, John Richards,Prasanna Sattigeri, Moninder Singh,Karthikeyan Shanmugam, Kush Varshney,Dennis Wei, Yunfeng Zhang, Ramya Raghavendra

IBM AI Open Source Tool Explainability Talk

Upcoming talk, looks to be quite interesting regarding AI explain-ability open source method.  The talk will be recorded and I will post its location afterwards.

 CSIG (Cognitive Systems Institute Group) Talk - Thursday Sep 5, 2019 - 10:30-11am US Eastern
Title: Al Explainability 360 Toolkit

Speakers: Vijay Arya & Amit Dhurandhar, IBM Research

As AI and ML algorithms make inroads into society, calls are increasing for algorithms to explain their outputs. Affected citizens, government regulators. domain experts. or system developers. present different requirements for explanations. To address these needs we introduce:

AI Explainability 360 (http://aix360.mybluemix.net/)  (good tutorials there) , an open-source software toolkit featuring 8 state-of-the-art explainability methods and 2 evaluation metrics. We provide a taxonomy to help entities require explanations to navigate the space of explanation methods, in the toolkit and in the broader literature.

We have implemented an extensible software architecture that organizes methods according to their place in the AI modeling pipeline. We discuss enhancements to bring research innovations closer to consumers of explanations. ranging from algorithms. to tutorials and an interactive web demo to introduce AI explainability to different and application domains. Together, the toolkit and taxonomy can help identify gaps where more are needed and provide a platform to incorporate them as they are developed. 

Zoom meeting Link: https://zoom.us/j/7371462221

Zoom Callin: (415) 762-9988 or (646) 568-7788 Meeting id 7371462221
Zoom International Numbers: https://zoom.us/zoomconference
Thu, Aug 2, 10:30am US Eastern https://zoom.us/j/7371462221
More Details and recording Here : http://cognitive-science.info/community/weekly-update/

More Aims at Explainable AI

More interesting moves in providing explanation to AI.  Here from Ga Tech.   As in the previous recent  post, this asks the question:   What does explainable mean?   And to whom in what context?  Simplicity is good, if everyone agrees to its truth and value.

A Breakthrough in Explainable AI   By Joe Dysart in CACM
A new artificial intelligence (AI) agent offers easy-to-understand English explanations of the AI's analysis and decisions.

Such tools of explanation are considered critical by those working in AI who fear users may be reluctant to embrace AI applications that make recommendations whose rationales are shrouded in mystery.

"As AI pervades all aspects of our lives, there is a distinct need for human-centered AI design that makes black-boxed AI systems explainable to everyday users," says Upol Ehsan, a doctoral student in the School of Interactive Computing at the Georgia Institute of Technology (Georgia Tech) and lead author of the study.  "Our work takes a formative step toward understanding the role of language-based explanations and how humans perceive them."

Adds Devi Parikh, an assistant professor in Georgia Tech's School of Interactive Computing, "Just like human teams are more effective when team members are transparent and can understand the rationale behind each others decisions or opinions, human-AI teams will be more effective if AI systems explain themselves and are somewhat interpretable to a human."

Eshan and his research team set out to solve the explainable AI problem by developing an AI agent that could offer easy-to-understand explanations to humans in certain settings.

For their research, the team—including researchers from Georgia Tech, Cornell University, and the University of Kentucky—decided to create an AI agent that would analyze and explain moves in the video game "Frogger." The game is an ideal choice for developing an AI agent, given its simplicity: the entire goal of the game is to move an animated frog across a screen, enabling it to dodge oncoming vehicles and other animated hazards.

The researchers trained their AI agent by first asking gamers to play Frogger as they explained the rationale behind each action they took, move by move."  ... ' 

IBM Explainable AI Toolkit

ZDnet poses a good question: Whats a good explanation?   Its  a matter of context.  Algorithms are often brought up because they seem simple  .... just a short statement of 'truth' that can be inserted anywhere.  But, they are usually based on considerable context, data, metadata ...   And can be much less than explainable .... in practice, the shorter they are the less they are usually clear to real decision makers

IBM offers explainable AI toolkit, but it’s open to interpretation in ZDNet
IBM’s latest foray into making A.I. more amenable to the world is a toolkit of algorithms that can be used to explain the decisions of machine learning programs. It raises a deep question: Just what is an explanation, and how can we find ones that we will accept?... 

By Tiernan Ray  .... '

Google What-IF Tool for Code Free ML Visualization

Like the idea of visual tools that map with specific process, resource needs and output results.  Leads to better understandable and resilient results.

Google's What-If Tool And The Future Of Explainable AI
Kalev Leetaru Contributor in Forbes
AI & Big Data

(Excerpt)

" ..... As deep learning has matured sufficiently to find widespread adoption in industry and as developers require increasingly greater understanding of their creations in order to pioneer new advances, the AI community has begun investing heavily in explainable AI as a way to render their black boxes transparent.

Google has been an early leader in emphasizing interpretability and how practitioners can build more understandable, representative and resilient AI solutions. Last year the company unveiled its What-If Tool, which offers a range of interactive visualizations and guided explorations of a TensorFlow model, allowing developers to explore how their model interpreted its training data and how subtle changes to a given input would change its classification, yielding insights into the model’s robustness. .... " 

Google's Description:

The What-If Tool: Code-Free Probing of Machine Learning Models
Tuesday, September 11, 2018
Posted by James Wexler, Software Engineer, Google AI

Building effective machine learning (ML) systems means asking a lot of questions. It's not enough to train a model and walk away. Instead, good practitioners act as detectives, probing to understand their model better: How would changes to a datapoint affect my model’s prediction? Does it perform differently for various groups–for example, historically marginalized people? How diverse is the dataset I am testing my model on?

Answering these kinds of questions isn’t easy. Probing “what if” scenarios often means writing custom, one-off code to analyze a specific model. Not only is this process inefficient, it makes it hard for non-programmers to participate in the process of shaping and improving ML models. One focus of the Google AI PAIR initiative is making it easier for a broad set of people to examine, evaluate, and debug ML systems.


Today, we are launching the What-If Tool, a new feature of the open-source TensorBoard web application, which let users analyze an ML model without writing code. Given pointers to a TensorFlow model and a dataset, the What-If Tool offers an interactive visual interface for exploring model results.  .... " 

Explainable AI from Kyndi is Funded

Quite a claim, like the components I see mentioned.  More explanation, logic, graphs, semantics ....  we explored and used them all,  is it enough for required transparency?  And how will such AI and explanation be maintained in changing contexts?   And resulting risks?

Kyndi Platform funding:

....  EXPLAINABLE ARTIFICIAL INTELLIGENCE ...

Kyndi is building the first Explainable AI platform for government, financial services, and life sciences. Our solutions are the antithesis of the old AI “black box.”
KYNDI ADDS $20M TO EXPAND TEAM AND ACCELERATE GROWTH
The Series B round was led by Intel Capital, with participation from UL Ventures, PivotNorth Capital, and existing investors.
THE KYNDI AI PLATFORM
Kyndi is an artificial intelligence company that’s building the first Explainable AI product and Intelligent Process Automation software for government, pharmaceutical, and financial services organizations. .... 

The Kyndi AI Platform uses machine learning to streamline regulated business processes and offer auditable AI systems for enterprises and government. Kyndi’s product exists because Deep Learning is a “black box” and cannot be used in regulated industries where organizations are required to explain the reasons for any decision.

Our platform uses a novel approach to AI, unifying probabilistic and logical methods. This enables organizations to analyze massive amounts of data to create actionable knowledge significantly faster and without having to sacrifice explainability. Kyndi’s Explainable AI™ Platform supports the following solutions: Intelligence, Defense, Compliance (i.e., for financial services and healthcare), and Research. Crucially, the Kyndi AI Platform also helps to mitigate the human bias that can arise in the process of extracting knowledge and answers from data.

EXPLAINABILITY
Explainable artificial intelligence achieves the level of trust that is so important for accelerated growth and acceptance of this revolutionary technology.
AI cannot be a “black box,” as it so often is today. Explainable AI™ means that our software’s reasoning is apparent to the user, and that the system can explain its rationale. This visibility allows you to have confidence in the system’s outputs, be aware of any uncertainties, anticipate how the software will work in the future, and know how to improve the system. Such knowledge is essential to confident analysis and decision making. Explainability is at the core of Kyndi’s products and solutions.

NATURAL LANGUAGE PROCESSING (NLP)
With Kyndi’s AI products, knowledge is accumulated and transferred through written forms of natural language.

Kyndi has developed a unique and effective approach to NLP to automate and scale knowledge consumption. First, our NLP solution tokenizes text and identifies parts of speech and sentence structure. Next, we identify named entities with real-world references and compute semantic distances between words using our proprietary Semantic Distance Field Model to show us how strongly any two entities are related. Semantic parsing and Relation Extraction allows us to formally name this relationship where both the entities and relationships create a proto-ontology that encodes and condenses the meaning of a document, a collection of documents, or a whole domain.

With this clear view of ideas, concepts, and relationships, we build a knowledge graph that scales with your business and that lets you query your collection and retrieve the information you want, not just the words you used to ask the question.

KNOWLEDGE GRAPHS
Once Kyndi’s NLP pipeline identifies the structure and contents of your data we create a graph representation, no matter the size.

With each node in the graph identifying an entity, the connections between the nodes are semantic vectors that signify the relationships and significance between entities. The feature-rich graph enables you to conduct a quick and accurate analysis by matching fragments and returning details relevant to your search queries. To achieve this functionality Kyndi has developed industry-leading technology that allows for sub-graph matching based on cognitive signatures.  ..... "