Cleaning Messy Data Tables

Looks to be efficient at least.  Check out the code.   Note the use of Bayesian reasoning.

Home/News/System Cleans Messy Data Tables Automatically/Full Text

ACM TECHNEWS  System Cleans Messy Data Tables Automatically By MIT News

A system developed by researchers at the Massachusetts Institute of Technology (MIT) automatically cleans "dirty data" of things such as typos, duplicates, missing values, misspellings, and inconsistencies.

PClean combines background information about the database and possible issues with common-sense probabilistic reasoning to make judgment calls for specific databases and error types. Its repairs are based on Bayesian reasoning, which applies probabilities based on prior knowledge to ambiguous data to determine the correct answer, and can provide calibrated estimates of its uncertainty.

The researchers found that PClean, with just 50 lines of code, outperformed benchmarks in both accuracy and runtime.

From MIT News

Intro to Bayesian

Good intro to the idea  of Bayesian computation and modeling,   But ultimately does include math.  We successfully used the method for simulating relatively complex process.   Or 'computing' probabilistically the real world based on available data.  This is an intro, and you are best advised to not implement it directly, but use a package as the basis for use.    But it is also good to use Bayesian methods to think about your problem and map it out usefully ... 

The ABCs of Approximate Bayesian Computation

An introduction into parameter inference using Approximate Bayesian Computational methods.  By Tom Leyshon in TowardsdataScience

What is Bayesian statistics?

Bayesian statistics are methods that allow for the systematic updating of prior beliefs in the evidence of new data [1]. The fundamental theorem that these methods are built upon is known as Bayes’ theorem.... " 

Neural Networks are Bayesian

 Just reading this, worth thinking about. Technical.

Neural networks are fundamentally Bayesian  in TowardsDataScience By Chris Mingard

Stochastic Gradient Descent approximates Bayesian sampling

Deep neural networks (DNNs) have been extraordinarily successful in many different situations — from image recognition and playing chess to driving cars and making medical diagnoses. However, in spite of this success, a good theoretical understanding of why they generalise (learn) so well is still lacking.

In this post, we summarise results from three papers, which provide a candidate for a theory of generalisation in DNNs [1,2,3].     .... "

Why We Need Bayesian

This link is first a reminder to myself that we need to continually promote means of risk and uncertainty awareness in models.    Meta-reasoning is always important.  Thinking about the context in which your models will be used.  If you don't do that you have missed something important.  Understanding the risks it will have in use.   The article gets quite technical, but the intros are worthwhile to read.  And there are links to good online courses, which  also have good intros.

Bayesian meta-learning
This story introduces bayesian meta-learning approaches, which covers bayesian black-box meta-learning, bayesian optimization-based meta-learning, ensembles of MAMLs and probabilistic MAML. This a short summary of the course ‘Stanford CS330: Multi-Task and Meta-Learning, 2019 | Lecture 5 — Bayesian Meta-Learning’.
By Qiurui Chen in TowardsDataScience

For meta-learning algorithms, 3 algorithmic properties are important: expressive power, consistency, and uncertainty awareness. Expressive power is the ability for f to represent a range of learning procedures, it measures scalability and applicability to a range of domains. Consistency means learned learning procedure will solve tasks with enough data, this property reduces reliance on meta-training tasks, which leads to good out-of-distribution performance. Uncertainty awareness is the ability to reason about ambiguity during learning. It allows us to think about how we might explore new environments in a reinforcement learning context in order to reduce our uncertainty. It also thinks about if we are in safety-critical settings, we want to calibrate uncertainty estimates. It also allows us to think about, from the Bayesian perspective of Meta-learning, what sort of principle approaches can be derived from those graphical models?

This story covers 1. Why be Bayesian? 2. Bayesian meta-learning approaches 3. How to evaluate Bayesians ... "   ...'

Classification with Naive Bayes

Good piece, behind a paywall but worth a look.  Technical.

Comparing a variety of Naive Bayes classification algorithms
Comprehensive list of formulas for text classification
Pavel Horbonos (Midvel Corp)

Naive Bayes algorithm is one of the well-known supervised classification algorithms. It bases on the Bayes theorem, it is very fast and good enough for text classification. I believe that there is no need to describe the theory behind it, nevertheless, we will cover a few concepts and after that focus on the comparing of different implementations.  .... "

Articles about Bayesian Methods and Networks

Just had a chance to reference this resource in DSC, good intro with a mix of technical and general introduction.

15 Great Articles about Bayesian Methods and Networks
Posted by Vincent Granville on March 15, 2019 

This resource is part of a series on specific topics related to data science: regression, clustering, neural networks, deep learning, decision trees, ensembles, correlation, Python, R, Tensorflow, SVM, data reduction, feature selection, experimental design, cross-validation, model fitting, and many more. To keep receiving these articles, sign up on DSC.  .... "

Bayesian Networks and Conditional Independence

Came up in a conversation about representing knowledge.

Conditional Independence — The Backbone of Bayesian Networks
Conditional Independence Intuition, Derivation, and Examples   By Aerin Kim 

Sometimes the explanation in Wikipedia is not the easiest to understand.
From https://en.wikipedia.org/wiki/Conditional_independence

Expectation Influences Perception

Known for some time.   Now how do we best  make use of this in AI interactions?   Can our brains be primed with signals to make them ready for interaction?  Can we measure what is needed using Bayesian methods?

How expectation influences perception

Neuroscientists find brain activity patterns that encode our beliefs and affect how we interpret the world around us.

MIT neuroscientists have identified patterns of brain activity that underlie our ability to interpret sensory input based on our expectations and past experiences.

By Anne Trafton | MIT News Office 

For decades, research has shown that our perception of the world is influenced by our expectations. These expectations, also called “prior beliefs,” help us make sense of what we are perceiving in the present, based on similar past experiences. Consider, for instance, how a shadow on a patient’s X-ray image, easily missed by a less experienced intern, jumps out at a seasoned physician. The physician’s prior experience helps her arrive at the most probable interpretation of a weak signal.

The process of combining prior knowledge with uncertain evidence is known as Bayesian integration and is believed to widely impact our perceptions, thoughts, and actions. Now, MIT neuroscientists have discovered distinctive brain signals that encode these prior beliefs. They have also found how the brain uses these signals to make judicious decisions in the face of uncertainty.

“How these beliefs come to influence brain activity and bias our perceptions was the question we wanted to answer,” says Mehrdad Jazayeri, the Robert A. Swanson Career Development Professor of Life Sciences, a member of MIT’s McGovern Institute for Brain Research, and the senior author of the study. .... "

Simplified Naive Bayes in One Image

Nicely done description I fired off to several people.

Naive Bayes in One Picture
Posted by Stephanie Glen in DSC

Naive Bayes is a deceptively simple way to find answers to probability questions that involve many inputs. For example, if you're a website owner, you might be interested to know the probability that a visitor will make a purchase. That question has a lot of "what-ifs", including time on page, pages visited, and prior visits. Naive Bayes essentially allows you to take the raw inputs (i.e. historical data), sort the data into more meaningful chunks, and input them into a formula.  .... '

Articles on Bayesian Theory and Methods

 Good introduction to these methods, including code.   Click through for links to each article.

15 Great Articles about Bayesian Methods and Networks
Posted by Vincent Granville on March 15, 2019

This resource is part of a series on specific topics related to data science: regression, clustering, neural networks, deep learning, decision trees, ensembles, correlation, Python, R, Tensorflow, SVM, data reduction, feature selection, experimental design, cross-validation, model fitting, and many more. To keep receiving these articles, sign up on DSC.

15 Great Articles about Bayesian Methods and Networks:

An Introduction to Bayesian Reasoning
Basics of Bayesian Decision Theory
How Bayesian Inference Works
Marketing Insight from Unsupervised Bayesian Belief Networks
Bayesian Nonparametric Models
Using Bayesian Kalman Filter to predict positions of moving particles
Naive Bayes Classification explained with Python code
Wheel Of Fortune - Bayesian Inference
Neural Networks from a Bayesian Perspective
A curated list of resources dedicated to bayesian deep learning
A quick introduction to PyMC3 and Bayesian models
Analysis of Perishable Products Sales Using Bayesian Inference
R and Stan: introduction to Bayesian modeling
And Monty Hall Went Bayesian...
Bayesian Probability

Towards Democratizing Data Science

Nice thought, though I wonder how the use/presentation of the results can spin the outcome.  Note the use of Bayesian methods, which we also used extensively to provide more explainable results.   Use of the Jupyter notebook is good for understanding.    Also related to some of our own work modeling data valuation.   Looking deeper

Democratizing Data Science 
MIT News   by Rob Matheson

Massachusetts Institute of Technology (MIT) researchers have developed a tool for nonstatisticians that automatically generates models for analyzing raw data. The tool takes in datasets and generates sophisticated statistical models normally used by experts to analyze, interpret, and predict underlying patterns in data. The tool currently resides on Jupyter Notebook, an open source Web framework that allows users to run programs interactively in browsers; users can write just a few lines of code to uncover insights into a range of topics. The system uses Bayesian modeling, a statistical method that continuously updates the probability of a variable as more information about the variable becomes available. The tool uses a modified version of "program synthesis," a technique that automatically creates computer programs given data and a language to work within. Said MIT’s Feras Saad, “People have a lot of datasets that are sitting around, and our goal is to build systems that let people automatically get models they can use to ask questions about that data.”

Bayesian Optimization

Under consideration for a project, how do we choose among multiple learning contexts and parameters?  When there is noise in the objective evaluations?   A means for automating machine learning?  This paper also makes the case.  See also full tutorial paper pointed to below, which is technical.

https://arxiv.org/abs/1807.02811   Abstract 

A Tutorial on Bayesian Optimization   By Peter I. Frazier

Bayesian optimization is an approach to optimizing objective functions that take a long time (minutes or hours) to evaluate. It is best-suited for optimization over continuous domains of less than 20 dimensions, and tolerates stochastic noise in function evaluations. It builds a surrogate for the objective and quantifies the uncertainty in that surrogate using a Bayesian machine learning technique, Gaussian process regression, and then uses an acquisition function defined from this surrogate to decide where to sample. In this tutorial, we describe how Bayesian optimization works, including Gaussian process regression and three common acquisition functions: expected improvement, entropy search, and knowledge gradient. We then discuss more advanced techniques, including running multiple function evaluations in parallel, multi-fidelity and multi-information source optimization, expensive-to-evaluate constraints, random environmental conditions, multi-task Bayesian optimization, and the inclusion of derivative information. We conclude with a discussion of Bayesian optimization software and future research directions in the field. Within our tutorial material we provide a generalization of expected improvement to noisy evaluations, beyond the noise-free setting where it is more commonly applied. This generalization is justified by a formal decision-theoretic argument, standing in contrast to previous ad hoc modifications.

Subjects: Machine Learning (stat.ML); Machine Learning (cs.LG); Optimization and Control (math.OC)   .... 

Bringing in Information Theory

Nice idea.   Its really all about the knowledge we have now, and what we can add to that by learning.  And ways we can usefully measure that.  Now how can we leverage this idea?  This is ultimately a 'computer science' technical idea, via Information Theory, but the author makes it as comprehensible as is possible.  Intro below, and then off you go.

When Bayes, Ockham, and Shannon come together to define machine learning  by Tirthajyoti Sarkar in TowardsDataScience

Editorial Associate "Towards Data Science" | Sr. Principal Engineer | Ph.D. in EE (U. of Iilinois)| AI/ML certification, Stanford, MIT | Open-source contributor

A beautiful idea, which binds together concepts from statistics, information theory, and philosophy.

Introduction

It is somewhat surprising that among all the high-flying buzzwords of machine learning, we don’t hear much about the one phrase which fuses some of the core concepts of statistical learning, information theory, and natural philosophy into a single three-word-combo.

Moreover, it is not just an obscure and pedantic phrase meant for machine learning (ML) Ph.Ds and theoreticians. It has a precise and easily accessible meaning for anyone interested to explore, and a practical pay-off for the practitioners of ML and data science.

I am talking about Minimum Description Length. And you may be thinking what the heck that is…

Let’s peal the layers off and see how useful it is…

Factor Analysis and Bayesian Networks

Factor analysis was a favorite statistical method we used to analyze complex influences.  Here is a link to a Bayesian approach. 

Factor Analysis Reinvented—Probabilistic Latent Factor Induction with Bayesian Networks and BayesiaLab

Bayesian networks have been gaining prominence among scientists over the last decade, and insights generated with this new paradigm can now be found in books and papers that circulate well beyond the academic community. Practitioners and managerial decision-makers see references to Bayesian networks in studies ranging from biostatistics to marketing analytics. Therefore, it is not surprising that the relatively new Bayesian network framework prompts comparisons with more conventional methods, such as Factor Analysis, which remains widely used in many fields of study.

The goal of this webinar is to compare a traditional statistical factor analysis with BayesiaLab's new workflow for Probabilistic Latent Factor Induction using a psychometric example.

More information, slides, presentation.

Definition and Use of Transfer Learning

Was pointed out to me that transfer learning was important to leveraging intelligence.    I had heard of it, but sought out a closer look at definition and areas of useful research, here is a start.  Not mentioned in any of the usual Quant methods in business books, but even starting with smaller or restricted prototypes can be seen as transfer learning.

Understanding Transfer Learning ....

Transfer learning,   From Wikipedia, the free encyclopedia: 
Transfer learning or inductive transfer is a research problem in machine learning that focuses on storing knowledge gained while solving one problem and applying it to a different but related problem.[1] For example, knowledge gained while learning to recognize cars could apply when trying to recognize trucks. This area of research bears some relation to the long history of psychological literature on transfer of learning, although formal ties between the two fields are limited.  .... " 

" ... The earliest cited work on transfer in machine learning is attributed to Lorien Pratt, who formulated the discriminability-based transfer (DBT) algorithm in 1993.[2] ... In 1997, the journal Machine Learning published a special issue devoted to transfer learning,[3] and by 1998, the field had advanced to include multi-task learning,[4] along with a more formal analysis of its theoretical foundations.[5] Learning to Learn,[6] edited by Pratt and Sebastian Thrun, is a 1998 review of the subject. .... Transfer learning has also been applied in cognitive science, with the journal Connection Science publishing a special issue on reuse of neural networks through transfer in 1996 ... " 

A Survey on Transfer Learning:

ACM article abstract.

A major assumption in many machine learning and data mining algorithms is that the training and future data must be in the same feature space and have the same distribution. However, in many real-world applications, this assumption may not hold. For example, we sometimes have a classification task in one domain of interest, but we only have sufficient training data in another domain of interest, where the latter data may be in a different feature space or follow a different data distribution. In such cases, knowledge transfer, if done successfully, would greatly improve the performance of learning by avoiding much expensive data-labeling efforts. 

In recent years, transfer learning has emerged as a new learning framework to address this problem. This survey focuses on categorizing and reviewing the current progress on transfer learning for classification, regression, and clustering problems. In this survey, we discuss the relationship between transfer learning and other related machine learning techniques such as domain adaptation, multitask learning and sample selection bias, as well as covariate shift. We also explore some potential future issues in transfer learning research ... " 

Update: Adversarial Risk Analysis Talk

 Talk given this week by Dr David Banks of Duke University and sponsored by Yichen Qin,  Assistant Professor, Department of Operations, Business Analytics, and Information Systems,  Lindner College of Business, University of Cincinnati,  on April 6, 2018

Adversarial Risk Analysis Talk, full announcement.

Speaker: Dr. David Banks, Duke University

Title: Adversarial Risk Analysis

Abstract: Adversarial Risk Analysis (ARA) is a Bayesian alternative to classical game theory. Rooted in decision theory, one builds a model for the decision-making of one's opponent, placing subjective distributions over all unknown quantities. Then one chooses the action that maximizes expected utility. This approach aligns with some perspectives in modern behavioral economics, and enables principled analysis of novel problems, such as a multiparty auction in which there is no common knowledge and different bidders have different opinion about each other.   .... " 

Here are the slides from the talk  (Technical)

Good talk. In particular because it dealt with  how people work in interactions.   Either versus nature, or versus other humans, here in some sort of competition.  The most common game example used was the Auction.  In the cases described these were adversarial.  Under the structure of this  'game' to win the auction.   Humans in any interaction build a model of who they are interacting with.   The methods proposed construct numerical methods to define the value of alternative strategies.

 But it immediately came to  mind that these methods don't need to be adversarial.   When people converse, or ask for help from an adviser  (Human or machine).  they are looking to maximize the value of the interaction.   In a chatbot, for example a person wants to solve a problem, the chatbot has defined knowledge.  How should the interaction proceed?   How should these advisory approaches be strategically designed?  Under constraints and costs?   How do we rate Assistants in an approach to provide value?  Whats the structure of that game?    Examining.

Talk: Adversarial Risk Analysis

Of interest in particular, due to the adversarial element.   Risk analysis does not often enough consider the intelligent nature of agents that create risk.   Competitors for example, and now even intelligent agents in the form of assistants or other machine driven systems.

The OBAIS department at the Lindner College of Business, University of Cincinnati, invites you to attend the Lindner Research Excellence Seminar and Professional Development Discussion:

Research Seminar:
Date and time: Friday, April 6, 2018, 10:30AM-12:00PM 
Location: Lindner Hall 218
Speaker: Dr. David Banks, Duke University
Title: Adversarial Risk Analysis

Abstract: Adversarial Risk Analysis (ARA) is a Bayesian alternative to classical game theory. Rooted in decision theory, one builds a model for the decision-making of one's opponent, placing subjective distributions over all unknown quantities. Then one chooses the action that maximizes expected utility. This approach aligns with some perspectives in modern behavioral economics, and enables principled analysis of novel problems, such as a multiparty auction in which there is no common knowledge and different bidders have different opinion about each other.

This presentation is based on:
Rios Insua, D., Rios, J., Banks, D. L., “Adversarial Risk Analysis,” Journal of the American Statistical Association, 2009.
https://www.tandfonline.com/doi/abs/10.1198/jasa.2009.0155
Banks, D. L., Rios, J., Rios Insua, D., “Adversarial Risk Analysis,” Chapman and Hall/CRC, 2015, ISBN 9781498712392.

https://www.crcpress.com/Adversarial-Risk-Analysis/Banks-Aliaga-Insua/p/book/9781498712392

Best wishes,
Yichen Qin,  Assistant Professor
Department of Operations, Business Analytics, and Information Systems
Lindner College of Business, University of Cincinnati
Website: http://business.uc.edu/academics/departments/obais/faculty/qinyn.html
Email: qinyn@ucmail.uc.edu

Optimizing Health Policies with Bayesian Networks

 Another excellent, mostly nontechnical presentation on the topic.   Interesting is the decision model itself, and the topic of health decisions.  Unlike most modeling methods, this approach embeds the details of the model into the decision process being modeled.  So you can visually see the details of what is being modeled and discuss it with decision makers.  Also, it directly models uncertainty involved, based on real known data.   We used these methods actively,  I but find them rarely applied in business.   Consider it.  ....

Presentation link and slides below: 

By Stefan Conrady

Managing Partner at Bayesia USA & Singapore: Bayesian Networks for Research, Analytics, and Reasoning

Optimizing Health Policies with Bayesian Networks

In case you missed today's webinar, here is the recording. Today's program was about developing a reasoning framework for health policies in developing nations with Bayesian networks. The specific study question was whether to implement a "test & treat" policy versus a presumptive treatment approach for malaria and bacterial pneumonia. https://bayesia.wistia.com/medias/16vb2vljlt

Decision and Reasoning Analysis with Uncertainty

Am a big proponent of linking models directly to decisions, and thus make them relevant to real business process.  Business is all about uncertain business decisions.  One way to do this is to uses Bayesian networks.  And the software Bayesia.  They have a contract with a previous employer for this purpose.  I saw it successfully used there.

They have just released another very accessible presentation on how to use Bayesian Networks.    Well worth understanding if you make, define, improve or maintain decisions.

Probabilistic Reasoning Under Uncertainty with Bayesian Networks  given by Stefan Conrady of Bayesia  ...

Bayesian Networks for Media Mix

Been pointing to some of the presentations by Bayesia Lab in their recent Paris show because I know there are readers out there who are interested in applications of Bayesian Networks.   Media Mix is a classical optimization problem in the CPG and marketing world.   How do I apportion advertising spending to various media?   Classic optimization.    You know the saw, "what part of my advertising spending is wasted?"  Did lots of this early on and intriqued by the connections:

Erin Barr: Media Mix Optimization Using Bayesian Belief Networks and BayesiaLab