EBook on Ensemble Learning

I see that Jason Brownlee has a new book on Ensemble Learning,  a method  I recently mentioned here,  “Ensemble Learning Algorithms With Python“   at the link a considerable look at it. Have not examined it myself as yet.  

…so What is Ensemble Learning?

Ensemble learning algorithms combine the predictions of two or more models.

The idea of ensemble learning is closely related to the idea of the “wisdom of crowds“. This is where many different independent decisions, choices or estimates are combined into a final outcome that is often more accurate than any single contribution.

This is the core idea behind major aspects of modern society, such as a scientific peer review, a jury of peers, and seeking a second opinion. It is an alternative to seeking out and taking the advice of an expert.

In applied machine learning, it means combining the predictions from multiple models trained on your dataset, instead of seeking the single best performing model. ... 

Considering Composite AI:

Could not agree more, but I am seeing practitioners wanting to stay with pure 'AI/Machine Learning'. It stays with the 'magic' of AI.  When hybrid methods should be used.  Why?  Because the other methods are seen as dated?  And require quite different training.  Are just classic analytics?  In typical analytics these are called 'Ensemble Methods', often mentioned here, we used them often, see the tags below.

Composite AI: What Is It, and Why You Need It    Alex Woodie in DataNami

You might have noticed a new term, “composite AI,” floating around the cybersphere. Don’t worry–it’s not a complex new technology that you must master. In fact, while the term may be new, the core idea behind it is not. Nevertheless, it’s likely a technique that you should be thinking about incorporating in your enterprise AI processes.

Gartner helped put composite AI on the map last summer, when it published its 2020 Hype Cycle for Emerging Technologies. Simply put, Composite AI refers to the “combination of different AI techniques to achieve the best result,” according to Gartner. That’s it. Simple enough, right?

So, what other AI techniques could that mean? It’s important here to keep in mind that AI is a very broad term. While some might believe that AI refers to the latest, greatest deep learning and neural network algorithms, AI actually covers much more under its sizable umbrella.

Machine learning and deep learning are types of AI. But there are many other types of AI that should be in your wheelhouse that fall outside of the machine learning/deep learning bubble. That includes traditional rules-based systems, natural language processing (NLP), optimization techniques, and graph techniques, according to Gartner.

A composite AI system is to be built atop a “composite architecture,” which Gartner identified as its number one Hype Cycle trend for 2020. A composite architecture (you might have guessed) incorporates packaged business capabilities that run atop a flexible data fabric, thereby enabling users to take be flexible and adaptable amidst rapidly changing systems and requirements.   ... "

Ensemble Methods for Combining

Ensemble methods are not new, they have long been around to use combine multiple kinds of techniques.  But we have more methods now.   Here a good mostly non technical view of how to combine methods. With straight forward code.  I like the HOW and WHY.

Model averaging methods: how and why to build ensemble models

Averaging predictions for fun and profit — and for dealing with the uncertainty of model selection. With examples in R!

Michael Mahoney in TowardsDatascience  ... " 

Occams Razor and Ensemble Learning

Thoughtful piece.  Much more at the link.   Here the intro:

Ensemble Learning Algorithm Complexity and Occam’s Razor   by Jason Brownlee on December 21, 2020 in Ensemble Learning 

Occam’s razor suggests that in machine learning, we should prefer simpler models with fewer coefficients over complex models like ensembles.

Taken at face value, the razor is a heuristic that suggests more complex hypotheses make more assumptions that, in turn, will make them too narrow and not generalize well. In machine learning, it suggests complex models like ensembles will overfit the training dataset and perform poorly on new data.

In practice, ensembles are almost universally the type of model chosen on projects where predictive skill is the most important consideration. Further, empirical results show a continued reduction in generalization error as the complexity of an ensemble learning model is incrementally increased. These findings are at odds with the Occam’s razor principle taken at face value.

In this tutorial, you will discover how to reconcile Occam’s Razor with ensemble machine learning.

After completing this tutorial, you will know:

heuristic that suggests choosing simpler machine learning models as they are expected to generalize better. The heuristic can be divided into two razors, one of which is true and remains a useful tool and the other that is false and should be abandoned.

Ensemble learning algorithms like boosting provide a specific case of how the second razor fails and added complexity can result in lower generalization error.

Let’s get started.  ...  "

Tutorial: Random Forest for Time Series

 Am a long  time proponent of ensemble methods.  Here Jason Brownlee provides a nice tutorial on an often powerful method everyone should know.  As usual, well done, minimal tech.

Random Forest for Time Series Forecasting   by Jason Brownlee

by Jason Brownlee  in Time Series

Random Forest is a popular and effective ensemble machine learning algorithm.

It is widely used for classification and regression predictive modeling problems with structured (tabular) data sets, e.g. data as it looks in a spreadsheet or database table.

Random Forest can also be used for time series forecasting, although it requires that the time series dataset be transformed into a supervised learning problem first. It also requires the use of a specialized technique for evaluating the model called walk-forward validation, as evaluating the model using k-fold cross validation would result in optimistically biased results.

In this tutorial, you will discover how to develop a Random Forest model for time series forecasting.

After completing this tutorial, you will know:

Random Forest is an ensemble of decision trees algorithms that can be used for classification and regression predictive modeling.  Time series datasets can be transformed into supervised learning using a sliding-window representation.How to fit, evaluate, and make predictions with an Random Forest regression model for time series forecasting.

Let’s get started.   ....  

On Ensemble Methods, the How and Why

Nicely done piece on the topic.  The general overiew motivates the use of all kinds of group solutions, even without analytics methods involved.  We often used them if we were at all uncertain of best solution methods.   They also could give better indication  of the breadth of value in a solution.  Worth reading.

How and why of the ensemble models  in TowardsDataScience  
why does crowd intelligence work?
Dr. Saptarsi Goswami

 There is a famous game show in India named “Kaun Banega Crorepati (KBC)” inspired by “Who wants to be a Millionaire”. It was kind of a quiz show with multiple choices of answers. If the participants can choose the right option for all questions, he or she can win 10 Million Indin Rupees. The participants had some options to resort to if he is unsure about any question. One such option was taking an audience poll and go by the majority choice.

The questions could be from Sports, Mythology, Politics, Music, Movies, Culture, Science, etc that is from a variety of subjects. The audience were no experts of any such subject. Interestingly, more often than not the majority opinion will turn out to be the correct answer. This is the central concept behind ensembling.

Seems like a Magic right, let’s look at the mathematical intuition of the same.
The idea is presented using the following simple diagram, in the context of ML. Now each person is equivalent to a classifier and like the audience is diverse, the more uncorrelated the classifiers better it is.  ... " 

Improving AI via Deep Learning and Process

Thoughts on how machine learning will improve.  I agree that in particular hybrid models that bring in process logic, extensions of  RPA methods,  and  simulation models of the 'twin' form will also become more important.

Deep learning is about to get easier — and more widespread
By Ben Dickson in Venturebeat

We’ve seen a big push in recent months to solve AI’s “big data problem.” And some interesting breakthroughs have begun to emerge that could make AI accessible to many more businesses and organizations.

What is the big data problem? It’s the challenge of getting enough data to enable deep learning, a very popular and promising AI technique that allows machines to find relationships and patterns in data by themselves. (For example, after being fed many images of cats, a deep learning program could create its own definition of what constitutes ‘cat’ and use that to identify future images as either ‘cat’ or ‘not cat’. If you change ‘cat’ to ‘customer,’ you can see why many companies are eager to test-drive this technology.)

Recommended Videos

Deep learning algorithms often require millions of training examples to perform their tasks accurately. But many companies and organizations don’t have access to such large caches of annotated data to train their models (getting millions of pictures of cats is hard enough; how do you get millions of properly annotated customer profiles — or, considering an application from the health care realm, millions of annotated heart failure events?). On top of that, in many domains, data is fragmented and scattered, requiring tremendous efforts and funding to consolidate and clean for AI training. In other fields, data is subject to privacy laws and other regulations, which may put it out of reach of AI engineers.

This is why AI researchers have been under pressure over the last few years to find workarounds for the enormous data requirements of deep learning. And it’s why there’s been a lot of interest in recent months as several promising solutions have emerged — two that would require less training data, and one that would allow organizations to create their own training examples.

Here’s an overview of those emerging solutions.

Zappos Uses Genetic Algorithms

Surprising application in place and an indication of multiple algorithms in parallel.

At Zappos, Algorithms Teach Themselves 
The Wall Street Journal  (with paywall)
Jared Council

Online shoe and clothing retailer Zappos sees promise in a self-learning algorithm's ability to address the problem of its search engine producing irrelevant results. Zappos' chief data scientist Ameen Kazerouni said several years ago his team began testing a genetic algorithm, which has since become critical to boosting the search engine's relevancy. Genetic algorithms generate various solutions to a problem, using natural-selection principles like reproduction and mutation to return the optimal or "fittest" solution. The algorithms were designed to parse out the intent of a search phrase, with those that perform best on an internal "relevance test," which models how users engage with search results, having the greatest odds of having their traits inherited by the next generation. Zappos uses three genetic algorithm engines in parallel to generate better search results.   .... " 

Handling Noise in Modeling

In the Statistics.com Blog.  Useful piece on handling the inevitable noise with models.  Directional rather than technical piece.

Handling the Noise - Boost It or Ignore It?

Posted on Mar 06, 2019 By: Peter Bruce

In most statistical modeling or machine learning prediction tasks, there will be cases that can be easily predicted based on their predictor values (signal), as well as cases where predictions are unclear (noise). Two statistical learning methods, boosting and ProfWeight, use those difficult cases in exactly opposite ways - boosting up-weights them, and ProfWeight down-weights them. ... "

Best Analysis with Ensembles

Promoting multiple (aka Ensemble) methods.  New book. Have for years used what are now called ensemble methods.  This piece gives good motivation for its use.  Passing it along.   Podcast and transcript.

Ensemble Models
How to Get the Best Results from Big Data Analysis
Author Scott E. Page, a complex systems expert, explains how applying multiple data analysis models greatly enhances decision making.

Scott E. Page, professor of complex systems, political science and economics at the University of Michigan, doesn’t want people to limit themselves to linear thinking. In his new book, The Model Thinker: What You Need to Know to Make Data Work for You, he explains how taking a multi-paradigm approach puts more power into solving problems, innovating and understanding the full range of consequences to complex actions. He believes using many models is the best way to make sense out of the reams of data available in today’s digital world. Page recently spoke on the Knowledge@Wharton radio show on Sirius XM about why it’s important to widen your data lens.  

 An edited transcript of the conversation follows.

Knowledge@Wharton: What is multi-model thinking?

Scott Page: We live in this time where there are two fundamental things going on. One is, there’s just a firehose or hairball of data, right? Tons of data out there. At the same time, we have this recognition that the problems and challenges that we confront are complex. And by that, I mean high-dimensional, lots of interdependencies, difficult to understand. So, what do we do? How do we use that data to confront the complexity?

The philosophy I’m putting forward goes as follows: You have to arrange that data on some sort of model. You want to think of a model as Charlie Munger, the famous investor, describes it — a latticework of understanding on which you can array the data.

But models by definition are simple, so there’s a disconnect. I’m trying to understand something complex with something that’s simple. What I’ve bought with that simplicity is logical coherence. But what I’ve lost in that simplicity is any notion of coverage because there’s too much stuff I’ve got to leave out.

Instead, what I propose you do is bring an ensemble of models to bear. This is a thing. People in machine learning have been doing this; all the fancy stuff’s going on in AI. If you really unpack what’s going on in those sophisticated algorithms, they really are ensembles of little algorithms and little rules. The idea is, any one model is going to be wrong, but many models are going to be not only a lot of coverage, but also a collection of coherent understandings of a complex phenomenon.

Knowledge@Wharton: Is this multi-model approach common in the business world? ... 
"

Ensemble Models for Deep Learning

Ensemble models are now commonly used in all sorts of analytics.  You use the results of multiple models and combine the results.  Jason Brownlee shows how this can be done for deep learning methods.  Good tutorial explanation.

How to Create a Random-Split, Cross-Validation, and Bagging Ensemble for Deep Learning in Keras  by Jason Brownlee in Better Deep Learning

Ensemble learning are methods that combine the predictions from multiple models.

It is important in ensemble learning that the models that comprise the ensemble are good, making different prediction errors. Predictions that are good in different ways can result in a prediction that is both more stable and often better than the predictions of any individual member model.

One way to achieve differences between models is to train each model on a different subset of the available training data. Models are trained on different subsets of the training data naturally through the use of resampling methods such as cross-validation and the bootstrap, designed to estimate the average performance of the model generally on unseen data. The models used in this estimation process can be combined in what is referred to as a resampling-based ensemble, such as a cross-validation ensemble or a bootstrap aggregation (or bagging) ensemble.

In this tutorial, you will discover how to develop a suite of different resampling-based ensembles for deep learning neural network models.   ... " 

AI Eye Tracking of 200 Eye Movements

What appears to be a deeper kind of Eye movement biometric analysis than is common.  Dealing with both enhrent traits, and also real time changes?  Note that the test mentioned included retail interaction.  Consider comparison and ensemble use with other measures.  Also note the specific use of decision tree to pre establish range of classification.

This New AI Can Track 200 Eye Movements to Determine Your Personality Traits   Business Insider   Katharina Mass in  Business Insider

Researchers at Saarland University's Max Planck Institute for Computer Science and the University of Stuttgart in Germany, along with colleagues from the University of South Australia, have created software that recognizes personality traits via eye tracking. The researchers equipped 50 student volunteers with devices that recorded their eye movements. The volunteers were then asked to walk across the campus for about 10 minutes and purchase something from one of the campus shops. The subjects also completed questionnaires commonly used to evaluate personality traits. The team used more than 200 markers, such as the frequency with which subjects blinked and how long they focused on something, to determine traits that are linked to certain eye movements. With this data, the researchers created "decision trees" for various personality traits, enabling the software to recognize designated characteristics. In addition, the team says the eye-tracking software can gauge whether someone is conscientious, sociable, and tolerant, and even to what extent they might be emotionally unstable.  ... "

A Guide to Ensemble Learning, with my Cautions

Nicely done.  Ensemble methodology means using a number of different solution methods to solve a problem, and finding the best individual or combination of methods.  Caution must be taken to ensure that the assumptions for each method is reasonable, that the right metadata is available to apply the solution, and the solution is stable under its assumptions. 

Often this means that the model needs to be simplified to some degree.  Make sure the simplification is reasonable for purpose and that owners of the data and process agree.   Have seen reasonability slip away under pressure for a best solution.

Guide to Ensemble Learning with Python code, considerable depth, via O'Reilly

A Comprehensive Guide to Ensemble Learning (with Python codes)
By  Aishwarya Singh in Vidhya Learning .... 

Model Ensembling to Reduce Error

Technical but nicely done piece, with code and intuitive arguments as to value.  Via O'Reilly:

An introduction to model ensembling by Jovan Sardinha

Model ensembling represents a family of techniques that help reduce generalization error in machine learning tasks. In this article, I will share some ways that ensembling has been employed and some basic intuition on why it works.  .... " 

Concise Algorithms for Machine Learning

Nicely done piece by Ronald van Loon,  The 10 Algorithms Machine Learning Engineers Need to Know . I like these concise descriptions of useful techniques. From an engineering rather than a mathematical perspective.  Its good to know these for sharing with decision makers.  Inclusion and descriptions are debatable, which is OK.  These also include explanatory visuals of each each technique.

R Connects to Watson for Decision Improvement

Every successful Cognitive/AI  project I have been involved in since the 80s has also used machine learning analytics techniques now readily available in R.   Machine learning data science can be seen as a way to add intelligence to a system, and prepare it for better decision based methods.

  I think we will continue to see this collaboration between methods.  This is a new kind of ensemble technique that will evolve.   Decision making can be directly involved. Will continue to follow this development.

" ... New R extension gives data scientists easy access to IBM's Watson   By Katherine Noyes  

Data scientists have a lot of tools at their disposal, but not all of them are equally accessible. Aiming to put IBM's Watson AI within closer reach, analytics firm Columbus Collaboratory on Thursday released a new open-source R extension called CognizeR.

R is an open-source language that's widely used by data scientists for statistical and analytics applications. Previously, data scientists would have had to exit R to tap Watson's capabilities, coding the calls to Watson's application programming interfaces (APIs) in another language, such as Java or Python.

Now, CognizeR lets them tap into Watson's so-called "cognitive" artificial-intelligence services without leaving their native development environment.

"Data scientists can now seamlessly tap into our cognitive services to unlock data that lives in unstructured forms like chats, emails, social media, images, and documents," wrote Rob High, vice president and CTO for Watson, in a blog post.  .. " 

Lecture on Decision Trees

Video  " ... Random forests, aka decision forests, and ensemble methods. Slides available at: http://www.cs.ubc.ca/~nando/540-2013/... Course taught in 2013 at UBC by Nando de Freitas ... " 

Ensemble Methods for Tuning Results

Good overview.  We used these methods before they were well understood.   Modern coding makes this much easier to do, and even to adapt as your data changes.  In Python or R.  Consider automating the method to tune your results to changing data.  Or to changes in targeted process.

Automating Multiple Streams of Biometric Analysis

I see that iMotions is out with a method that can analytically combine eye tracking, facial expression, ECG and more.   Have not used their devices and methods, but combining streams of data would be interesting.  This is analogous to ensemble methods in analytics.  Does this further need a cognitive, in a true sense, method to combine results?    More here.

Gathering in Random Forests

Random Forests. A notable ensemble-style method for analytic classification.  Particularly interesting to me because it uses more easily understood decision trees.     Some relationship to other clustering methods like K Means.  As it is available in R  Note that ensemble methods use multiple kinds (or parameters) of analytical models and compare their value on a given problem.  The approach is becoming increasingly used for tough problems.