Decision Trees Explained

 Nicely done,  only somewhat technical   ...  worth understanding, intro below, more at link.  Its often very useful to construct a decision tree to understand a process being modelled before going deeper.  

Towards Data Science by Shailey Dash / November 02, 2022

Decision Trees Explained — Entropy, Information Gain, Gini Index, CCP Pruning

Though Decision Trees look simple and intuitive, there is nothing very simple about how the algorithm goes about the process deciding on splits and how tree pruning occurs. In this post I take you through a simple example to understand the inner workings of Decision Trees.

Decision Trees are a popular and surprisingly effective technique, particularly for classification problems. But, the seemingly intuitive interface hides complexities. The criterion for selecting variables and hierarchy can be tricky to get, not to mention Gini index, Entropy ( wait, isn’t that physics?) and information gain (isn’t that information theory?). As you can see there are lots of tricky problems on which you can get stuck on. The best way to understand Decision Trees is to work through a small example which has sufficient complexity to be able to demonstrate some of the common points one suddenly goes, ‘ not sure what happens here…?’.

This post is therefore more like a tutorial or a demo where I will work through a toy dataset that I have created to understand the following:  ... .' 

Decision Trees

Quick overview if decision trees. 

Decision Tree Algorithm, Explained

All you need to know about decision trees and how to build and optimize decision tree classifier.

By Nagesh Singh Chauhan, Data Science Enthusiast on February 9, 2022 in Machine Learning

Introduction

Classification is a two-step process, learning step and prediction step, in machine learning. In the learning step, the model is developed based on given training data. In the prediction step, the model is used to predict the response for given data. Decision Tree is one of the easiest and popular classification algorithms to understand and interpret.

Decision Tree Algorithm

Decision Tree algorithm belongs to the family of supervised learning algorithms. Unlike other supervised learning algorithms, the decision tree algorithm can be used for solving regression and classification problems too.

The goal of using a Decision Tree is to create a training model that can use to predict the class or value of the target variable by learning simple decision rules inferred from prior data(training data).

In Decision Trees, for predicting a class label for a record we start from the root of the tree. We compare the values of the root attribute with the record’s attribute. On the basis of comparison, we follow the branch corresponding to that value and jump to the next node.

Types of Decision Trees

Types of decision trees are based on the type of target variable we have. It can be of two types:

Building Decision Trees by Hand without Code

 Cool little piece that is instructional and even inspirational.   Math and process seen together to produce real value.

Random Forest and Decision Trees by hand — no coding

By Thomas Le Menestrel,      Computational Engineering student at Stanford | Passionate about Machine Learning and Data Science  in TowardsDataScience 

Introduction

In this article, we will discuss Decision Trees and Random Forest, two algorithms used in Machine Learning for classification and regression tasks.

I will show how to build a Decision Tree from scratch using a pen and paper and how to generalise this and build a Random Forest model.

Dataset

Let’s see how this work in practice with a simple dataset. .... ( much more )

Intro to Decision Trees

Good general piece from KDNuggets,  in Basics of Decision Trees

Decision Tree Algorithm, Explained

All you need to know about decision trees and how to build and optimize decision tree classifier.

By Nagesh Singh Chauhan, Data Science Enthusiast on February 9, 2022 in Machine Learning

Introduction

Classification is a two-step process, learning step and prediction step, in machine learning. In the learning step, the model is developed based on given training data. In the prediction step, the model is used to predict the response for given data. Decision Tree is one of the easiest and popular classification algorithms to understand and interpret.

Decision Tree Algorithm

Decision Tree algorithm belongs to the family of supervised learning algorithms. Unlike other supervised learning algorithms, the decision tree algorithm can be used for solving regression and classification problems too.

The goal of using a Decision Tree is to create a training model that can use to predict the class or value of the target variable by learning simple decision rules inferred from prior data(training data).

In Decision Trees, for predicting a class label for a record we start from the root of the tree. We compare the values of the root attribute with the record’s attribute. On the basis of comparison, we follow the branch corresponding to that value and jump to the next node.

Types of Decision Trees

Types of decision trees are based on the type of target variable we have. It can be of two types:  .... See the complete docatthe link.  

Molecular Device Reconfigured for Computational Tasks

Intriguing direction.  How many such computational tasks? 

Molecular Device Can Be Reconfigured for Different Computational Tasks, By Texas A&M University, September 9, 2021  in CACM.

An international team of researchers has described a novel molecular device with exceptional computing prowess. 

The device can be reconfigured on the fly for different computational tasks by simply changing applied voltages. The same device can also retain information for future retrieval and processing.

It is described in "Decision Trees Within a Molecular Memristor,"   published in the journal Nature.

"We have now created a molecular device with dramatic reconfigurability, which is achieved not by changing physical connections like in the brain, but by reprogramming its logic," says R. Stanley Williams, professor in the Department of Electrical and Computer Engineering at Texas A&M University.

The molecular device might in the future help design next-generation processing chips with enhanced computational power and speed, but consuming significantly reduced energy, says Thirumalai Venkatesan, director of the Center for Quantum Research and Technology at the University of Oklahoma, and adjunct professor at the National University of Singapore.

From Texas A&M University, View Full Article

Decision Trees and Random forests

This is fundamental stuff, that every practitioner should know well.  Its the simplest way that machines can be made to learn. And provides a good way to deliver real results.  Nice description... most is quite non-technical.  You should also know when this wont work, the data required, cautions to take .... but this is a great start

How do Decision Trees and Random Forests Work?

Part 1: Decision Trees

By Dick Brown  in TowardsDatascience

 Decision trees and random forests are two commonly used algorithms in predictive modeling. In this article, I’m going to discuss the process behind decision trees. I’m planning to follow this up with a second part that discusses random forests, and then compare the two.

First off: decision trees. A decision tree is named for the shape of the plot that comes out. The image below shows a decision tree for deciding what factors affected survival from the Titanic disaster.  ... " 

Using VisiRule to Create Chatbots

Had been looking for an easy way to create chatbots that include FAQ's. Recalled a company called  VisiRule that supported rule-based expert systems.  Decision tree style chatbots.    We looked at them some time ago.    Below are some examples of their work, with a number of chatbot examples.  VisiRule has been mentioned many times in this blog, see their tag below.

ChatBot Demos

This page contains various ChatBot demos which have been automatically generated from a VisiRule chart. This allows authors to quickly draw the intended conversation structure visually and easily. In addition, common questions can be answered by attaching a FAQ KB in the form of question/response. VisiRule contains a NLU component to help match user input text to these known questions. The combination of a structured decision tree flowchart conversation with general purpose information retrieval makes for a more rounded and holistic user experience.  ... "

Making Decision Trees Accurate and Explainable

Explaining AI, Decision Trees

Berkeley Artificial Intelligence Research
Making Decision Trees Accurate Again: Explaining What Explainable AI Did Not   By Alvin Wan  

The interpretability of neural networks is becoming increasingly necessary, as deep learning is being adopted in settings where accurate and justifiable predictions are required. These applications range from finance to medical imaging. However, deep neural networks are notorious for a lack of justification. Explainable AI (XAI) attempts to bridge this divide between accuracy and interpretability, but as we explain below, XAI justifies decisions without interpreting the model directly.

What is “Interpretable”?
Defining explainability or interpretability for computer vision is challenging: What does it even mean to explain a classification for high-dimensional inputs like images? As we discuss below, two popular definitions involve saliency maps and decision trees, but both approaches have their weaknesses. .... " 

Decision Trees for Shopping

Perhaps simplistic, but interesting.   Even if you have to ultimately complicate such a model, its useful to start with something this simple and see how well it can predict, then add more complexity later.  In the enterprise, we did lots of that.  It can also act as a benchmark for more complex models.  Decision trees are also nicely transparent.

Decision Trees for Online Shopping Analysis
Towards Data Science by Chathuranga Siriwardhana  

Nowadays there is a trend to use online shopping solutions like Amazon, eBay, AliExpress. These websites provide a platform for the sellers to sell their products to a large number of customers. Since many delivery services are connected with these online shopping platforms, customers from different countries buy products. Unlike the traditional shops, the ratings and the good-name is directly represented on the shopping platform for each seller. Therefore the sellers have let the customers return their bought items if they don’t like the product or there is any defect of the item. Some sellers refund the whole amount if the customers complain that the items are not delivered within the promised period. Some customers are misusing these facilities and fraud to the sellers. Therefore, the sellers on the online shopping platforms experience a huge loss of profits. Let’s discuss how we can spot these types of customers by developing a simple Machine Learning model; a Decision Tree.

Have a look at this medium post on Decision Trees if you are not familiar with them. For a quick recap, a decision tree is a model in machine learning which includes the conditions on which we are categorizing the data (for labelling problem). As an example, think about a simple situation where a man is happy is the weather is sunny or he is on vacation. This scenario is modelled below. Note that you can use weather and vacation status to predict the man’s happiness with this model.  ... "

Unifying Logical and Statistical AI With Markov Logic

As AI practitioners in the enterprise we understood this early on.   You need to know the results of statistical analysis AND the ability to link them usefully to logical decision making.   Sometimes easy,  sometimes not   Thus approaches like decision trees based on statistical data became popular for our team.   We understood too that Markov methods could provide the framework for providing this, so we experimented with them.  In both cases the results were also relatively transparent.    This unification can also outline way that humans will interact with the AI.  Research on the idea was going on then and is still now.  Below gives you a good update.  Starts basic and gets technical.

Unifying Logical and Statistical AI with Markov Logic
By Pedro Domingos, Daniel Lowd 
Communications of the ACM, July 2019, Vol. 62 No. 7, Pages 74-83    10.1145/3241978

For many years, the two dominant paradigms in artificial intelligence (AI) have been logical AI and statistical AI. Logical AI uses first-order logic and related representations to capture complex relationships and knowledge about the world. However, logic-based approaches are often too brittle to handle the uncertainty and noise present in many applications. Statistical AI uses probabilistic representations such as probabilistic graphical models to capture uncertainty. However, graphical models only represent distributions over propositional universes and must be customized to handle relational domains. As a result, expressing complex concepts and relationships in graphical models is often difficult and labor-intensive.  .... "   (  Full Technical paper)

Video intro to the concept (technical): 





Alchemy Language, mentioned in the above talk:

https://alchemy.cs.washington.edu/
Alchemy: Open Source AI
Welcome to the Alchemy system! Alchemy is a software package providing a series of algorithms for statistical relational learning and probabilistic logic inference, based on the Markov logic representation. Alchemy allows you to easily develop a wide range of AI applications, including: .... " 

A View of Decision Trees

A considerable and detailed look at decision trees, and multiple applications.   We used DT methods extensively in the enterprise.  We implemented it within our standard AI methods.   Had been used for a long time because the specifics of the approach allow for complete transparency.   It as much machine learning as neural nets are.    Its not considered enough today because its not seen as sexy enough.

The Complete Guide to Decision Trees  (Long article that provides a good overview, then gets technical) 

Posted by Diego Lopez Yse in DSC

Everything you need to know about a top algorithm in Machine Learning

In the beginning, learning Machine Learning (ML) can be intimidating. Terms like “Gradient Descent”, “Latent Dirichlet Allocation” or “Convolutional Layer” can scare lots of people. But there are friendly ways of getting into the discipline, and I think starting with Decision Trees is a wise decision.

Decision Trees (DTs) are probably one of the most useful supervised learningalgorithms out there. As opposed to unsupervised learning (where there is no output variable to guide the learning process and data is explored by algorithms to find patterns), in supervised learning your existing data is already labelled and you know which behaviour you want to predict in the new data you obtain. This is the type of algorithms that autonomous cars use to recognize pedestrians and objects, or organizations exploit to estimate customers lifetime value and their churn rates.

In a way, supervised learning is like learning with a teacher, and then apply that knowledge to new data.

DTs are ML algorithms that progressively divide data sets into smaller data groups based on a descriptive feature, until they reach sets that are small enough to be described by some label. They require that you have data that is labelled (tagged with one or more labels, like the plant name in pictures of plants), so they try to label new data based on that knowledge. .... " 

Classification and Regression Trees

Good piece about a favorite method, in part because its results and methods are easy to explain.  Used it many times in the enterprise, even developed semi automated similar methods.

Classification and Regression Trees in DSC by Packt

Learn about CART in this guest post by Jillur Quddus, a lead technical architect, polyglot software engineer and data scientist with over 10 years of hands-on experience in architecting and engineering distributed, scalable, high-performance, and secure solutions used to combat serious organized crime, cybercrime, and fraud.

Although both linear regression models allow and logistic regression models allow us to predict a categorical outcome, both of these models assume a linear relationship between variables. Classification and Regression Trees (CART) overcome this problem by generating Decision Trees. These decision trees can then be traversed to come to a final decision, where the outcome can either be numerical (regression trees) or categorical (classification trees). A simple classification tree used by a mortgage lender is illustrated in the following diagram:  .... " 

Practicality of Decision Trees

Good piece on the pros and Cons and design of Decision trees, which we often selected as a first approach when we addressed a problem.  Not because we were certain it was the best approach, but because it started with something that was almost always included, a human decision.   Even if there was no obvious human decision being made, we made it a point to look for one, either before the machine decision, in constructing data, or afterwards in using the analytical results.  And, as the piece below suggests, its the easiest to explain.

Building Blocks of Decision Trees
Posted by Divya Singh in DSC .... 

Decision Trees with Python

With a full conceptual implementation in Python:

A breath of fresh air with Decision Trees in Medium

A very versatile decision support tool, capable of fitting complex algorithms, that can perform both classification and regression tasks, and even multi output tasks.

Trees are very interesting beings… they can start from a single branch and develop into a very complex network of branches with millions of leaves at their ends. It’s curious that a great number of technologies and methodologies are created based on what we see in Nature. Machine Learning Decision Tree algorithm is one of those cases!

A decision tree is a Supervised Machine Learning algorithm. This non-parametric system, contrary to Linear Regression models (which assume linearity), makes no underlying assumptions about the distribution of the errors or the data. It is a flowchart-like structure, composed of several questions (node) and depending on the answers (branch) given it will lead to a class label or value (leaf) when applied to any observation.  ... "

Unsupervised Decision Trees

Nicely done piece.   Big supporter of decision trees in general, since they have a basic element of transparency.

Have You Heard About Unsupervised Decision Trees

By William Vorhies in DSC

Summary: Unless you’re involved in anomaly detection you may never have heard of Unsupervised Decision Trees.  It’s a very interesting approach to decision trees that on the surface doesn’t sound possible but in practice is the backbone of modern intrusion detection.

I was at a presentation recently that focused on stream processing but the use case presented was about anomaly detection.  When they started talking about unsupervised decision trees my antenna went up.  What do you mean unsupervised decision trees?  What would they split on?
It turns out that if you’re in the anomaly detection world unsupervised decision trees are pretty common.  Since I’m not in that world and I suspect few of us are, I thought I’d share what I found. .... "

Decision Trees in Practice

They can also work because they are transparent.     But that transparency can also show when they are difficult to use operationally.   Which can be a good thing.

Why do Decision Trees Work?   in DSC
Posted by Amelia Matteson 

This article is from Win-Vector LLC: 

In this article we will discuss the machine learning method called “decision trees”, moving quickly over the usual “how decision trees work” and spending time on “why decision trees work.” We will write from a computational learning theory perspective, and hope this helps make both decision trees and computational learning theory more comprehensible. The goal of this article is to set up terminology so we can state in one or two sentences why decision trees tend to work well in practice.  .... " 

Articles about Decision Trees

Good list from DSC of articles about decision trees, provided by Vincent Granville.    We found much value in the enterprise of these methods because their output was explainable to decision makers.   In addition we were able to use these results directly plugged into rule based expert systems to implement AI.  I still believe there is value in such rule bases to implement knowledge in simple logic directly.  Such systems still exist, for example, Visirule, recently updated, which we examined as early as 2009.

Machine Learning and Hidden Decision Trees

Had posted about this once before.    We used decision trees effectively in the enterprise, and they can be useful due to their transparency.   In particular because it has a built in relationship to process.   Use it to work with decision makers to take next steps with a process improvement idea.  In DSC, revisiting hidden decision trees.     Also consider how you can use basic excel to do machine learning.

Case Based Reasoning Driving Chatbots

I was reminded about the possibility of using, even powering chatbots using case based reasoning (CBR) methods and intelligence architecture.  We had looked at this for some simple cases by constructing decision trees and building on those structures.  This is also of value because CBR data is easier to maintain and update with new learning.    Earlier post on this topic.  Anyone done something related?

Price Optimization Using Decision Trees

 Interesting example, a technique I also experimented with for Retail.  Good because it relates to specific business process, and be can tested in that context.  Easy to understand the idea of a decision tree.  And building or using the model is also training in the business process.  But the methods involved are not new data science, and have been  around for decades.

Price Optimisation Using Decision Tree (Regression Tree) - Machine Learning  by Bernard Antwi Adabankah  

The research was conducted to find out what price  maximises profit without sacrificing the high demand for the product due to the price being too high nor sacrificing the margins on the product due to the price being too low. 

The goal is to experiment with different price levels for the same product in one market place and country to see how sales volumes change with prices and which volume level of products we can be sold for that optimal price range.  ... "