Cross Industry Standard Process for Data Mining

CRISP-DM ( Cross Industry Standard Process for Data Mining) which we plugged into as far back as 07, seems to have disappeared, along with its web site.  Data mining also appears to have been folded into the study and methods of data science.  In particular the decision tree aspects.  That is too bad, because the decision oriented pieces were particularly understandable to executives.    We found the tree methods indispensable.  A WP article still covers the basics . The six fundamental aspects of the process are:

  1.) Business Understanding
  2.) Data Understanding
  3.) Data Preparation
  4.) Modeling
  5.) Evaluation
  6.) Deployment

With lots of details included in each segment. Each element makes excellent points about what is to be done, what resources are needed, who is responsible and where the results go.    I see that SPSS Modeler (formerly Clementine) still embraces the concept.   Does anyone still teach this?

Insight to Action Seminar: Linking Predictive and Prescriptive Analytics

Yesterday saw a very good DSC seminar by Mikhail Lakirovich of IBM.  This presented how IBM is integrating machine learning methods and decision optimization.  They demonstrated SPSS Modeler, which we used widely when it used to be called Clementine,   A good, very visual way to think about predictive/prescriptive analytics.  That also creates flow documentation of approaches used.  And integrates intelligence about mapping solutions to business problems.  The number of analytical templates available in the system is impressive. A mostly non technical seminar, suitable for the details-oriented analytical executive.

" ... DSC Webinar Series: From Insight to Action – Predictive and Prescriptive Analytics from IBM
Predictive analytics has become an imperative for organizations as they strive to incorporate data-driven decision making into their processes by understanding potential future outcomes. Prescriptive analytics can then build on this foundation by answering the eternal question "What should we do about this?"  Recording of the seminar.

In today’s webinar you will learn how IBM SPSS Predictive Analytics can help you uncover patterns and trends in your data and how IBM Decision Optimization can be leveraged to incorporate those insights into optimal decisions and how IBM analytics lets your understand your data with ease and speed and translate that understanding into value.

Speaker: Mikhail Lakirovich of IBM
Hosted by: Tim Matteson, Co-Founder DSC

SPSS as an Enhancement to R

It was just pointed out to me that SPSS can be linked to R.  Had known this but not the details. Worth a closer look.  Upcoming Webinar on May 26 on the topic.   Points out a number of limitations of R and why its useful to connect it to a Statistical platform.

ExperienceOne Customer Engagement Solution

Just brought to my attention:

IBM just released ExperienceOne - an analytics and automation-based customer engagement solution that ties together marketing, sales and service.   Site. 

Have taken only a cursory look so far but the integration of social analytics and SPSS Modeler methods is interesting.  We used SPSS Modeler integrated methods for internal decision methods.

Alternative Machine Learning Environments

I am starting to examine Machine Learning Platforms for an application.  This post is mostly for my own and a few collaborators use.  Not complete or just open source.  Do not claim to be expert in any of these methods, but have encountered them in a number of very differing machine learning contexts. In some cases used for building a system from the ground up, in other cases reviewing or extending existing modeling efforts.  R is not included here, it is more of a programming language than a learning platform.  All these are worth examining.  Open to alternative ideas.

IBM SPSS Modeler (Which I formerly used as Clementine):  http://www-01.ibm.com/software/analytics/spss/products/modeler/

Weka:  http://www.cs.waikato.ac.nz/ml/weka/

Also Orange Data Mining   http://orange.biolab.si/

And BigML Data Mining:  https://bigml.com/how_it_works

Techniques in Sentiment Analysis

In CACM:   Excellent overview piece on the subject.  The abstract itself has interest: " ... Sentiment analysis (or opinion mining) is defined as the task of finding the opinions of authors about specific entities. The decision-making process of people is affected by the opinions formed by thought leaders and ordinary people. When a person wants to buy a product online he or she will typically start by searching for reviews and opinions written by other people on the various offerings. Sentiment analysis is one of the hottest research areas in computer science. Over 7,000 articles have been written on the topic. 

    Hundreds of startups are developing sentiment analysis solutions and major statistical packages such as SAS and SPSS include dedicated sentiment analysis modules. There is a huge explosion today of 'sentiments' available from social media including Twitter, Facebook, message boards, blogs, and user forums. These snippets of text are a gold mine for companies and individuals that want to monitor their reputation and get timely feedback about their products and actions. Sentiment analysis offers these organizations the ability to monitor the different social media sites in real time and act accordingly. Marketing managers, PR firms, campaign managers, politicians, and even equity investors and online shoppers are the direct beneficiaries of sentiment analysis technology .... " 

Business Analytics, Past and Future

I have been a long time practitioner of business analytics, the use of quantitative methods to improve business processes. Starting at the Defense Department, where we built military simulation models, and ending up at P&G.  My first project there, when I arrived in 1977, was to improve warehouse efficiency using IBM's mathematical programming system: MPSX.  That followed with using the same mathematical optimization approaches for scheduling, supply chain siting and executive decision making.

So I was happy to see in the July/August Issue of Analytics Magazine an article by Arnold Greenland on the history and current state of business analytics at IBM.    This allows me to reflect on the growth and impact of analytical methods at IBM and elsewhere.  Since then the original MPSX package has disappeared, and has been replaced by the acquired Ilog/Cplex, which goes far beyond the original package, adding nonlinear methods as well.

Shortly after being introduced to MPSX, in 1980, we addressed the analysis of unstructured data, typically consumer comments in unstructured text, using recently developed methods called  'Content Analysis'.  These permitted the semantic analysis of multiple human languages.  An early attempt to look at and understand unstructured 'Big Data'.

During all of this time we also used a number of statistical methods throughout the enterprise to explore and improve systems.  SPSS and SAS were in frequent use.  A package called Clementine allowed us to use advanced logical methods, like artificial neural nets, to implement what were essentially statistical methods, to store and implement specific decision rules.   These methods could then be inserted in both software and hardware processes.  Clementine also permitted the structural exploration of the decision process.  Ultimately Clementine was acquired by SPSS, and were eventually absorbed into Modeler.

During the 1990's, a heady time for artificial intelligence, we implemented expert systems using a now defunct language called M1.  Which were successful for a number of complex industrial management processes.  Some of the same capabilities can be seen in JRules.  That work has been extended into recent AI explorations like that of IBM's Watson.

What is further interesting now is that many of these methods are now available to the small and medium sized business.  We live in a time where the tools are available,  just go out and use them.

Process: Making Analytics Work for Every Company

When I worked with analytical methods in the big enterprise, it was often the case that work had already been done to map out the business processes involved.   In the mid 1980s we were already using methods like expert systems to improve these processes. Some of the research there came from academic work in artificial intelligence. At the time IBM was also starting to test the idea of 'expert systems' to model and improve internal processes. This often included the requirement to integrate systems like optimization, statistical modeling and correct and complete database designs and contents to make sure the decisions were represented correctly.

Frankly it would have been difficult for the small to medium size company to do the same thing at that time. In my consulting experience, I have discovered that their methods are not mapped as completely, and the methods needed  have historically been too difficult and expensive to use.

Since then I have followed methods like IBM's Watson Project, which uses many of the same expertise capturing capabilities.  And SPSS, which embodies statistical and logical data mining methods we used within the former Clementine package. These methods help determine where the key parts of the business process reside. So many of the methods that we had to cobble together are there today for ready use. It is still true that expertise is required to connect these ideas together. And specialized expertise and techniques need to be leveraged to scope an accurate model of the business process. This day is here.

The pieces are ready for any size enterprise. Be glad to discuss in additional detail.

This post was written as part of the IBM for Midsize Business program, which provides midsize businesses with the tools, expertise and solutions they need to become engines  of a smarter planet. More on that here: goo.gl/VQ40Cg 

Predictive Analytics with IBM SPSS

After attending the predictive analytics seminar last week I have been doing some examination of the current elements of the SPSS product suite.  I found this particular useful recent writeup by the Ironside Group.    This relates well to our use in the enterprise of the 'Clementine'  package, the predecessor of the Modeler capabilities.  Easy to use and visually powerful for selling a solution:

" ... IBM Business Analytics software is designed to deliver complete, insightful, and accurate information for decision makers to make more effective decisions. As part of this portfolio, IBM SPSS Predictive Analytics software specializes in predictive analytics which organizations can rely on to predict future outcomes based on patterns found in historical data. Organizations in almost all industries of every size can benefit from the IBM SPSS technology to provide clear and actionable insight into current performance as well as use the information to drive better outcomes for future business. While Business Intelligence can help to monitor the current and past performance of the business operation, Predictive Analytics can answer questions on why the business is performing in a certain manner and how it can be improved in the future .... "

Predictive Perspectives Seminar

I attended the IBM/SPSS seminar today.   Jason Verlen,  Director of IBM/SPSS gave the keynote address that well positioned the use of predictive techniques in business based on the SPSS suite of tools. This links well with IBM's Smarter Planet effort.

I have previously written about our own use of the Clementine tool set,  from which the SPSS Modeler tools arose.   We used those tools for supply chain, Marketing,  R&D and HR applications in the enterprise.  The tools were useful because they not only gathered and ran models, but also graphically documented the models themselves.  This allows the models to be both more easily re-used, and effectively explained to management.  I was glad to see that a number of new capabilities have been added to the Clementine suite that make it easier to use and leverage.  It is well worth taking a close look at it with you own data. 

Verlen made the case that we live in an increasingly instrumented and interconnected world that begs for intelligent, data-based solutions.  Too often we do not have the insight, have inefficient access to data, and have the inability to predict outcomes.  Every business has big opportunities to apply predictive  and often real-time analytics today.  Our own analytics team did this for many years, and could have well used these tools to make the job easier and more effective. 

Overall the seminar was very well done and enjoyed.    Here is an overview they provided.

I also liked their white paper: Seven Reasons Why You Need Predictive Analytics Today, a comprehensive, business oriented paper.  Requires registration.

Predictive Perspectives with SPSS

Free, Cincinnati local, half day IBM SPSS seminar on predictive analytics.

More here, and registration link.

SEATING IS LIMITED, REGISTER NOW!

Predictive Perspectives Cincinnati September 13, 2011 8:00 a.m. – 1:30 p.m.   Cincinnati Marriott at RiverCenter 10 West RiverCenter Boulevard Covington