A Guide to Data Analysis

Nicely done, non technical, but considerable and basic and step by step guide to data analysis. Gives lots of suggestions to be complete about the process.  From Geckoboard, a company that has a dashboard product.

A Guide to Basic Data Analysis
Trying to find the cause of a problem in your business? Data can help, but sometimes knowing how to explore and interpret it can be intimidating. We've put together this no nonsense data analysis guide to walk you through a simple process so you can confidently use data to find answers and make smart decisions.  .... " 

Curiosity and AI

How do you build curiosity in a system.  And the ability to act on curiosity?  In a simplistic sense, mining for data and then ingesting it?   If you are a robot or drone with sensors your have an element of being able to reach out for it as well.  And measure the value of what you find.  Back to the 'Explore/Exploit' dilemma.  Mentioned by Brian Christian in his recent talk.   Good piece in AAAS:

Scientists imbue robots with curiosity  By Matthew Hutson 

In a twist on artificial intelligence (AI), computer scientists have programmed machines to be curious—to explore their surroundings on their own and learn for the sake of learning. The new approach could allow robots to learn even faster than they can now. Someday they might even surpass human scientists in forming hypotheses and pushing the frontiers of what’s known.

“Developing curiosity is a problem that’s core to intelligence,” says George Konidaris, a computer scientist who runs the Intelligent Robot Lab at Brown University and was not involved in the research. “It’s going to be most useful when you’re not sure what your robot is going to have to do in the future.”  .... " 

Over the years, scientists have worked on algorithms for curiosity, but copying human inquisitiveness has been tricky. For example, most methods aren’t capable of assessing artificial agents’ gaps in knowledge to predict what will be interesting before they see it. (Humans can sometimes judge how interesting a book will be by its cover.)

Todd Hester, a computer scientist currently at Google DeepMind in London hoped to do better. “I was looking for ways to make computers learn more intelligently, and explore as a human would,” he says. “Don’t explore everything, and don’t explore randomly, but try to do something a little smarter.”

Data Exploration Tutorial

A Comprehensive Guide to Data Exploration

Posted by Emmanuelle Rieuf ... 

This article on a complete tutorial on data exploration, was posted by Sunil Ray. Sunil is a Business Analytics and Intelligence professional with deep experience in the Indian Insurance industry.

Introduction
There are no shortcuts for data exploration. If you are in a state of mind, that machine learning can sail you away from every data storm, trust me, it won’t. After some point of time, you’ll realize that you are struggling at improving model’s accuracy. In such situation, data exploration techniques will come to your rescue.   .... " 

Large Scale Graphics Visualization

Just noticed this new system.  I have been working with GePhi:

An interview with Leo Meyerovich:

Who is behind Graphistry?
Graphistry spun out of UC Berkeley’s Parallel Computing lab last year. It stems from my Ph.D. on the first parallel web browser (Mozilla etc. are building new browsers around those ideas) and from Matt Torok (my RA), who built Superconductor, a GPU scripting language for big interactive data visualizations. 

What does Graphistry do?
Graphistry scales and streamlines visual analysis of big graphs.  Think answering questions about people (intelligence, sales, marketing), about things (data centers, sensors), and combinations of them (e.g., financial transactions). For example, we used it to crack a 70K+ node botnet a couple days ago. Our tool immediately revealed the accounts involved, their different roles, especially key accounts, and, after 30min of interactive analysis & googling, the credit card & passport theft operation it funneled to. Most tools can only sensibly show hundreds of nodes,  and a couple open source ones handle tens of thousands, but we’re already pushing 100X more than that. ... " 

Graph Analytics to Understand Bigger Data

A way to visually and systematically examine how your Big Data is interconnected.    Here recently in Infoworld,   We used this for a number of projects in the enterprise.  The software is catching up with the volume and complexity of the data.   Again, I point to the free open source system Gephi, a good place to start, see the text link below to much more.

Sometimes a key aspect of what is called exploratory data analysis. 
Its not just about the data, its about how it interacts with other data and its context. The network becomes a form of anaytical metadata.

" ... How graph analytics deliver deeper understanding .. Graph analysis will make big data even bigger

What's the fastest growing use case for big data analytics?

By mapping relationships among high volumes of highly connected data, graph analytics unlocks more insightful questions and produces more accurate outcomes ... 

As the sources, types, and amounts of data continue to expand, so will the need for different kinds of analytics to make something of that data. Unfortunately, there is not a one-size-fits-all approach to analytics -- no magic pill that will get your organization the insight it needs to stay competitive. Graph analytics has emerged as the new hot topic, but to what end? What is the impact of graph analytics technology on organizations seeking to discover the cause, effect, and influence of events on business outcomes?  ... " 

Graph Databases and Modeling the Internet of Things

Have had a number of conversations on the use of graphical databases lately.  It is a good technology to understand.    It works well with anything that is interconnected.  As are people and any communicating things.  Obviously:  Its exploratory data analysis and resulting discovery.   In SiliconAngle:

" .... As a result, the Internet of Things should perhaps instead be called the “Internet of Connected Things.” This emphasizes the network itself, and the many interaction points between individual devices, people, apps, and locations; and how they can and should (or should not) interact. Understanding and managing these connections will be at least as important for businesses as understanding and managing the devices themselves.

Imagination is key to unlocking the value of connected things. For example, in a telecommunications or aviation network, the questions, “What cell tower is experiencing problems?” and “Which plane will arrive late?” can be answered much more accurately by understanding how the individual components are connected and impact one another.

Understanding connections is also key to understanding dependencies and uncovering cascading impacts. Such insight allows businesses to identify opportunities for new services and products that make the most of the IoT. To identify these opportunities, businesses need tools that can show these connections quickly and easily. ... " 

Graph Analytics and Gephi

In Forbes, an article on Graph Analytics.    Good overview,  it will become an important Bigdata concept, but no mention of Gephi,  an Open Graph Viz Platform.   Its free for experimentation.  It is not the same thing that Forbes describes, but is an excellent starting point.  A means to produce and visualize very complex, data based graph structures. Have used it for supply chain and physical system problems and mentioned it a number of times in this blog.

(Brought to my attention by @KirkDBorne)

Revealing Relationships with Graphical Discovery: Teradata Aster

Graphical analysis has been covered here a number of times.  In particular using the GePhi package.  We see that this is written about in the Teradata blog, using their Aster package.  Had not looked at that approach, but will explore now.  

" ... Teradata® Aster Discovery Platform 6 is sparking a sharp interest in graph discovery and the unique value it can deliver to businesses. Graph reporting, a precursor to graph discovery, that helps find basic connections such as an individual’s friends in a social network, has been available for years but has limited analytic value.

The graph analytics capabilities enabled by Teradata Aster Discovery Platform 6 are different. The discovery platform uses algorithms that process an entire graph to extract deep insights, such as identifying the biggest influencers in a social network, and provide answers to complicated analytics problems in a timely manner—something graph reporting databases can’t do well. ... " 

Topology Mapping for Analytics

Seems Vincent Granville has been thinking similarly lately.  See his article and code in Data Science Central:   A little known component that should be part of most data science algorithms. Beware, this has a general introduction, and then gets technical with R code. And goes further:

" ...  This is a component often missing, yet valuable for most systems, algorithms and architectures that are dealing with online or mobile data, known as digital data: be it transaction scoring, fraud detection, online marketing, marketing mix and advertising optimization, online search, plagiarism and spam detection, etc. .... I will call it an Internet Topology Mapping. It might not be stored as a traditional database (it could be a graph database, a file system, or a set of look-up tables). It must be pre-built (e.g. as look-up tables, with regular updates) to be efficiently used. ... "

I came to think of this when I experimented with the free, open source GePhi network management system and reported on it here. Though the most common network we might work topologically is the Internet, or a social network. These are not the only networks, things like influence maps come to mind.     Have not thought how R in particular links with GePhi, but exploring.  I have a real application in mind. Any thoughts out there?

GEPhi Graph Visualization for Exploratory Data Analysis

This method was brought to my attention my a colleague.  We had looked at it some time ago and it has come back as a possible exploratory solution.

The Open Graph Viz Platform
" ... Gephi is an interactive visualization and exploration platform for all kinds of networks and complex systems, dynamic and hierarchical graphs .... 

Runs on Windows, Linux and Mac OS X. Gephi is open-source and free.

Gephi is a tool for people that have to explore and understand graphs. Like Photoshop but for data, the user interacts with the representation, manipulate the structures, shapes and colors to reveal hidden properties. The goal is to help data analysts to make hypothesis, intuitively discover patterns, isolate structure singularities or faults during data sourcing. It is a complementary tool to traditional statistics, as visual thinking with interactive interfaces is now recognized to facilitate reasoning. This is a software for Exploratory Data Analysis, a paradigm appeared in the Visual Analytics field of research.... "