On Complex Network Analysis

Nicely done overview.  And its not only about Python applications, includes an intro look at accessible and free systems like Gephi.   Many network analysis concepts are described, gives you a view of the ultimate power of the idea.   It automatically visualizes the problem, should always be thought of  Right now looking at how knowledge graphs can be used to prepopulate assistant style systems with context.

Complex Network Analysis in Python: Recognize - Construct - Visualize - Analyze - Interpret 1st Edition   by Dmitry Zinoviev

They write: 

" ... Construct, analyze, and visualize networks with networkx, a Python language module. Network analysis is a powerful tool you can apply to a multitude of datasets and situations. Discover how to work with all kinds of networks, including social, product, temporal, spatial, and semantic networks. Convert almost any real-world data into a complex network--such as recommendations on co-using cosmetic products, muddy hedge fund connections, and online friendships. Analyze and visualize the network, and make business decisions based on your analysis. If you're a curious Python programmer, a data scientist, or a CNA specialist interested in mechanizing mundane tasks, you'll increase your productivity exponentially.  ... " 

Free O'Reilly Book on Graph Databases

This approach to do analytics is too little used. We always called it Graph Analytics.   This free ebook, sponsored by Neo4j, is a good introduction.  Polinode is known for its ease of integration for  HR survey data.  There are also free, open source software for this kind of analysis, see my text tag links for examples.

GePhi Updates to version 0.9

I see that the Gephi Graph Analytics system has been updated to version 0.9  ... Creeping ever so slowly up to version 1.0   Used it for several structural analyses of supply chain designs. The popular, open source, free package is worth exploring.   Worth updating as older versions had issues with Java.  " .... Gephi is an interactive visualization and exploration platform for all kinds of networks and complex systems, dynamic and hierarchical graphs. ... Runs on Windows, Mac OS X and Linux. Gephi is open-source and free. ... " 

Graph Query Language is Open Source

Neo4j:  Brought to my attention.  Have announced open source.  This is known to be used by Wal-Mart and Ebay.   Case studies at their site.   " ... An Intuitive Approach to Data Problems ... Graphs are the most efficient and natural way of working with data. ... They are deeply intuitive, and mimic the interconnectedness of concepts and ideas in the human mind.  ... " .   I see they have an apparently free community edition, and a number of other resources at their site.     See also their blog.

An Example of Network Graph Analysis

From DSC, Visualize your Social Media Analytics  by Manoj Sharma

" ... Network Graph Analysis can be used to identify the most important participants based on their degree of interaction with other social media users for the topic of interest. Centrality in a graph can be used to identify the most vertices in a graph which translates into the most influential person in a social network.  .... 

For example, in the Twitter network graph (built using Python NetworkX and d3.js) shown here for @Walmart, we see a strong tweet connection with Huggies. Network graphs like the one above can be used to study interactions in social media which may lead to insights that can be useful in the real world. ... "   (click below for image enlargement)

Linkurious: New Tool for Graph Data Visualization

An interesting new tool I had not heard of, it competes with tools like Gephi (Open source, free) and Polinode,  which I have used.  Have not had experience with this one as yet.

Linkurious unveils enterprise graph data visualization tool
Data visualizations represent one of the easiest ways to explore business data, revealing all kinds of hidden nuggets that businesses can derive actionable insights from. Now, data visualization outfit Linkurious has come up with a way to help businesses to ‘see’ their graph-based data more easily with the launch of its enterprise platform, Linkurious Enterprise, designed to search, explore and visualize connections in graph data.  ... "

Update: Polinode Visualizes Organizations

Continue to follow Polinode as a means to model and analyze organizations.  The Australian company has added some new features that are of interest.  This is a survey based system that allows you to construct a visual graph model of your organization.  Or model an organization you might be merging with. Lots of possibilities.  Future versions will allow means to merge the surveys with other meta organization data. Also output the data to other graphics engines like Gephi.  Apply for a free beta trial at the link.

Polinode: Leveraging Visual Enterprise Networks

Polinode was brought to my attention because we also looked at using visualized networks to understand how our company worked, and could be simulated in its adaptation to changes over time and context, both slow and rapid.   A kind of dynamic, simulation of  corporate people architecture.   The thought was to further add the interaction of people with their data.  We also used highly visual network representations.  Polinode writes:

" Polinode is a powerful and easy-to-use platform for mapping, visualizing and analyzing relationships across organizations  .... We help organizations become more Agile and Innovative ... 
Polinode is a flexible tool that helps cut through complexity. At its core is the ability to map, visualize and analyze relationships. Applications range from identifying change agents and finding critical links through to promoting diversity and improving workplace layouts....plus a lot more.... " 

Check out Polinode here, you can apply for trying a closed Beta.
You can also see a video demo here.   I am looking further.

Large Scale Graphics Visualization

Just noticed this new system.  I have been working with GePhi:

An interview with Leo Meyerovich:

Who is behind Graphistry?
Graphistry spun out of UC Berkeley’s Parallel Computing lab last year. It stems from my Ph.D. on the first parallel web browser (Mozilla etc. are building new browsers around those ideas) and from Matt Torok (my RA), who built Superconductor, a GPU scripting language for big interactive data visualizations. 

What does Graphistry do?
Graphistry scales and streamlines visual analysis of big graphs.  Think answering questions about people (intelligence, sales, marketing), about things (data centers, sensors), and combinations of them (e.g., financial transactions). For example, we used it to crack a 70K+ node botnet a couple days ago. Our tool immediately revealed the accounts involved, their different roles, especially key accounts, and, after 30min of interactive analysis & googling, the credit card & passport theft operation it funneled to. Most tools can only sensibly show hundreds of nodes,  and a couple open source ones handle tens of thousands, but we’re already pushing 100X more than that. ... " 

Graph Analytics to Understand Bigger Data

A way to visually and systematically examine how your Big Data is interconnected.    Here recently in Infoworld,   We used this for a number of projects in the enterprise.  The software is catching up with the volume and complexity of the data.   Again, I point to the free open source system Gephi, a good place to start, see the text link below to much more.

Sometimes a key aspect of what is called exploratory data analysis. 
Its not just about the data, its about how it interacts with other data and its context. The network becomes a form of anaytical metadata.

" ... How graph analytics deliver deeper understanding .. Graph analysis will make big data even bigger

What's the fastest growing use case for big data analytics?

By mapping relationships among high volumes of highly connected data, graph analytics unlocks more insightful questions and produces more accurate outcomes ... 

As the sources, types, and amounts of data continue to expand, so will the need for different kinds of analytics to make something of that data. Unfortunately, there is not a one-size-fits-all approach to analytics -- no magic pill that will get your organization the insight it needs to stay competitive. Graph analytics has emerged as the new hot topic, but to what end? What is the impact of graph analytics technology on organizations seeking to discover the cause, effect, and influence of events on business outcomes?  ... " 

Graph Databases and Modeling the Internet of Things

Have had a number of conversations on the use of graphical databases lately.  It is a good technology to understand.    It works well with anything that is interconnected.  As are people and any communicating things.  Obviously:  Its exploratory data analysis and resulting discovery.   In SiliconAngle:

" .... As a result, the Internet of Things should perhaps instead be called the “Internet of Connected Things.” This emphasizes the network itself, and the many interaction points between individual devices, people, apps, and locations; and how they can and should (or should not) interact. Understanding and managing these connections will be at least as important for businesses as understanding and managing the devices themselves.

Imagination is key to unlocking the value of connected things. For example, in a telecommunications or aviation network, the questions, “What cell tower is experiencing problems?” and “Which plane will arrive late?” can be answered much more accurately by understanding how the individual components are connected and impact one another.

Understanding connections is also key to understanding dependencies and uncovering cascading impacts. Such insight allows businesses to identify opportunities for new services and products that make the most of the IoT. To identify these opportunities, businesses need tools that can show these connections quickly and easily. ... " 

Graph Analytics and Gephi

In Forbes, an article on Graph Analytics.    Good overview,  it will become an important Bigdata concept, but no mention of Gephi,  an Open Graph Viz Platform.   Its free for experimentation.  It is not the same thing that Forbes describes, but is an excellent starting point.  A means to produce and visualize very complex, data based graph structures. Have used it for supply chain and physical system problems and mentioned it a number of times in this blog.

(Brought to my attention by @KirkDBorne)

Revealing Relationships with Graphical Discovery: Teradata Aster

Graphical analysis has been covered here a number of times.  In particular using the GePhi package.  We see that this is written about in the Teradata blog, using their Aster package.  Had not looked at that approach, but will explore now.  

" ... Teradata® Aster Discovery Platform 6 is sparking a sharp interest in graph discovery and the unique value it can deliver to businesses. Graph reporting, a precursor to graph discovery, that helps find basic connections such as an individual’s friends in a social network, has been available for years but has limited analytic value.

The graph analytics capabilities enabled by Teradata Aster Discovery Platform 6 are different. The discovery platform uses algorithms that process an entire graph to extract deep insights, such as identifying the biggest influencers in a social network, and provide answers to complicated analytics problems in a timely manner—something graph reporting databases can’t do well. ... " 

Topology Mapping for Analytics

Seems Vincent Granville has been thinking similarly lately.  See his article and code in Data Science Central:   A little known component that should be part of most data science algorithms. Beware, this has a general introduction, and then gets technical with R code. And goes further:

" ...  This is a component often missing, yet valuable for most systems, algorithms and architectures that are dealing with online or mobile data, known as digital data: be it transaction scoring, fraud detection, online marketing, marketing mix and advertising optimization, online search, plagiarism and spam detection, etc. .... I will call it an Internet Topology Mapping. It might not be stored as a traditional database (it could be a graph database, a file system, or a set of look-up tables). It must be pre-built (e.g. as look-up tables, with regular updates) to be efficiently used. ... "

I came to think of this when I experimented with the free, open source GePhi network management system and reported on it here. Though the most common network we might work topologically is the Internet, or a social network. These are not the only networks, things like influence maps come to mind.     Have not thought how R in particular links with GePhi, but exploring.  I have a real application in mind. Any thoughts out there?

Mathematical Shape of Things to Come

New challenges on how to deal with bigger, more complex, dynamic and disorganized data.  Some great nontechnical examples here that can give you an appreciation  of the problems being addressed. Will the mathematics of topology point the way?  See   my recent posts on GePhi as one way to explore these problems. Visualization is always the place to start doing analytics.

" ... Scientific data sets are becoming more dynamic, requiring new mathematical techniques on par with the invention of calculus.... 

In Quanta:

 ... DeDeo is not the only researcher grappling with these challenges. Across every discipline, data sets are getting bigger and more complex, whether one is dealing with medical records, genomic sequencing, neural networks in the brain, astrophysics, historical archives or social networks. Alessandro Vespignani, a physicist at Northeastern University who specializes in harnessing the power of social networking to model disease outbreaks, stock market behavior, collective social dynamics, and election outcomes, has collected many terabytes of data from social networks such as Twitter, nearly all of it raw and unstructured. “We didn’t define the conditions of the experiments, so we don’t know what we are capturing,” he said. ... " 

GePhi Open Graphics Blog

Started to follow the GePhi open graphics blog.  Its all about networks, big and small.    Free and open research software.   Examining now.  " ... Networks are everywhere: email systems, financial transaction systems and gene-protein interaction networks are just a few examples. Gephi began as a university student project four years ago and has quickly become an open source software leader in the visualization and analysis of large networks. It is an important contribution to the ecosystem of tools used by researchers and big data analysts to explore and extract value from the deluge of relational data and disseminate a better understanding for people to think about a “connected” world.  ... " 

GEPhi Graph Visualization for Exploratory Data Analysis

This method was brought to my attention my a colleague.  We had looked at it some time ago and it has come back as a possible exploratory solution.

The Open Graph Viz Platform
" ... Gephi is an interactive visualization and exploration platform for all kinds of networks and complex systems, dynamic and hierarchical graphs .... 

Runs on Windows, Linux and Mac OS X. Gephi is open-source and free.

Gephi is a tool for people that have to explore and understand graphs. Like Photoshop but for data, the user interacts with the representation, manipulate the structures, shapes and colors to reveal hidden properties. The goal is to help data analysts to make hypothesis, intuitively discover patterns, isolate structure singularities or faults during data sourcing. It is a complementary tool to traditional statistics, as visual thinking with interactive interfaces is now recognized to facilitate reasoning. This is a software for Exploratory Data Analysis, a paradigm appeared in the Visual Analytics field of research.... "