Data Catalogs vs Discovery

Depends... Internally controlled vs externally, other ...   Metadata also a key component. Governance too.

Data Catalogs Are Dead; Long Live Data Discovery  in Towardsdatascience

Why we need to rethink our approach to metadata management and data governance

By Barr Moses... 

Pharma and the Sharing of Data in Drug Discovery

Not quite what I call crowdsourcing, but rather a  means to make it easier to share and fund expensive research efforts.  Ultimately the very expensive genomic research efforts have to be funded too.   Good to see this direction being explored.

Big Pharma Buys into Crowdsourcing for Drug Discovery
By Menaka Wilhelm  in Wired

" .... Part of the problem is simply that drug design is hard. But many researchers point to the systems of paywalls and patents that lock up data, slowing the flow of information. So a nonprofit called the Structural Genomics Consortium is countering with a strategy of extreme openness. They’re partnering with nine pharmaceutical companies and labs at six universities, including Oxford, the University of Toronto, and UNC Chapel Hill. They’re pledging to share everything with each other—drug wish lists, results in open access journals, and experimental samples—hoping to speed up the long, expensive drug design process for tough diseases like Huntington’s. .... "

Intelligence Analysis

New Book of Of interest.  How are these kinds of structure embedded into AI?

Intelligence Analysis as Discovery of Evidence, Hypotheses, and Arguments: Connecting the Dots
by Gheorghe Tecuci, David A. Schum, Dorin Marcu, Mihai Boicu 

This unique book on intelligence analysis covers several vital but often overlooked topics. It teaches the evidential and inferential issues involved in "connecting the dots" to draw defensible and persuasive conclusions from masses of evidence: from observations we make, or questions we ask, we generate alternative hypotheses as explanations or answers; we make use of our hypotheses to generate new lines of inquiry and discover new evidence; and we test the hypotheses with the discovered evidence. 

To facilitate understanding of these issues and enable the performance of complex analyses, the book introduces an intelligent analytical tool, called Disciple-CD. Readers will practice with Disciple-CD and learn how to formulate hypotheses; develop arguments that reduce complex hypotheses to simpler ones; collect evidence to evaluate the simplest hypotheses; and assess the relevance and the believability of evidence, which combine in complex ways to determine its inferential force and the probabilities of the hypotheses. .... " 

Discovery in Big Data

Had not looked at Platfora ... Here integrated with Spark.   I see discovery as different than typical machine learning.  Not too dissimilar from data visualization, where the human visual system does the pre discovery, and potential linking to useful business decision results.  it is worth making the distinction.

Boosting Scientific Discovery CSI Talk

Dear Professors, Students, IBMers and others, 

Just a reminder about our Cognitive Systems Institute Group Speaker Series on Thursday April 28, 2016 at 10:30 am ET US (7:30 am PT US).  Our presenter this week is Xiaolin (Andy) Li,  from the University of Florida, who will present “CognitiveEngine: Boosting Scientific Discovery.”  

Please point your web browser to https://apps.na.collabserv.com/meetings/join?id=2894-8491password=cognitive.   

Use audio on computer or 855-233-7153 in the US (other countries numberhere) PIN Code: 43179788    Non-IBMers and Non-Members, please use the "guest" option instead of entering your email.   

Please find the schedule of presenters herefor the next several calls and please sign up for making a presentation by sending a note to me (fodell@us.ibm.com).   A link to slides and a recording of each call should be available on the CSIG website (http://cognitive-science.info/community/weekly-update/).   

We encourage those who join the calls to add questions and comments to the https://www.linkedin.com/groups/Cognitive-Systems-Institute-6729452 on LinkedIn and please ask questions at the end of the call.

Thank you and please let me know if you would like to be a presenter on an upcoming call!  
Dianne Fodell

Model Discovery, Process Mining with Fluxicon

This software, Fluxicon Disco, was new to me.  Described in KDNuggets.   I recall doing something similar in R, but this appears to cover the whole realm of process flow modeling.    Discovers a model from data, as opposed to BPM,  which has you create the model.

  " ... Process mining can analyze your process in a bottom-up fashion. You don't need to have a model of your process to analyze it — Process mining uses the history data in your IT systems.

Your IT system already records all steps of your process in execution. With process mining, you get a process model from these data. This way, your real process, and actual business rules, can be discovered automatically. ... "

Discovery Analytics

Bill Franks of Teradata on Discovery analytics.  We used to call it  exploratory analytics, and every project started with some of that.  Which is why data visualization was so useful, even before we had so many tool options.   Good thoughts, he writes:  "  ...  I spend a lot of time these days talking with companies about the need for a formal approach to enabling what is often called “discovery analytics” or “exploratory analytics.” What I find is that many people have a fundamental misunderstanding of what discvery analytics is all about. There is one analogy that I have found to be effective in getting people to better understand the concept. In this blog, I’ll walk you through that analogy.  ... "      I agree.   Worth reading his complete argument.

How does this differ from the Watson Discovery Advisor, which I have written about before?  See the Watson Discovery Advisor tag.   The Discovery Advisor is more knowledge management than analytics oriented.  But it would make sense to combine the two for many applications.

Automated Hypothesis Generation

Took a brief look at this at the request of a client, it is still in evolution.    The Watson Chef is an interesting example, but it is overwhelmed today by dozens of recipe search engines that are 'good enough'.   More generally hypothesis generation is a very valuable direction.

In the Economist:  A new type of software helps researchers decide what they should be looking for

" .... Devising new recipes sounds a trivial use for a multimillion-dollar piece of kit. But Dr Gordon’s culinary experiment neatly demonstrates the idea of automated hypothesis generation—and the possible uses of that are certainly not trivial. More than 90 groups of scientists are now developing hypothesis-generation software. They hope to use it not on recipe books but on the vast corpus of scientific literature (by one tally at least 50m scientific papers) that has piled up in public databases. ... 

It all, then, looks rather promising, both for science and for IBM—which launched a commercial version of its automated-hypothesis-generation software in August. Dr Gordon hopes Discovery Advisor, as this service is known, will be a money-spinner for the firm. If that does happen, it will probably be because it has also proved an ideas-spinner for science. ... "

A Discovery Advisor

A press release of nearly a year ago that I reviewed for a meeting with a client in the last few weeks. What does it mean for a company to do discovery on a combination of its own knowledge and data and public information?  This press release inspired a number of people to ask questions like:   How do we test this process?  Re-evaluating that in the new year.

" ... IBM is unveiling IBM Watson Discovery Advisor. The new service will help industries such as pharmaceutical and publishing do more with their R&D investments while empowering research teams to accomplish more in far less time. The Discovery Advisor is designed to reduce the time researchers need to formulate conclusions that can advance their work, from months to days and days to just hours. 

The Discovery Advisor will call upon Watson’s cognitive intelligence to save researchers the time needed to pore though millions of articles, journals and studies. After quickly reading through, determining context and synthesizing vast amounts of data, it will help users pinpoint connections within the data that can strengthen and accelerate their work.  .. " 

Signals Group Product Intelligence

Just brought to my attention:  Signals Group

" ... Powerful product intelligence driven by data, geared towards action. We saw a need to reduce the risk and increase probability for success in building and launching new products. We believed that faster time-to-insight and evidence-based decisions are not just buzzwords.

As natural problem solvers, intelligence officers and geeks with a passion for data, we believed that the right combination of technology, data, and subject matter expertise could overcome this challenge.... 

We created a single platform capable of managing different types of data, performing analyses, and linking those insights to real new product development decisions.

We enable corporations and their product teams to build better, more successful products by transforming multiple external big data sources into actionable intelligence findings. ....  "  

Watson Discovery Advisor Updated

Continued interest in how IBM Watson will be applied to real business problems.  There will be a demonstration today in NYC that we are following.   Quite an interesting claim to be able  find interesting patterns without direct queries.   More to follow.

" Watson Discovery Advisor has been upgraded to seek out hidden relations in data ... The service will provide you leads "when you don't know the question to ask, and for when you want to uncover and discover in the data new insights and patterns," said Steve Gold, IBM vice president for the Watson platform.

Many fields of expertise could benefit from the service, particularly those that collect large amounts of data that require analysis, such as law, medicine and finance, he said.

"It turns out there is a huge appetite in industry for this type of capability," Gold said. "Many of these industries, through traditional approaches, have found it time consuming and tiring to get through the information and find new insights." ... " 

More background here.

15 sec video clip showing chemical structure discovery w/Watson Discovery Advisor. 

Updated:  Specific examples of application and more background.   

Business Intelligence to Address Data Discovery

In CWorld: Divining Data.

Just a few months ago I would not have directly connected Business Intelligence (BI) with what is usually called Data Discovery.  The later is usually connected to methods that find patterns in data using advanced analytical methods.  Also known more technically as Data Mining, finding the nuggets in vast streams of data being generated by companies today to leverage for value.  It is true that 'discovery' is a process that should be first done visually by people who are familiar with the data itself, and its context.  So in that sense Business Intelligence is Discovery as well and information delivery.  It is just not that in BI's usual implementation.

Discovery Driven Prototyping

In Computing Now:     An area we examined.  Here an abstract of the paper at the link:

" ... A new prototyping technique uses discovery-driven prototypes to let users control what they can and would like to do with new technologies. The authors describe this technique and examine its effectiveness, reviewing a series of in situ user studies.

It has always been challenging for designers and developers to readily pinpoint potentially viable opportunities for emerging technologies in people's daily lives. In ubiquitous computing in particular, the uncontrollable dynamics of a user's context makes this issue even more prominent. In this article, the authors assume they can't hypothesize what users of new technologies will desire. To address this challenge, they developed discovery-driven prototyping, which lets users control what they will do with new technologies. Here, the authors describe DDP and its effectiveness through a series of in- situ user studies. .. " 

Will Personalization Kill Discovery?

In Marketing Ying and Yang:  Will personalization kill discovery, choice and diversity of thought? blogs Yvette London.  She makes some useful points.  I worried about this in the early days of matching goods and  but have found that the personalization was never precise enough to remove potential expansion for creativity.

Data Mining Conference

Via Cacm: A data mining conference whose outcomes will be interesting.  Have followed and attended the KDD for years. Will seek to report information that emerges: " ... This week, Chicago hosts the 19th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. More than 1,200 researchers and practitioners across academia and industry are converging to discuss, debate, and advance the field of knowledge discovery from data. A blend of theoretical and algorithmic advances alongside very novel applications and in-the-trenches experiences on applying data mining to solve real world problems showcase the diversity of this vibrant community. .... " 

Telling a Story with Data

Quite useful view of this interaction.  Data analysis is ultimately about story telling, creating drama and then leading to decision value.  Plus modifying the story to the needs of a particular group.    I like the notion of  'dynamic persuasion', but also 'augmented discovery'  here.   " ... An excellent visualization, according to Edward Tufte, expresses "complex ideas communicated with clarity, precision and efficiency." I would add that an excellent visualization also tells a story through the graphical depiction of statistical information. As I discussed in an earlier post, visualization in its educational or confirmational role is really a dynamic form of persuasion. Few forms of communication are as persuasive as a compelling narrative. To this end, the visualization needs to tell a story to the audience. Storytelling helps the viewer gain insight from the data. (For a great example, how much do you think steroids have influenced baseball?) .... " 

Global Intelligence and Advanced Data Discovery

Manny Aparicio of Saffron Technology sends along this note and links to supporting video:

"At this year’s Gartner BI Summit in Los Angeles, Rita Sallam identified a new era of Hybrid (Big) Data Analytics. Hybrid data comes from structured sources like enterprise applications and unstructured sources (mainly text), like news feeds, social media, and documents. Next generation BI tools unify this data so it can be used for advanced data discovery, pattern detection and decision-making. What was previously only possible in Hollywood films is now business reality..... "

Previously about Saffron Technology.

Data Discovery

Data discovery is not a term I have heard much recently.  We experimented with the topic as part of a broader artificial intelligence program.  It linked well with what was also called 'case based reasoning' and 'associative memories'.  Saving the experiences and interactions of the company in data form. For examination and re-use later.   Bottom line:  How do we get useful patterns from data?  Now we have lots more usable data, from increasing sources.  So it makes sense to revisit the topic.  Good non technical introductory view here from the HBR.  Emphasizing how it can be integrated into the organzation.