Hadoop, Spark and Map Reduce

Very good, not too technical , good selection and useful articles from DSC.  Join the group to get resources like this ongoing:

11 Great Hadoop, Spark and Map-Reduce Articles     Posted by Vincent Granville  

This reference is a part of a new series of DSC articles, offering selected tutorials, references/resources, and interesting articles on subjects such as deep learning, machine learning, data science, deep data science, artificial intelligence, Internet of Things, algorithms, and related topics. It is designed for the busy reader who does not have a lot of time digging into long lists of advanced publications.  .... " 

Measuring Relatedness of Knowledge Graphs

Interesting and technical talk on how research labs are using Map-Reduce to examine the closeness of terms in text.   Looks at the efficiency of doing this for very large examples of knowledge bases.   Creates a measure of relatedness of interest for practical applications.

" ... speaker this week is Yosi Mass from IBM Haifa Research Lab, who will present "A scalable graph representation of knowledge bases and its uses for semantic document relatedness."  ... '

Slides here.
Audio will be placed here (Scroll down)

Hadoop and Metadata

Excerpts from recent research about meta data use in the enterprise.  From Hortonworks and more in Infoq:

" ... One of the most attractive qualities of Hadoop is its flexibility to work with semi-structured and unstructured data without schemas.  Unstructured data represents 80% of  the overall data in most organizations and is growing at 10-50x  structured data. Indeed, Hadoop excels at extracting  structured data from unstructured data.  HCatalog helps Hadoop deliver value from the output of its labor, by providing access to mined, structured data by those who would consume it: analysts, systems and applications.

HCatalog is a metadata and table management system for Hadoop. It is based on the metadata layer found in Hive and provides a relational view through a SQL like language to data within Hadoop. HCatalog allows users to share data and metadata across Hive, Pig, and MapReduce. It also allows users to write their applications without being concerned with how or where the data is stored, and insulates users from schema and storage format changes. ... " 

Essential Hadoop Tools

Some essential Hadoop tools.  An introductory slideshow, which is useful for the beginner. And a useful explanatory overview " ... .While many refer to the entire constellation of map and reduce tools as Hadoop, there's still one small pile of code at the center known as Hadoop. The Java-based code synchronizes worker nodes in executing a function on data stored locally. Results from these worker nodes are aggregated and reported. The first step is known as "map"; the second, "reduce."

Hadoop offers a thin abstraction over local data storage and synchronization, allowing programmers to concentrate on writing code for analyzing the data. Hadoop handles the rest. The job is split up and scheduled by Hadoop. Errors or failures are expected, and Hadoop is designed to work around faults by individual machines. ... " 

Hadoop and a Pending Metadata Mess

 An intriguing point about the tool and its use, and general documentation of data, its origins and measurements (metadata) :  " ... Why is Hadoop so popular? There are many reasons. First of all it is not so much a product as an ecosystem, with many components: MapReduce, HBase, HCatalog, Pig, Hive, Sqoop, Mahout and quite a few more. That makes it versatile, and all these components are open source, so most of them improve with each release cycle. .... 

 ... So if someone in the company wants some external data or even internal data captured for later use, Hadoop can just sit there and drink it up. And that’s fine as long as you don’t lose track of what the data in the lake actually is. But this is where the devil crawls into the detail. You can scale Hadoop out so it becomes just one very large data lake and sits there gulping down all the data it can drink. You can also instantiate multiple instances of Hadoop, each devoted to a specific kind of usage, but we do not often hear about IT sites doing that – after all Hadoop scales out to the edge of the solar system, does it not? ... "  

Reducing MapReduce

Vincent Granville on What MapReduce can't do.    " ... We discuss here a large class of big data problems where MapReduce can't be used - not in a straightforward way at least - and we propose a rather simple analytic, statistical solution. ... MapReduce is a technique that splits big data sets into many smaller ones, process each small data set separately (but simultaneously) on different servers or computers, then gather and aggregate the results of all the sub-processes to produce the final answer. Such a distributed architecture allows you to process big data sets 1,000 times faster than traditional (non-distributed) designs, if you use 1,000 severs and split the main process into 1,000 sub-processes..... "