Dealing with Tsunami of Data

Continues.

RisingWave Emerges to Tackle Tsunami of Real-Time Data

Alex Woodie in Datanami

Only the most advanced companies have overcome the technical complexity involved with processing streaming data in real time. One of the vendors aiming to reduce this complexity and make stream data processing available to the masses is RisingWave Labs, which today announced $36 million in financing.

The early days of stream data processing brought us stand-alone systems that were capable of acting upon vast streams of data, and doing so with low latency and reliability. Stream processing frameworks like Apache Storm made headway in addressing these challenges and led the way to more sophisticated frameworks like Apache Flink and others.

Things got significantly more complex when companies realized they needed to know something about the past to take the best action on the newest data, which necessitated the integration of stream processing frameworks with databases or data lakes, where the historical record lived as persisted data. Architectural blueprints, such as the Lambda and Kappa architectures, were proposed to address this unique challenge, but the technical complexity in keeping these dual-path systems running are immense.

Today we’re seeing the emergence of a new category of product–the streaming database–aimed at solving this problem. Instead of running data through a dedicated stream processing framework like Storm or Flink, the backers of streaming databases think that all the data processing–including the business end of a streaming big data pipeline like Kafka, Kinesis, or Pulsar–can be handled by the SQL query engine contained in a relational database.

RisingWave is a Postgres-compatible database developed to process data streams in the cloud (Image courtesy RisingWave Labs)

That’s the approach taken with RisingWave, a new open source streaming database that emerged just over a year ago. Yingjun Wu, a former AWS and IBM engineer, created RisingWave as a cloud-native database with the goal of providing the bernefits of stream processing without the technical complexity inherent with stream processing frameworks.

“Existing open-source systems are very costly to deploy, maintain, and use in the modern cloud environment,” Wu, who is the CEO of RisingWave Labs, says in a press release today. “Our goal is not to build yet another streaming system that is 10X faster than existing systems, but to deliver a simple and cost-effective system that allows everyone to benefit from stream processing.”

Developed in Rust, RisingWave is a Postgres-compatible database can do many of the things that stream processing frameworks do, but within the context and control of a familiar relational database running in the cloud and the SQL language, according to Wu, who has a PhD from National University of Singapore and was also a visiting PhD at Carnegie Mellon University.

“[RisingWave] consumes streaming data, performs continuous queries, and maintains results dynamically in the form of a materialized view,” Wu says in a blog post earlier this year. “Processing data streams inside a database is quite different from that inside a stream computation engine: streaming data are instantly ingested into data tables; queries over streaming and historical data are simply modeled as table joins; query results are directly maintained and updated inside the database, without pushing into a downstream system.”

The open source project, which available on GitHub via an Apache 2.0 license, is being adopted by organizations for a range of uses, including real-time analytics and alerting; IoT device tracking; monitoring user activity; and online application data serving. The company, which changed its name from Singularity Data three weeks ago, recently unveiled the beta of a hosted commercial version of RisingWave; it’s slated to become generally available next year.  ... ' 

Cloud Based Reactions to Natural Disasters

Basic idea is good,  support the system with useful and readily available knowledge to support operations.  Probably harder though to actually make decisions based on quality of data available.

Cloud-Based Electronic System May Help First Responders Better React to Natural Disasters
Purdue University News
By Chris Adam

Purdue University researchers have developed a cloud-based supply chain management system to help emergency responders track inventory and distribution in countries hit by natural disasters. Purdue's Yuehwern Yih and colleagues worked with first responders to understand their requirements when responding to areas struck by catastrophic events; the system is designed to replace paper forms and ad hoc spreadsheets often used in developing nations to track vital resources. Data can be entered offline for syncing to the system through Wi-Fi or network connections, and the full life cycle of items in the system can be traced. Yih said, "The Purdue system can provide real-time data to allow better tracking of supplies and aid so that help reaches those most in need."  ... '

Real Time Data, Real Time Decisions

Useful introduction.  Note the usefulness in many domains, from supply chain to manufacturing to finance.    There were some also some recent suggestions that this might be neeed because data cannot be held because of regulatory changes, and needs to be used immediately.  Whats the best decision you can make in limited time, before the data mush be released?

From big data to fast data
Designing application architectures for real-time decisions.
By Raul Estrada in O'Reilly

For more about building fast-data architectures that meet your real-time decision data needs, download a free copy of an excerpt from Designing Data-Intensive Applications, compliments of Mesosphere.

Enterprise data needs change constantly but at inconsistent rates, and in recent years change has come at an increasing clip. Tools once considered useful for big data applications are not longer sufficient. When batch operations predominated, Hadoop could handle most of an organization’s needs. Development in other IT areas (think IoT, geolocation, etc.) have changed the way data is collected, stored, distributed, processed and analyzed. Real-time decision needs complicate this scenario and new tools and architectures are needed to handle these challenges efficiently.  ... "

Will GDPR Drive us Towards Real-time Analytics?

Some good thoughts on the implications.   At its simplest will this mean we will need to use and release data more quickly?   Thus data will become real time?  At what cost of accuracy?

How GDPR Drives Real-Time Analytics  In DataFloq

New reforms under the General Data Protection Regulation (GDPR) started as an attempt to standardise data protection regulations in 2012. The European Union intends to make Europe “fit for the digital age.” It took four years to finalise the agreements and reach a roadmap on how the laws will be enforced.

The GDPR presents new opportunities as well as difficulties for businesses, digital companies, data collectors, and digital marketers. On the one hand, these regulations will make it more difficult for businesses and data mining firms to collect and analyse customer data for marketers, while on the other, they will present an opportunity for data collectors to innovate and enhance their techniques. This will lead to a better collection of more meaningful data, as customers will be directly involved. .... "

Embedding Analytics into Applications

Nice piece.  Good idea.   At one level a very obvious idea.  All analytics should be embedded into business applications.  And those applications are always a kind of business process, else they are not worth doing.    Sometimes they exist within existing computing processes.   In all of these cases the overall system may be more or less real-time.   So know your business and its context, consider it early and often.

The business advantages of embedding analytics into applications
Access to critical data in real time enables workers to generate insights from large amounts of information.  ....    By Matthew Sarrel  

Real Time Aviation Data Analytics

More and more data is being gathered and acted upon in real time. Here is a nice example where students are participating, from the SAScom Blog.  With video.

Students visualize aircraft data in real time using SAS  0
By Katie Howard on SAS Voices May 8, 2017 Data Visualization 

What do you get when you combine analytics, aviation and the Internet of Things? A learning experience that leaves everyone flying high! .... "

Oracle and Real Time Anaytics in the Cloud

Cloud solutions continue to roll. I see less about the specifics of solutions here, except a bare listing.  You do need a cloud to provide solutions, but also workable solutions.  Like the mention of more real-time capabilities, as delivered in the cloud.

Oracle rolls out cloud-based data integration system for real-time analytics   by Mike Heatley in SiliconAngle: 

Oracle Corp.’s bid to supplant Amazon Web Services at the top of the public cloud summit moved a step forward with the introduction of a new service that aims to integrate enterprise customer’s data with real-time analytics.

On Monday, the company rolled out its Data Integrator Cloud Service that’s designed to accelerate support for real-time analytics for enterprise customers. The idea is to address the challenge of delivering insights from data analytics to the appropriate applications and employees who use them. ... "

Visualization Insight Tools for Netflix

In the Netflix Techblog, some good details.

Improving Netflix’s Operational Visibility with Real-Time Insight Tools
By Ranjit Mavinkurve, Justin Becker and Ben Christensen

For Netflix to be successful, we have to be vigilant in supporting the tens of millions of connected devices that are used by our 40+ million members throughout 40+ countries. These members consume more than one billion hours of content every month and account for nearly a third of the downstream Internet traffic in North America during peak hours.

From an operational perspective, our system environments at Netflix are large, complex, and highly distributed. And at our scale, humans cannot continuously monitor the status of all of our systems. To maintain high availability across such a complicated system, and to help us continuously improve the experience for our customers, it is critical for us to have exceptional tools coupled with intelligent analysis to proactively detect and communicate system faults and identify areas of improvement. 

In this post, we will talk about our plans to build a new set of insight tools and systems that create grea er visibility into our increasingly complicated and evolving world.  .... " 

Creating Models of Virtual Changes from Physical Interactions

A interesting effect.   Here you create a model of how a real world object reacts to physical interactions.  Say using a videotape of the object in motion under influences.  Then you use that model to have it react the same way when interacted with artificially. A kind of virtual interaction reality.

To get an appreciation, watch the linked video.  Possible applications in gaming, animation, engineering models.   Further imagining this with alternate forms of interactions.   Say of a visualization of data being influenced by context?  Thought provoking.

Bluemix Supports Hangout Analytics

Nice to see some now sevices in Bluemix Cloud that support real time stream analytics.  This is notably of interest to applications like Retail.   Updates in their blog give some detailed outlines and descriptions of analysis value.  Possibility of use with Beacons, Supply chain decisions, IOT and others.   Ultimately this looks at the dynamics of all sorts of geospatial data.

" ... Hangout is a concept in geospatial analysis where an entity is said to be hanging out or in a hangout if it has been dwelling in the same region for a specified period of time.  Once an entity (for example, a smartphone or a vehicle) has been in a region longer than a specified dwell time, it is considered to be hanging out in that region.  You can use hangout detection in a variety of mobile device and Internet of Things (IoT) scenarios, including:

Marketing: Offering real-time promotions to people who remain in a particular area for a significant period of time.

Intelligence: Are two ships passing in the night, or are they hanging out together, possibly indicating some nefarious activity?

Transportation: Has a vehicle in your fleet made its stops at the anticipated locations?  ...  "

Google and Real Time Ads

In AdExchanger:

" ... Google sounds more and more like Twitter, heralding ways marketers can tap into live events for engagement.

As such, the company on Wednesday rolled out a new ad product called Real-Time Ads at a press event in New York.

With Real-Time Ads, Google aims to capitalize on the ripple effect of live events, such as sports events, political rallies or awards shows, along with all of the consumer and brand conversations that arise because of them.

The format can be “dynamically inserted” across YouTube, “hundreds of thousands of apps” and about two million sites across the Google Display Network, according to Tara Walpert Levy, managing director of agency sales for Google. ... " 

A Translating Megaphone Assists

A simple example of an assistant.  A megaphone at Tokyo Airport translates.  We have started to see examples of realtime translation. But imagine the same device being augmented with additional contextual information.   An ideal avenue of assistance?  Or does the need for common sense reasoning override the application?

November December Analytics Magazine on Real Time

November/December Analytics Magazine is now Online

From real-time analytics to data reduction to the ongoing debate over whether humans make better decisions than machines, the November/December issue of Analytics Magazine, now online, has something for everyone ... 
  
Cover Story: 
The Reality of Real Time
by Larry Skowronek
Marketing and customer interaction: Four important areas where real-time analytics can provide real value.

Clorox Mapping Flu Real Time with Social

This relates to some work we did in the 90s on detecting epidemics and bioterror.  See the tag,

Clorox analyzes social media to predict flu outbreaks 
Clorox is promoting its sanitation products with an online push called Clorox Cold & Flu Pulse, which gives consumers a real-time flu report for their locality based on analysis of social media conversations. Clorox's tool is the first to use social chatter for flu predictions, rather than mapping outbreaks as they happen, said marketing professor Jonah Berger. Consumers can also add their own information using the hashtag #StoptheSpread. MediaPost Communications (10/28)

Teradata Listener

Analyzing real time streams of data,  from real systems, or from simulations or prototypes or models, is a useful concept.

Teradata doubles down on IoT with two new tools .... 
" ... The Internet of Things is a big part of what makes big data so big, and on Monday, analytics provider Teradata unleashed two new tools to help make sense of the vast troves of data it produces.

Teradata Listener and Teradata Aster Analytics on Hadoop help users "listen" to massive streams of IoT data in real time and then use analytics to find the distinctive underlying patterns.

Teradata Listener is self-service software for ingesting and distributing individual or multiple data streams from sources including sensors, telematics, mobile events, click streams, social media feeds and IT server logs. ... " 

Self Service Time Series Analysis

Building self-service tools to monitor high-volume time-series data
The O'Reilly Data Show Podcast: Phil Liu on the evolution of metric monitoring tools and cloud computing.   ....  "

IBM Pushing Spark for Big Data

IBM Wants to Push Spark, Real-Time Big Data Tool, Into Mainstream

 In the WSJ: IBM ... has thrown its weight behind Spark, an increasingly popular tool that is used to analyze large amounts of data in real time. Spark opens up all sorts of emerging business applications, such as the ability to quickly target ads toward people passing in front of a digital billboard.

The number of such uses is expected to greatly expand as Internet-connected sensors become pervasive in physical objects, a phenomenon known as the Internet of Things. By working to push Spark into the mainstream now, IBM hopes to catch an emerging market at the very early stages. ... " 

See the tag link Spark below for more notes on Apache Spark use, including introductory posts.  Also looking forward to see how this can be closely linked to Watson cognitive efforts.   Note also the mention of real time retail interactions like in-store displays or smartphones.

Internet of Things Transparency Ad Targeting

See EvryThng: The Internet of Things Smart Products Platform

EVRYTHNG is the award-winning IoT cloud platform that connects any consumer product to the Web and manages real-time data to drive applications. Smart products don’t just deliver connected experiences and support services, they share data with enterprise systems and other device clouds for smarter ROI    .... " 

In Adage:
Internet of Things Data Could Fuel Ad Targeting
Next Home for Connected Product Data Is the Marketing Database

The Internet of Things has promised to turn our everyday interactions with stuff into data for logistical and marketing applications.

But now that more and more corporations, including Diageo and Mondelez, have tested actual web-connected products in the market, the industry is approaching the next stage of connected appliances and food packaging. That means figuring out where all that information will go and how it will be used. IoT platform company Evrythng sees a home for data generated by connected thermostats, bottles of booze, designer handbags and washing machines in first-party marketing databases.... " 

A new realm of marketing and retail transparency? We looked at a number of web connected packages, displays, shelves, kitchens, rooms and appliances.

A Fast Data Challenge

From the ACM, viewed yesterday, technical, note its application to the IOT, which will produce lots of fast data.  This all gets fast, and thus interesting above 100K messages per second.  The first four chapters of the free e-Book linked to below are a nice, non-technical intro to the concept and values of fast data.

The Fast Data Challenge and Picking the Right Database: Why One Size Doesn’t Fit All
Interacting with fast data, data that is in motion, is a fundamentally different process than interacting with big data that is at rest. And few businesses have the ability to extract the value of that data when it matters most — at the moment it arrives — because traditional database technology simply hasn't kept pace.

Dr. Michael Stonebraker, Professor at MIT and co-founder of VoltDB, has long held the belief that, without the right database architecture in place, today's organizations run the risk of being left behind in a world that's smarter and faster than what legacy systems can handle.

It’s time to rethink the technology stack needed to enable fast data applications. What’s needed is a framework for making technology choices and architecting fast data applications, the fast data stack.  The fast data stack has three levels: data ingestion, real-time analytics and decisions, and data export.   ....

You may have to register for this.   View here,  I understand this will be present until May 2016.    All Seminar Slides.

And a free background e-Book.    About the open source VoltDB system.

Micro Moments and Marketing

Via Google.  New idea to me.  The Micro Moment as describing how people use mobile.  By my observation, these are not always that 'micro'.   The concept is an interesting one.    How much are they changing the rules of marketing?

" .... Consumer behavior has changed forever. Today's battle for hearts, minds, and dollars is won (or lost) in micro-moments—intent-driven moments of decision-making and preference-shaping that occur throughout the entire consumer journey.   Read more about this new mental model for marketing. .. As mobile has become an indispensable part of our daily lives, we're witnessing a fundamental change in the way people consume media. What used to be our predictable, daily sessions online have been replaced by many fragmented interactions that now occur instantaneously. Many of these moments have become quite routine as we check the time or text a friend. ... "