Dealing with Tsunami of Data

Continues.

RisingWave Emerges to Tackle Tsunami of Real-Time Data

Alex Woodie in Datanami

Only the most advanced companies have overcome the technical complexity involved with processing streaming data in real time. One of the vendors aiming to reduce this complexity and make stream data processing available to the masses is RisingWave Labs, which today announced $36 million in financing.

The early days of stream data processing brought us stand-alone systems that were capable of acting upon vast streams of data, and doing so with low latency and reliability. Stream processing frameworks like Apache Storm made headway in addressing these challenges and led the way to more sophisticated frameworks like Apache Flink and others.

Things got significantly more complex when companies realized they needed to know something about the past to take the best action on the newest data, which necessitated the integration of stream processing frameworks with databases or data lakes, where the historical record lived as persisted data. Architectural blueprints, such as the Lambda and Kappa architectures, were proposed to address this unique challenge, but the technical complexity in keeping these dual-path systems running are immense.

Today we’re seeing the emergence of a new category of product–the streaming database–aimed at solving this problem. Instead of running data through a dedicated stream processing framework like Storm or Flink, the backers of streaming databases think that all the data processing–including the business end of a streaming big data pipeline like Kafka, Kinesis, or Pulsar–can be handled by the SQL query engine contained in a relational database.

RisingWave is a Postgres-compatible database developed to process data streams in the cloud (Image courtesy RisingWave Labs)

That’s the approach taken with RisingWave, a new open source streaming database that emerged just over a year ago. Yingjun Wu, a former AWS and IBM engineer, created RisingWave as a cloud-native database with the goal of providing the bernefits of stream processing without the technical complexity inherent with stream processing frameworks.

“Existing open-source systems are very costly to deploy, maintain, and use in the modern cloud environment,” Wu, who is the CEO of RisingWave Labs, says in a press release today. “Our goal is not to build yet another streaming system that is 10X faster than existing systems, but to deliver a simple and cost-effective system that allows everyone to benefit from stream processing.”

Developed in Rust, RisingWave is a Postgres-compatible database can do many of the things that stream processing frameworks do, but within the context and control of a familiar relational database running in the cloud and the SQL language, according to Wu, who has a PhD from National University of Singapore and was also a visiting PhD at Carnegie Mellon University.

“[RisingWave] consumes streaming data, performs continuous queries, and maintains results dynamically in the form of a materialized view,” Wu says in a blog post earlier this year. “Processing data streams inside a database is quite different from that inside a stream computation engine: streaming data are instantly ingested into data tables; queries over streaming and historical data are simply modeled as table joins; query results are directly maintained and updated inside the database, without pushing into a downstream system.”

The open source project, which available on GitHub via an Apache 2.0 license, is being adopted by organizations for a range of uses, including real-time analytics and alerting; IoT device tracking; monitoring user activity; and online application data serving. The company, which changed its name from Singularity Data three weeks ago, recently unveiled the beta of a hosted commercial version of RisingWave; it’s slated to become generally available next year.  ... '