New Algorithm for Stream Data

Almost every business deals with streams of data.   Determining patterns in a stream is essential, one of recent clients makes most of their profit in it. here in Quanta Mag, some new methods for addressing the problem. Not technically deep, but takes some thought.   How do we test this today?

 Best-Ever Algorithm Found for Huge Streams of Data by Kevin Hartnett in Quanta Mag

To efficiently analyze a firehose of data, scientists first have to break big numbers into bits.

' .... “We developed a new algorithm that is simultaneously the best” on every performance dimension, said Jelani Nelson, a computer scientist at Harvard University and a co-author of the work with Kasper Green Larsen of Aarhus University in Denmark, Huy Nguyen of Northeastern University and Mikkel Thorup of the University of Copenhagen.

This best-in-class streaming algorithm works by remembering just enough of what it’s seen to tell you what it’s seen most frequently. It suggests that compromises that seemed intrinsic to the analysis of streaming data are not actually necessary. It also points the way forward to a new era of strategic forgetting.  .... ' 

" ... Trend Spotting  ... Streaming algorithms are helpful in any situation where you’re monitoring a database that’s being updated continuously. This could be AT&T keeping tabs on data packets or Google charting the never-ending flow of search queries. In these situations it’s useful, even necessary, to have a method for answering real-time questions about the data without re-examining or even remembering every piece of data you’ve ever seen. ... "