An article of mine, “Data in Flight,” is published in this month’s Communications of the ACM. In it, I took the time to explain, in layman’s terms, why I think streaming database technology is a game-changer.
Many pundits have latched on to the term CEP (Complex Event Processing) to describe this technology. CEP is a legitimate and important application, and I believe that streaming SQL is a good way to solve it, but the article tries to put a bit of space between the two concepts. There are so many problems that benefit from the declarative, relational approach but where the data arrives incrementally and the problem can be solved much more efficiently by a streaming engine working (mainly) in memory than a database, and CEP is just one application area. My article describes a few of those problems.
I’m all fired up about streaming databases, just as I was when I co-founded SQLstream. I’ve worked in the database field for over 20 years, and I think it’s the most exciting thing to happen in databases in a generation. (Yes, it’s more important than data warehousing and, cough, object databases.)
Streaming SQL technology is rapidly becoming part of the standard toolkit for solving data management problems. If you’re not familiar with the technology, reading the article is a good way to come up to speed. Enjoy!