4 ms·
So tapping into the change-logs of the underlying data-sources and providing a stream processing layer that's expressible in a stream SQL dialect? btw, not bei
by chrisjc 7y ago
So tapping into the change-logs of the underlying data-sources and providing a stream processing layer that's expressible in a stream SQL dialect?
btw, not being critical of your project, just trying to understand it.
- cube2222 7y agoMainly thinking of explicitly stream oriented data sources like kafka, but yeah, change logs are really solved with an analogous abstraction. There's a great paper on that: "One SQL to Rule Them All", check it out. We also want to scale well from single computer one-of data exploration queries, to full blown clustered long-term stateful stream processing. The point is to provide a well thought out SQL interface to as many data sources as possible, and like drill does, push down as much computation as possible. We actually learned about drill only after creating OctoSQL, but that's another story. (We're definitely less mature currently and support fewer datasources)
- chrisjc 7y agoHa! I was going to link that to you once I understood what you meant. https://news.ycombinator.com/item?id=20059006#20062821 https://news.ycombinator.com/item?id=20059006#20062821 I believe there is a similar effort going on in the Flink project. They're slowly merging in work from the Blink project to provide a unified SQL paradigm (batch + stream).
- cube2222 7y agoYup, Akidaus book is a great resource too. We're basically aiming for a middle ground between Flink, Presto and Drill.
- deleted 7y ago[deleted]