Underneath this is the hard truth that streaming is hard and costly. I've run into many people with optimistic dreams around how to take an SQL or other query l

Underneath this is the hard truth that streaming is hard and costly. I've run into many people with optimistic dreams around how to take an SQL or other query languages and then make it update based on data changes rather than periodic polling. These systems are always expensive and yield marginal improvements when compared to a dumb publish subscribe system. I've got notes on publish subscribe systems, and I am publishing a paper next month about the end result system that I built over five years. For a clue about the midpoint, my SRECON talk addresses this: https://www.usenix.org/conference/srecon17americas/program/presentation/erlich https://www.usenix.org/conference/srecon17americas/program/p... A key note is that publish/subscribe over-commits the problem space (i.e. the problem space can potentially be quadratic depending on use), but if you give up on durability then your costs can go way-way-way down. The lack of durability is a problem, BUT it can be leveraged to discover non-marginal use-cases such that further investments make sense. Unfortunately, once you invest in the imperfect system such that it operates at five+ nines during normal operation, then very few query based systems are competitive.