4 ms·
I came here to say that one of the few I can think of is KSQL. The downside is you pretty much have to turn your entire table into an event/message system. Ho
by sumtechguy 5y ago
I came here to say that one of the few I can think of is KSQL. The downside is you pretty much have to turn your entire table into an event/message system.
However, once you add a event/message system into place you could just have a topic you hang out on and just redo the changes on the query once the event pops. Something like select col1,col2 from table where lastchangetime >= mylastquerytime. Instead of polling with the same thing. Downsides to this is the extra index needed and polling overhead to name the biggest issues.
KSQL however is one of those things that looks really cool when you look at it. But if you dig a bit deeper, it makes less and less sense to use. For example in this case if you are using KSQL instead of normal streams/topic pumps you have a bit of overhead of SQL and yet another server cluster in the mix instead of the avro/json that is already built in. Also if you have already decomposed your data into messages into topics you can just as easily just wait on the topic for new data to show up. As that is something kafka does very easily and is built in already. Then also it does not work cross DB (say broker->sqlserver). Then also you now have a 4th type of code path in your system (kafka topics, kafka streams, SQL queries, and KSQL streams).