4 ms·

Streams are a fascinating topic, but this is not in any way shape or form a substitute for a real database. Take a look at the docs for joins: https://docs.con

Streams are a fascinating topic, but this is not in any way shape or form a substitute for a real database. Take a look at the docs for joins: https://docs.confluent.io/current/ksql/docs/developer-guide/join-streams-and-tables.html https://docs.confluent.io/current/ksql/docs/developer-guide/... It’s a long list of caveats about how your queries will silently fail if you don’t guarantee all sorts of properties of the data. Anyone considering making Kafka the center of your data infrastructure should also consider using a conventional column-store database instead. The only advantage of Kafka in this comparison is better latency (seconds vs minutes). If you can live with a minute or two of latency, the advantages of a real database are MANY.

dig1 7y ago
There is major difference between Kafka and database(s): Kafka is used for passing data through and databases are for storing the data. Also, KSQL/kdbSQL are not intended to be general tool for transforming all data, but to assist you without writing custom services (unless I'm getting things wrong). Now, should Kafka be central place for routing data? I think that is the main reason why it exists. Should it be used for permanently storing data like conventional database? Absolutely not. By design, it will temporarily store it in case consumer goes offline, but you can also use it to implement backpressure/queue for slow receivers that are not capable to catch up with fast data ingestion (e.g. syslog -> kafka -> logstash).