6 ms·
There is major difference between Kafka and database(s): Kafka is used for passing data through and databases are for storing the data. Also, KSQL/kdbSQL are no
by dig1 7y ago
There is major difference between Kafka and database(s): Kafka is used for passing data through and databases are for storing the data. Also, KSQL/kdbSQL are not intended to be general tool for transforming all data, but to assist you without writing custom services (unless I'm getting things wrong).
Now, should Kafka be central place for routing data? I think that is the main reason why it exists. Should it be used for permanently storing data like conventional database? Absolutely not.
By design, it will temporarily store it in case consumer goes offline, but you can also use it to implement backpressure/queue for slow receivers that are not capable to catch up with fast data ingestion (e.g. syslog -> kafka -> logstash).