4 ms·

Can you talk more about the use case? My question would be, why use that setup if you don't have to? That setup is not something I would reach for unless I re

by borplk 8y ago

Can you talk more about the use case? My question would be, why use that setup if you don't have to? That setup is not something I would reach for unless I really really had to (and when you think you really really have to, check again, then check again, you probably don't really really have to). How about a RDBMS like Postgres setup with streaming synchronous replication and durability-tweaked parameters?

walrus1066 8y ago
In my opinion, using RMDBS as source of truth makes more sense, because these systems were optimised for reliability, durability & data retention. To be completely honest, some have drunk the proverbial Confluent koolaid, with kafka being the hammer to hit all nails. I just want to communicate the risks & tradeoffs with taking this approach. Problem is, the theory sort-of holds up, if configured correctly, kafka should be as reliable/durable as PG/SqlServer/MySQL cluster. But I can't find any material on companies experience running kafka this way, I.e. 'we ran this setup for a year in prod, one of our brokers disks got corrupted, but we were able to get back with no data loss, via X,Y,Z.' or 'we couldn't recover and had to accept data loss'. EDIT: in short, are the claims made by this article reliable: https://www.confluent.io/blog/okay-store-data-apache-kafka/ https://www.confluent.io/blog/okay-store-data-apache-kafka/