3 ms·
Who is running single az deployments who also cares about data loss and availability? Seriously? I’ve personally supported 1000s of kafka deploys and this isn’t
by ceencee 4y ago
Who is running single az deployments who also cares about data loss and availability? Seriously? I’ve personally supported 1000s of kafka deploys and this isn’t a thing in the cloud at least. There is no call for wanting fsync per message, it is an anti pattern and isn’t done because it isn’t necessary. Data loss in kafka isn't a real problem that hurts real world users at all.
- jakewins 4y agoI was grabbing beer with a buddy who has ran some large - petabytes per month - Kafka deployments, and his experience was very much that Kafka will lose acked writes if not very carefully configured. He had direct experiences with data loss from JVM GC creating terrible flaky cluster conditions and, separately, from running out of disk on one cluster machine
- nemothekid 4y agoI can't list names about the "unserious" people who aren't running multi-AZ, but this is the approach to durability that MongoDB took ~15 years ago and they have never lived it down. It may just be that data reliability isn't a huge concern for messaging queues, so it's less of an issue, but pretending the risk isn't there doesn't help anyone.
- dilyevsky 4y ago> There is no call for wanting fsync per message, it is an anti pattern and isn’t done because it isn’t necessary 1. Don't have to do it by message 2. It's used by many distributed db engines, kafka and (i think) zk are the outliers here, not the other way around
- eternalban 4y agoKafka is not a "db engine". zk is a "db engine" in the same way 'DNS' is a "db engine".
- ahachete 4y agoOh, DNS is definitely a database engine [1] ;) [1]: https://dyna53.io https://dyna53.io
- dilyevsky 4y agoAh yes, the semantic argument. Fyi - pulsar and etcd do use fsync
- eternalban 4y agoNo one is arguing with you. You were making an argument based on a misinformed software category assertion and the error was pointed out. So r/fyi/til maybe?