4 ms·
I heard several times that Kafka was put in front of elasticsearch clusters for handling traffic burst. You can also use Redpanda, Pulsar, NATS and other distri
by francoismassot 3y ago
I heard several times that Kafka was put in front of elasticsearch clusters for handling traffic burst. You can also use Redpanda, Pulsar, NATS and other distributed queues.
One thing that is also very interesting with Kafka is that you can achieve exactly-once semantic without too much efforts: by keeping track of the positions of partitions in your own database and carefully acknowledging them when you are sure data is safely stored in your db. That's what we did with our engine Quickwit, so far it's the most efficient way to index data in it.
One obvious drawback with Kafka is that it's one more piece to maintain... and it's not a small one.
- pranay01 3y agoHave you done/seen any benchmarks between Redpanda/NATS and Kafka for this use case? Some folks in SigNoz community have also suggested NATS for this, but I have not deep dived into benchmarks/features yet
- francoismassot 3y agoUnfortunately no :/
- radicality 3y agoExactly-once semantics of what specifically? Or do you mean at-least-once ?
- francoismassot 3y agoExactly-once semantic between Kafka and the observability engine.
- richieartoul 3y agoYou have to do a bit more than that if you want exactly once end-to-end (I.E if Kafka itself can contain duplicates). One of my former colleagues did a good write up on how Husky does it: https://www.datadoghq.com/blog/engineering/husky-deep-dive/ https://www.datadoghq.com/blog/engineering/husky-deep-dive/
- francoismassot 3y agoYeah, I was only talking about exactly once semantic between Kafka and Quickwit.
- viraptor 3y ago> exactly-once semantic without too much efforts: by keeping track of the positions of partitions in your own database and carefully acknowledging them when you are sure data is safely stored in your db That's not really "exactly once". What happens when your system dies after it made sure the data is safely stored in the db and before ack-ing?
- Svenskunganka 3y agoDepending on how you use the database it is. If you write the data as well as the offset to the DB in the same transaction, you can then seek to the offset stored in the DB after application restart and continue from there.
- deleted 3y ago[deleted]
- viraptor 3y ago> after application restart and continue from there. What if the application doesn't restart before the queue decides the message was lost and resends?
- hashhar 3y agoIn Kafka the "queue" is dumb, it doesn't lose messages (it's an append only durable log) nor does it resend anything unless the consumer requests it.
- viraptor 3y agoThere has to be a retry system somewhere, otherwise you'd end up with a 0-or-more delivery system if the app crashes after picking up from the queue, but never processing or ack-ing.
- mirekrusin 3y agoYou should drop "(...) and carefully acknowledging them when you are sure data is safely stored in your db (...)" part then, because it means it's not necessary, you don't rely on it. One-or-more semantics + local deduplication gives one-and-only semantics. In this case you're optimising local deduplication with strictly monotonic index. One downside is that you leak internals of other system (partitions). The other is that it implies serialised processing - you can't process anything in parallel as you have single index threshold that defines what has been and what has yet not been processed.
- foota 3y agoIsn't exactly once delivery the kind of problem like the CAP thereom where it's not possible? You can make the downstream idemptoent wrt what the queue is delivering, but the queue might still redeliver things.
- ankitnayan 3y agohttps://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ https://www.confluent.io/blog/exactly-once-semantics-are-pos...