4 ms·

Depending on how you use the database it is. If you write the data as well as the offset to the DB in the same transaction, you can then seek to the offset stor

Depending on how you use the database it is. If you write the data as well as the offset to the DB in the same transaction, you can then seek to the offset stored in the DB after application restart and continue from there.

deleted 3y ago
[deleted]
viraptor 3y ago
> after application restart and continue from there. What if the application doesn't restart before the queue decides the message was lost and resends?
- hashhar 3y ago
  In Kafka the "queue" is dumb, it doesn't lose messages (it's an append only durable log) nor does it resend anything unless the consumer requests it.
  - viraptor 3y ago
    There has to be a retry system somewhere, otherwise you'd end up with a 0-or-more delivery system if the app crashes after picking up from the queue, but never processing or ack-ing.
mirekrusin 3y ago
You should drop "(...) and carefully acknowledging them when you are sure data is safely stored in your db (...)" part then, because it means it's not necessary, you don't rely on it. One-or-more semantics + local deduplication gives one-and-only semantics. In this case you're optimising local deduplication with strictly monotonic index. One downside is that you leak internals of other system (partitions). The other is that it implies serialised processing - you can't process anything in parallel as you have single index threshold that defines what has been and what has yet not been processed.
- Svenskunganka 3y ago
  I'm not the one who wrote the original comment, so I can't modify it. But one should still commit offsets because it is the happy-path; DB transaction successful? Commit offset. If the latter fails due to e.g application crash and you seek at startup to the partition offset stored in the DB + 1, you get exactly-once semantics. There's some more details, e.g you'd have to do the same during consumer group rebalance, and topic configuration also plays a role, for example if the topic is a compacted topic or not, and if you write tombstones, what its retention policy is. edit: You added some more to your comment after I posted this one, so I'll try to cover them as well: > One downside is that you leak internals of other system (partitions). Yeah, sure. > The other is that it implies serialised processing - you can't process anything in parallel as you have single index threshold that defines what has been and what has yet not been processed. It doesn't imply serialised processing. It depends on the use-case, if each record in a topic has to be processed serially, you can't parallelize full-stop; number of partitions equals 1. But if each record can be individually processed you get parallelism equal to the number of partitions the topic has configured. You also achieve parallelism in the same way if only some records in a topic needs to be processed serially, at which point you can use the same key for the records needing to be serially processed and they will end up in the same partition, for example recording the coordinates of a plane - each plane can be processed in parallel, but an individual plane's coordinates need to be processed serially - just use the planes unique identifier as key and the coordinates for the same plane will be appended to the log of the same partition.
  - mirekrusin 3y ago
    Yes, it's good option but it requires serialised processing in partition scope, which may or may not be desirable. If one-and-only-one semantics are needed and processing should be parallel, other methods have to be used.
    - robertlagrant 3y ago
      Presumably every exactly once processing scenario needs you to squeeze things through a serial pipe at some point, or you could have 2 messages with the same ID come in and be processed in parallel?

deleted 3y ago

[deleted]

viraptor 3y ago

> after application restart and continue from there. What if the application doesn't restart before the queue decides the message was lost and resends?

hashhar 3y ago
In Kafka the "queue" is dumb, it doesn't lose messages (it's an append only durable log) nor does it resend anything unless the consumer requests it.
- viraptor 3y ago
  There has to be a retry system somewhere, otherwise you'd end up with a 0-or-more delivery system if the app crashes after picking up from the queue, but never processing or ack-ing.

mirekrusin 3y ago

You should drop "(...) and carefully acknowledging them when you are sure data is safely stored in your db (...)" part then, because it means it's not necessary, you don't rely on it. One-or-more semantics + local deduplication gives one-and-only semantics. In this case you're optimising local deduplication with strictly monotonic index. One downside is that you leak internals of other system (partitions). The other is that it implies serialised processing - you can't process anything in parallel as you have single index threshold that defines what has been and what has yet not been processed.

Svenskunganka 3y ago
I'm not the one who wrote the original comment, so I can't modify it. But one should still commit offsets because it is the happy-path; DB transaction successful? Commit offset. If the latter fails due to e.g application crash and you seek at startup to the partition offset stored in the DB + 1, you get exactly-once semantics. There's some more details, e.g you'd have to do the same during consumer group rebalance, and topic configuration also plays a role, for example if the topic is a compacted topic or not, and if you write tombstones, what its retention policy is. edit: You added some more to your comment after I posted this one, so I'll try to cover them as well: > One downside is that you leak internals of other system (partitions). Yeah, sure. > The other is that it implies serialised processing - you can't process anything in parallel as you have single index threshold that defines what has been and what has yet not been processed. It doesn't imply serialised processing. It depends on the use-case, if each record in a topic has to be processed serially, you can't parallelize full-stop; number of partitions equals 1. But if each record can be individually processed you get parallelism equal to the number of partitions the topic has configured. You also achieve parallelism in the same way if only some records in a topic needs to be processed serially, at which point you can use the same key for the records needing to be serially processed and they will end up in the same partition, for example recording the coordinates of a plane - each plane can be processed in parallel, but an individual plane's coordinates need to be processed serially - just use the planes unique identifier as key and the coordinates for the same plane will be appended to the log of the same partition.
- mirekrusin 3y ago
  Yes, it's good option but it requires serialised processing in partition scope, which may or may not be desirable. If one-and-only-one semantics are needed and processing should be parallel, other methods have to be used.
  - robertlagrant 3y ago
    Presumably every exactly once processing scenario needs you to squeeze things through a serial pipe at some point, or you could have 2 messages with the same ID come in and be processed in parallel?