3 ms·
I think "exactly once" doesn't imply "non-reprocessing" (sorry for the double negative). Meaning, you want "exactly once" and you don't want duplicates, yes. B
by ecesena 9y ago
I think "exactly once" doesn't imply "non-reprocessing" (sorry for the double negative).
Meaning, you want "exactly once" and you don't want duplicates, yes. But you allow for reprocessing, provided that you have a way for deduplicating.
You want a guarantee that if the producer (at the top of your data processing pipeline) sends a message, then this message eventually corresponds to exactly 1 record in your final storage(s).
One easy-to-understand-yet-simplistic example is: send a message to kafka, use topic+partition+offset as primary key, store in a RDBMS. This is widely accepted as "exactly once", but clearly you may have multiple attempts to save the message into the db, which will fail due to the primary key integrity constrain.
- StavrosK 9y agoSo basically what's called "at least once with deduplicating". The parent comment addresses that.