2 ms·
So, a combination of a best effort "at least once" messaging with deduplication near the receiving edge. Fairly standard, honestly. There is still a potential
by falcolas 9y ago
So, a combination of a best effort "at least once" messaging with deduplication near the receiving edge. Fairly standard, honestly.
There is still a potential for problems in the message delivery to the endpoints (malformed messages, Kafka errors, messages not being consumed fast enough and lost), or duplication at that level (restart a listener on the Kafka stream with the wrong message ID) as well.
This is based on my own pains with Kinesis and Lambda (which, I know, isn't Kafka).
In my experience, better to just allow raw "at least once" messaging and perform idempotant actions based off the messages. It's not always possible (and harder when it is possible), but its tradeoffs mean you're less likely to lose messages.
- caust1c 9y agoThis is generally better, but we're delivering these messages to integrations which don't necessarily take idempotent actions.