4 ms·

This article is incorrect. > The way we achieve exactly-once delivery in practice is by faking it. Either the messages themselves should be idempotent, meaning

This article is incorrect. > The way we achieve exactly-once delivery in practice is by faking it. Either the messages themselves should be idempotent, meaning they can be applied more than once without adverse effects, or we remove the need for idempotency through deduplication. The author is wildly overestimating the complexity of the solution by ignoring the existence of a very simple but critical concept: UUIDs If the client generates and assigns a UUID to each message it sends, the receiver can easily check if a specific message was already received before by comparing it against previously received UUIDs and can discard duplicates. The title of the article is misleading because exactly-once delivery is in fact possible by deduplication. The deduplication can occur automatically (without having to know any context about the message beyond its UUID); it can occur BEFORE the delivery of the message to the consumer... So the title and entire premise of the article is incorrect. What the author should really be saying is that more than one copy of the same message may occasionally pass through the network due to failures... But they are misleading in suggesting that it's not possible to deliver exactly one copy or that deduplication is a herculean task when in fact it's trivial.

birdyrooster 4y ago
If all of your agents are seeing the UUID for the first time, they cannot deduplicate amongst themselves without quorum or shared state which has race conditions itself. It’s non trivial.
- vlovich123 4y ago
  But not impossible.
  - birdyrooster 4y ago
    Depending on your ability to share state between these agents it could very well be impossible. It depends on other factors.
- EdwardDiego 4y ago
  Yeah, non trivial is correct. A common Kafka approach is to partition by key, so that a given UUID will only be placed on one partition, and we're guaranteed that any further messages with that key will also be placed on that partition, so handled by the same consumer. Then create a change-log topic that's co-partitioned by key with the input topic. And then funk around with partition assignment strategies so if consumer X is assigned partition 10 of the input topic, then it's also assigned partition 10 of the change-log topic. Then add RocksDB as a fast state store on a persistent volume claim, as restoring state fresh from your change-log topic turns out to take about 7 minutes. And then realise you've just reimplemented bits of Kafka Streams poorly.
- jongjong 4y ago
  There's no need to. Different agents will have different copies of the same message, that's a given. If the sender ensures that a UUID is attached to a single specific message and never re-used (though they may retry to send the exact same message multiple times), then there is no risk that multiple receivers would have different versions of a message and therefore consensus is guaranteed without even having to talk to each other. If you assume that the sender cannot be trusted to not reuse a single UUID for multiple messages (e.g. to trick the receiver), you can still work around that limitation by computing and comparing message hashes (e.g. sha256) on the receiver side... Don't even need UUIDs. Every time you receive a message, you can hash it and check if you're received a message with this hash before (storing the hash of each message as it is received). You can use the hash in place of a UUID though in this case you probably need to add some index to each message to ensure that each hash is unique over time (since a message with the exact same payload will be counted as the same message, even if broadcast a long time apart).
  - preseinger 4y ago
    exactly once delivery is not satisfied if the same message is delivered to the recipient multiple times
    - jongjong 4y ago
      It depends on your definition of 'recipient' and 'delivery'. If you assume that the receiver is a specific database instance (or shard) or a specific data store (on the receiver side), then it's totally possible to have exactly once delivery in the sense that each message sent would end up being inserted into the database/data store exactly once without duplicates.
      - preseinger 4y ago
        but you can't make those assumptions, in general
gmcquistin 4y ago
> the receiver can easily check if a specific message was already received It seems like you're conflating exactly-once delivery with exactly-once processing. Your proposed solution is only necessary _because_ we can't have exactly-once delivery as TFA states.
- jongjong 4y ago
  I don't agree with that contrived definition of delivery. As a developer, what I care about is delivery of the message to my function/handler. E.g. so that I can store it in my local database/store on the receiver side exactly once and ultimately show it to the user exactly once. Definitely possible.
  - deleted 4y ago
    [deleted]
mankyd 4y ago
>> we remove the need for idempotency through deduplication. > If the client generates and assigns a UUID to each message it sends, the receiver can easily check if a specific message was already received before by comparing it against previously received UUIDs and can discard duplicates. You quoted the part of the article that suggests exactly your solution: deduplication.
- jongjong 4y ago
  Yes but the author suggests that it's difficult to perform or requires constructing messages in a complicated way which is not the case. Generating and attaching a UUID to a message is very simple, so is checking whether or not a UUID has already been encountered on the receiver side. The author's definition of 'message delivery' is incorrect and so is the title which is clickbait.
  - greggyb 4y ago
    Perhaps adding a UUID is too much overhead? If you have a very high throughput system with messages of less than 128 bits, then the addition of a UUID to every message would effectively cut throughput in half. I do not have such a use case in mind, but the description above does not sound terribly unrealistic to me.
    - datadata 4y ago
      Also in order to track UUIDs that have been received, you would either need an infinite amount of memory, or have the system bounded in terms of how many messages out of order a message be delivered and still be processed correctly.
      - jongjong 4y ago
        You can use a database. Practically, most systems will keep a record of each message on disk in any case (at least they will have some logs) so it's not that outrageous. Safe to assume that the receiver can discard non-authenticated messages to avoid spam.
prng2021 4y ago
You’re completely missing the point. Many multi master databases exist. Network partitions happen. Bugs in multi master databases during network partitions have happened and can happen. This is like distributed systems 101. So given those facts, your design of using UUIDs is absolutely not full proof if the database where you stored your UUID is not in a consistent state. Here’s some more useful info for your perusal (https://ucare.cs.uchicago.edu/pdf/socc14-cbs.pdf https://ucare.cs.uchicago.edu/pdf/socc14-cbs.pdf) “ Data consistency means that all nodes or replicas agree on the same value of a data (or eventually agree in the context of eventual consistency). In reality, there are several cases (5%) where data consistency is violated and users get stale data or the system’s behavior becomes erratic. The root causes mainly come from logic bugs in operational protocols (43%), data races (29%) and failure handling problems (10%).1 Be- low we expand these problems.”
- KptMarchewa 4y ago
  Your database does not have to be networked, it can be on-disk KV store like RocksDB, which is what things like Flink state backend use.

birdyrooster 4y ago

If all of your agents are seeing the UUID for the first time, they cannot deduplicate amongst themselves without quorum or shared state which has race conditions itself. It’s non trivial.

vlovich123 4y ago
But not impossible.
- birdyrooster 4y ago
  Depending on your ability to share state between these agents it could very well be impossible. It depends on other factors.
EdwardDiego 4y ago
Yeah, non trivial is correct. A common Kafka approach is to partition by key, so that a given UUID will only be placed on one partition, and we're guaranteed that any further messages with that key will also be placed on that partition, so handled by the same consumer. Then create a change-log topic that's co-partitioned by key with the input topic. And then funk around with partition assignment strategies so if consumer X is assigned partition 10 of the input topic, then it's also assigned partition 10 of the change-log topic. Then add RocksDB as a fast state store on a persistent volume claim, as restoring state fresh from your change-log topic turns out to take about 7 minutes. And then realise you've just reimplemented bits of Kafka Streams poorly.
jongjong 4y ago
There's no need to. Different agents will have different copies of the same message, that's a given. If the sender ensures that a UUID is attached to a single specific message and never re-used (though they may retry to send the exact same message multiple times), then there is no risk that multiple receivers would have different versions of a message and therefore consensus is guaranteed without even having to talk to each other. If you assume that the sender cannot be trusted to not reuse a single UUID for multiple messages (e.g. to trick the receiver), you can still work around that limitation by computing and comparing message hashes (e.g. sha256) on the receiver side... Don't even need UUIDs. Every time you receive a message, you can hash it and check if you're received a message with this hash before (storing the hash of each message as it is received). You can use the hash in place of a UUID though in this case you probably need to add some index to each message to ensure that each hash is unique over time (since a message with the exact same payload will be counted as the same message, even if broadcast a long time apart).
- preseinger 4y ago
  exactly once delivery is not satisfied if the same message is delivered to the recipient multiple times
  - jongjong 4y ago
    It depends on your definition of 'recipient' and 'delivery'. If you assume that the receiver is a specific database instance (or shard) or a specific data store (on the receiver side), then it's totally possible to have exactly once delivery in the sense that each message sent would end up being inserted into the database/data store exactly once without duplicates.
    - preseinger 4y ago
      but you can't make those assumptions, in general

gmcquistin 4y ago

> the receiver can easily check if a specific message was already received It seems like you're conflating exactly-once delivery with exactly-once processing. Your proposed solution is only necessary _because_ we can't have exactly-once delivery as TFA states.

jongjong 4y ago
I don't agree with that contrived definition of delivery. As a developer, what I care about is delivery of the message to my function/handler. E.g. so that I can store it in my local database/store on the receiver side exactly once and ultimately show it to the user exactly once. Definitely possible.
- deleted 4y ago
  [deleted]

mankyd 4y ago

>> we remove the need for idempotency through deduplication. > If the client generates and assigns a UUID to each message it sends, the receiver can easily check if a specific message was already received before by comparing it against previously received UUIDs and can discard duplicates. You quoted the part of the article that suggests exactly your solution: deduplication.

jongjong 4y ago
Yes but the author suggests that it's difficult to perform or requires constructing messages in a complicated way which is not the case. Generating and attaching a UUID to a message is very simple, so is checking whether or not a UUID has already been encountered on the receiver side. The author's definition of 'message delivery' is incorrect and so is the title which is clickbait.
- greggyb 4y ago
  Perhaps adding a UUID is too much overhead? If you have a very high throughput system with messages of less than 128 bits, then the addition of a UUID to every message would effectively cut throughput in half. I do not have such a use case in mind, but the description above does not sound terribly unrealistic to me.
  - datadata 4y ago
    Also in order to track UUIDs that have been received, you would either need an infinite amount of memory, or have the system bounded in terms of how many messages out of order a message be delivered and still be processed correctly.
    - jongjong 4y ago
      You can use a database. Practically, most systems will keep a record of each message on disk in any case (at least they will have some logs) so it's not that outrageous. Safe to assume that the receiver can discard non-authenticated messages to avoid spam.

prng2021 4y ago

You’re completely missing the point. Many multi master databases exist. Network partitions happen. Bugs in multi master databases during network partitions have happened and can happen. This is like distributed systems 101. So given those facts, your design of using UUIDs is absolutely not full proof if the database where you stored your UUID is not in a consistent state. Here’s some more useful info for your perusal (https://ucare.cs.uchicago.edu/pdf/socc14-cbs.pdf https://ucare.cs.uchicago.edu/pdf/socc14-cbs.pdf) “ Data consistency means that all nodes or replicas agree on the same value of a data (or eventually agree in the context of eventual consistency). In reality, there are several cases (5%) where data consistency is violated and users get stale data or the system’s behavior becomes erratic. The root causes mainly come from logic bugs in operational protocols (43%), data races (29%) and failure handling problems (10%).1 Be- low we expand these problems.”

KptMarchewa 4y ago
Your database does not have to be networked, it can be on-disk KV store like RocksDB, which is what things like Flink state backend use.