3 ms·

It's fake because the deduplication isn't a property of the messaging system, it's a property of the system consuming the messages. "Your system" is providing a

by whakim 4y ago

It's fake because the deduplication isn't a property of the messaging system, it's a property of the system consuming the messages. "Your system" is providing a workaround (duplication) for at-least-once delivery, not the message system. I think it's also worth noting that your suggested workaround has actually recreated the problem it's intending to solve. If you mark the UUID as "received" as soon as you get a message, how do you deal with duplicates if the processing fails? If you mark the UUID as "received" when you're done processing, how do you deal with the possibility that you'll receive the message multiple times? This can get hairy very quickly.

iainmerrick 4y ago
The solution to that last problem is a transaction log.
strken 4y ago
We come back to the original argument, which is that at-least-once delivery of idempotent operations is how you represent things that should happen exactly once in a system, and in a saner world this could reasonably be called exactly-once delivery. Every distributed systems engineer knows what "exactly-once delivery" is asking for, and in plain English it's valid to conflate the two, but for some reason the field has decided to treat the phrase as an annoying semantic pit trap for the unwary. Want to add an ID to your transactions to make them idempotent? Well, even though your transactions are now recorded exactly once, that wasn't technically delivery! Gotcha!
- Kalium 4y ago
  The issue is precision in terminology and resulting understanding. At-least-once delivery with idempotency imposes certain rules on the processing of messages. As you say, every distributed systems engineer knows this. When you call something exactly-once, people who are perhaps not distributed systems engineers make the reasonable assumption that this means exactly what it says. They will engineer around this reasonable assumption based on a clear technical description and get something hilariously broken in non-obvious ways. This will have happened because jargon ("exactly-once delivery") has been confused for a technical description of a delivery system's properties. Not everyone in this series of comments is a distributed systems engineer. Never mind everyone using a messaging system.
- whakim 4y ago
  If you zoom out enough then every system looks like a black box and the only thing that matters is inputs and outputs. Messaging systems are no different in that respect from literally everything else. If you're building a messaging system, there's a world of difference in terms of how other people integrate with your software if you say "exactly-once" vs "at-least-once."

by whakim 4y ago

iainmerrick 4y ago

The solution to that last problem is a transaction log.

strken 4y ago

We come back to the original argument, which is that at-least-once delivery of idempotent operations is how you represent things that should happen exactly once in a system, and in a saner world this could reasonably be called exactly-once delivery. Every distributed systems engineer knows what "exactly-once delivery" is asking for, and in plain English it's valid to conflate the two, but for some reason the field has decided to treat the phrase as an annoying semantic pit trap for the unwary. Want to add an ID to your transactions to make them idempotent? Well, even though your transactions are now recorded exactly once, that wasn't technically delivery! Gotcha!

Kalium 4y ago
The issue is precision in terminology and resulting understanding. At-least-once delivery with idempotency imposes certain rules on the processing of messages. As you say, every distributed systems engineer knows this. When you call something exactly-once, people who are perhaps not distributed systems engineers make the reasonable assumption that this means exactly what it says. They will engineer around this reasonable assumption based on a clear technical description and get something hilariously broken in non-obvious ways. This will have happened because jargon ("exactly-once delivery") has been confused for a technical description of a delivery system's properties. Not everyone in this series of comments is a distributed systems engineer. Never mind everyone using a messaging system.
whakim 4y ago
If you zoom out enough then every system looks like a black box and the only thing that matters is inputs and outputs. Messaging systems are no different in that respect from literally everything else. If you're building a messaging system, there's a world of difference in terms of how other people integrate with your software if you say "exactly-once" vs "at-least-once."