4 ms·
Exactly-Once Delivery May Not Be What You Want
- akkartik 12y agoMight be a straight-forward application of Buridan's Principle: http://research.microsoft.com/en-us/um/people/lamport/pubs/buridan.pdf http://research.microsoft.com/en-us/um/people/lamport/pubs/b...
- autarch 12y agoThe point about unique IDs is well taken, but there's a cost to tracking these unique IDs too. For example, if you generate hundreds of message a second, you quickly end up having to track millions of unique ids. Then you have to manage the storage of these IDs somehow, so you end up implementing a system to wipe out any ID older than a certain time, and so now you not only have track the IDs you've seen, but when you've seen them. That said, this is probably better than making an incredibly complex message queue system just to support exactly once delivery.
- alexk 12y agoCassandra is very good at this, works perfectly in similar case of tracking unique keys with TTL.
- vishnugupta 12y ago> One is that end-to-end system semantics matter much more than the semantics of an individual building block, and sometimes what seems like a very desirable semantic for a building block may end up making the end-to-end problem harder. The other is that simple, practical, solutions like unique IDs can make really hard problems much easier, and allow us to build and ship real systems that work in predictable ways. These assertions that I tend to follow most of the time. I keep warning my team members to never assume that a request for payment/fulfillment/some-business-function is going to arrive exactly once. If they build systems assuming that underlying scaffolding is going to behave in certain way then they won't be designing for failure. In any distributed system it's very important to define correctness from the end user's perspective. And then design for failure ensuring that the correctness guarantees are not violated.
- EGreg 12y agoObviously, exactly-once delivery is a side effect of possible overproduction and then filtering for uniquenessz Some people might also say that creativity is a side effect of brainstorming and idea selection. But it's more complex than that - over time our brains work out heuristics and figure out patterns that work well, and use that as a language to construct other things. That's why people who have been doing something for years can do it on a much higher level.
- ChuckMcM 12y agoThis is a fantastic summary of the challenge of distributed execution. I also really enjoyed the quote "there is always room to slot in another turtle" I am definitely going to keep that one handy.
- kiyoto 12y agoAs a maintainer of logging middleware (https://www.fluentd.org https://www.fluentd.org) I am asked all the time whether we have "exactly once" semantics. I try to be honest and say we err on the side of "at least once" message delivery, but even then, there are edge cases where messages can be lost. It really vexes me to see how a lot of distributed systems projects claim to support "exactly once", and it makes me happy to see a grounded, intelligent summary like this one.
- bkirwi 12y agoThis is a nice article, but the title strays into 'Fox and the Grapes' territory. Exactly-once delivery is pretty much always what you want -- in contexts where all manner of delivery guarantees are possible, from Java's concurrent queues to Go's channels, pretty much everyone delivers a message precisely once. Of course, you can't always get what you want, and in a distributed system exactly-once delivery is not possible in general. But in the cases where it is possible, it's often worthwhile: it's just a whole lot easier to reason about than a system where messages can be arbitrarily re-delivered.
- covi 12y ago> One is that end-to-end system semantics matter much more than the semantics of an individual building block, and sometimes what seems like a very desirable semantic for a building block may end up making the end-to-end problem harder. This is a wisdom whispered 3 decades ago. The end-to-end argument haunts us again [1]! [1] END-TO-END ARGUMENTS IN SYSTEM DESIGN http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf http://web.mit.edu/Saltzer/www/publications/endtoend/endtoen...
- akkartik 12y agoI made a more superficial version of this argument a year ago: http://akkartik.name/post/readable-bad http://akkartik.name/post/readable-bad Edit: "Safety is a characteristic of systems and not of their components" -- http://www.ctlab.org/documents/How%20Complex%20Systems%20Fail.pdf http://www.ctlab.org/documents/How%20Complex%20Systems%20Fai...