8 ms·

High-performance, exactly-once, failure-oblivious distributed programming (2018)

129 points by cmeiklejohn 7y ago

what_a_waow 7y ago
This is fail-of-course article
cfontes 7y ago
Kafka exactly-once semantics addresses the main issue of the article I think. It's now relatively simple for a developer to implement a system with exactly once guarantee as long as you take care of the world that is not inside a Kafka transaction (integrations with third parties and such), which is still not super easy sometimes, but less so then the distributed transaction that will happen inside Kafka. Kafka hides the complexity really well from my use of it so far is very reliable with the "new" semantics.
- pdpi 7y ago
  I'm getting mighty tired of the "exactly-once" thing. Everybody and their uncles seem to have picked up on this trend of advertising at-least-once systems as exactly-once, then burying somewhere in the docs that you're expected to guarantee idempotency yourself to get the appearance of exactly-once. That was the state of the art decades years ago, it's the state of the art now, and it's pretty damn dishonest to sell quality-of-life improvements as a fundamental shift in the guarantees/properties of these systems.
  - dualogy 7y ago
    > at-least-once systems with idempotency to get the appearance of exactly-once What baffles me even more is why the above is apparently not generally considered good-enough, elegant-enough --- and as a bonus, not violating the laws of physics either? Both sides of the coin are quite tameable and implementable. And together deliver what was wanted in the first place, and effectively. Curious in any subtle edge-cases I might have missed here!
    - pdpi 7y ago
      The problem is that there is an audience for whom exactly-once sounds like it makes their non-specialist lives much simpler compared to an at-least-once system because they can offload the necessary distributed systems expertise to somebody else. People insist on this messaging precisely because that crowd is somewhat vulnerable to this sort of shenanigans.
    - hosh 7y ago
      It requires teaching the engineering team to reason and prove idempotency guarantees. I've seen teams try to continue programming the way they always have and try to throw difficult problems over the wall. It is exactly-once in a non-threaded, monolithic, stateless web app, so why shouldn't it always be like that? It's part of a programmer's mindset to try to create abstractions and reason within a simpler problem space.
- i0exception 7y ago
  Kafka can probably guarantee exactly ones semantics on publishing (conditions apply). It definitely cannot guarantee exactly once semantics on the consumer and processing side. Imagine a scenario where you receive a message from Kafka and process it, but the processor crashes or has a network partition right after. There's no way for the message to be acknowledged and you either have to design your system to be idempotent or handle exactly-once semantics further down the stack. Databases have been handling exactly-once semantics for decades now. What Kafka is doing is not new and actually gives you a false sense of security when it comes to these kinds of things.
- docker_up 7y ago
  Kafka exactly-once semantics is just that, they are "semantics". What Kafka supports is exactly-once processing which has been supported in other stream processing frameworks such as Apache Storm years before Confluent's marketing. Duplicates are possible in Kafka with the current implementation of exactly-once, if one uses Kafka's consumer api it will de-dedupe on the processing side. So no, there is no such thing as exactly-once in distributed systems.
farazbabar 7y ago
Exactly once processing is not possible in distributed systems. Anyone that tries to sell that snake oil is dishonest and anyone who buys it should not be making purchasing decisions. The definition and requirement of idempotent processing means systems must be able to handle messages delivered more than once, which irrefutably proves there is no such thing as exactly once. Even within centralized, monolithic systems with no outside interaction, laws of physics and reality still apply - a pull on the cord, an earthquake, a flooding or a myriad of other things may interrupt message processing resulting in either exactly zero or more times a message will be processed even if the message were being processed within the confines of an embedded micro-controller using hand crafted assembler code.
- docker_up 7y ago
  This is what I've been told as well from distributed systems experts. It's more "exactly-once most of the time" but believing that it will always be exactly once in all failure situations is delusional.
- SpicyLemonZest 7y ago
  Agreed. It’s not that I don’t get it; people have always built systems where even idempotency is impossible, and it’s a lot easier to convince business-side stakeholders that “exactly once” can’t be compromised on than that non-idempotency is a critical design flaw. But it’s easier because the term substantially misleads them about what you’re getting.
- zzzcpan 7y ago
  There is no literal exactly once of course, because it's not physically possible, but "exactly-once" semantics are possible in distributed systems. Data can be resynchronized, processes can be restarted with the side effects removed, etc.
  - farazbabar 7y ago
    And in the absence of correctly designed idempotent behaviors across the processing pipeline, it becomes unbelievably complex to handle all the corner cases correctly. The problem with marketing folks parroting and promoting the exactly once semantics is that any weak link in the processing chain compromises the correctness of entire system and what is worse is that the implications of subtle errors may not become evident for years to come (history is rife with legal cases and negative consequences of getting it wrong when it comes to folks' money). In critical sectors like banking, finance and payments, designing systems in well understood and boring manner is absolutely critical while hoping for the best based on shiny brochures and marketechture is a sure recipe for disaster.
kerblang 7y ago
> Many cloud service designs today rely on durable queues, such as Event Hub or Kafka. AFAIK nowhere in the Kafka documentation does it use the term "queue", and unless you only have one consumer per consumer group it's impossible to guarantee FIFO behavior. Maybe call me a nitpicker but I've seen this "queue" language lead to completely wrong assumptions about kafka.
- frankmcsherry 7y ago
  I googled "kafka documention", went to https://kafka.apache.org/documentation/ https://kafka.apache.org/documentation/, searched for "queue" and found 29 matches.
hosh 7y ago
I've been recently reading the papers coming out of the Berkley Disroderly Lab -- Bloom(L) languages, lattices, composing eventually-consistent, coordination-free systems. It's interesting to read this article with that lens. There are some properties that are similar, but this one looks like it is designed to let people continue programming the way they are at the cost of increased coordination with other systems. The idea of a replayable log seems to be able to convert a disordered sequence of events into something that is ordered. Whereas, the Bloom(L) stuff constructs algorithms that only requires partial order. An event stream can be disordered because the functions being used are monotonic, and the compositions of the data structure uses operators that are commutative, associative, and idempotent. (Thus, there is no requirement for exactly-once guarantee, or an ordered event stream).

129 points by cmeiklejohn 7y ago

This is fail-of-course article

Kafka exactly-once semantics addresses the main issue of the article I think. It's now relatively simple for a developer to implement a system with exactly once guarantee as long as you take care of the world that is not inside a Kafka transaction (integrations with third parties and such), which is still not super easy sometimes, but less so then the distributed transaction that will happen inside Kafka. Kafka hides the complexity really well from my use of it so far is very reliable with the "new" semantics.

pdpi 7y ago
I'm getting mighty tired of the "exactly-once" thing. Everybody and their uncles seem to have picked up on this trend of advertising at-least-once systems as exactly-once, then burying somewhere in the docs that you're expected to guarantee idempotency yourself to get the appearance of exactly-once. That was the state of the art decades years ago, it's the state of the art now, and it's pretty damn dishonest to sell quality-of-life improvements as a fundamental shift in the guarantees/properties of these systems.
- dualogy 7y ago
  > at-least-once systems with idempotency to get the appearance of exactly-once What baffles me even more is why the above is apparently not generally considered good-enough, elegant-enough --- and as a bonus, not violating the laws of physics either? Both sides of the coin are quite tameable and implementable. And together deliver what was wanted in the first place, and effectively. Curious in any subtle edge-cases I might have missed here!
  - pdpi 7y ago
    The problem is that there is an audience for whom exactly-once sounds like it makes their non-specialist lives much simpler compared to an at-least-once system because they can offload the necessary distributed systems expertise to somebody else. People insist on this messaging precisely because that crowd is somewhat vulnerable to this sort of shenanigans.
  - hosh 7y ago
    It requires teaching the engineering team to reason and prove idempotency guarantees. I've seen teams try to continue programming the way they always have and try to throw difficult problems over the wall. It is exactly-once in a non-threaded, monolithic, stateless web app, so why shouldn't it always be like that? It's part of a programmer's mindset to try to create abstractions and reason within a simpler problem space.
i0exception 7y ago
Kafka can probably guarantee exactly ones semantics on publishing (conditions apply). It definitely cannot guarantee exactly once semantics on the consumer and processing side. Imagine a scenario where you receive a message from Kafka and process it, but the processor crashes or has a network partition right after. There's no way for the message to be acknowledged and you either have to design your system to be idempotent or handle exactly-once semantics further down the stack. Databases have been handling exactly-once semantics for decades now. What Kafka is doing is not new and actually gives you a false sense of security when it comes to these kinds of things.
docker_up 7y ago
Kafka exactly-once semantics is just that, they are "semantics". What Kafka supports is exactly-once processing which has been supported in other stream processing frameworks such as Apache Storm years before Confluent's marketing. Duplicates are possible in Kafka with the current implementation of exactly-once, if one uses Kafka's consumer api it will de-dedupe on the processing side. So no, there is no such thing as exactly-once in distributed systems.

farazbabar 7y ago

Exactly once processing is not possible in distributed systems. Anyone that tries to sell that snake oil is dishonest and anyone who buys it should not be making purchasing decisions. The definition and requirement of idempotent processing means systems must be able to handle messages delivered more than once, which irrefutably proves there is no such thing as exactly once. Even within centralized, monolithic systems with no outside interaction, laws of physics and reality still apply - a pull on the cord, an earthquake, a flooding or a myriad of other things may interrupt message processing resulting in either exactly zero or more times a message will be processed even if the message were being processed within the confines of an embedded micro-controller using hand crafted assembler code.

docker_up 7y ago
This is what I've been told as well from distributed systems experts. It's more "exactly-once most of the time" but believing that it will always be exactly once in all failure situations is delusional.
SpicyLemonZest 7y ago
Agreed. It’s not that I don’t get it; people have always built systems where even idempotency is impossible, and it’s a lot easier to convince business-side stakeholders that “exactly once” can’t be compromised on than that non-idempotency is a critical design flaw. But it’s easier because the term substantially misleads them about what you’re getting.
zzzcpan 7y ago
There is no literal exactly once of course, because it's not physically possible, but "exactly-once" semantics are possible in distributed systems. Data can be resynchronized, processes can be restarted with the side effects removed, etc.
- farazbabar 7y ago
  And in the absence of correctly designed idempotent behaviors across the processing pipeline, it becomes unbelievably complex to handle all the corner cases correctly. The problem with marketing folks parroting and promoting the exactly once semantics is that any weak link in the processing chain compromises the correctness of entire system and what is worse is that the implications of subtle errors may not become evident for years to come (history is rife with legal cases and negative consequences of getting it wrong when it comes to folks' money). In critical sectors like banking, finance and payments, designing systems in well understood and boring manner is absolutely critical while hoping for the best based on shiny brochures and marketechture is a sure recipe for disaster.

kerblang 7y ago

> Many cloud service designs today rely on durable queues, such as Event Hub or Kafka. AFAIK nowhere in the Kafka documentation does it use the term "queue", and unless you only have one consumer per consumer group it's impossible to guarantee FIFO behavior. Maybe call me a nitpicker but I've seen this "queue" language lead to completely wrong assumptions about kafka.

frankmcsherry 7y ago
I googled "kafka documention", went to https://kafka.apache.org/documentation/ https://kafka.apache.org/documentation/, searched for "queue" and found 29 matches.

hosh 7y ago

I've been recently reading the papers coming out of the Berkley Disroderly Lab -- Bloom(L) languages, lattices, composing eventually-consistent, coordination-free systems. It's interesting to read this article with that lens. There are some properties that are similar, but this one looks like it is designed to let people continue programming the way they are at the cost of increased coordination with other systems. The idea of a replayable log seems to be able to convert a disordered sequence of events into something that is ordered. Whereas, the Bloom(L) stuff constructs algorithms that only requires partial order. An event stream can be disordered because the functions being used are monotonic, and the compositions of the data structure uses operators that are commutative, associative, and idempotent. (Thus, there is no requirement for exactly-once guarantee, or an ordered event stream).