3 ms·

Providing API for building applications that have transactions and help with idempotent producing is really exciting. This will help lower a lot of pains associ

by sks 9y ago

Providing API for building applications that have transactions and help with idempotent producing is really exciting. This will help lower a lot of pains associated with building stream processing systems. Doing it with very little performance overhead is amazing. I do feel that calling it "Exactly-once delivery with kafka" is slightly misleading as this requires the applications to be written in a certain way. The title makes is sound too general and borders on claiming something that is close to impossible. I dont want to be too critical here as the author was very honest with what this means in the blog post. Regardless of the title this is an amazing feature.

sulam 9y ago
Not "close" to impossible -- actually impossible.
- gumbo 9y ago
  The reasons this is "currently" impossible are: - We cannot build communication infrastructures that are 100% reliable - We build apps that actually crash. I would not venture as far as to say this is impossible. If one day we are able to leverage quantum entanglement phenomenon in telecommunications network this would certainly be very much possible.
  - 1812Overture 9y ago
    The machine that reads the quantum entagled particles will still occasionally break or have coding errors.
    - staticassertion 9y ago
      I don't really know a thing about that sort of network, but if you can assume a reliable network, the machine reading the packets going down shouldn't matter. The sender won't get an "ACK", will know the other machine is at fault and not the network, and can resend. Given a "perfect network" I think exactly-once-delivery is possible, right?
      - mac01021 9y ago
        What if the sender fails before it can fully process the ACK? When it comes online again it won't know that the last message sent was delivered.
        staticassertion 9y ago
        Transactions.
  - sulam 9y ago
    You've been downvoted, but the trick would be to convert asynchronous messaging to synchronous processing that proceeds at a predictable tick. That would make it possible to bypass the FLP result. I'm willing to imagine quantum entanglement could pull this off if we were able to entangle actual transistors. Dunno how plausible that is, but points for creativity if nothing else.
  - paulddraper 9y ago
    > We build apps that actually crash. And disks that fail and memory that bitflips and communication methods that are susceptible to random noise and machines dependent on failable power sources.
- mason55 9y ago
  Also known as the Two Generals Problem
falcolas 9y ago
> calling it "Exactly-once delivery with kafka" is slightly misleading Indeed. Idempotent operations is the mainstay of managing "at least once" message delivery. If you had a true "exactly once" system, idempotency would be unnecessary. > Regardless of the title this is an amazing feature. I feel it's a good iterative step forward, but nothing near the "revolutionary" that's being boasted.

by sks 9y ago

sulam 9y ago

Not "close" to impossible -- actually impossible.

gumbo 9y ago
The reasons this is "currently" impossible are: - We cannot build communication infrastructures that are 100% reliable - We build apps that actually crash. I would not venture as far as to say this is impossible. If one day we are able to leverage quantum entanglement phenomenon in telecommunications network this would certainly be very much possible.
- 1812Overture 9y ago
  The machine that reads the quantum entagled particles will still occasionally break or have coding errors.
  - staticassertion 9y ago
    I don't really know a thing about that sort of network, but if you can assume a reliable network, the machine reading the packets going down shouldn't matter. The sender won't get an "ACK", will know the other machine is at fault and not the network, and can resend. Given a "perfect network" I think exactly-once-delivery is possible, right?
    - mac01021 9y ago
      What if the sender fails before it can fully process the ACK? When it comes online again it won't know that the last message sent was delivered.
      - staticassertion 9y ago
        Transactions.
- sulam 9y ago
  You've been downvoted, but the trick would be to convert asynchronous messaging to synchronous processing that proceeds at a predictable tick. That would make it possible to bypass the FLP result. I'm willing to imagine quantum entanglement could pull this off if we were able to entangle actual transistors. Dunno how plausible that is, but points for creativity if nothing else.
- paulddraper 9y ago
  > We build apps that actually crash. And disks that fail and memory that bitflips and communication methods that are susceptible to random noise and machines dependent on failable power sources.
mason55 9y ago
Also known as the Two Generals Problem

falcolas 9y ago

> calling it "Exactly-once delivery with kafka" is slightly misleading Indeed. Idempotent operations is the mainstay of managing "at least once" message delivery. If you had a true "exactly once" system, idempotency would be unnecessary. > Regardless of the title this is an amazing feature. I feel it's a good iterative step forward, but nothing near the "revolutionary" that's being boasted.