7 ms·

I used to work in the orbit of a distinguished eng at AWS who was famous for saying something to the effect of, "At the bottom of any scaled distributed system

by mgdev 4y ago

I used to work in the orbit of a distinguished eng at AWS who was famous for saying something to the effect of, "At the bottom of any scaled distributed system is either Paxos, or a bug."

ChadNauseam 4y ago
It's a fun quote, it reminds me of the "every sufficiently advanced program contains a bug-ridden implementation of half of common lisp", or something along those lines. But there really is a wide world of distributed consensus systems out there (although Paxos is easily the most elegant).
- arthurcolle 4y ago
  There's a variation of this on the subject of distributed systems re: Erlang/BEAM VM
  - qohen 4y ago
    Yes, from Robert Virding, one of Erlang's co-creators, from the Erlang mailing-list, from 2008 [0]: After reading the blogs about how good Erlang's concurrency model is and how we just just made a super implementation of it in XXX I have been led to formulate Virding's First Rule of Programming: Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang. This is, of course, a mild travesty of Greenspun (*) but I think it is fundamental enough to be my first rule, not the tenth. [0] http://erlang.org/pipermail/erlang-questions/2008-January/032226.html http://erlang.org/pipermail/erlang-questions/2008-January/03...
- winkywooster 4y ago
  You mean “Greenspun's tenth rule” https://en.m.wikipedia.org/wiki/Greenspun%27s_tenth_rule https://en.m.wikipedia.org/wiki/Greenspun%27s_tenth_rule
gamegoblin 4y ago
When I was at AWS I heard the same phrase from a DE, probably the same guy, and once heard him also say "Raft is just a special case of Paxos to try to simplify it, but regular Paxos isn't actually that hard, just use Paxos". I was fairly junior at the time, and Raft seemed much more approachable, but after really forcing myself to read and understand the Paxos paper, I see what he meant. I am pretty sure most of the love for Raft was that the original whitepaper was just a better presentation. The actual Paxos algorithm is quite simple. If you go into Raft already having mastered Paxos (as this DE was doing), it's clear that Raft is basically isomorphic to a special case of Paxos.
- jsmith45 4y ago
  https://arxiv.org/abs/2004.05074 https://arxiv.org/abs/2004.05074 This paper argues basically argues that raft has a different leadership election mechanism than paxos, but that if you tweak some terminology, and make a few relatively reasonable implementation choices for paxos they are otherwise pretty equivalent. It even gives a raft style single page description of paxos (using raft style terminology), and shows how little it differs from the equivalent single page summary of raft. The main implementation choices they use are: - combined roles into a single server role - enforce that log messages are decided in sequence (largely to avoid the having to specify the behavior of newly elected leader to propose operations for the gaps (possibly no-ops)) - numeric ballot number, rather than lexicographical pair (but this changes nothing except making the summary slightly easier to express)
roughly 4y ago
“…but I repeat myself.”
rubiquity 4y ago
Assuming we're thinking of the same person, I believe it went: "There are three types of consistent distributed systems: paxos, broken protocols, and single points of failures."
- dastbe 4y ago
  i don't understand why people won't just say alv. he won't hurt you.
  - rubiquity 4y ago
    You're assuming the name matters more than the content.
    - dastbe 4y ago
      i guess he never explained a false dichotomy to you :). knowing who said this allows readers to be able to look up more of the (unfortunately scant) content publicly available from him.
  - mgdev 4y ago
    Really sucks the mystery out of the room though, doesn't it? It's almost more impressive that people know who it is without saying the name.
dtjohnnymonkey 4y ago
I just watched a talk on this the other day (assuming it's same person since he's saying the same thing) the other day: https://youtu.be/QVvFVwyElLY?t=2370 https://youtu.be/QVvFVwyElLY?t=2370
toomim 4y ago
For some reason, people considering PAXOS/RAFT don't tend to consider CRDT/OT synchronization. I think this is a big oversight. We should start considering CRDT/OT/VCS/Diffsync approaches to distributed systems as well. They present a very nice alternative approach: whereas PAXOS/RAFT implement a consistent "distributed state machine", a CRDT, OT, VCS, or Diffsync system implements consistent "distributed state", upon which one can build a machine as a function of the state. This latter approach is actually simpler, IMO, because it encapsulates all the challenge of distributed consistency within a smaller subset of the problem — state synchronization. This makes it more generally re-usable. When you create a system, you can just use an off-the-shelf library & algorithm to synchronize your data over a network, and then write synchronous functions on top of that to represent the system you want, however you want, without having to understand PAXOS/RAFT.
- diroussel 4y ago
  Would this approach be resistant to a rogue actor. If one actor bad data would all the other actors still be able to reach consensus? Paxos is complicated, but it’s well studied and proven.
  - suchire 4y ago
    Neither OT/CRDT nor Paxos/Raft are meant to resist rogue actors. They are not Byzantine fault tolerant.
YZF 4y ago
There's also ZooKeeper with its own consensus protocol.

by mgdev 4y ago

I used to work in the orbit of a distinguished eng at AWS who was famous for saying something to the effect of, "At the bottom of any scaled distributed system is either Paxos, or a bug."

ChadNauseam 4y ago

It's a fun quote, it reminds me of the "every sufficiently advanced program contains a bug-ridden implementation of half of common lisp", or something along those lines. But there really is a wide world of distributed consensus systems out there (although Paxos is easily the most elegant).

arthurcolle 4y ago
There's a variation of this on the subject of distributed systems re: Erlang/BEAM VM
- qohen 4y ago
  Yes, from Robert Virding, one of Erlang's co-creators, from the Erlang mailing-list, from 2008 [0]: After reading the blogs about how good Erlang's concurrency model is and how we just just made a super implementation of it in XXX I have been led to formulate Virding's First Rule of Programming: Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang. This is, of course, a mild travesty of Greenspun (*) but I think it is fundamental enough to be my first rule, not the tenth. [0] http://erlang.org/pipermail/erlang-questions/2008-January/032226.html http://erlang.org/pipermail/erlang-questions/2008-January/03...
winkywooster 4y ago
You mean “Greenspun's tenth rule” https://en.m.wikipedia.org/wiki/Greenspun%27s_tenth_rule https://en.m.wikipedia.org/wiki/Greenspun%27s_tenth_rule

gamegoblin 4y ago

When I was at AWS I heard the same phrase from a DE, probably the same guy, and once heard him also say "Raft is just a special case of Paxos to try to simplify it, but regular Paxos isn't actually that hard, just use Paxos". I was fairly junior at the time, and Raft seemed much more approachable, but after really forcing myself to read and understand the Paxos paper, I see what he meant. I am pretty sure most of the love for Raft was that the original whitepaper was just a better presentation. The actual Paxos algorithm is quite simple. If you go into Raft already having mastered Paxos (as this DE was doing), it's clear that Raft is basically isomorphic to a special case of Paxos.

jsmith45 4y ago
https://arxiv.org/abs/2004.05074 https://arxiv.org/abs/2004.05074 This paper argues basically argues that raft has a different leadership election mechanism than paxos, but that if you tweak some terminology, and make a few relatively reasonable implementation choices for paxos they are otherwise pretty equivalent. It even gives a raft style single page description of paxos (using raft style terminology), and shows how little it differs from the equivalent single page summary of raft. The main implementation choices they use are: - combined roles into a single server role - enforce that log messages are decided in sequence (largely to avoid the having to specify the behavior of newly elected leader to propose operations for the gaps (possibly no-ops)) - numeric ballot number, rather than lexicographical pair (but this changes nothing except making the summary slightly easier to express)

roughly 4y ago

“…but I repeat myself.”

rubiquity 4y ago

Assuming we're thinking of the same person, I believe it went: "There are three types of consistent distributed systems: paxos, broken protocols, and single points of failures."

dastbe 4y ago
i don't understand why people won't just say alv. he won't hurt you.
- rubiquity 4y ago
  You're assuming the name matters more than the content.
  - dastbe 4y ago
    i guess he never explained a false dichotomy to you :). knowing who said this allows readers to be able to look up more of the (unfortunately scant) content publicly available from him.
- mgdev 4y ago
  Really sucks the mystery out of the room though, doesn't it? It's almost more impressive that people know who it is without saying the name.

dtjohnnymonkey 4y ago

I just watched a talk on this the other day (assuming it's same person since he's saying the same thing) the other day: https://youtu.be/QVvFVwyElLY?t=2370 https://youtu.be/QVvFVwyElLY?t=2370

toomim 4y ago

For some reason, people considering PAXOS/RAFT don't tend to consider CRDT/OT synchronization. I think this is a big oversight. We should start considering CRDT/OT/VCS/Diffsync approaches to distributed systems as well. They present a very nice alternative approach: whereas PAXOS/RAFT implement a consistent "distributed state machine", a CRDT, OT, VCS, or Diffsync system implements consistent "distributed state", upon which one can build a machine as a function of the state. This latter approach is actually simpler, IMO, because it encapsulates all the challenge of distributed consistency within a smaller subset of the problem — state synchronization. This makes it more generally re-usable. When you create a system, you can just use an off-the-shelf library & algorithm to synchronize your data over a network, and then write synchronous functions on top of that to represent the system you want, however you want, without having to understand PAXOS/RAFT.

diroussel 4y ago
Would this approach be resistant to a rogue actor. If one actor bad data would all the other actors still be able to reach consensus? Paxos is complicated, but it’s well studied and proven.
- suchire 4y ago
  Neither OT/CRDT nor Paxos/Raft are meant to resist rogue actors. They are not Byzantine fault tolerant.

YZF 4y ago

There's also ZooKeeper with its own consensus protocol.