4 ms·

If you have enough background in distributed systems to choose a consensus algorithm you are definitely capable of implementing one yourself and even invent you

If you have enough background in distributed systems to choose a consensus algorithm you are definitely capable of implementing one yourself and even invent your own. It's not a bad idea either, most existing consensus-based systems are absolute garbage. People just hugely overestimate how much those algorithms matter for production systems. EDIT: Note, I'm not talking about blockchains and byzantine fault tolerance, but in context of Paxos/Raft.

eloff 8y ago
That's like saying if you have some crypto background you can implement your own crypto library. It's TERRIBLE advice. Read the Google paper on their chubby distributed locking system using paxos. They had a hard time getting it right - unless you think you're so much better than them. IIRC Leslie Lamport works (worked?) for them. Use an existing, well used, well scrutinized implementation of a known algorithm.
- kjeetgill 8y ago
  Are you sure? I thought Lamport has been at Microsoft Research since forever.
  - eloff 8y ago
    Ah the vagaries of the human memory.
  - therein 8y ago
    You're correct. Lamport has joined MSR in 2001 right after publishing "Time, Clocks, and the Ordering of Events in a Distributed System" and has been there ever since. I am not sure if he is still at the MV office but he was one of the reasons why MSR kept a presence in MV for a while.
    - lou1306 8y ago
      Actually "Time, Clocks..." is a paper from the late 70s. Maybe you were thinking of the Paxos paper, "The Part-Time Parliament" (1998)?
      - therein 8y ago
        Yeah, copy-pasted the wrong title and by the time I noticed, it was too late to edit. :)
- dlubarov 8y ago
  > Use an existing, well used, well scrutinized implementation of a known algorithm. Keep in mind that this is BFT consensus. Until recently, it was a niche area of research with very little real world usage. Even if an existing algorithm like PBFT met their needs, there are no well tested PBFT libraries.
- drfuchs 8y ago
  No, Lamport does not now, and never has, worked at Google.
- apersona 8y ago
  If you're a normal software engineer, I agree, you shouldn't implement it yourself. But if you're a researcher/had a CS research background, I don't see why not. By definition research results in novel approaches (ideally better) than the current ones. You have to be willing to challenge the consensus. PhD students who can prove that their algs work should work on it, like the results of this paper: https://raft.github.io https://raft.github.io

eloff 8y ago

That's like saying if you have some crypto background you can implement your own crypto library. It's TERRIBLE advice. Read the Google paper on their chubby distributed locking system using paxos. They had a hard time getting it right - unless you think you're so much better than them. IIRC Leslie Lamport works (worked?) for them. Use an existing, well used, well scrutinized implementation of a known algorithm.

kjeetgill 8y ago
Are you sure? I thought Lamport has been at Microsoft Research since forever.
- eloff 8y ago
  Ah the vagaries of the human memory.
- therein 8y ago
  You're correct. Lamport has joined MSR in 2001 right after publishing "Time, Clocks, and the Ordering of Events in a Distributed System" and has been there ever since. I am not sure if he is still at the MV office but he was one of the reasons why MSR kept a presence in MV for a while.
  - lou1306 8y ago
    Actually "Time, Clocks..." is a paper from the late 70s. Maybe you were thinking of the Paxos paper, "The Part-Time Parliament" (1998)?
    - therein 8y ago
      Yeah, copy-pasted the wrong title and by the time I noticed, it was too late to edit. :)
dlubarov 8y ago
> Use an existing, well used, well scrutinized implementation of a known algorithm. Keep in mind that this is BFT consensus. Until recently, it was a niche area of research with very little real world usage. Even if an existing algorithm like PBFT met their needs, there are no well tested PBFT libraries.
drfuchs 8y ago
No, Lamport does not now, and never has, worked at Google.
apersona 8y ago
If you're a normal software engineer, I agree, you shouldn't implement it yourself. But if you're a researcher/had a CS research background, I don't see why not. By definition research results in novel approaches (ideally better) than the current ones. You have to be willing to challenge the consensus. PhD students who can prove that their algs work should work on it, like the results of this paper: https://raft.github.io https://raft.github.io