5 ms·

Since the article mentions Google as the outlier preferring Paxos, I may be able to shed some light from a few years ago. The Paxos, paxosdb, and related libra

Since the article mentions Google as the outlier preferring Paxos, I may be able to shed some light from a few years ago. The Paxos, paxosdb, and related libraries (despite the name, all are multi-paxos) are solid and integrated directly into a number of products (Borg, Chubby, CFS, Spanner, etc.). There are years of engineering effort and unit tests behind the core Paxos library and so it makes sense to keep using and improving it instead of going off to Raft. As far as I am aware the Google Paxos implementation predates Raft by quite a while. I think in general if most other people use Raft it's better for the community to have single, stable, and well-tested shared implementations for much the same reason it's good for Google to stick with Paxos.

tyingq 5y ago
This makes sense to me. Very few of us have the resources to maintain, for example, the kind of globally synced (way beyond typical NTP) clock infrastructure that Google has (TrueTime[1]). [1] https://cloud.google.com/spanner/docs/true-time-external-consistency https://cloud.google.com/spanner/docs/true-time-external-con...
- gravypod 5y ago
  If you want to know just how hard this can be with bare metal that you control, take a look at this: https://signalsandthreads.com/clock-synchronization/ https://signalsandthreads.com/clock-synchronization/
  - jeffbee 5y ago
    On the other hand, if you run in Google Compute Engine you get accurate clock discipline for free.
    - electricshampo1 5y ago
      This is just best effort on google's end right? Don't think anything is documented/guaranteed such that you would be able to, for ex. rely on it like spanner's use of true time.
      - jeffbee 5y ago
        Sure, you'd have to invent the rest, but TrueTime isn't about having perfect clocks, it's about estimating the error of your peer's clocks. Having a reasonable platform clock is a good starting point, and addresses the problems discussed in the Jane Street article.
fred256 5y ago
> The Paxos, paxosdb, and related libraries (despite the name, all are multi-paxos) are solid and integrated directly into a number of products (Borg, Chubby, CFS, Spanner, etc.). Maybe things have changed, but I thought the bottom turtle for pretty much any infrastructure system at Google was Chubby. I didn't realize Borg now directly does Paxos.
- isubasinghe 5y ago
  What do you mean by bottom turtle? Chubby itself needs something like Paxos for state replication.
  - fred256 5y ago
    What I meant was that I thought any system that required some form of distributed locking or consensus, did so by building on top of Chubby (which does Paxos), not by implementing Paxos directly.
- justicezyx 5y ago
  BorgMaster had its own consensus protocol prior to paxos lib. It was switched to paxos in 2010? (Don't quote me on this) Chubby uses paxos. Iirc, the history is that Chubby comes first, then it's paxos code got pulled out into paxos lib, which proliferate inside Google. And yes, Google had no other choice than Paxos at its days.

tyingq 5y ago

This makes sense to me. Very few of us have the resources to maintain, for example, the kind of globally synced (way beyond typical NTP) clock infrastructure that Google has (TrueTime[1]). [1] https://cloud.google.com/spanner/docs/true-time-external-consistency https://cloud.google.com/spanner/docs/true-time-external-con...

gravypod 5y ago
If you want to know just how hard this can be with bare metal that you control, take a look at this: https://signalsandthreads.com/clock-synchronization/ https://signalsandthreads.com/clock-synchronization/
- jeffbee 5y ago
  On the other hand, if you run in Google Compute Engine you get accurate clock discipline for free.
  - electricshampo1 5y ago
    This is just best effort on google's end right? Don't think anything is documented/guaranteed such that you would be able to, for ex. rely on it like spanner's use of true time.
    - jeffbee 5y ago
      Sure, you'd have to invent the rest, but TrueTime isn't about having perfect clocks, it's about estimating the error of your peer's clocks. Having a reasonable platform clock is a good starting point, and addresses the problems discussed in the Jane Street article.

fred256 5y ago

> The Paxos, paxosdb, and related libraries (despite the name, all are multi-paxos) are solid and integrated directly into a number of products (Borg, Chubby, CFS, Spanner, etc.). Maybe things have changed, but I thought the bottom turtle for pretty much any infrastructure system at Google was Chubby. I didn't realize Borg now directly does Paxos.

isubasinghe 5y ago
What do you mean by bottom turtle? Chubby itself needs something like Paxos for state replication.
- fred256 5y ago
  What I meant was that I thought any system that required some form of distributed locking or consensus, did so by building on top of Chubby (which does Paxos), not by implementing Paxos directly.
justicezyx 5y ago
BorgMaster had its own consensus protocol prior to paxos lib. It was switched to paxos in 2010? (Don't quote me on this) Chubby uses paxos. Iirc, the history is that Chubby comes first, then it's paxos code got pulled out into paxos lib, which proliferate inside Google. And yes, Google had no other choice than Paxos at its days.