4 ms·

I was watching a video by one of Amazon’s distinguished engineers a while ago and for the life of me I can’t find it now but the thing that stuck with me from i

I was watching a video by one of Amazon’s distinguished engineers a while ago and for the life of me I can’t find it now but the thing that stuck with me from it is, “There are only three modes for a distributed system - it implements paxos/raft itself, it relies on a data store that implements paxos/raft or it’s wrong.” Obviously a bit of an over simplification since there have been some advancements in that area but not far off. This article introduces an approach that falls under the second mode and I think it’s a good way to build a distributed system without having to re invent the wheel especially if you can just rely on the data store for the consensus and make your brokers and nodes relatively dumb. Look at k8s and etcd for a very widely used example of this approach as well. I’m also currently building a distributed search engine using etcd for the service registry, broker peer announcements, and worker queue and it’s been a good experience so far.

hardwaresofton 4y ago
Well in some situations you can also get away with CRDTs or Gossip (SWIM), but yeah in the general sense Paxos (Raft is a Paxos-family algo). There are lots of paxos family algos to choose from[0]. > I’m also currently building a distributed search engine using etcd for the service registry, broker peer announcements, and worker queue and it’s been a good experience so far. As a random stranger on the internet, please build an abstraction layer around etcd. Even if there's only one implementation, I've found that so many distributed projects that just never reach the scale that etcd is built for would benefit from the option of writing their own drivers (you don't need a full plugin system just a regular abstraction layer). One example is Kine[0] for k8s -- if k8s had a built in option for writing/reading from something like Postgres from the beginning it would have been a better project for it, IMO. If you do build a plugin system though, you can pass off the work of maintaining the other implementations! [0]: https://vadosware.io/post/paxosmon-gotta-concensus-them-all https://vadosware.io/post/paxosmon-gotta-concensus-them-all [1]: https://github.com/k3s-io/kine https://github.com/k3s-io/kine
- ianbutler 4y ago
  I think that's a great idea, I already have the beginnings of a plugin system since local non distributed is a mode you can run in and I had to abstract etcd away to do that. Will add it to the list!
  - hardwaresofton 4y ago
    Thanks! Also I’m pretty curious where I can find your project…
    - ianbutler 4y ago
      Hey at the moment it’s not public but I’m getting to the point where it will be. Happy to ping you when it is, also planning to do a show HN post here as well with it.
      - hardwaresofton 4y ago
        Please do! email in bio and also I'll try to look out for the Show HN!
rubiquity 4y ago
> “There are only three modes for a distributed system - it implements paxos/raft itself, it relies on a data store that implements paxos/raft or it’s wrong.” That sounds like something Al Vermeulen would say. Though my recollection of the quote is more along the lines of: "There are three types of consistent distributed systems: paxos, broken protocols, and single points of failures." I'm guessing the talk it came from is Under The Covers: Core Distributed Systems Primitives that Power Our Platform[0]. That presentation is full of great insight into how AWS thinks about distributed systems. The other presenter, Swami, is the VP of AWS Databases now but started off getting Amazon's initial Paxos implementation going. I was lucky enough to work on those core building blocks and it was an incredible experience. 0 - https://www.youtube.com/watch?v=QVvFVwyElLY https://www.youtube.com/watch?v=QVvFVwyElLY
- ianbutler 4y ago
  Yup that's the one! Thank you for digging it up. That work must have been really gratifying especially since it underpins so much now.

hardwaresofton 4y ago

Well in some situations you can also get away with CRDTs or Gossip (SWIM), but yeah in the general sense Paxos (Raft is a Paxos-family algo). There are lots of paxos family algos to choose from[0]. > I’m also currently building a distributed search engine using etcd for the service registry, broker peer announcements, and worker queue and it’s been a good experience so far. As a random stranger on the internet, please build an abstraction layer around etcd. Even if there's only one implementation, I've found that so many distributed projects that just never reach the scale that etcd is built for would benefit from the option of writing their own drivers (you don't need a full plugin system just a regular abstraction layer). One example is Kine[0] for k8s -- if k8s had a built in option for writing/reading from something like Postgres from the beginning it would have been a better project for it, IMO. If you do build a plugin system though, you can pass off the work of maintaining the other implementations! [0]: https://vadosware.io/post/paxosmon-gotta-concensus-them-all https://vadosware.io/post/paxosmon-gotta-concensus-them-all [1]: https://github.com/k3s-io/kine https://github.com/k3s-io/kine

ianbutler 4y ago
I think that's a great idea, I already have the beginnings of a plugin system since local non distributed is a mode you can run in and I had to abstract etcd away to do that. Will add it to the list!
- hardwaresofton 4y ago
  Thanks! Also I’m pretty curious where I can find your project…
  - ianbutler 4y ago
    Hey at the moment it’s not public but I’m getting to the point where it will be. Happy to ping you when it is, also planning to do a show HN post here as well with it.
    - hardwaresofton 4y ago
      Please do! email in bio and also I'll try to look out for the Show HN!

rubiquity 4y ago

> “There are only three modes for a distributed system - it implements paxos/raft itself, it relies on a data store that implements paxos/raft or it’s wrong.” That sounds like something Al Vermeulen would say. Though my recollection of the quote is more along the lines of: "There are three types of consistent distributed systems: paxos, broken protocols, and single points of failures." I'm guessing the talk it came from is Under The Covers: Core Distributed Systems Primitives that Power Our Platform[0]. That presentation is full of great insight into how AWS thinks about distributed systems. The other presenter, Swami, is the VP of AWS Databases now but started off getting Amazon's initial Paxos implementation going. I was lucky enough to work on those core building blocks and it was an incredible experience. 0 - https://www.youtube.com/watch?v=QVvFVwyElLY https://www.youtube.com/watch?v=QVvFVwyElLY

ianbutler 4y ago
Yup that's the one! Thank you for digging it up. That work must have been really gratifying especially since it underpins so much now.