4 ms·

One point the article didn't cover is clients. Making a good Zookeeper client is hard, for two reasons: 1. The protocol is difficult to implement. In theory yo

One point the article didn't cover is clients. Making a good Zookeeper client is hard, for two reasons: 1. The protocol is difficult to implement. In theory you could just use Jute to codegen this part, but that assumes Jute supports the language you need. Doozer improves on this with a simple text-based protocol, and etcd goes a little further with a HTTP API. 2. The primitives Zookeeper exposes are very primitive. Implementing higher level abstractions such as locking or leader election on top of znodes is easy to get wrong. Just this year Curator has fixed critical bugs in both of those algorithms: https://github.com/Netflix/curator/blob/master/CHANGES.txt https://github.com/Netflix/curator/blob/master/CHANGES.txt My feeling is that etcd and doozer fare a little better on #2 just because their primitives are slightly easier to understand, but fundamentally the problem still exists. I'm looking forward to seeing more innovation in this area.

philsnow 13y ago
re #1, as I said elsewhere, at youtube we use zkocc to proxy readonly connections to zookeeper, and zkocc supports e.g. bson-over-http (and in any case it would be easier to add new client protocol support to zkocc than to zookeeper, I think. maybe I just think that because I don't want to java.)
- SEJeff 13y ago
  You work for youtube and use zookeeper?
tomjohnson3 13y ago
one other thing to keep in mind is that the underlying algorithms (zab and raft) provide different guarantees. for example, zookeeper/zab allows reading directly from followers with a guarantee to get at least a past value that won't be rolled back. this was one reason zookeer didn't use paxos: https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab+vs.+Paxos https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab+vs... in my understanding, raft doesn't allow reading directly from followers, because the follower logs may get repaired/rolled-back when a new leader is elected. (though i'm sure an implementation can tweak the protocol to provide this support.) that said, raft has a lot of interesting applications, and, in my opinion, is definitely more understandable than the many versions of paxos. (implementing zab yourself, at this point, would be a futile exercise.) i found the videos from the raft user study to be very well done (and easier to understand than even their paper): raft: http://www.youtube.com/watch?v=JEpsBg0AO6o http://www.youtube.com/watch?v=JEpsBg0AO6o paxos: http://www.youtube.com/watch?v=YbZ3zDzDnrw http://www.youtube.com/watch?v=YbZ3zDzDnrw ...however, i think they did paxos a disadvantage by not just focusing on multi-paxos (which is probably the most common implementation). but, it's certainly fair to say that info about paxos is spread out far and wide...with perhaps too many knobs to turn and implementation-related details to fill in yourself. as a side note: i've just started implementing raft in a set of libraries (multiple languages) that will be open source - along with other protocols.
- ideal0227 13y ago
  One thing is not correct. Raft allows to read from followers. You will never be able to read uncommitted value in raft.
- babo 13y ago
  > raft doesn't allow reading directly from followers, In raft all client connections to followers redirected to use the current master. > i think they did paxos a disadvantage by not just focusing on multi-paxos Raft is equivalent to (multi-)Paxos
  - nieksand 13y ago
    Having followers redirect to the leader is how the paper describes the algorithm. But I don't think there is anything stopping you from having followers service committed log entry reads, provided you're willing to live with being out-of-date.
    - tomjohnson3 13y ago
      i don't think this is the case. here is a link to a video by one of the authors about log repair during leader election: http://www.youtube.com/watch?feature=player_detailpage&v=YbZ3zDzDnrw#t=1481 http://www.youtube.com/watch?feature=player_detailpage&v=YbZ... log entries on a follower may get rolled back - and thrown out - since they were not accepted on a majority of followers.
  - codexon 13y ago
    Does this mean that there is no read scaling? The master server must be able to handle the entire read load?
- benbjohnson 13y ago
  I'm the original author of go-raft (the implementation used in etcd) so I'll try to address some of the points in this thread. Raft only updates the state of the system once log entries are committed to a quorum the local state will never be rolled back. Log entries can be thrown out but they haven't been committed to the local state so it doesn't matter. You can read from the leader if you need to ensure linearizability but that will kill your read scaling. Another approach is to read locally and check that the local raft node isn't in a "candidate" state (which would mean that it hasn't received a heartbeat from the master within the last 150ms). That approach works for a lot of cases. As far as implementing multi-paxos, the authors behind Google Chubby have talked about how there is a large divide between theoretical multi-paxos and actually implementing multi-paxos. Also, there aren't any standalone multi-paxos Go libraries available. I wrote go-raft at the time because there wasn't an alternative distributed consensus library in Go at the time. Let me know if you need an extra pair of eyes on your Raft implementation or if you have any questions (ben@skylandlabs.com).
voidmain 13y ago
I agree. At FoundationDB, we're writing a coordination tool (working name "beastmaster") that provides service discovery, locking, leader election, etc on top of our transactional key/value store. It is higher level than these tools; our idea is to try to make it useful from a command line or DNS rather than have to be baked into every piece of software that needs service discovery. beast service --lock name=mailserver.foundationdb.com --run mailserver.sh & beast service name=webserver.foundationdb.com --run webserver.sh & ping mailserver.foundationdb.com wget http://webserver.foundationdb.com/
- enigmo 13y ago
  That's how our Chubby clone works... it uses the Route53 API to publish service endpoints and leaders.
  - fintler 13y ago
    I'm also in the club of people building one of these things (mine is on top of LevelDB and written in ANSI C with a lock-free design). My most recent blog post is on getting the thing to bootstrap: http://www.bringhurst.org/2013/09/09/consensus-quorum-bootstrapping-101.html http://www.bringhurst.org/2013/09/09/consensus-quorum-bootst...
    - enigmo 13y ago
      Ours is on top of LevelDB too but "lock free" in Haskell using the STM.
      - jeremyjh 13y ago
        Is this open source?
        enigmo 13y ago
        Not yet... it'll land here when it's ready https://github.com/alphaHeavy https://github.com/alphaHeavy
kapilvt 13y ago
Another issue with zookeeper, the c bindings which are the basis for alot of language bindings, is both buggy and often out of date, esp wrt to individual language bindings. I've often see internal cpu spiked loads on c lib's internal io thread, ie pathological behavior. The recent support for dynamic quorums or transactions has yet to make it to any of the distributed language bindings. At least for python the folks at mozilla put together a nice from scratch library in the form of kazoo, at the cost of reimplementing the entire protocol. Compare that to the simplicity of etcd's, where you can just curl/wget a request.

philsnow 13y ago

re #1, as I said elsewhere, at youtube we use zkocc to proxy readonly connections to zookeeper, and zkocc supports e.g. bson-over-http (and in any case it would be easier to add new client protocol support to zkocc than to zookeeper, I think. maybe I just think that because I don't want to java.)

SEJeff 13y ago
You work for youtube and use zookeeper?

tomjohnson3 13y ago

one other thing to keep in mind is that the underlying algorithms (zab and raft) provide different guarantees. for example, zookeeper/zab allows reading directly from followers with a guarantee to get at least a past value that won't be rolled back. this was one reason zookeer didn't use paxos: https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab+vs.+Paxos https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab+vs... in my understanding, raft doesn't allow reading directly from followers, because the follower logs may get repaired/rolled-back when a new leader is elected. (though i'm sure an implementation can tweak the protocol to provide this support.) that said, raft has a lot of interesting applications, and, in my opinion, is definitely more understandable than the many versions of paxos. (implementing zab yourself, at this point, would be a futile exercise.) i found the videos from the raft user study to be very well done (and easier to understand than even their paper): raft: http://www.youtube.com/watch?v=JEpsBg0AO6o http://www.youtube.com/watch?v=JEpsBg0AO6o paxos: http://www.youtube.com/watch?v=YbZ3zDzDnrw http://www.youtube.com/watch?v=YbZ3zDzDnrw ...however, i think they did paxos a disadvantage by not just focusing on multi-paxos (which is probably the most common implementation). but, it's certainly fair to say that info about paxos is spread out far and wide...with perhaps too many knobs to turn and implementation-related details to fill in yourself. as a side note: i've just started implementing raft in a set of libraries (multiple languages) that will be open source - along with other protocols.

ideal0227 13y ago
One thing is not correct. Raft allows to read from followers. You will never be able to read uncommitted value in raft.
babo 13y ago
> raft doesn't allow reading directly from followers, In raft all client connections to followers redirected to use the current master. > i think they did paxos a disadvantage by not just focusing on multi-paxos Raft is equivalent to (multi-)Paxos
- nieksand 13y ago
  Having followers redirect to the leader is how the paper describes the algorithm. But I don't think there is anything stopping you from having followers service committed log entry reads, provided you're willing to live with being out-of-date.
  - tomjohnson3 13y ago
    i don't think this is the case. here is a link to a video by one of the authors about log repair during leader election: http://www.youtube.com/watch?feature=player_detailpage&v=YbZ3zDzDnrw#t=1481 http://www.youtube.com/watch?feature=player_detailpage&v=YbZ... log entries on a follower may get rolled back - and thrown out - since they were not accepted on a majority of followers.
- codexon 13y ago
  Does this mean that there is no read scaling? The master server must be able to handle the entire read load?
benbjohnson 13y ago
I'm the original author of go-raft (the implementation used in etcd) so I'll try to address some of the points in this thread. Raft only updates the state of the system once log entries are committed to a quorum the local state will never be rolled back. Log entries can be thrown out but they haven't been committed to the local state so it doesn't matter. You can read from the leader if you need to ensure linearizability but that will kill your read scaling. Another approach is to read locally and check that the local raft node isn't in a "candidate" state (which would mean that it hasn't received a heartbeat from the master within the last 150ms). That approach works for a lot of cases. As far as implementing multi-paxos, the authors behind Google Chubby have talked about how there is a large divide between theoretical multi-paxos and actually implementing multi-paxos. Also, there aren't any standalone multi-paxos Go libraries available. I wrote go-raft at the time because there wasn't an alternative distributed consensus library in Go at the time. Let me know if you need an extra pair of eyes on your Raft implementation or if you have any questions (ben@skylandlabs.com).

voidmain 13y ago

I agree. At FoundationDB, we're writing a coordination tool (working name "beastmaster") that provides service discovery, locking, leader election, etc on top of our transactional key/value store. It is higher level than these tools; our idea is to try to make it useful from a command line or DNS rather than have to be baked into every piece of software that needs service discovery. beast service --lock name=mailserver.foundationdb.com --run mailserver.sh & beast service name=webserver.foundationdb.com --run webserver.sh & ping mailserver.foundationdb.com wget http://webserver.foundationdb.com/

enigmo 13y ago
That's how our Chubby clone works... it uses the Route53 API to publish service endpoints and leaders.
- fintler 13y ago
  I'm also in the club of people building one of these things (mine is on top of LevelDB and written in ANSI C with a lock-free design). My most recent blog post is on getting the thing to bootstrap: http://www.bringhurst.org/2013/09/09/consensus-quorum-bootstrapping-101.html http://www.bringhurst.org/2013/09/09/consensus-quorum-bootst...
  - enigmo 13y ago
    Ours is on top of LevelDB too but "lock free" in Haskell using the STM.
    - jeremyjh 13y ago
      Is this open source?
      - enigmo 13y ago
        Not yet... it'll land here when it's ready https://github.com/alphaHeavy https://github.com/alphaHeavy

kapilvt 13y ago

Another issue with zookeeper, the c bindings which are the basis for alot of language bindings, is both buggy and often out of date, esp wrt to individual language bindings. I've often see internal cpu spiked loads on c lib's internal io thread, ie pathological behavior. The recent support for dynamic quorums or transactions has yet to make it to any of the distributed language bindings. At least for python the folks at mozilla put together a nice from scratch library in the form of kazoo, at the cost of reimplementing the entire protocol. Compare that to the simplicity of etcd's, where you can just curl/wget a request.