4 ms·

How does etcd compare to zookeeper? What made etcd the choice for kubernetes and cncf?

by polskibus 8y ago

guipsp 8y ago
zookeeper uses a coordination kernel, while etcd uses a replicated state machine.
- polskibus 8y ago
  What are the pros and cons? Should etcd be the default whenever one is thinking about using something like that?
  - dkoston 8y ago
    It really depends on your use case but one of the main "pros" of etcd is the narrow latency band when writing. This article is likely biased to the good parts of etcd as it's written by coreOS but you can see how the latency of writes in etcd is very consistent compared to the wide range of latencies experienced writing to ZooKeeper or Consul: https://coreos.com/blog/performance-of-etcd.html https://coreos.com/blog/performance-of-etcd.html There are other "pros" related to the fact that it's been designed for "cloud native" architectures like kubernetes. For example, FoundationDB can perform on average at sub-milisecond latency for writes (https://apple.github.io/foundationdb/benchmarking.html https://apple.github.io/foundationdb/benchmarking.html) versus 1.6ms on etcd however configuring FoudationDB to run programmatically is challenging as it was designed in an environment where ops people rack physical servers. All key/value stores have good points and bad points but that's in relation to your use case. If write or read throughput isn't the most important, say it's consistency or availability, you may make a different choice about what are "pros" and what are "cons". Another "pro" or "con" may be the language its written in or how it runs or deploys. If you run a Java shop and have tons of experience writing and deploying Java code, it may be in your best interest to be able to have more control by using a project written in Java. conversely, if you have all go engineers, you may want a project written in go. If you only have junior engineers, you may want whatever is easiest to operate and deploy.
philips 8y ago
A few reasons we didn't use ZK at the time (some of these are out of date). Zookeeper had: - No TLS security story - An abandoned RPC/serialization system that was hard to use in other languages - A consensus algorithm that differed from systems described in literature - A large RAM footprint Awhile ago some etcd engineers made an experiment in fact to try and run ZK client protocol on etcd with a proxy: https://github.com/etcd-io/zetcd https://github.com/etcd-io/zetcd https://coreos.com/blog/introducing-zetcd https://coreos.com/blog/introducing-zetcd Today, etcd performs much better than ZK and I believe it is much more widely deployed with a wider set of engaged users. https://coreos.com/blog/performance-of-etcd.html https://coreos.com/blog/performance-of-etcd.html
- smarterclayton 8y ago
  Pre 3.5.0 zookeeper reconfiguration of a running cluster was also much harder - that was a significant discussion point on Kubernetes when we had the etcd vs (anything) discussions early after open sourcing. I still think etcd total ordering over history also made reasoning about changes in the system while we were writing the first versions of the controllers and caches and list-watch loops. ZK had partial order, and I was leery of that at the time.

How does etcd compare to zookeeper? What made etcd the choice for kubernetes and cncf?

guipsp 8y ago

zookeeper uses a coordination kernel, while etcd uses a replicated state machine.

polskibus 8y ago
What are the pros and cons? Should etcd be the default whenever one is thinking about using something like that?
- dkoston 8y ago
  It really depends on your use case but one of the main "pros" of etcd is the narrow latency band when writing. This article is likely biased to the good parts of etcd as it's written by coreOS but you can see how the latency of writes in etcd is very consistent compared to the wide range of latencies experienced writing to ZooKeeper or Consul: https://coreos.com/blog/performance-of-etcd.html https://coreos.com/blog/performance-of-etcd.html There are other "pros" related to the fact that it's been designed for "cloud native" architectures like kubernetes. For example, FoundationDB can perform on average at sub-milisecond latency for writes (https://apple.github.io/foundationdb/benchmarking.html https://apple.github.io/foundationdb/benchmarking.html) versus 1.6ms on etcd however configuring FoudationDB to run programmatically is challenging as it was designed in an environment where ops people rack physical servers. All key/value stores have good points and bad points but that's in relation to your use case. If write or read throughput isn't the most important, say it's consistency or availability, you may make a different choice about what are "pros" and what are "cons". Another "pro" or "con" may be the language its written in or how it runs or deploys. If you run a Java shop and have tons of experience writing and deploying Java code, it may be in your best interest to be able to have more control by using a project written in Java. conversely, if you have all go engineers, you may want a project written in go. If you only have junior engineers, you may want whatever is easiest to operate and deploy.

philips 8y ago

A few reasons we didn't use ZK at the time (some of these are out of date). Zookeeper had: - No TLS security story - An abandoned RPC/serialization system that was hard to use in other languages - A consensus algorithm that differed from systems described in literature - A large RAM footprint Awhile ago some etcd engineers made an experiment in fact to try and run ZK client protocol on etcd with a proxy: https://github.com/etcd-io/zetcd https://github.com/etcd-io/zetcd https://coreos.com/blog/introducing-zetcd https://coreos.com/blog/introducing-zetcd Today, etcd performs much better than ZK and I believe it is much more widely deployed with a wider set of engaged users. https://coreos.com/blog/performance-of-etcd.html https://coreos.com/blog/performance-of-etcd.html

smarterclayton 8y ago
Pre 3.5.0 zookeeper reconfiguration of a running cluster was also much harder - that was a significant discussion point on Kubernetes when we had the etcd vs (anything) discussions early after open sourcing. I still think etcd total ordering over history also made reasoning about changes in the system while we were writing the first versions of the controllers and caches and list-watch loops. ZK had partial order, and I was leery of that at the time.