5 ms·

Really? I've found it a lot easier to manage than etcd. The fact that you manually specify all of your nodes in the config file removes a whole host of errors y

Really? I've found it a lot easier to manage than etcd. The fact that you manually specify all of your nodes in the config file removes a whole host of errors you can create in etcd with its fancy discovery stuff. What's your set up? How many zookeeper nodes do you run? What problems have you run into?

kelseyhightower 12y ago
You bring up a really good point. Many of the docs on etcd promote the use of the discovery service[1] as a convenient way of bootstrapping an etcd cluster -- really useful when you don't know the IP addresses of each cluster member up front. The discovery bootstrap method is also great for demos and testing environments, but as you correctly highlight this is not the ideal way to run a production setup. With the release of etcd 2.0, I strongly recommend using the static bootstrap[2] method for provisioning an etcd cluster. The static bootstrap method provides the key documentation clues that help you reason about an etcd installation. Finally, etcd 2.0 introduces support for bootstrapping using DNS[3], which provides the convenience of the discovery bootstrap, and the explicitness of the static bootstrap. [1] https://coreos.com/docs/cluster-management/setup/cluster-discovery/ https://coreos.com/docs/cluster-management/setup/cluster-dis... [2] https://github.com/coreos/etcd/blob/master/Documentation/clustering.md#static https://github.com/coreos/etcd/blob/master/Documentation/clu... [3] https://github.com/coreos/etcd/blob/master/Documentation/clustering.md#dns-discovery https://github.com/coreos/etcd/blob/master/Documentation/clu...
carlivar 12y ago
We would get increased latency interacting with Zookeeper over time until eventually it would completely fail. The log messages when this would happen (either latency or failure) were extremely unhelpful. The logging server-side for Zookeeper in fact I found downright terrible. We wound up proactively restarting the ZK cluster regularly, which improved stability. Granted, it was our own software written to use it, and we suspect there were problems with the way it was written. It was easier for us to just rip it out than debug, however. I find it overly complicated to write against given the need for thick clients. Consul phrases it well (https://consul.io/intro/vs/zookeeper.html https://consul.io/intro/vs/zookeeper.html): "ZooKeeper provides ephemeral nodes which are K/V entries that are removed when a client disconnects. These are more sophisticated than a heartbeat system, but also have inherent scalability issues and add client side complexity. All clients must maintain active connections to the ZooKeeper servers, and perform keep-alives. Additionally, this requires "thick clients", which are difficult to write and often result in difficult to debug issues." Edit: in retrospect our problems might have been solved by turning syncing off, as described here: http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/zookeeper_psuedo_scalability_and_absolute http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/zo... But you can figure out from the above how scalable Zookeeper is... not very. We run in physical datacenters and I certainly wouldn't be thrilled about building out snowflake RAID systems just for our ZK clusters (we generally try to use whitebox commodity hardware and we want individual nodes to be as disposable as possible). It's ironic that ZK requires such consistency when the goals of Mesos are exactly the opposite.
- mdellabitta 12y ago
  > It's ironic that ZK requires such consistency when the goals of Mesos are exactly the opposite. Is it ironic? Seems to me that if you want to have a distributed cluster that can deal with worker failure and still be useful, you need to rely on something to durably maintain your state at a lower level.
  - carlivar 12y ago
    I agree and I want that thing not to be ZK. Consul or etcd would be fine.
    - mdellabitta 12y ago
      Well, you've described how zk doesn't meet the needs of your software as written, but I'm not sure you've established why Mesos would be better off without it...
      - carlivar 12y ago
        Yes, I have. Zookeeper's logging (and thus troubleshooting ability) is atrocious.
        mdellabitta 12y ago
        And yet somehow they seem to have gotten it to work. I run a few Zookeeper-based systems where ZK just seems to work and I don't have to look at logging statements, either. I've never dealt with Zookeeper downtime that wasn't Amazon's fault.
        carlivar 12y ago
        So you are having ZK downtime? Such a system shouldn't ever really go down, even with Amazon problems. We strive for software in which individual nodes can be flakey or down without impacting the uptime of the whole cluster. ZK doesn't fit this criteria, as you have just stated. I prefer gossip-based Raft systems, for example (which is why I like Consul so much. We're also big Riak users).

kelseyhightower 12y ago

You bring up a really good point. Many of the docs on etcd promote the use of the discovery service[1] as a convenient way of bootstrapping an etcd cluster -- really useful when you don't know the IP addresses of each cluster member up front. The discovery bootstrap method is also great for demos and testing environments, but as you correctly highlight this is not the ideal way to run a production setup. With the release of etcd 2.0, I strongly recommend using the static bootstrap[2] method for provisioning an etcd cluster. The static bootstrap method provides the key documentation clues that help you reason about an etcd installation. Finally, etcd 2.0 introduces support for bootstrapping using DNS[3], which provides the convenience of the discovery bootstrap, and the explicitness of the static bootstrap. [1] https://coreos.com/docs/cluster-management/setup/cluster-discovery/ https://coreos.com/docs/cluster-management/setup/cluster-dis... [2] https://github.com/coreos/etcd/blob/master/Documentation/clustering.md#static https://github.com/coreos/etcd/blob/master/Documentation/clu... [3] https://github.com/coreos/etcd/blob/master/Documentation/clustering.md#dns-discovery https://github.com/coreos/etcd/blob/master/Documentation/clu...

carlivar 12y ago

We would get increased latency interacting with Zookeeper over time until eventually it would completely fail. The log messages when this would happen (either latency or failure) were extremely unhelpful. The logging server-side for Zookeeper in fact I found downright terrible. We wound up proactively restarting the ZK cluster regularly, which improved stability. Granted, it was our own software written to use it, and we suspect there were problems with the way it was written. It was easier for us to just rip it out than debug, however. I find it overly complicated to write against given the need for thick clients. Consul phrases it well (https://consul.io/intro/vs/zookeeper.html https://consul.io/intro/vs/zookeeper.html): "ZooKeeper provides ephemeral nodes which are K/V entries that are removed when a client disconnects. These are more sophisticated than a heartbeat system, but also have inherent scalability issues and add client side complexity. All clients must maintain active connections to the ZooKeeper servers, and perform keep-alives. Additionally, this requires "thick clients", which are difficult to write and often result in difficult to debug issues." Edit: in retrospect our problems might have been solved by turning syncing off, as described here: http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/zookeeper_psuedo_scalability_and_absolute http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/zo... But you can figure out from the above how scalable Zookeeper is... not very. We run in physical datacenters and I certainly wouldn't be thrilled about building out snowflake RAID systems just for our ZK clusters (we generally try to use whitebox commodity hardware and we want individual nodes to be as disposable as possible). It's ironic that ZK requires such consistency when the goals of Mesos are exactly the opposite.

mdellabitta 12y ago
> It's ironic that ZK requires such consistency when the goals of Mesos are exactly the opposite. Is it ironic? Seems to me that if you want to have a distributed cluster that can deal with worker failure and still be useful, you need to rely on something to durably maintain your state at a lower level.
- carlivar 12y ago
  I agree and I want that thing not to be ZK. Consul or etcd would be fine.
  - mdellabitta 12y ago
    Well, you've described how zk doesn't meet the needs of your software as written, but I'm not sure you've established why Mesos would be better off without it...
    - carlivar 12y ago
      Yes, I have. Zookeeper's logging (and thus troubleshooting ability) is atrocious.
      - mdellabitta 12y ago
        And yet somehow they seem to have gotten it to work. I run a few Zookeeper-based systems where ZK just seems to work and I don't have to look at logging statements, either. I've never dealt with Zookeeper downtime that wasn't Amazon's fault.
        carlivar 12y ago
        So you are having ZK downtime? Such a system shouldn't ever really go down, even with Amazon problems. We strive for software in which individual nodes can be flakey or down without impacting the uptime of the whole cluster. ZK doesn't fit this criteria, as you have just stated. I prefer gossip-based Raft systems, for example (which is why I like Consul so much. We're also big Riak users).