6 ms·

Docker Containers at Scale: Our Take on Docker Swarm

95 points by presspot 12y ago

rcarmo 12y ago
I'm a bit concerned about the network layer (or, rather, lack thereof). I've looked at one or two of the "overlay network" approaches (that essentially have a container act as a router/tunneller between hosts), and wish there was something in Swarm that let me do basic port-mapping/load balancing across containers on multiple hosts (preferably with auto-scaling) without funkiness and undue overhead.
carlivar 12y ago
If only Mesos would drop the Zookeeper requirement. We've found Zookeeper to be overly complicated and difficult to troubleshoot.
- lclarkmichalek 12y ago
  Really? I've found it a lot easier to manage than etcd. The fact that you manually specify all of your nodes in the config file removes a whole host of errors you can create in etcd with its fancy discovery stuff. What's your set up? How many zookeeper nodes do you run? What problems have you run into?
  - kelseyhightower 12y ago
    You bring up a really good point. Many of the docs on etcd promote the use of the discovery service[1] as a convenient way of bootstrapping an etcd cluster -- really useful when you don't know the IP addresses of each cluster member up front. The discovery bootstrap method is also great for demos and testing environments, but as you correctly highlight this is not the ideal way to run a production setup. With the release of etcd 2.0, I strongly recommend using the static bootstrap[2] method for provisioning an etcd cluster. The static bootstrap method provides the key documentation clues that help you reason about an etcd installation. Finally, etcd 2.0 introduces support for bootstrapping using DNS[3], which provides the convenience of the discovery bootstrap, and the explicitness of the static bootstrap. [1] https://coreos.com/docs/cluster-management/setup/cluster-discovery/ https://coreos.com/docs/cluster-management/setup/cluster-dis... [2] https://github.com/coreos/etcd/blob/master/Documentation/clustering.md#static https://github.com/coreos/etcd/blob/master/Documentation/clu... [3] https://github.com/coreos/etcd/blob/master/Documentation/clustering.md#dns-discovery https://github.com/coreos/etcd/blob/master/Documentation/clu...
  - carlivar 12y ago
    We would get increased latency interacting with Zookeeper over time until eventually it would completely fail. The log messages when this would happen (either latency or failure) were extremely unhelpful. The logging server-side for Zookeeper in fact I found downright terrible. We wound up proactively restarting the ZK cluster regularly, which improved stability. Granted, it was our own software written to use it, and we suspect there were problems with the way it was written. It was easier for us to just rip it out than debug, however. I find it overly complicated to write against given the need for thick clients. Consul phrases it well (https://consul.io/intro/vs/zookeeper.html https://consul.io/intro/vs/zookeeper.html): "ZooKeeper provides ephemeral nodes which are K/V entries that are removed when a client disconnects. These are more sophisticated than a heartbeat system, but also have inherent scalability issues and add client side complexity. All clients must maintain active connections to the ZooKeeper servers, and perform keep-alives. Additionally, this requires "thick clients", which are difficult to write and often result in difficult to debug issues." Edit: in retrospect our problems might have been solved by turning syncing off, as described here: http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/zookeeper_psuedo_scalability_and_absolute http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/zo... But you can figure out from the above how scalable Zookeeper is... not very. We run in physical datacenters and I certainly wouldn't be thrilled about building out snowflake RAID systems just for our ZK clusters (we generally try to use whitebox commodity hardware and we want individual nodes to be as disposable as possible). It's ironic that ZK requires such consistency when the goals of Mesos are exactly the opposite.
    - mdellabitta 12y ago
      > It's ironic that ZK requires such consistency when the goals of Mesos are exactly the opposite. Is it ironic? Seems to me that if you want to have a distributed cluster that can deal with worker failure and still be useful, you need to rely on something to durably maintain your state at a lower level.
      - carlivar 12y ago
        I agree and I want that thing not to be ZK. Consul or etcd would be fine.
- redmar 12y ago
  etcd support for Mesos is almost there. see: https://issues.apache.org/jira/browse/MESOS-1806 https://issues.apache.org/jira/browse/MESOS-1806 the code is already reviewable so it won't take long before this gets released. AFAIK it allows you to replace ZK for etcd making it a lot easier to run this on top of coreos as a working etcd cluster is a fact in a working coreos cluster.
phildougherty 12y ago
If you've spent any significant amount of time with Mesos and Marathon, you've probably found that it's buggy (Marathon at least), complicated, and hard to work with. Zookeeper is just one part of the complicated web of infrastructure you need to stand up. To top it off you end up needing to run something like Netflix' Exhibitor to have any shred of confidence that if ZK goes away the state of your cluster will be recoverable. It definitely "works", but certainly leaves something to be desired in the stability and ease of use department.
- otterley 12y ago
  I'm curious about just how difficult managing ZK is. My initial experience was pretty pleasant. Sure, it doesn't have a pretty admin UI (Exhibitor fills in that gap), but it's not like the consensus is that ZK is unreliable or doesn't live up to its utility claims. So, what exactly am I missing here?
  - phildougherty 12y ago
    I wasn't picking on ZK in particular. I was more pointing out that the system as a whole is complicated. It might be fine for "enterprise deployments" where a company has a lot of time, money and resources to throw at the implementation. In my opinion the average end user is going to get lost in the details of how these systems interconnect and work together. Debugging some of the more complex issues that they will face will be challenging. Even Exhibitor is not a silver bullet for managing ZK. It helps with rolling restarts and lets you do a sort of MySQL-esque bin-log replay of your state, but even with Exhibitor, it's not going to make for a smooth recovery at 4:00AM when PagerDuty rings. The underlying design of how Mesos and Marathon communicate create situations where they have differing views of the state of the cluster, and you end up with "orphans" which are tasks that Mesos is aware of but Marathon knows nothing about. In my opinion, the system feels like it was designed to be something else, and then all this functionality was tacked on later. Coincidentally that is exactly what the case is with Mesos. I think there is a lot of room for an improved user experience that these tools will struggle to provide.
    - 12y ago
afarrell 12y ago
Why is it called Docker Swarm rather than Docker Fleet or something similarly logistics-related?
- rubiquity 12y ago
  CoreOS already has a project named Fleet[0]. [0] - https://github.com/coreos/fleet https://github.com/coreos/fleet
polynomial 12y ago
If you're in NYC, there will be a talk tonight on Building & Deploying Applications to Apache Mesos at the Digital Ocean meetup: http://eventhunt.io/node/5984 http://eventhunt.io/node/5984.

95 points by presspot 12y ago

I'm a bit concerned about the network layer (or, rather, lack thereof). I've looked at one or two of the "overlay network" approaches (that essentially have a container act as a router/tunneller between hosts), and wish there was something in Swarm that let me do basic port-mapping/load balancing across containers on multiple hosts (preferably with auto-scaling) without funkiness and undue overhead.

carlivar 12y ago

If only Mesos would drop the Zookeeper requirement. We've found Zookeeper to be overly complicated and difficult to troubleshoot.

lclarkmichalek 12y ago
Really? I've found it a lot easier to manage than etcd. The fact that you manually specify all of your nodes in the config file removes a whole host of errors you can create in etcd with its fancy discovery stuff. What's your set up? How many zookeeper nodes do you run? What problems have you run into?
- kelseyhightower 12y ago
  You bring up a really good point. Many of the docs on etcd promote the use of the discovery service[1] as a convenient way of bootstrapping an etcd cluster -- really useful when you don't know the IP addresses of each cluster member up front. The discovery bootstrap method is also great for demos and testing environments, but as you correctly highlight this is not the ideal way to run a production setup. With the release of etcd 2.0, I strongly recommend using the static bootstrap[2] method for provisioning an etcd cluster. The static bootstrap method provides the key documentation clues that help you reason about an etcd installation. Finally, etcd 2.0 introduces support for bootstrapping using DNS[3], which provides the convenience of the discovery bootstrap, and the explicitness of the static bootstrap. [1] https://coreos.com/docs/cluster-management/setup/cluster-discovery/ https://coreos.com/docs/cluster-management/setup/cluster-dis... [2] https://github.com/coreos/etcd/blob/master/Documentation/clustering.md#static https://github.com/coreos/etcd/blob/master/Documentation/clu... [3] https://github.com/coreos/etcd/blob/master/Documentation/clustering.md#dns-discovery https://github.com/coreos/etcd/blob/master/Documentation/clu...
- carlivar 12y ago
  We would get increased latency interacting with Zookeeper over time until eventually it would completely fail. The log messages when this would happen (either latency or failure) were extremely unhelpful. The logging server-side for Zookeeper in fact I found downright terrible. We wound up proactively restarting the ZK cluster regularly, which improved stability. Granted, it was our own software written to use it, and we suspect there were problems with the way it was written. It was easier for us to just rip it out than debug, however. I find it overly complicated to write against given the need for thick clients. Consul phrases it well (https://consul.io/intro/vs/zookeeper.html https://consul.io/intro/vs/zookeeper.html): "ZooKeeper provides ephemeral nodes which are K/V entries that are removed when a client disconnects. These are more sophisticated than a heartbeat system, but also have inherent scalability issues and add client side complexity. All clients must maintain active connections to the ZooKeeper servers, and perform keep-alives. Additionally, this requires "thick clients", which are difficult to write and often result in difficult to debug issues." Edit: in retrospect our problems might have been solved by turning syncing off, as described here: http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/zookeeper_psuedo_scalability_and_absolute http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/zo... But you can figure out from the above how scalable Zookeeper is... not very. We run in physical datacenters and I certainly wouldn't be thrilled about building out snowflake RAID systems just for our ZK clusters (we generally try to use whitebox commodity hardware and we want individual nodes to be as disposable as possible). It's ironic that ZK requires such consistency when the goals of Mesos are exactly the opposite.
  - mdellabitta 12y ago
    > It's ironic that ZK requires such consistency when the goals of Mesos are exactly the opposite. Is it ironic? Seems to me that if you want to have a distributed cluster that can deal with worker failure and still be useful, you need to rely on something to durably maintain your state at a lower level.
    - carlivar 12y ago
      I agree and I want that thing not to be ZK. Consul or etcd would be fine.
redmar 12y ago
etcd support for Mesos is almost there. see: https://issues.apache.org/jira/browse/MESOS-1806 https://issues.apache.org/jira/browse/MESOS-1806 the code is already reviewable so it won't take long before this gets released. AFAIK it allows you to replace ZK for etcd making it a lot easier to run this on top of coreos as a working etcd cluster is a fact in a working coreos cluster.

phildougherty 12y ago

If you've spent any significant amount of time with Mesos and Marathon, you've probably found that it's buggy (Marathon at least), complicated, and hard to work with. Zookeeper is just one part of the complicated web of infrastructure you need to stand up. To top it off you end up needing to run something like Netflix' Exhibitor to have any shred of confidence that if ZK goes away the state of your cluster will be recoverable. It definitely "works", but certainly leaves something to be desired in the stability and ease of use department.

otterley 12y ago
I'm curious about just how difficult managing ZK is. My initial experience was pretty pleasant. Sure, it doesn't have a pretty admin UI (Exhibitor fills in that gap), but it's not like the consensus is that ZK is unreliable or doesn't live up to its utility claims. So, what exactly am I missing here?
- phildougherty 12y ago
  I wasn't picking on ZK in particular. I was more pointing out that the system as a whole is complicated. It might be fine for "enterprise deployments" where a company has a lot of time, money and resources to throw at the implementation. In my opinion the average end user is going to get lost in the details of how these systems interconnect and work together. Debugging some of the more complex issues that they will face will be challenging. Even Exhibitor is not a silver bullet for managing ZK. It helps with rolling restarts and lets you do a sort of MySQL-esque bin-log replay of your state, but even with Exhibitor, it's not going to make for a smooth recovery at 4:00AM when PagerDuty rings. The underlying design of how Mesos and Marathon communicate create situations where they have differing views of the state of the cluster, and you end up with "orphans" which are tasks that Mesos is aware of but Marathon knows nothing about. In my opinion, the system feels like it was designed to be something else, and then all this functionality was tacked on later. Coincidentally that is exactly what the case is with Mesos. I think there is a lot of room for an improved user experience that these tools will struggle to provide.
  - 12y ago

afarrell 12y ago

Why is it called Docker Swarm rather than Docker Fleet or something similarly logistics-related?

rubiquity 12y ago
CoreOS already has a project named Fleet[0]. [0] - https://github.com/coreos/fleet https://github.com/coreos/fleet

polynomial 12y ago

If you're in NYC, there will be a talk tonight on Building & Deploying Applications to Apache Mesos at the Digital Ocean meetup: http://eventhunt.io/node/5984 http://eventhunt.io/node/5984.