12 ms·

The Basics: Kubernetes, Mesosphere, and Docker Swarm

301 points by CrankyBear 10y ago

stuntkite 10y ago
I've since moved on to Kubernetes and SaltStack, but I got into production Docker containers with Deis[1] a year ago. With full Heroku buildpack support, Kubernetes backed, and great container tools. I feel like it's sad that this project doesn't get more love. It's a 20 minute deploy to GKE or AWS (I recommend GKE). If you run OSX, TheNewNormal's solo and cluster Kube setup is a breeze and has the Deis Workflow as a one command install, then you have full local parity with deployment. When I started with straight Docker figuring out the transition to the real world was be really daunting. Getting into a real production environment that I could compost quickly as I learned learn the paradigms was invaluable. Articles like this seem to buzzword their way into making the path muddier than it actually is. Another big help for me was any time I needed to set something up, first thing, I'd look at how the bleeding edge was being done with Alpine. Practically you might not always run that light, but that community has all the answers. [1]https://deis.com/ https://deis.com/ [2]https://github.com/TheNewNormal/kube-solo-osx https://github.com/TheNewNormal/kube-solo-osx [3]https://github.com/TheNewNormal/kube-cluster-osx https://github.com/TheNewNormal/kube-cluster-osx
- tedmiston 10y ago
  When I first heard about Deis it was heavily compared to Flynn [1]. It's a little hard to tell from the outside if the two are still competitors or if Deis is focusing on a lower level of abstraction. https://flynn.io/ https://flynn.io/
  - hackerboos 10y ago
    They both support Heroku-like buildpacks. Major difference IMO is that Flynn supports databases and other stateful services. I evaluated Flynn, Deis and Dokku and wasn't entirely satisfied with any of them opting for Heroku instead. I've promised people a blogpost on this and I will do it once I get time to evaluate RancherOS too.
    - baronseng 10y ago
      Can you evaluate Kontena as well while you are at it?
  - tuananh 10y ago
    Flynn was much less polished than Deis. At least, that was when i tried it last year. I also started with Deis and move on to k8s later on.
- bogomipz 10y ago
  >"I've since moved on to Kubernetes and SaltStack" I curious how those relate to each other. Are you using SaltStack to bootstrap Kubernetes?
  - lobster_johnson 10y ago
    Not the OP, but we use Google Container Engine (hosted Kubernetes), with Salt for the non-GKE VMs. This is needed because K8s is not mature enough to host all the things. In particular, stateful sets are still in beta. Not sure if I would trust K8s to run our production databases even with stateful sets. We've had K8s kill pods for unknown reasons, for example, and the volume handling has also been historically a bit flaky. Fine for completely redundant, stateless containers, less fine for stateless ones.
    - bogomipz 10y ago
      Sure that makes sense. Statefuf especially databases would make me nerovous as well. am curious, what are the advantages of GKE over deploying K8 on AWS? Or were you already GCE?
      - lobster_johnson 10y ago
        Our production setup is actually still on Digital Ocean; we're currently testing GCP (GKE + Salt-managed VMs) with a staging cluster to see how it works in practice. Before GCP, we set up a staging cluster on AWS. It was okay. The biggest pain point is that AWS's VPC does not match Kubernetes' requirement for per-pod IPs, so anyone who installs on AWS ends up setting up their own overlay network (such as Flannel or Calico). That's a big downside, because VPC isn't fun to deal with. You don't really notice how antiquated AWS is until you move to GCP. Everything — networking, UI, CLI tools, etc. — feels more modern, sleeker and less creaky. One area where GCP is particularly superior is networking. Google gives you a Layer 3 SDN (virtual network) that's more flexible than the rigid subnet-carving you need to do with AWS VPC. The tag-based firewall rules are also a breath of fresh air after AWS's weird, aging "security group" model. It's not all innovation, of course. Some services are carbon-copy clones of AWS counterparts: Pub/Sub is essentially SQS, Cloud Storage is S3, and so on, with only minor improvements along the way. Cloud Storage, for example, doesn't fix S3's lack of queryability. I've also been distinctly unimpressed with the "StackDriver"-branded suite of services, which do things like logging and metrics. I don't know why anyone in this day and age would bother making something that doesn't compare favourably to Prometheus. I should add that the security situation on GKE could be better: * GKE's Docker containers run in privileged mode. * There's still no role-based authentication. * Containers end up getting access to a lot of privileged APIs because they inherit the same IAM role as the machine. * You can't disable the automatic K8s service account mount [1]. Another scary thing, unrelated to GKE, is that the VMs run a daemon that automatically creates users with SSH keys. As a team member, you can SSH into any box with any user name. Not sure if real security weakness, but I don't like it. I love the CLI tools ("gcloud" and so on), much nicer than awscli etc, and hopefully much friendlier to junior devs. [1] You can disable the mount by mounting an "emptyDir" on top, but that makes K8s think the pod has local data, which causes the autoscaler to refuse to tear down a node. Fortunately there's an option coming to disable the service account.
lowbloodsugar 10y ago
Unfortunately, Mesosphere is no longer focusing on Marathon as an Apache Mesos framework and is instead integrating it into DC/OS, their commercial offering. Open-source Mesos+Marathon no longer appears to be an option.
- nemothekid 10y ago
  Isn't DC/OS now opensource? (https://github.com/dcos/dcos https://github.com/dcos/dcos)
  - ssk2 10y ago
    Yup! https://mesosphere.com/blog/2016/04/19/open-source-dcos/ https://mesosphere.com/blog/2016/04/19/open-source-dcos/ (I am a Mesosphere employee.)
    - SEJeff 10y ago
      Will any of the commercial only stuff (like the distributed firewall or ldap authentication / authorization bits) ever be open sourced, or will DC/OS forever be open core crippleware?
      - ssk2 10y ago
        That's a little harsh. I don't think the intention is to be open core crippleware - there are certainly a good number of users using (open) DC/OS in production without the enterprise features that you mention. I can't speak to the specific roadmap since I am uninformed but generally I believe the trend is for most enterprise features to trickle down into open over time.
    - poooogles 10y ago
      Care to comment on GP?
      - ssk2 10y ago
        We're very much committed to the Marathon open source community and keeping them happy. I'm afraid I don't know what the plans are for the project (not working on Marathon myself) but being the most popular open source project run and maintained by Mesosphere, I have no doubt that we will continue adding new features and functionality.
tom_pulo 10y ago
With all of this talk about docker, Kubernetes and the rest I feel peer pressured into ditching my monolithic heroku rails app and switching to the distributed services heaven that docker seems too advertise. Can anybody that has made the switch give me a convincing argument about why I should switch to (or not)? My feeling that docker is great if you are VP of Engineering at Netflix, but is probably not the best thing if you are starting a startup and just need to get things done. Disclaimer: I'm not religious about this and I'm totally open to being convinced that I'm wrong.
- wstrange 10y ago
  It sounds like your current setup is working just fine, so I don't see a compelling reason to switch. When you start to deploy many applications, and they need to talk to each other, and you need service discovery, automatic restart, rolling upgrades, horizontal scaling, etc - then Kubernetes brings a lot of value.
- ekidd 10y ago
  > With all of this talk about docker, Kubernetes and the rest I feel peer pressured into ditching my monolithic heroku rails app and switching to the distributed services heaven that docker seems too advertise. I successfully run lots of Docker microservices in production, and I strongly advise you to keep your app on Heroku as long as you can. :-) Microservices make sense in two circumstances: 1. You have multiple teams of developers, and you want them to have different release cycles and loosely-coupled APIs. In this case, you can let each team have a microservice. 2. There's a module in your app which is self-contained and naturally isolated, with a stable API. You could always just make this a separate Heroku app with its own REST API. But in general, microservices add complexity and make it harder to do certain kinds of refactorings. You can make them work, if you know what you're doing. For example, ECS +RDS+ALBs is halfway civilized, especially if you manage the configuration with Terraform, and set up a CI server to build Docker images and run tests. But it's still a lot more complex than a single, well-refactored app on Heroku.
- ktta 10y ago
  If you're a startup, then I'd look at the growth you're expecting. Containers scale well, and when you're big, maintaining multiple heroku apps eats into developer time that can be better spent somewhere else. Of course, if you've just started, and are getting an MVP out the door, don't worry about docker just yet. And also don't listen to the microservices people. It'll be like putting the cart before the horse.
jaaron 10y ago
Title really shouldn't include "Mesosphere." It should be "Marathon" or "DC/OS". Or, if you keep Mesosphere, it should be "Google, Mesosphere, and Docker." Aside from that, it's not a bad overview. I particularly like this quote: “Mesos and Kubernetes are largely aimed at solving similar problems of running clustered applications; they have different histories and different approaches to solving the problem. Mesos focuses its energy on very generic scheduling and plugging in multiple different schedulers.” In contrast, says Burns, “Kubernetes was designed from the ground up to be an environment for building distributed applications from containers. It includes primitives for replication and service discovery as core primitives, whereas such things are added via frameworks in Mesos. … Swarm is an effort by Docker to extend the existing Docker API to make a cluster of machines look like a single Docker API.” Mesos is useful when you have requirements that don't fit in k8s' rather opinionated world view. If you can use k8s, it's great, but it's also changing rapidly, so be aware you're on the bleeding edge. DockerSwarm is Docker finding a business model. (I'm biased here)
- lowbloodsugar 10y ago
  Better bleeding edge than end-of-life. We're using Marathon, and suddenly the entire UI is end-of-life, and your options are "use Marathon back-end only without a UI" or "migrate to DC/OS" which is even more opinionated than K8s. I think the real tell is that Mesosphere don't have the cash/people to maintain a non-DC/OS UI for marathon, and there isn't a community able/willing to do it for them.
  - XorNot 10y ago
    About a year ago there was a decision made in our project to "just use Marathon" (our technical lead has a bit of a tendency to declare things as simple, without thinking too hard about operational problems) - a year later (...don't ask...) and I very much have the feeling we've backed the wrong horse. Conversely...I'm deeply suspicious of excited discussions of distributed systems these days because it doesn't seem like a lot of people are truly running them "at scale" - there's not a lot of write-ups about how people handle 300+ node clusters, and probably more importantly how they handle them (how do their internal users use them).
technologyvault 10y ago
Great article! I just made a point to get caught up Docker and related elements of the containerization world. Mesosphere looks like it works similar to Nanobox (http://nanobox.io http://nanobox.io)
bogomipz 10y ago
>" I’ve spoken with several DevOps who argue that compared with running a Kubernetes or Mesos cluster, Docker Swarm is a snap." I'm always leary of statements like that. My experience with cluster managers(Yarn, Mesos) and distributed systems in general is that they are almost never "a snap" to run once you move past trivial workloads and requirements.
- atomi 10y ago
  Not many startups move past trivial workloads. And if you do, it's a good problem to have. If swarm can get your mvp up quickly, it might be a good choice.
- btmiller 10y ago
  Yep, that's my exact experience with Swarm (1.13). I want the higher-level abstractions like K8s Pods and Deployments, but I absolutely cannot wait around for Swarm to stabilize those kind of things.
- Axsuul 10y ago
  My experience with Docker Swarm has been great. I came from Kubernetes but found it way too complex for my needs.
- amouat 10y ago
  The author is (mis)quoting me there. I meant the initial process of installing and getting Swarm running is much easier than k8s or Mesos, not that on-going maintenance was easier. Also, since I wrote that, k8s has done a lot of work to simplify installation with kubeadm and other improvements.
  - bogomipz 10y ago
    Ah OK. Yeah doing a greenfield deployment is one thing, "operationalizing" it is often more involved. For Mesos anyone if is interested and you are Ansivle shop you can look at Ansible Shipyard to simplify an install. https://github.com/AnsibleShipyard https://github.com/AnsibleShipyard Kubeadm looks interesting. Thanks.
ovidiup 10y ago
I built Jollyturns (https://jollyturns.com https://jollyturns.com) which has a fairly large server-side component. Before starting the work on Jollyturns I worked at Google (I left in 2010) on a variety of infrastructure projects, where I got to use first hand Borg and all the other services running on it. When I started more than 5 years ago few of the options mentioned in the article were available, so I just used Xen running on bare metal. I just wanted to get things done as opposed to forever experimenting with infrastructure. Not to mention the load was inexistent, so everything was easy to manage. After I launched the mobile app, I decided to spend some time on the infrastructure. 2 years ago I experimented with Marathon, which was originally developed at Twitter, probably by a bunch of former Google employees. The reason is Marathon felt very much like Borg: you could see the jobs you launched in a pretty nice Web interface, including their log files, resource utilization and so on. Deploying it on bare metal machines however was an exercise in frustration since Marathon relied heavily on Apache Mesos. For a former Googler, Mesos had some weird terminology and was truly difficult to understand what the heck was going on. Running Marathon on top of it had major challenges: when long-running services failed you could not figure out why things were not working. So after about 2 weeks I gave up on it, and went back to manually managed Xen VMs. Around the same time I spent few days with Docker Swarm. If you're familiar with Borg/Kubernetes, Swarm is very different from it since - at least when it started, you had to allocate services to machines by hand. I wrote it off quickly since allocating dockerized services on physical machines was not my idea of cluster management. Last year I switched to Kubernetes (version 1.2) since it's the closest to what I expect from a cluster management system. The version I've been using in production has a lot of issues: high availability (HA) for its components is almost non-existent. I had to setup Kube in such a way that its components have some resemblance of HA. In the default configuration, Kubernetes installs its control components on a single machine. If that machine fails or is rebooted your entire cluster disappears. Even with all these issues, Kubernetes solves a lot of problems for you. The flannel networking infrastructure greatly simplifies the deployment of docker containers, since you don't need to worry about routing traffic between your containers. Even now Kubernetes doesn't do HA: https://github.com/kubernetes/kubernetes/issues/26852 https://github.com/kubernetes/kubernetes/issues/26852 https://github.com/kubernetes/kubernetes/issues/18174 https://github.com/kubernetes/kubernetes/issues/18174 Don't be fooled by the title of the bug report, the same component used in kubectl to implement HA could be used inside Kube' servers for the same reason. I guess these days the Google engineers working on Kubernetes have no real experience deploying large services on Borg inside Google. Such a pity!
webo 10y ago
We've been running Kubernetes in production for over a year. It has a steep learning curve and it takes a lot of time to fine-tune the underlying infrastructure (we use CoreOs) to make it production-ready. There still seems to be a lot of shortcomings of k8s that seem like should have been addressed by now: - it is impossible to trigger a rescheduling / rebalancing. When a new node comes in (via AutoScaling policy or whatever), kubernetes doesn't do anything. Thus, the new nodes can be sitting there doing nothing. - once a pod has been scheduled onto a node, it never reschedules it anywhere else. The node may be experiencing problems, thus the pod is affected. k8s doesn't do anything to heal that -- it could simply delete the pod so it is rescheduled somewhere else. - docker itself constantly ships with a lot of bugs. To this day (we are on docker 1.12.6), we constantly have problems with the docker daemon hanging or becoming unresponsive. I'm not sure if k8s can do much about this, but I feel like it should since we don't directly control docker. - doesn't integrate more tightly with the OS / cloud provider. For example, it could perform health checks and decide if the node should be restarted, terminated, or idle. All of our services are stateless, so it would be nice to have the option for all the above, especially k8s started as being the solution for stateless apps.
- alpb 10y ago
  > - once a pod has been scheduled onto a node, it never reschedules it anywhere else. The node may be experiencing problems, thus the pod is affected. k8s doesn't do anything to heal that -- it could simply delete the pod so it is rescheduled somewhere else. Can you please elaborate this? When you are using replication controllers or deployments, don’t they drive the state to the desired/goal state, which is N replicas of a pod? So when the node is shut down, I guess it should be rescheduling those dead pods somewhere else to satisfy the goal state?
  - webo 10y ago
    You may have misunderstood me. The case I'm talking is, the node reports Ready, but the pod itself is not functioning properly. One common issue we have is the pod gets stuck in a restart loop (for whatever reason, including starvation of resources). k8s just keeps restarting it for days on that node, instead of simply rescheduling it after X restarts or some other condition. https://github.com/kubernetes/kubernetes/issues/13385 https://github.com/kubernetes/kubernetes/issues/13385
sitepodmatt 10y ago
Quoting from the disadvantages section: "Second, Kubernetes excels at automatically fixing problems. But it’s so good at it that containers can crash and be restarted so fast you don’t notice your containers are crashing. " A key part of Kubernetes is to the bring the state of the system to that of the spec, through a series of watches - and in some places polling (config maps watching in kubectl as of 1.4.x) - this may look like magic, but it's not fixing problems per se its that your desired state of the system (the spec) is different to what is observed (the status). This is not a disadvantage. Kubernetes if not for someone sitting at the terminal expecting a bell or stop the world when problems happens, although I guess you could configure it that way and fight against the system.

301 points by CrankyBear 10y ago

I've since moved on to Kubernetes and SaltStack, but I got into production Docker containers with Deis[1] a year ago. With full Heroku buildpack support, Kubernetes backed, and great container tools. I feel like it's sad that this project doesn't get more love. It's a 20 minute deploy to GKE or AWS (I recommend GKE). If you run OSX, TheNewNormal's solo and cluster Kube setup is a breeze and has the Deis Workflow as a one command install, then you have full local parity with deployment. When I started with straight Docker figuring out the transition to the real world was be really daunting. Getting into a real production environment that I could compost quickly as I learned learn the paradigms was invaluable. Articles like this seem to buzzword their way into making the path muddier than it actually is. Another big help for me was any time I needed to set something up, first thing, I'd look at how the bleeding edge was being done with Alpine. Practically you might not always run that light, but that community has all the answers. [1]https://deis.com/ https://deis.com/ [2]https://github.com/TheNewNormal/kube-solo-osx https://github.com/TheNewNormal/kube-solo-osx [3]https://github.com/TheNewNormal/kube-cluster-osx https://github.com/TheNewNormal/kube-cluster-osx

tedmiston 10y ago
When I first heard about Deis it was heavily compared to Flynn [1]. It's a little hard to tell from the outside if the two are still competitors or if Deis is focusing on a lower level of abstraction. https://flynn.io/ https://flynn.io/
- hackerboos 10y ago
  They both support Heroku-like buildpacks. Major difference IMO is that Flynn supports databases and other stateful services. I evaluated Flynn, Deis and Dokku and wasn't entirely satisfied with any of them opting for Heroku instead. I've promised people a blogpost on this and I will do it once I get time to evaluate RancherOS too.
  - baronseng 10y ago
    Can you evaluate Kontena as well while you are at it?
- tuananh 10y ago
  Flynn was much less polished than Deis. At least, that was when i tried it last year. I also started with Deis and move on to k8s later on.
bogomipz 10y ago
>"I've since moved on to Kubernetes and SaltStack" I curious how those relate to each other. Are you using SaltStack to bootstrap Kubernetes?
- lobster_johnson 10y ago
  Not the OP, but we use Google Container Engine (hosted Kubernetes), with Salt for the non-GKE VMs. This is needed because K8s is not mature enough to host all the things. In particular, stateful sets are still in beta. Not sure if I would trust K8s to run our production databases even with stateful sets. We've had K8s kill pods for unknown reasons, for example, and the volume handling has also been historically a bit flaky. Fine for completely redundant, stateless containers, less fine for stateless ones.
  - bogomipz 10y ago
    Sure that makes sense. Statefuf especially databases would make me nerovous as well. am curious, what are the advantages of GKE over deploying K8 on AWS? Or were you already GCE?
    - lobster_johnson 10y ago
      Our production setup is actually still on Digital Ocean; we're currently testing GCP (GKE + Salt-managed VMs) with a staging cluster to see how it works in practice. Before GCP, we set up a staging cluster on AWS. It was okay. The biggest pain point is that AWS's VPC does not match Kubernetes' requirement for per-pod IPs, so anyone who installs on AWS ends up setting up their own overlay network (such as Flannel or Calico). That's a big downside, because VPC isn't fun to deal with. You don't really notice how antiquated AWS is until you move to GCP. Everything — networking, UI, CLI tools, etc. — feels more modern, sleeker and less creaky. One area where GCP is particularly superior is networking. Google gives you a Layer 3 SDN (virtual network) that's more flexible than the rigid subnet-carving you need to do with AWS VPC. The tag-based firewall rules are also a breath of fresh air after AWS's weird, aging "security group" model. It's not all innovation, of course. Some services are carbon-copy clones of AWS counterparts: Pub/Sub is essentially SQS, Cloud Storage is S3, and so on, with only minor improvements along the way. Cloud Storage, for example, doesn't fix S3's lack of queryability. I've also been distinctly unimpressed with the "StackDriver"-branded suite of services, which do things like logging and metrics. I don't know why anyone in this day and age would bother making something that doesn't compare favourably to Prometheus. I should add that the security situation on GKE could be better: * GKE's Docker containers run in privileged mode. * There's still no role-based authentication. * Containers end up getting access to a lot of privileged APIs because they inherit the same IAM role as the machine. * You can't disable the automatic K8s service account mount [1]. Another scary thing, unrelated to GKE, is that the VMs run a daemon that automatically creates users with SSH keys. As a team member, you can SSH into any box with any user name. Not sure if real security weakness, but I don't like it. I love the CLI tools ("gcloud" and so on), much nicer than awscli etc, and hopefully much friendlier to junior devs. [1] You can disable the mount by mounting an "emptyDir" on top, but that makes K8s think the pod has local data, which causes the autoscaler to refuse to tear down a node. Fortunately there's an option coming to disable the service account.

lowbloodsugar 10y ago

Unfortunately, Mesosphere is no longer focusing on Marathon as an Apache Mesos framework and is instead integrating it into DC/OS, their commercial offering. Open-source Mesos+Marathon no longer appears to be an option.

nemothekid 10y ago
Isn't DC/OS now opensource? (https://github.com/dcos/dcos https://github.com/dcos/dcos)
- ssk2 10y ago
  Yup! https://mesosphere.com/blog/2016/04/19/open-source-dcos/ https://mesosphere.com/blog/2016/04/19/open-source-dcos/ (I am a Mesosphere employee.)
  - SEJeff 10y ago
    Will any of the commercial only stuff (like the distributed firewall or ldap authentication / authorization bits) ever be open sourced, or will DC/OS forever be open core crippleware?
    - ssk2 10y ago
      That's a little harsh. I don't think the intention is to be open core crippleware - there are certainly a good number of users using (open) DC/OS in production without the enterprise features that you mention. I can't speak to the specific roadmap since I am uninformed but generally I believe the trend is for most enterprise features to trickle down into open over time.
  - poooogles 10y ago
    Care to comment on GP?
    - ssk2 10y ago
      We're very much committed to the Marathon open source community and keeping them happy. I'm afraid I don't know what the plans are for the project (not working on Marathon myself) but being the most popular open source project run and maintained by Mesosphere, I have no doubt that we will continue adding new features and functionality.

tom_pulo 10y ago

With all of this talk about docker, Kubernetes and the rest I feel peer pressured into ditching my monolithic heroku rails app and switching to the distributed services heaven that docker seems too advertise. Can anybody that has made the switch give me a convincing argument about why I should switch to (or not)? My feeling that docker is great if you are VP of Engineering at Netflix, but is probably not the best thing if you are starting a startup and just need to get things done. Disclaimer: I'm not religious about this and I'm totally open to being convinced that I'm wrong.

wstrange 10y ago
It sounds like your current setup is working just fine, so I don't see a compelling reason to switch. When you start to deploy many applications, and they need to talk to each other, and you need service discovery, automatic restart, rolling upgrades, horizontal scaling, etc - then Kubernetes brings a lot of value.
ekidd 10y ago
> With all of this talk about docker, Kubernetes and the rest I feel peer pressured into ditching my monolithic heroku rails app and switching to the distributed services heaven that docker seems too advertise. I successfully run lots of Docker microservices in production, and I strongly advise you to keep your app on Heroku as long as you can. :-) Microservices make sense in two circumstances: 1. You have multiple teams of developers, and you want them to have different release cycles and loosely-coupled APIs. In this case, you can let each team have a microservice. 2. There's a module in your app which is self-contained and naturally isolated, with a stable API. You could always just make this a separate Heroku app with its own REST API. But in general, microservices add complexity and make it harder to do certain kinds of refactorings. You can make them work, if you know what you're doing. For example, ECS +RDS+ALBs is halfway civilized, especially if you manage the configuration with Terraform, and set up a CI server to build Docker images and run tests. But it's still a lot more complex than a single, well-refactored app on Heroku.
ktta 10y ago
If you're a startup, then I'd look at the growth you're expecting. Containers scale well, and when you're big, maintaining multiple heroku apps eats into developer time that can be better spent somewhere else. Of course, if you've just started, and are getting an MVP out the door, don't worry about docker just yet. And also don't listen to the microservices people. It'll be like putting the cart before the horse.

jaaron 10y ago

Title really shouldn't include "Mesosphere." It should be "Marathon" or "DC/OS". Or, if you keep Mesosphere, it should be "Google, Mesosphere, and Docker." Aside from that, it's not a bad overview. I particularly like this quote: “Mesos and Kubernetes are largely aimed at solving similar problems of running clustered applications; they have different histories and different approaches to solving the problem. Mesos focuses its energy on very generic scheduling and plugging in multiple different schedulers.” In contrast, says Burns, “Kubernetes was designed from the ground up to be an environment for building distributed applications from containers. It includes primitives for replication and service discovery as core primitives, whereas such things are added via frameworks in Mesos. … Swarm is an effort by Docker to extend the existing Docker API to make a cluster of machines look like a single Docker API.” Mesos is useful when you have requirements that don't fit in k8s' rather opinionated world view. If you can use k8s, it's great, but it's also changing rapidly, so be aware you're on the bleeding edge. DockerSwarm is Docker finding a business model. (I'm biased here)

lowbloodsugar 10y ago
Better bleeding edge than end-of-life. We're using Marathon, and suddenly the entire UI is end-of-life, and your options are "use Marathon back-end only without a UI" or "migrate to DC/OS" which is even more opinionated than K8s. I think the real tell is that Mesosphere don't have the cash/people to maintain a non-DC/OS UI for marathon, and there isn't a community able/willing to do it for them.
- XorNot 10y ago
  About a year ago there was a decision made in our project to "just use Marathon" (our technical lead has a bit of a tendency to declare things as simple, without thinking too hard about operational problems) - a year later (...don't ask...) and I very much have the feeling we've backed the wrong horse. Conversely...I'm deeply suspicious of excited discussions of distributed systems these days because it doesn't seem like a lot of people are truly running them "at scale" - there's not a lot of write-ups about how people handle 300+ node clusters, and probably more importantly how they handle them (how do their internal users use them).

technologyvault 10y ago

Great article! I just made a point to get caught up Docker and related elements of the containerization world. Mesosphere looks like it works similar to Nanobox (http://nanobox.io http://nanobox.io)

bogomipz 10y ago

>" I’ve spoken with several DevOps who argue that compared with running a Kubernetes or Mesos cluster, Docker Swarm is a snap." I'm always leary of statements like that. My experience with cluster managers(Yarn, Mesos) and distributed systems in general is that they are almost never "a snap" to run once you move past trivial workloads and requirements.

atomi 10y ago
Not many startups move past trivial workloads. And if you do, it's a good problem to have. If swarm can get your mvp up quickly, it might be a good choice.
btmiller 10y ago
Yep, that's my exact experience with Swarm (1.13). I want the higher-level abstractions like K8s Pods and Deployments, but I absolutely cannot wait around for Swarm to stabilize those kind of things.
Axsuul 10y ago
My experience with Docker Swarm has been great. I came from Kubernetes but found it way too complex for my needs.
amouat 10y ago
The author is (mis)quoting me there. I meant the initial process of installing and getting Swarm running is much easier than k8s or Mesos, not that on-going maintenance was easier. Also, since I wrote that, k8s has done a lot of work to simplify installation with kubeadm and other improvements.
- bogomipz 10y ago
  Ah OK. Yeah doing a greenfield deployment is one thing, "operationalizing" it is often more involved. For Mesos anyone if is interested and you are Ansivle shop you can look at Ansible Shipyard to simplify an install. https://github.com/AnsibleShipyard https://github.com/AnsibleShipyard Kubeadm looks interesting. Thanks.

ovidiup 10y ago

I built Jollyturns (https://jollyturns.com https://jollyturns.com) which has a fairly large server-side component. Before starting the work on Jollyturns I worked at Google (I left in 2010) on a variety of infrastructure projects, where I got to use first hand Borg and all the other services running on it. When I started more than 5 years ago few of the options mentioned in the article were available, so I just used Xen running on bare metal. I just wanted to get things done as opposed to forever experimenting with infrastructure. Not to mention the load was inexistent, so everything was easy to manage. After I launched the mobile app, I decided to spend some time on the infrastructure. 2 years ago I experimented with Marathon, which was originally developed at Twitter, probably by a bunch of former Google employees. The reason is Marathon felt very much like Borg: you could see the jobs you launched in a pretty nice Web interface, including their log files, resource utilization and so on. Deploying it on bare metal machines however was an exercise in frustration since Marathon relied heavily on Apache Mesos. For a former Googler, Mesos had some weird terminology and was truly difficult to understand what the heck was going on. Running Marathon on top of it had major challenges: when long-running services failed you could not figure out why things were not working. So after about 2 weeks I gave up on it, and went back to manually managed Xen VMs. Around the same time I spent few days with Docker Swarm. If you're familiar with Borg/Kubernetes, Swarm is very different from it since - at least when it started, you had to allocate services to machines by hand. I wrote it off quickly since allocating dockerized services on physical machines was not my idea of cluster management. Last year I switched to Kubernetes (version 1.2) since it's the closest to what I expect from a cluster management system. The version I've been using in production has a lot of issues: high availability (HA) for its components is almost non-existent. I had to setup Kube in such a way that its components have some resemblance of HA. In the default configuration, Kubernetes installs its control components on a single machine. If that machine fails or is rebooted your entire cluster disappears. Even with all these issues, Kubernetes solves a lot of problems for you. The flannel networking infrastructure greatly simplifies the deployment of docker containers, since you don't need to worry about routing traffic between your containers. Even now Kubernetes doesn't do HA: https://github.com/kubernetes/kubernetes/issues/26852 https://github.com/kubernetes/kubernetes/issues/26852 https://github.com/kubernetes/kubernetes/issues/18174 https://github.com/kubernetes/kubernetes/issues/18174 Don't be fooled by the title of the bug report, the same component used in kubectl to implement HA could be used inside Kube' servers for the same reason. I guess these days the Google engineers working on Kubernetes have no real experience deploying large services on Borg inside Google. Such a pity!

webo 10y ago

We've been running Kubernetes in production for over a year. It has a steep learning curve and it takes a lot of time to fine-tune the underlying infrastructure (we use CoreOs) to make it production-ready. There still seems to be a lot of shortcomings of k8s that seem like should have been addressed by now: - it is impossible to trigger a rescheduling / rebalancing. When a new node comes in (via AutoScaling policy or whatever), kubernetes doesn't do anything. Thus, the new nodes can be sitting there doing nothing. - once a pod has been scheduled onto a node, it never reschedules it anywhere else. The node may be experiencing problems, thus the pod is affected. k8s doesn't do anything to heal that -- it could simply delete the pod so it is rescheduled somewhere else. - docker itself constantly ships with a lot of bugs. To this day (we are on docker 1.12.6), we constantly have problems with the docker daemon hanging or becoming unresponsive. I'm not sure if k8s can do much about this, but I feel like it should since we don't directly control docker. - doesn't integrate more tightly with the OS / cloud provider. For example, it could perform health checks and decide if the node should be restarted, terminated, or idle. All of our services are stateless, so it would be nice to have the option for all the above, especially k8s started as being the solution for stateless apps.

alpb 10y ago
> - once a pod has been scheduled onto a node, it never reschedules it anywhere else. The node may be experiencing problems, thus the pod is affected. k8s doesn't do anything to heal that -- it could simply delete the pod so it is rescheduled somewhere else. Can you please elaborate this? When you are using replication controllers or deployments, don’t they drive the state to the desired/goal state, which is N replicas of a pod? So when the node is shut down, I guess it should be rescheduling those dead pods somewhere else to satisfy the goal state?
- webo 10y ago
  You may have misunderstood me. The case I'm talking is, the node reports Ready, but the pod itself is not functioning properly. One common issue we have is the pod gets stuck in a restart loop (for whatever reason, including starvation of resources). k8s just keeps restarting it for days on that node, instead of simply rescheduling it after X restarts or some other condition. https://github.com/kubernetes/kubernetes/issues/13385 https://github.com/kubernetes/kubernetes/issues/13385

sitepodmatt 10y ago

Quoting from the disadvantages section: "Second, Kubernetes excels at automatically fixing problems. But it’s so good at it that containers can crash and be restarted so fast you don’t notice your containers are crashing. " A key part of Kubernetes is to the bring the state of the system to that of the spec, through a series of watches - and in some places polling (config maps watching in kubectl as of 1.4.x) - this may look like magic, but it's not fixing problems per se its that your desired state of the system (the spec) is different to what is observed (the status). This is not a disadvantage. Kubernetes if not for someone sitting at the terminal expecting a bell or stop the world when problems happens, although I guess you could configure it that way and fight against the system.