6 ms·

Kubernetes (k8s) makes for an amazing developer story. Mesos is much more bare metal, but the scheduler scales a loooot better than the still relatively immatur

by SEJeff 11y ago

Kubernetes (k8s) makes for an amazing developer story. Mesos is much more bare metal, but the scheduler scales a loooot better than the still relatively immature k8s scheduling component. One of the original authors of mesos wrote a paper on scheduling: https://www.cs.berkeley.edu/~alig/papers/drf.pdf https://www.cs.berkeley.edu/~alig/papers/drf.pdf. Mesos is one of the first "two level" schedulers. I very highly recommend that you also read this article for an idea of how this is a good idea: http://www.umbrant.com/blog/2015/mesos_omega_borg_survey.html http://www.umbrant.com/blog/2015/mesos_omega_borg_survey.htm... The k8s upstream was forward thinking enough to make the scheduler parts of it pluggable, which allow the (imo) holy grail of something like this https://github.com/mesosphere/kubernetes-mesos https://github.com/mesosphere/kubernetes-mesos. This gives you the nice logical "pod" description for services, running on the battle tested mesos scheduler. There are many 10k+ node bare metal mesos deployments (apple, twitter, etc). There aren't yet many kubernetes deployments of that scale. They truly are mutually beneficial. Mesos makes ops happy, and k8s makes devs happy. Together you have a relatively easy to setup internal PaaS (your own heroku with not a ton of work) more or less. Disclaimer: I'm a heavy mesos and apache aurora user.

jaytaylor 11y ago
Thanks for sharing all that terrific information! The only thing I'll add is that k8s isn't targeting the same scale as Mesos. Their current goal is to support up to 400-500 nodes, max. Source: One of the core k8s developers I met at a CoreOS meetup in SF earlier this year. They said if I needed to go beyond 500 nodes that I should probably look at something else.
- josephjacks 11y ago
  This is categorically incorrect. K8S will ultimately scale to N number of nodes. Within 2015, it will scale to 1K+ nodes, as per the roadmap. Being modeled after Google's Borg system, I encourage curious/interested folks to read at the recent Borg paper [0] which also outlines lessons learned in running Borg at Google for nearly 15 years and managing many millions of machines. [0] http://research.google.com/pubs/pub43438.html http://research.google.com/pubs/pub43438.html Disclaimer: I work for Kismatic.
  - cthalupa 11y ago
    Calling that categorically incorrect is pretty disingenuous when we have Google engineers who are working on the project saying that it currently supports 100 nodes with ease, and that they /expect/ it to handle more in the (very near) future. It might not be correct for much longer, but if it is the case now, how can you say it's categorically incorrect?
    - josephjacks 11y ago
      To clarify, this entire statement is categorically incorrect: "The only thing I'll add is that k8s isn't targeting the same scale as Mesos. Their current goal is to support up to 400-500 nodes, max."
      - jaytaylor 11y ago
        My apologies -- I should have phrased as "as recently as 2-3 months ago".
iyn 11y ago
Can you suggest best resources (text/video) for learning about Kubernetes & Mesos? I use Docker & CoreOS all the time (love it) and I'm always trying to improve/learn something new. Can you tell how do you use Apache Aurora? What other interesting projects are worth learning about?
- mmccaff 11y ago
  I recently spent some time playing with Mesos, Marathon (the web ui + api for scheduling long-running jobs / services) and Chronos (the web ui + api for scheduling cron / batch jobs). I did this on my Macbook, using Virtualbox, Vagrant, and this: https://github.com/mesosphere/playa-mesos https://github.com/mesosphere/playa-mesos ^ I started with that, and then installed Chronos with apt-get in addition. Specifically, for launching Docker containers, this was useful: https://docs.mesosphere.com/tutorials/launch-docker-container-on-mesosphere/ https://docs.mesosphere.com/tutorials/launch-docker-containe... I didn't try Aurora but it seems it'd be an alternative to Marathon + Chronos (Mesos calls all of these "frameworks").
- Hortinstein 11y ago
  how do you currently do service discovery w/ Docker and CoreOS without MESOS or Kubernetes?
  - iyn 11y ago
    Honest answer: I don't. In short: most of the things have fixed config that is loaded into etcd cluster and different services in Docker containers use it to communicate with other containers/services (something like {rabbitmq_host: "host address"}. In the project I'm working on right now I have just 10 boxes which will probably grow to 20-30 in the coming months. It's nothing, I know, and as you can tell from the hacky nature of my setup I'm learning as I go about this, but I'm trying to incrementally improve different parts. Something like Kubernetes/Mesos seems like a next step.
    - shykes 11y ago
      In case you're interested, the next version of Docker (1.7) supports multi-host networking and dynamic service discovery out of the box. The whole thing is pluggable and can use various distributed state backends (etcd, zookeeper etc) or IP connectivity backends (veth, macvlan, vxlan, openvpn etc) without changing your application. Service discovery uses DNS so you don't need to modify your application to take advantage of it. It's probably the most significant change to Docker in the last year. This will make the integration with Kubernetes smoother. Currently Google is forced to rip out Docker's native networking stack because it is not flexible enough for their opinionated networking model. This causes many Docker applications to break in Kubernetes today. That problem should go away with Docker 1.7+ because Google-style networking can be expressed as a Docker plugin, which Kubernetes can load programmatically as part of its orchestration. An added benefit is that you can augment Docker with Google-style networking even if you use Kubernetes competitors like Mesos, Swarm, Cloudfoundry etc. (EDIT added details more relevant to Kubernetes)
sagichmal 11y ago
> the battle tested mesos scheduler. Well, let's not get ahead of ourselves.
rco8786 11y ago
> mesos and apache aurora ohh i bet we're in the same building right now
lobster_johnson 11y ago
Thanks for the explanation. Sounds like Kubernetes should work just fine for small (<20 nodes) clusters, though. I'm still not quite understanding what utility Kubernetes brings to the table if you can also use it with Mesos. If you use Mesos, why involve Kubernetes at all, and not some Mesos-specific framework like Marathon or Aurora? Is Kubernetes simply a competitor to those frameworks? My concern about Mesos is mainly footprint and complexity. You need to run ZooKeeper, the master, the slaves, and then each framework. Only Mesos itself is written in C++, everything else is JVM, which is a pretty significant memory hog. By installing Mesos you just increased the complexity of the deployment/ops stack by a huge margin; you reap many benefits, of course, but Mesos is a lot more opaque and complex than a few daemons and some SSH-based scripts.
- thockingoog 11y ago
  Kubernetes supports 100 nodes with ease, and we expect to handle much more than that very quickly. We just had to pick some target to start with.
- SEJeff 11y ago
  Zookeeper is a hog, no one will disagree. But etcd is a very awesome yet very new technology. Even the protocol it implements (which is awesome), raft, is very new as a distributed consensus protocol. I'm not in any remote means throwing cold water on k8s, it is fantastic stuff. I only know that there are very very large production mesos clusters today, and the same can not be said (yet) for k8s. Read those two links I posted in the parent though if you have the time. It will make a ton more sense. That being said, you k8s is sexy stuff, it just ties you to docker, and I believe soon to be rocket. When I first started evaluating both (around docker 1.2.x), docker was not super viable and was pretty buggy. With 1.6.x and newer, most of my original concerns cease to matter. They are both excellent technologies, use whatever works for your environment.
lobster_johnson 11y ago
Thanks for the explanation. Sounds like Kubernetes should work just fine for small (<20 nodes) clusters, though. I'm still not quite understanding what utility Kubernetes brings to the table if you can also use it with Mesos. If you use Mesos, why involve Kubernetes at all, and not some Mesos-specific framework like Marathon or Aurora? Is Kubernetes simply a competitor to those frameworks? My concern about Mesos is mainly footprint and complexity. You need to run ZooKeeper, the master, the slaves, and then each framework. Only Mesos itself is written in C++, everything else is JVM, which is a pretty significant memory hog. (Kubernetes, by comparison, is written in Go and would presumably be more lightweight; its only dependencies are etcd and Docker.) By installing Mesos you just increased the complexity of the deployment/ops stack by a huge margin; you reap many benefits, of course, but Mesos is a lot more opaque and complex than a few daemons and some SSH-based scripts.
jacques_chester 11y ago
> (your own heroku with not a ton of work) I've worked on Cloud Foundry, which is ostensibly a Heroku competitor. The idea that you can replicate Heroku's full functionality "easily" is just silly. Full-feature PaaSes do a lot of things, including a whole bunch of tedious nitty-gritty details. We're well into the days of early maturity on PaaS products. You can install Cloud Foundry or OpenShift, or host on Heroku. Writing your own PaaS at this point is a bit like writing a custom operating system circa 1995. Unless you have a compelling reason to do so, you'd be utterly crazy to.

by SEJeff 11y ago

jaytaylor 11y ago

Thanks for sharing all that terrific information! The only thing I'll add is that k8s isn't targeting the same scale as Mesos. Their current goal is to support up to 400-500 nodes, max. Source: One of the core k8s developers I met at a CoreOS meetup in SF earlier this year. They said if I needed to go beyond 500 nodes that I should probably look at something else.

josephjacks 11y ago
This is categorically incorrect. K8S will ultimately scale to N number of nodes. Within 2015, it will scale to 1K+ nodes, as per the roadmap. Being modeled after Google's Borg system, I encourage curious/interested folks to read at the recent Borg paper [0] which also outlines lessons learned in running Borg at Google for nearly 15 years and managing many millions of machines. [0] http://research.google.com/pubs/pub43438.html http://research.google.com/pubs/pub43438.html Disclaimer: I work for Kismatic.
- cthalupa 11y ago
  Calling that categorically incorrect is pretty disingenuous when we have Google engineers who are working on the project saying that it currently supports 100 nodes with ease, and that they /expect/ it to handle more in the (very near) future. It might not be correct for much longer, but if it is the case now, how can you say it's categorically incorrect?
  - josephjacks 11y ago
    To clarify, this entire statement is categorically incorrect: "The only thing I'll add is that k8s isn't targeting the same scale as Mesos. Their current goal is to support up to 400-500 nodes, max."
    - jaytaylor 11y ago
      My apologies -- I should have phrased as "as recently as 2-3 months ago".

iyn 11y ago

Can you suggest best resources (text/video) for learning about Kubernetes & Mesos? I use Docker & CoreOS all the time (love it) and I'm always trying to improve/learn something new. Can you tell how do you use Apache Aurora? What other interesting projects are worth learning about?

mmccaff 11y ago
I recently spent some time playing with Mesos, Marathon (the web ui + api for scheduling long-running jobs / services) and Chronos (the web ui + api for scheduling cron / batch jobs). I did this on my Macbook, using Virtualbox, Vagrant, and this: https://github.com/mesosphere/playa-mesos https://github.com/mesosphere/playa-mesos ^ I started with that, and then installed Chronos with apt-get in addition. Specifically, for launching Docker containers, this was useful: https://docs.mesosphere.com/tutorials/launch-docker-container-on-mesosphere/ https://docs.mesosphere.com/tutorials/launch-docker-containe... I didn't try Aurora but it seems it'd be an alternative to Marathon + Chronos (Mesos calls all of these "frameworks").
Hortinstein 11y ago
how do you currently do service discovery w/ Docker and CoreOS without MESOS or Kubernetes?
- iyn 11y ago
  Honest answer: I don't. In short: most of the things have fixed config that is loaded into etcd cluster and different services in Docker containers use it to communicate with other containers/services (something like {rabbitmq_host: "host address"}. In the project I'm working on right now I have just 10 boxes which will probably grow to 20-30 in the coming months. It's nothing, I know, and as you can tell from the hacky nature of my setup I'm learning as I go about this, but I'm trying to incrementally improve different parts. Something like Kubernetes/Mesos seems like a next step.
  - shykes 11y ago
    In case you're interested, the next version of Docker (1.7) supports multi-host networking and dynamic service discovery out of the box. The whole thing is pluggable and can use various distributed state backends (etcd, zookeeper etc) or IP connectivity backends (veth, macvlan, vxlan, openvpn etc) without changing your application. Service discovery uses DNS so you don't need to modify your application to take advantage of it. It's probably the most significant change to Docker in the last year. This will make the integration with Kubernetes smoother. Currently Google is forced to rip out Docker's native networking stack because it is not flexible enough for their opinionated networking model. This causes many Docker applications to break in Kubernetes today. That problem should go away with Docker 1.7+ because Google-style networking can be expressed as a Docker plugin, which Kubernetes can load programmatically as part of its orchestration. An added benefit is that you can augment Docker with Google-style networking even if you use Kubernetes competitors like Mesos, Swarm, Cloudfoundry etc. (EDIT added details more relevant to Kubernetes)

sagichmal 11y ago

> the battle tested mesos scheduler. Well, let's not get ahead of ourselves.

rco8786 11y ago

> mesos and apache aurora ohh i bet we're in the same building right now

lobster_johnson 11y ago

Thanks for the explanation. Sounds like Kubernetes should work just fine for small (<20 nodes) clusters, though. I'm still not quite understanding what utility Kubernetes brings to the table if you can also use it with Mesos. If you use Mesos, why involve Kubernetes at all, and not some Mesos-specific framework like Marathon or Aurora? Is Kubernetes simply a competitor to those frameworks? My concern about Mesos is mainly footprint and complexity. You need to run ZooKeeper, the master, the slaves, and then each framework. Only Mesos itself is written in C++, everything else is JVM, which is a pretty significant memory hog. By installing Mesos you just increased the complexity of the deployment/ops stack by a huge margin; you reap many benefits, of course, but Mesos is a lot more opaque and complex than a few daemons and some SSH-based scripts.

thockingoog 11y ago
Kubernetes supports 100 nodes with ease, and we expect to handle much more than that very quickly. We just had to pick some target to start with.
SEJeff 11y ago
Zookeeper is a hog, no one will disagree. But etcd is a very awesome yet very new technology. Even the protocol it implements (which is awesome), raft, is very new as a distributed consensus protocol. I'm not in any remote means throwing cold water on k8s, it is fantastic stuff. I only know that there are very very large production mesos clusters today, and the same can not be said (yet) for k8s. Read those two links I posted in the parent though if you have the time. It will make a ton more sense. That being said, you k8s is sexy stuff, it just ties you to docker, and I believe soon to be rocket. When I first started evaluating both (around docker 1.2.x), docker was not super viable and was pretty buggy. With 1.6.x and newer, most of my original concerns cease to matter. They are both excellent technologies, use whatever works for your environment.

lobster_johnson 11y ago

Thanks for the explanation. Sounds like Kubernetes should work just fine for small (<20 nodes) clusters, though. I'm still not quite understanding what utility Kubernetes brings to the table if you can also use it with Mesos. If you use Mesos, why involve Kubernetes at all, and not some Mesos-specific framework like Marathon or Aurora? Is Kubernetes simply a competitor to those frameworks? My concern about Mesos is mainly footprint and complexity. You need to run ZooKeeper, the master, the slaves, and then each framework. Only Mesos itself is written in C++, everything else is JVM, which is a pretty significant memory hog. (Kubernetes, by comparison, is written in Go and would presumably be more lightweight; its only dependencies are etcd and Docker.) By installing Mesos you just increased the complexity of the deployment/ops stack by a huge margin; you reap many benefits, of course, but Mesos is a lot more opaque and complex than a few daemons and some SSH-based scripts.

jacques_chester 11y ago

> (your own heroku with not a ton of work) I've worked on Cloud Foundry, which is ostensibly a Heroku competitor. The idea that you can replicate Heroku's full functionality "easily" is just silly. Full-feature PaaSes do a lot of things, including a whole bunch of tedious nitty-gritty details. We're well into the days of early maturity on PaaS products. You can install Cloud Foundry or OpenShift, or host on Heroku. Writing your own PaaS at this point is a bit like writing a custom operating system circa 1995. Unless you have a compelling reason to do so, you'd be utterly crazy to.