9 ms·

Kubernetes: The Future of Deployment

159 points by bashtoni 11y ago

lobster_johnson 11y ago
I'd love for someone to explain how Kubernetes compares to Mesos. Every article I find on the subject says they are mutually beneficial, not competitors — that you would typically run Kubernetes as a Mesos framework — yet Kubernetes also seems like it duplicates much of Mesos' functionality on its own.
- SEJeff 11y ago
  Kubernetes (k8s) makes for an amazing developer story. Mesos is much more bare metal, but the scheduler scales a loooot better than the still relatively immature k8s scheduling component. One of the original authors of mesos wrote a paper on scheduling: https://www.cs.berkeley.edu/~alig/papers/drf.pdf https://www.cs.berkeley.edu/~alig/papers/drf.pdf. Mesos is one of the first "two level" schedulers. I very highly recommend that you also read this article for an idea of how this is a good idea: http://www.umbrant.com/blog/2015/mesos_omega_borg_survey.html http://www.umbrant.com/blog/2015/mesos_omega_borg_survey.htm... The k8s upstream was forward thinking enough to make the scheduler parts of it pluggable, which allow the (imo) holy grail of something like this https://github.com/mesosphere/kubernetes-mesos https://github.com/mesosphere/kubernetes-mesos. This gives you the nice logical "pod" description for services, running on the battle tested mesos scheduler. There are many 10k+ node bare metal mesos deployments (apple, twitter, etc). There aren't yet many kubernetes deployments of that scale. They truly are mutually beneficial. Mesos makes ops happy, and k8s makes devs happy. Together you have a relatively easy to setup internal PaaS (your own heroku with not a ton of work) more or less. Disclaimer: I'm a heavy mesos and apache aurora user.
  - jaytaylor 11y ago
    Thanks for sharing all that terrific information! The only thing I'll add is that k8s isn't targeting the same scale as Mesos. Their current goal is to support up to 400-500 nodes, max. Source: One of the core k8s developers I met at a CoreOS meetup in SF earlier this year. They said if I needed to go beyond 500 nodes that I should probably look at something else.
    - josephjacks 11y ago
      This is categorically incorrect. K8S will ultimately scale to N number of nodes. Within 2015, it will scale to 1K+ nodes, as per the roadmap. Being modeled after Google's Borg system, I encourage curious/interested folks to read at the recent Borg paper [0] which also outlines lessons learned in running Borg at Google for nearly 15 years and managing many millions of machines. [0] http://research.google.com/pubs/pub43438.html http://research.google.com/pubs/pub43438.html Disclaimer: I work for Kismatic.
      - cthalupa 11y ago
        Calling that categorically incorrect is pretty disingenuous when we have Google engineers who are working on the project saying that it currently supports 100 nodes with ease, and that they /expect/ it to handle more in the (very near) future. It might not be correct for much longer, but if it is the case now, how can you say it's categorically incorrect?
        josephjacks 11y ago
        To clarify, this entire statement is categorically incorrect: "The only thing I'll add is that k8s isn't targeting the same scale as Mesos. Their current goal is to support up to 400-500 nodes, max."
        jaytaylor 11y ago
        My apologies -- I should have phrased as "as recently as 2-3 months ago".
  - iyn 11y ago
    Can you suggest best resources (text/video) for learning about Kubernetes & Mesos? I use Docker & CoreOS all the time (love it) and I'm always trying to improve/learn something new. Can you tell how do you use Apache Aurora? What other interesting projects are worth learning about?
    - mmccaff 11y ago
      I recently spent some time playing with Mesos, Marathon (the web ui + api for scheduling long-running jobs / services) and Chronos (the web ui + api for scheduling cron / batch jobs). I did this on my Macbook, using Virtualbox, Vagrant, and this: https://github.com/mesosphere/playa-mesos https://github.com/mesosphere/playa-mesos ^ I started with that, and then installed Chronos with apt-get in addition. Specifically, for launching Docker containers, this was useful: https://docs.mesosphere.com/tutorials/launch-docker-container-on-mesosphere/ https://docs.mesosphere.com/tutorials/launch-docker-containe... I didn't try Aurora but it seems it'd be an alternative to Marathon + Chronos (Mesos calls all of these "frameworks").
    - Hortinstein 11y ago
      how do you currently do service discovery w/ Docker and CoreOS without MESOS or Kubernetes?
      - iyn 11y ago
        Honest answer: I don't. In short: most of the things have fixed config that is loaded into etcd cluster and different services in Docker containers use it to communicate with other containers/services (something like {rabbitmq_host: "host address"}. In the project I'm working on right now I have just 10 boxes which will probably grow to 20-30 in the coming months. It's nothing, I know, and as you can tell from the hacky nature of my setup I'm learning as I go about this, but I'm trying to incrementally improve different parts. Something like Kubernetes/Mesos seems like a next step.
        shykes 11y ago
        In case you're interested, the next version of Docker (1.7) supports multi-host networking and dynamic service discovery out of the box. The whole thing is pluggable and can use various distributed state backends (etcd, zookeeper etc) or IP connectivity backends (veth, macvlan, vxlan, openvpn etc) without changing your application. Service discovery uses DNS so you don't need to modify your application to take advantage of it. It's probably the most significant change to Docker in the last year. This will make the integration with Kubernetes smoother. Currently Google is forced to rip out Docker's native networking stack because it is not flexible enough for their opinionated networking model. This causes many Docker applications to break in Kubernetes today. That problem should go away with Docker 1.7+ because Google-style networking can be expressed as a Docker plugin, which Kubernetes can load programmatically as part of its orchestration. An added benefit is that you can augment Docker with Google-style networking even if you use Kubernetes competitors like Mesos, Swarm, Cloudfoundry etc. (EDIT added details more relevant to Kubernetes)
  - sagichmal 11y ago
    > the battle tested mesos scheduler. Well, let's not get ahead of ourselves.
  - rco8786 11y ago
    > mesos and apache aurora ohh i bet we're in the same building right now
  - lobster_johnson 11y ago
    Thanks for the explanation. Sounds like Kubernetes should work just fine for small (<20 nodes) clusters, though. I'm still not quite understanding what utility Kubernetes brings to the table if you can also use it with Mesos. If you use Mesos, why involve Kubernetes at all, and not some Mesos-specific framework like Marathon or Aurora? Is Kubernetes simply a competitor to those frameworks? My concern about Mesos is mainly footprint and complexity. You need to run ZooKeeper, the master, the slaves, and then each framework. Only Mesos itself is written in C++, everything else is JVM, which is a pretty significant memory hog. By installing Mesos you just increased the complexity of the deployment/ops stack by a huge margin; you reap many benefits, of course, but Mesos is a lot more opaque and complex than a few daemons and some SSH-based scripts.
    - thockingoog 11y ago
      Kubernetes supports 100 nodes with ease, and we expect to handle much more than that very quickly. We just had to pick some target to start with.
    - SEJeff 11y ago
      Zookeeper is a hog, no one will disagree. But etcd is a very awesome yet very new technology. Even the protocol it implements (which is awesome), raft, is very new as a distributed consensus protocol. I'm not in any remote means throwing cold water on k8s, it is fantastic stuff. I only know that there are very very large production mesos clusters today, and the same can not be said (yet) for k8s. Read those two links I posted in the parent though if you have the time. It will make a ton more sense. That being said, you k8s is sexy stuff, it just ties you to docker, and I believe soon to be rocket. When I first started evaluating both (around docker 1.2.x), docker was not super viable and was pretty buggy. With 1.6.x and newer, most of my original concerns cease to matter. They are both excellent technologies, use whatever works for your environment.
  - lobster_johnson 11y ago
    Thanks for the explanation. Sounds like Kubernetes should work just fine for small (<20 nodes) clusters, though. I'm still not quite understanding what utility Kubernetes brings to the table if you can also use it with Mesos. If you use Mesos, why involve Kubernetes at all, and not some Mesos-specific framework like Marathon or Aurora? Is Kubernetes simply a competitor to those frameworks? My concern about Mesos is mainly footprint and complexity. You need to run ZooKeeper, the master, the slaves, and then each framework. Only Mesos itself is written in C++, everything else is JVM, which is a pretty significant memory hog. (Kubernetes, by comparison, is written in Go and would presumably be more lightweight; its only dependencies are etcd and Docker.) By installing Mesos you just increased the complexity of the deployment/ops stack by a huge margin; you reap many benefits, of course, but Mesos is a lot more opaque and complex than a few daemons and some SSH-based scripts.
  - jacques_chester 11y ago
    > (your own heroku with not a ton of work) I've worked on Cloud Foundry, which is ostensibly a Heroku competitor. The idea that you can replicate Heroku's full functionality "easily" is just silly. Full-feature PaaSes do a lot of things, including a whole bunch of tedious nitty-gritty details. We're well into the days of early maturity on PaaS products. You can install Cloud Foundry or OpenShift, or host on Heroku. Writing your own PaaS at this point is a bit like writing a custom operating system circa 1995. Unless you have a compelling reason to do so, you'd be utterly crazy to.
- tristanz 11y ago
  They are competitors, but life isn't simple. The reality is many frameworks currently only run on Mesos or YARN, and Kubernetes has not reached V1, so larger installations typically need multiple frameworks. Mesos is a proven way to run multiple frameworks side-by-side. Ebay's YARN on Mesos is another example. But where this all leads remains to be seen.
- cmcluck 11y ago
  (disclaimer: i work at Google and was one of the founders of the project) when we were looking at building k8s our mission was to help the world move forwards to a more cloud native approach to development. by cloud native i mean container packaged, dynamically scheduled, micro-services oriented. we figured that in the end our data centers are going to be well suited to run cloud native apps, since they were designed from the ground up for this approach to management, and will offer performance and efficiency advantages over the alternatives. we also however recognized that no matter how cheap, fast and reliable the hosting offering is, most folks don't want to be locked into a single provider and Google in particular. we needed to do what we were doing in the open, and the thing that we built needed to be pattern compatible with our approach to management and quite frankly address some of the mistakes we had in previous frameworks (Borg mostly as a first system). we looked really closely at Apache Mesos and liked a lot of what we saw, but there were a couple of things that stopped us just jumping on it. (1) it was written in C++ and the containers world was moving to Go -- we knew we planned to make a sustained and considerable investment in this and knew first hand that Go was more productive (2) we wanted something incredibly simple to showcase the critical constructs (pods, labels, label selectors, replication controllers, etc) and to build it directly with the communities support and mesos was pretty large and somewhat monolithic (3) we needed what Joe Beda dubbed 'over-modularity' because we wanted a whole ecosystem to emerge, (4) we wanted 'cluster environment' to be lightweight and something you could easily turn up or turn down, kinda like a VM; the systems integrators i knew who worked with mesos felt that it was powerful but heavy and hard to setup (though i will note our friends at Mesosphere are helping to change this). so we figured doing something simple to create a first class cluster environment for native app management, 'but this time done right' as Tim Hockin likes to say everyday. now we really like the guys at Mesosphere and we respect the fact that Mesos runs the vast majority of existing data processing frameworks. by adding k8s on mesos you get the next-generation cloud native scheduler and the ability to run existing workloads. by running k8s by itself you get a lightweight cluster environment for running next gen cloud native apps. -- craig
  - peteridah 11y ago
    Thanks for this, it cleared up some confusion in my mind. A blogpost capturing these thoughts would be great.
dchuk 11y ago
What's the deal with the name "Kubernetes"? Does it mean anything, or have some tech significance, or is it really just because it basically means "ruler" in Greek?
- irickt 11y ago
  It is also related to the source of the word "cybernetics".
  - fixermark 11y ago
    Correct, which is how it slants to the "Borg cube" pun. Cybernetics was the term chosen by Norbert Wiener in the book "Cybernetics," and he traced the word's origin to the greek "kubernetes;" it related to his first example of a cybernetic system, the self-correcting steam-controlled rudder on a ship [http://en.wikipedia.org/wiki/Steering_engine http://en.wikipedia.org/wiki/Steering_engine]. (Why the pun? Kubernetes was heavily inspired / guided by Google's internal scheduling tool, which was named Borg (http://blog.kubernetes.io/2015/04/borg-predecessor-to-kubernetes.html http://blog.kubernetes.io/2015/04/borg-predecessor-to-kubern...).)
- thebeardisred 11y ago
  It means "Helmsman" in ancient Greek. Similarly it's related to the word "Governor" e.g: "kubernan" in ancient greek means to steer "kubernetes" is helmsman "gubernare" means to steer or to govern in Latin "gubernator" is "governor" in Latin Which then leads into the modern word "Gubernatorial", et al.
  - henrikschroder 11y ago
    It's also a pun on Borg Cubes.
    - deleted 11y ago
      [deleted]
  - rantanplan 11y ago
NotOscarWilde 11y ago
A slightly off-topic comment, but being an early-stage PhD in theoretical CS with my thesis topic on approximation algorithms for scheduling, I would like to know whether there are some theoretical problems related to these VM schedulers used in practice. If there is somebody knowledgeable about what is theoretically open (unknown tight approximation ratio, for instance) AND very useful to people building Kubernetes et al, I would be really happy to learn more. (The natural advice is to "hit the books", actually read the papers related to Kubernetes and find out what is both theoretical and useful to this area. I intend to do that soon, but sifting through "practical papers" and looking for something interesting in theory is a lot of work, and I just hoped there might be somebody who could provide a shortcut.)
- philip1209 11y ago
  It was more of an amusement, but I used integer programming at a company hackathon to build a better image scheduler: https://engineering.opendns.com/2015/05/06/docker-container-scheduling-as-a-bin-packing-problem/ https://engineering.opendns.com/2015/05/06/docker-container-... With a large, fairly homogenous environment it didn't outperform random assignment that well, though. It worked best with small, inhomogeneous loads.
  - NotOscarWilde 11y ago
    Right, ILP is a great tool for solving NP-complete problems relatively fast (depending on the solver, but there are some very good ones out there). However, as a theoretical tool it probably is not that exciting unless you're ready to tackle P vs. NP this way. (Unless you move to semidefinite programming and the SDP hierarchies, where the progress is very exciting but not yet that applicable to scheduling, to the best of my knowledge.) > With a large, fairly homogenous environment it didn't outperform random assignment that well, though. It worked best with small, inhomogeneous loads. Yes, that's probably a piece of the puzzle that I don't have yet -- to know a theoretical model that is both useful in practice and at the same time greedy/randomized assignment is not "good enough" for practical uses.
  - seanp2k2 11y ago
NateDad 11y ago
Wow, you could totally s/kubernetes/juju/ in this article and still be 100% correct. (https://jujucharms.com https://jujucharms.com for those not aware of juju)
- saryant 11y ago
  Not cool. At least disclose that you're one of the devs behind Juju.
  - NateDad 11y ago
    Sorry, I didn't think of it. I've mentioned it multiple times on HN, but you're right, I should have added a disclosure. Seriously didn't mean to be pushing Juju, I was just surprised at how similar it was to Juju. I had always sort of assumed it was Google cloud only, and/or containers only, etc.
falcolas 11y ago
Worth remembering, Kubernetes was built to Google's needs, and Google runs with a shared network space on any given VM and assigns an entire /24 to the VM running docker. Each container gets one of those addresses. [1] This probably won't work for everyone - be sure to read into the fine grained details before drinking the koolade. They're also at least two build versions behind Docker.[2] [1] https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/networking.md https://github.com/GoogleCloudPlatform/kubernetes/blob/maste... [2] https://github.com/GoogleCloudPlatform/kubernetes/blob/master/cluster/saltbase/salt/docker/init.sls https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...
- wyc 11y ago
  Agreed. I have more specifics of Google-centralism in my comment here: https://news.ycombinator.com/item?id=9330049 https://news.ycombinator.com/item?id=9330049
  - crb 11y ago
    In that post, you asked "Do you see any other providers here? https://github.com/GoogleCloudPlatform/kubernetes/tree/master/pkg/volume" https://github.com/GoogleCloudPlatform/kubernetes/tree/maste... Your implication was, at the time, there was only a GCE Persistent Disk provider. Today, there's that, plus AWS EBS volume, git repo, GlusterFS, NFS, Ceph block device, iSCSI, as well as host path and empty directory. Sounds like the product has evolved with a broad spectrum of support to me!
    - wyc 11y ago
      Yes, it has! Again, I don't think the design decisions were made to lock out other vendors. It's very reasonable to me that the first platform to be supported by Google engineers is Google's platform. Side note: I should be a better citizen and link to specific commits next time.
- josephjacks 11y ago
  I see Docker 1.6 in your [2] link, where do you see they are two versions behind Docker?
sunyc 11y ago
bundling with (unholy-ly immature) SDN is the most damning things for its adoption. It is thought to be needed for "live migration", but I don't see me needing that anytime soon because we run on virtual machines anyway? Iaas provider is not going away,paying for the cost of SDN now for features that doesn't even exists yet, is insane.
- brendandburns 11y ago
  (kubernetes contributor here) SDN isn't required for k8s, what is required is that each Pod (group of containers) get it's own IP address, and that the IP address is routeable in the cluster. In many cases, the easiest way to achieve this is via an SDN, but it is also achievable by programming traditional routers. The reason for wanting an IP address per pod is that it eliminates the need for port mangling, which dramatically simplifies wiring applications together.
  - sunyc 11y ago
    All applications was already desinged to be port based. I don't see how this would drastically change that.
    - brendandburns 11y ago
      the problem with port mangling is that your application starts running on random ports, so in addition to requiring discovery for IP addresses, you now also have to do discovery for ports, which pretty much requires custom code and infrastructure linked into your binaries (how do you convince nginx/redis/... to use your lookup service for ports?) And ports are different between different replicas of your service, since they're chosen at random during scheduling. It also makes ACLs and QoS harder to define for the network, since you don't have a clean network identity (e.g IP Address) for each application.
sferoze 11y ago
An interesting sidenote. The Meteor development group is contributing to Kubernetes and will be using it to help scale Meteor with their upcoming paid service Galaxy.
- josephjacks 11y ago
  This is great to see independent software companies like Meteor embrace Kubernetes as the platform on which to build their next-generation services [0]. [0] http://info.meteor.com/blog/meteor-and-a-galaxy-of-containers-with-kubernetes http://info.meteor.com/blog/meteor-and-a-galaxy-of-container...
dcosson 11y ago
Does anyone have resources about security/isolation best practices for running multiple applications on Kubernetes (or Mesos or similar)? For instance in a non cloud-native app that runs in VM's, you might have one app per VM and have firewalls between different VM's that don't need to talk to each other. Then if a non-critical app got compromised and an attacker got remote execution or SQL injection or something they can't get to your other app servers or databases. If all your apps are in a cluster, the non-critical compromised app might be running on the same host as a critical app, in which case the only thing keeping the attacker from your database credentials or other secrets is the docker container isolation which if I understand correctly is not assumed to be secure the way VM isolation is. What are people doing to address this? Or are my assumptions wrong and it's not actually a problem to worry about? My initial impression with mesos was that you'd only use it if you're at big enough scale that you're running a huge number of instances of the same app or you're running a lot of different data processing tasks that all access the same data so no isolation is needed between them. Now I feel like I see Kubernetes being discussed frequently as a great way to run all your different microservices at any scale (e.g. "The Future of Deployment"), but I've never seen this aspect of security discussed.
- jacques_chester 11y ago
  You might prefer Cloud Foundry, which is switching its underlying container scheduling fabric to Lattice[1]. In particular, Cloud Foundry has more advanced security groups features, because it's mostly being marketed to enterprise customers. Disclaimer: I have worked on CF and I work for a company which is a major contributor to CF. [1] http://lattice.cf/ http://lattice.cf/
  - dcosson 11y ago
    Lattice looks interesting, looking forward to checking it out more
- brendandburns 11y ago
  Please check out the Secrets object in Kubernetes: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/secrets.md https://github.com/GoogleCloudPlatform/kubernetes/blob/maste... which is designed to address some of this.

159 points by bashtoni 11y ago

I'd love for someone to explain how Kubernetes compares to Mesos. Every article I find on the subject says they are mutually beneficial, not competitors — that you would typically run Kubernetes as a Mesos framework — yet Kubernetes also seems like it duplicates much of Mesos' functionality on its own.

SEJeff 11y ago
Kubernetes (k8s) makes for an amazing developer story. Mesos is much more bare metal, but the scheduler scales a loooot better than the still relatively immature k8s scheduling component. One of the original authors of mesos wrote a paper on scheduling: https://www.cs.berkeley.edu/~alig/papers/drf.pdf https://www.cs.berkeley.edu/~alig/papers/drf.pdf. Mesos is one of the first "two level" schedulers. I very highly recommend that you also read this article for an idea of how this is a good idea: http://www.umbrant.com/blog/2015/mesos_omega_borg_survey.html http://www.umbrant.com/blog/2015/mesos_omega_borg_survey.htm... The k8s upstream was forward thinking enough to make the scheduler parts of it pluggable, which allow the (imo) holy grail of something like this https://github.com/mesosphere/kubernetes-mesos https://github.com/mesosphere/kubernetes-mesos. This gives you the nice logical "pod" description for services, running on the battle tested mesos scheduler. There are many 10k+ node bare metal mesos deployments (apple, twitter, etc). There aren't yet many kubernetes deployments of that scale. They truly are mutually beneficial. Mesos makes ops happy, and k8s makes devs happy. Together you have a relatively easy to setup internal PaaS (your own heroku with not a ton of work) more or less. Disclaimer: I'm a heavy mesos and apache aurora user.
- jaytaylor 11y ago
  Thanks for sharing all that terrific information! The only thing I'll add is that k8s isn't targeting the same scale as Mesos. Their current goal is to support up to 400-500 nodes, max. Source: One of the core k8s developers I met at a CoreOS meetup in SF earlier this year. They said if I needed to go beyond 500 nodes that I should probably look at something else.
  - josephjacks 11y ago
    This is categorically incorrect. K8S will ultimately scale to N number of nodes. Within 2015, it will scale to 1K+ nodes, as per the roadmap. Being modeled after Google's Borg system, I encourage curious/interested folks to read at the recent Borg paper [0] which also outlines lessons learned in running Borg at Google for nearly 15 years and managing many millions of machines. [0] http://research.google.com/pubs/pub43438.html http://research.google.com/pubs/pub43438.html Disclaimer: I work for Kismatic.
    - cthalupa 11y ago
      Calling that categorically incorrect is pretty disingenuous when we have Google engineers who are working on the project saying that it currently supports 100 nodes with ease, and that they /expect/ it to handle more in the (very near) future. It might not be correct for much longer, but if it is the case now, how can you say it's categorically incorrect?
      - josephjacks 11y ago
        To clarify, this entire statement is categorically incorrect: "The only thing I'll add is that k8s isn't targeting the same scale as Mesos. Their current goal is to support up to 400-500 nodes, max."
        jaytaylor 11y ago
        My apologies -- I should have phrased as "as recently as 2-3 months ago".
- iyn 11y ago
  Can you suggest best resources (text/video) for learning about Kubernetes & Mesos? I use Docker & CoreOS all the time (love it) and I'm always trying to improve/learn something new. Can you tell how do you use Apache Aurora? What other interesting projects are worth learning about?
  - mmccaff 11y ago
    I recently spent some time playing with Mesos, Marathon (the web ui + api for scheduling long-running jobs / services) and Chronos (the web ui + api for scheduling cron / batch jobs). I did this on my Macbook, using Virtualbox, Vagrant, and this: https://github.com/mesosphere/playa-mesos https://github.com/mesosphere/playa-mesos ^ I started with that, and then installed Chronos with apt-get in addition. Specifically, for launching Docker containers, this was useful: https://docs.mesosphere.com/tutorials/launch-docker-container-on-mesosphere/ https://docs.mesosphere.com/tutorials/launch-docker-containe... I didn't try Aurora but it seems it'd be an alternative to Marathon + Chronos (Mesos calls all of these "frameworks").
  - Hortinstein 11y ago
    how do you currently do service discovery w/ Docker and CoreOS without MESOS or Kubernetes?
    - iyn 11y ago
      Honest answer: I don't. In short: most of the things have fixed config that is loaded into etcd cluster and different services in Docker containers use it to communicate with other containers/services (something like {rabbitmq_host: "host address"}. In the project I'm working on right now I have just 10 boxes which will probably grow to 20-30 in the coming months. It's nothing, I know, and as you can tell from the hacky nature of my setup I'm learning as I go about this, but I'm trying to incrementally improve different parts. Something like Kubernetes/Mesos seems like a next step.
      - shykes 11y ago
        In case you're interested, the next version of Docker (1.7) supports multi-host networking and dynamic service discovery out of the box. The whole thing is pluggable and can use various distributed state backends (etcd, zookeeper etc) or IP connectivity backends (veth, macvlan, vxlan, openvpn etc) without changing your application. Service discovery uses DNS so you don't need to modify your application to take advantage of it. It's probably the most significant change to Docker in the last year. This will make the integration with Kubernetes smoother. Currently Google is forced to rip out Docker's native networking stack because it is not flexible enough for their opinionated networking model. This causes many Docker applications to break in Kubernetes today. That problem should go away with Docker 1.7+ because Google-style networking can be expressed as a Docker plugin, which Kubernetes can load programmatically as part of its orchestration. An added benefit is that you can augment Docker with Google-style networking even if you use Kubernetes competitors like Mesos, Swarm, Cloudfoundry etc. (EDIT added details more relevant to Kubernetes)
- sagichmal 11y ago
  > the battle tested mesos scheduler. Well, let's not get ahead of ourselves.
- rco8786 11y ago
  > mesos and apache aurora ohh i bet we're in the same building right now
- lobster_johnson 11y ago
  Thanks for the explanation. Sounds like Kubernetes should work just fine for small (<20 nodes) clusters, though. I'm still not quite understanding what utility Kubernetes brings to the table if you can also use it with Mesos. If you use Mesos, why involve Kubernetes at all, and not some Mesos-specific framework like Marathon or Aurora? Is Kubernetes simply a competitor to those frameworks? My concern about Mesos is mainly footprint and complexity. You need to run ZooKeeper, the master, the slaves, and then each framework. Only Mesos itself is written in C++, everything else is JVM, which is a pretty significant memory hog. By installing Mesos you just increased the complexity of the deployment/ops stack by a huge margin; you reap many benefits, of course, but Mesos is a lot more opaque and complex than a few daemons and some SSH-based scripts.
  - thockingoog 11y ago
    Kubernetes supports 100 nodes with ease, and we expect to handle much more than that very quickly. We just had to pick some target to start with.
  - SEJeff 11y ago
    Zookeeper is a hog, no one will disagree. But etcd is a very awesome yet very new technology. Even the protocol it implements (which is awesome), raft, is very new as a distributed consensus protocol. I'm not in any remote means throwing cold water on k8s, it is fantastic stuff. I only know that there are very very large production mesos clusters today, and the same can not be said (yet) for k8s. Read those two links I posted in the parent though if you have the time. It will make a ton more sense. That being said, you k8s is sexy stuff, it just ties you to docker, and I believe soon to be rocket. When I first started evaluating both (around docker 1.2.x), docker was not super viable and was pretty buggy. With 1.6.x and newer, most of my original concerns cease to matter. They are both excellent technologies, use whatever works for your environment.
- lobster_johnson 11y ago
  Thanks for the explanation. Sounds like Kubernetes should work just fine for small (<20 nodes) clusters, though. I'm still not quite understanding what utility Kubernetes brings to the table if you can also use it with Mesos. If you use Mesos, why involve Kubernetes at all, and not some Mesos-specific framework like Marathon or Aurora? Is Kubernetes simply a competitor to those frameworks? My concern about Mesos is mainly footprint and complexity. You need to run ZooKeeper, the master, the slaves, and then each framework. Only Mesos itself is written in C++, everything else is JVM, which is a pretty significant memory hog. (Kubernetes, by comparison, is written in Go and would presumably be more lightweight; its only dependencies are etcd and Docker.) By installing Mesos you just increased the complexity of the deployment/ops stack by a huge margin; you reap many benefits, of course, but Mesos is a lot more opaque and complex than a few daemons and some SSH-based scripts.
- jacques_chester 11y ago
  > (your own heroku with not a ton of work) I've worked on Cloud Foundry, which is ostensibly a Heroku competitor. The idea that you can replicate Heroku's full functionality "easily" is just silly. Full-feature PaaSes do a lot of things, including a whole bunch of tedious nitty-gritty details. We're well into the days of early maturity on PaaS products. You can install Cloud Foundry or OpenShift, or host on Heroku. Writing your own PaaS at this point is a bit like writing a custom operating system circa 1995. Unless you have a compelling reason to do so, you'd be utterly crazy to.
tristanz 11y ago
They are competitors, but life isn't simple. The reality is many frameworks currently only run on Mesos or YARN, and Kubernetes has not reached V1, so larger installations typically need multiple frameworks. Mesos is a proven way to run multiple frameworks side-by-side. Ebay's YARN on Mesos is another example. But where this all leads remains to be seen.
cmcluck 11y ago
(disclaimer: i work at Google and was one of the founders of the project) when we were looking at building k8s our mission was to help the world move forwards to a more cloud native approach to development. by cloud native i mean container packaged, dynamically scheduled, micro-services oriented. we figured that in the end our data centers are going to be well suited to run cloud native apps, since they were designed from the ground up for this approach to management, and will offer performance and efficiency advantages over the alternatives. we also however recognized that no matter how cheap, fast and reliable the hosting offering is, most folks don't want to be locked into a single provider and Google in particular. we needed to do what we were doing in the open, and the thing that we built needed to be pattern compatible with our approach to management and quite frankly address some of the mistakes we had in previous frameworks (Borg mostly as a first system). we looked really closely at Apache Mesos and liked a lot of what we saw, but there were a couple of things that stopped us just jumping on it. (1) it was written in C++ and the containers world was moving to Go -- we knew we planned to make a sustained and considerable investment in this and knew first hand that Go was more productive (2) we wanted something incredibly simple to showcase the critical constructs (pods, labels, label selectors, replication controllers, etc) and to build it directly with the communities support and mesos was pretty large and somewhat monolithic (3) we needed what Joe Beda dubbed 'over-modularity' because we wanted a whole ecosystem to emerge, (4) we wanted 'cluster environment' to be lightweight and something you could easily turn up or turn down, kinda like a VM; the systems integrators i knew who worked with mesos felt that it was powerful but heavy and hard to setup (though i will note our friends at Mesosphere are helping to change this). so we figured doing something simple to create a first class cluster environment for native app management, 'but this time done right' as Tim Hockin likes to say everyday. now we really like the guys at Mesosphere and we respect the fact that Mesos runs the vast majority of existing data processing frameworks. by adding k8s on mesos you get the next-generation cloud native scheduler and the ability to run existing workloads. by running k8s by itself you get a lightweight cluster environment for running next gen cloud native apps. -- craig
- peteridah 11y ago
  Thanks for this, it cleared up some confusion in my mind. A blogpost capturing these thoughts would be great.

dchuk 11y ago

What's the deal with the name "Kubernetes"? Does it mean anything, or have some tech significance, or is it really just because it basically means "ruler" in Greek?

irickt 11y ago
It is also related to the source of the word "cybernetics".
- fixermark 11y ago
  Correct, which is how it slants to the "Borg cube" pun. Cybernetics was the term chosen by Norbert Wiener in the book "Cybernetics," and he traced the word's origin to the greek "kubernetes;" it related to his first example of a cybernetic system, the self-correcting steam-controlled rudder on a ship [http://en.wikipedia.org/wiki/Steering_engine http://en.wikipedia.org/wiki/Steering_engine]. (Why the pun? Kubernetes was heavily inspired / guided by Google's internal scheduling tool, which was named Borg (http://blog.kubernetes.io/2015/04/borg-predecessor-to-kubernetes.html http://blog.kubernetes.io/2015/04/borg-predecessor-to-kubern...).)
thebeardisred 11y ago
It means "Helmsman" in ancient Greek. Similarly it's related to the word "Governor" e.g: "kubernan" in ancient greek means to steer "kubernetes" is helmsman "gubernare" means to steer or to govern in Latin "gubernator" is "governor" in Latin Which then leads into the modern word "Gubernatorial", et al.
- henrikschroder 11y ago
  It's also a pun on Borg Cubes.
  - deleted 11y ago
    [deleted]
- rantanplan 11y ago

NotOscarWilde 11y ago

A slightly off-topic comment, but being an early-stage PhD in theoretical CS with my thesis topic on approximation algorithms for scheduling, I would like to know whether there are some theoretical problems related to these VM schedulers used in practice. If there is somebody knowledgeable about what is theoretically open (unknown tight approximation ratio, for instance) AND very useful to people building Kubernetes et al, I would be really happy to learn more. (The natural advice is to "hit the books", actually read the papers related to Kubernetes and find out what is both theoretical and useful to this area. I intend to do that soon, but sifting through "practical papers" and looking for something interesting in theory is a lot of work, and I just hoped there might be somebody who could provide a shortcut.)

philip1209 11y ago
It was more of an amusement, but I used integer programming at a company hackathon to build a better image scheduler: https://engineering.opendns.com/2015/05/06/docker-container-scheduling-as-a-bin-packing-problem/ https://engineering.opendns.com/2015/05/06/docker-container-... With a large, fairly homogenous environment it didn't outperform random assignment that well, though. It worked best with small, inhomogeneous loads.
- NotOscarWilde 11y ago
  Right, ILP is a great tool for solving NP-complete problems relatively fast (depending on the solver, but there are some very good ones out there). However, as a theoretical tool it probably is not that exciting unless you're ready to tackle P vs. NP this way. (Unless you move to semidefinite programming and the SDP hierarchies, where the progress is very exciting but not yet that applicable to scheduling, to the best of my knowledge.) > With a large, fairly homogenous environment it didn't outperform random assignment that well, though. It worked best with small, inhomogeneous loads. Yes, that's probably a piece of the puzzle that I don't have yet -- to know a theoretical model that is both useful in practice and at the same time greedy/randomized assignment is not "good enough" for practical uses.
- seanp2k2 11y ago

NateDad 11y ago

Wow, you could totally s/kubernetes/juju/ in this article and still be 100% correct. (https://jujucharms.com https://jujucharms.com for those not aware of juju)

saryant 11y ago
Not cool. At least disclose that you're one of the devs behind Juju.
- NateDad 11y ago
  Sorry, I didn't think of it. I've mentioned it multiple times on HN, but you're right, I should have added a disclosure. Seriously didn't mean to be pushing Juju, I was just surprised at how similar it was to Juju. I had always sort of assumed it was Google cloud only, and/or containers only, etc.

falcolas 11y ago

Worth remembering, Kubernetes was built to Google's needs, and Google runs with a shared network space on any given VM and assigns an entire /24 to the VM running docker. Each container gets one of those addresses. [1] This probably won't work for everyone - be sure to read into the fine grained details before drinking the koolade. They're also at least two build versions behind Docker.[2] [1] https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/networking.md https://github.com/GoogleCloudPlatform/kubernetes/blob/maste... [2] https://github.com/GoogleCloudPlatform/kubernetes/blob/master/cluster/saltbase/salt/docker/init.sls https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...

wyc 11y ago
Agreed. I have more specifics of Google-centralism in my comment here: https://news.ycombinator.com/item?id=9330049 https://news.ycombinator.com/item?id=9330049
- crb 11y ago
  In that post, you asked "Do you see any other providers here? https://github.com/GoogleCloudPlatform/kubernetes/tree/master/pkg/volume" https://github.com/GoogleCloudPlatform/kubernetes/tree/maste... Your implication was, at the time, there was only a GCE Persistent Disk provider. Today, there's that, plus AWS EBS volume, git repo, GlusterFS, NFS, Ceph block device, iSCSI, as well as host path and empty directory. Sounds like the product has evolved with a broad spectrum of support to me!
  - wyc 11y ago
    Yes, it has! Again, I don't think the design decisions were made to lock out other vendors. It's very reasonable to me that the first platform to be supported by Google engineers is Google's platform. Side note: I should be a better citizen and link to specific commits next time.
josephjacks 11y ago
I see Docker 1.6 in your [2] link, where do you see they are two versions behind Docker?

sunyc 11y ago

bundling with (unholy-ly immature) SDN is the most damning things for its adoption. It is thought to be needed for "live migration", but I don't see me needing that anytime soon because we run on virtual machines anyway? Iaas provider is not going away,paying for the cost of SDN now for features that doesn't even exists yet, is insane.

brendandburns 11y ago
(kubernetes contributor here) SDN isn't required for k8s, what is required is that each Pod (group of containers) get it's own IP address, and that the IP address is routeable in the cluster. In many cases, the easiest way to achieve this is via an SDN, but it is also achievable by programming traditional routers. The reason for wanting an IP address per pod is that it eliminates the need for port mangling, which dramatically simplifies wiring applications together.
- sunyc 11y ago
  All applications was already desinged to be port based. I don't see how this would drastically change that.
  - brendandburns 11y ago
    the problem with port mangling is that your application starts running on random ports, so in addition to requiring discovery for IP addresses, you now also have to do discovery for ports, which pretty much requires custom code and infrastructure linked into your binaries (how do you convince nginx/redis/... to use your lookup service for ports?) And ports are different between different replicas of your service, since they're chosen at random during scheduling. It also makes ACLs and QoS harder to define for the network, since you don't have a clean network identity (e.g IP Address) for each application.

sferoze 11y ago

An interesting sidenote. The Meteor development group is contributing to Kubernetes and will be using it to help scale Meteor with their upcoming paid service Galaxy.

josephjacks 11y ago
This is great to see independent software companies like Meteor embrace Kubernetes as the platform on which to build their next-generation services [0]. [0] http://info.meteor.com/blog/meteor-and-a-galaxy-of-containers-with-kubernetes http://info.meteor.com/blog/meteor-and-a-galaxy-of-container...

dcosson 11y ago

Does anyone have resources about security/isolation best practices for running multiple applications on Kubernetes (or Mesos or similar)? For instance in a non cloud-native app that runs in VM's, you might have one app per VM and have firewalls between different VM's that don't need to talk to each other. Then if a non-critical app got compromised and an attacker got remote execution or SQL injection or something they can't get to your other app servers or databases. If all your apps are in a cluster, the non-critical compromised app might be running on the same host as a critical app, in which case the only thing keeping the attacker from your database credentials or other secrets is the docker container isolation which if I understand correctly is not assumed to be secure the way VM isolation is. What are people doing to address this? Or are my assumptions wrong and it's not actually a problem to worry about? My initial impression with mesos was that you'd only use it if you're at big enough scale that you're running a huge number of instances of the same app or you're running a lot of different data processing tasks that all access the same data so no isolation is needed between them. Now I feel like I see Kubernetes being discussed frequently as a great way to run all your different microservices at any scale (e.g. "The Future of Deployment"), but I've never seen this aspect of security discussed.

jacques_chester 11y ago
You might prefer Cloud Foundry, which is switching its underlying container scheduling fabric to Lattice[1]. In particular, Cloud Foundry has more advanced security groups features, because it's mostly being marketed to enterprise customers. Disclaimer: I have worked on CF and I work for a company which is a major contributor to CF. [1] http://lattice.cf/ http://lattice.cf/
- dcosson 11y ago
  Lattice looks interesting, looking forward to checking it out more
brendandburns 11y ago
Please check out the Secrets object in Kubernetes: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/secrets.md https://github.com/GoogleCloudPlatform/kubernetes/blob/maste... which is designed to address some of this.