14 ms·

Kubernetes V1 Released

354 points by Oletros 11y ago

Oletros 11y ago
And related to that, CoreOS has launcehd a preview of its Kubernetes comercial platform [0] [0] http://techcrunch.com/2015/07/21/coreos-launches-preview-of-tectonic-its-commercial-kubernetes-platform/?ncid=rss http://techcrunch.com/2015/07/21/coreos-launches-preview-of-...
StevePerkins 11y ago
Can anyone recommend a good "Consultant's", "Solutions Architect's", or "Top-Right Quadrant on a Silly Gardner Chart" overview of how Kubernetes competes or cooperates with Mesos? It feels like there's quite a bit of dense conceptual reading you have to plow through before you can even start to talk about what these things do.
- ghshephard 11y ago
  Add to that, how does this relate to Docker-Swarm & Core-OS Fleet. StackOverflow has at least one quick briefer, but wow, this field is growing quickly: http://stackoverflow.com/questions/27640633/docker-swarm-kubernetes-mesos-core-os-fleet http://stackoverflow.com/questions/27640633/docker-swarm-kub...
  - waffle_ss 11y ago
    A couple things I've found when evaluating these: Docker Swarm seems to be the only one that supports one-off interactive containers that bind a TTY, like a Rails console (i.e. does the cluster support `docker run -it --rm busybox sh`). But its scheduling strategies[1] aren't as sophisticated as the others. Marathon doesn't support linked containers[2], so if you're using Mesos and need linked containers, you probably will want to run Kubernetes on it and use pods. [1]: https://github.com/docker/swarm/tree/b2182d080956040730cc76c31f322e637c85548a/scheduler/strategy https://github.com/docker/swarm/tree/b2182d080956040730cc76c... [2]: https://support.mesosphere.com/hc/en-us/articles/205006415-How-to-manage-applications-with-multiple-components-processes-on-Marathon- https://support.mesosphere.com/hc/en-us/articles/205006415-H...
    - throwaway1979 11y ago
      I've been playing with Swarm and like what I've seen so far. WRT Kubernetes, on a platform other than GCE, is Flannel required for networking?
      - amouat 11y ago
        I think you just need to provide a flat networking infrastructure where all nodes get their own IP and can reach each other. So you can swap flannel for project calico or your own setup if you like.
        eepyaich 11y ago
        Yep - see http://www.projectcalico.org/calico-networking-for-kubernetes/ http://www.projectcalico.org/calico-networking-for-kubernete.... [Disclosure - I'm part of the Project Calico team.]
        dward 11y ago
        Weave and Openvswitch are two other options. Many IaaS (gcp, aws) provide the required nobs and dials to configure this "natively" using their API so no extra SDN required if you are already running on a compatible cloud provider.
        throwaway1979 11y ago
        I thought each host needs an entire /24 subnet, and the pods that run on the host get individual IP addresses. Am I mistaken?
        TheIronYuppie 11y ago
        No, there's a private network that's shared between all the containers running on the pods, but as long as the nodes can see each other, you're good. Full disclosure: I work at Google on Kubernetes
      - TheIronYuppie 11y ago
        Networking on AWS works fine with no additional networking as well. Overlay networking is not required if you're running within a bunch of nodes that can see each other. Only if you get more complex will you require something, and there are quite a few solutions (Flannel, Weave, Calico, etc) Full disclosure: I work at Google on Kubernetes
- WestCoastJustin 11y ago
  At their most basic level, Kubernetes [1] and Mesos [2] both use a client/server type architecture, where you install a client on many compute nodes, then the server part farms out jobs (in the form of containers) to these client compute nodes. Mesos does not do scheduling, and is pretty low-level in terms of interfaces, so you would typically run some type of software which talks to it via API, like Marathon [3], Aurora [4], or Hadoop [5]. Then Marathon/Aurora/Hadoop/etc tells Mesos to farm out compute jobs to these end client nodes (aka schedules). Complexity can quickly go up depending on your hosting environment, scale, and HA requirements. There is actually a really good high-level overview diagram of what a mesos/hadoop/zookeeper setup looks like here [6]. The stack looks something like this for Kubernetes/Mesos (top down): - Kubernetes and Mesos (client/server packages depending on node type) - Docker (container engine) - OS (Ubuntu/RHEL/etc) What are some use-cases? - you have more work than can fit into one server - need to distribute load across N+ nodes - Google heavily uses containers (not k8s, but that inspired these patterns) - Gmail/Search/etc all run in containers [7] - Apple, Twitter, and Airbnb are running Mesos today [8, 9] There are a bunch of revolving services, like: - distributed key/values stores (etcd/zookeeper) - load balancers - image registries - user interfaces - cli tools - logging/monitoring/alerting - etc But, to answer your question, the main difference between Kubernetes and Mesos, is that Kubernetes offers an opinionated workflow, built-in scheduler, and patterns for how containers are deployed into this cluster of compute notes. The pattern is baked in from the start via Pods, Labels, Services, and Replication Controllers. It also helps to know that Kubernetes comes from Google, where they have been running containers in-house, so much of this workflow (pods, services, replication controllers, etc), comes from their internal use-cases. That's the 10,000 foot view. [1] https://github.com/GoogleCloudPlatform/kubernetes https://github.com/GoogleCloudPlatform/kubernetes [2] http://mesos.apache.org/ http://mesos.apache.org/ [3] https://mesosphere.github.io/marathon/ https://mesosphere.github.io/marathon/ [4] http://aurora.apache.org/ http://aurora.apache.org/ [5] https://hadoop.apache.org/ https://hadoop.apache.org/ [6] http://mesos.apache.org/assets/img/documentation/architecture3.jpg http://mesos.apache.org/assets/img/documentation/architectur... [7] http://www.wired.com/2013/03/google-borg-twitter-mesos/ http://www.wired.com/2013/03/google-borg-twitter-mesos/ [8] http://www.infoq.com/news/2015/05/mesos-powers-apple-siri http://www.infoq.com/news/2015/05/mesos-powers-apple-siri [9] https://www.youtube.com/watch?v=E4lxX6epM_U https://www.youtube.com/watch?v=E4lxX6epM_U
- mml 11y ago
  Kubernetes can run on Mesos as a Mesos service. Thereby giving you a more opinionated layout for your compute cluster, while running other peer Mesos services, such as Hadoop, Chronos And Marathon. https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/mesos.md https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...
  - ninkendo 11y ago
    I'm afraid answers like this actually make the confusion problem worse (nothing against your comment, just an observation in general.) If you're confused about the differences between similar-sounding products X and Y, the fact that "X runs on Y" or "Y supports X" has never made the situation any better, it only makes the line between X and Y even more blurred. I think this is especially true of Mesos, because people have a tendency to attribute qualities to Mesos that are actually qualities of a particular Mesos framework like Marathon or Aurora. As it is, Mesos is more of an SDK than anything, giving you the tools to write an orchestration system. It comes with built-in ways to communicate with nodes over a message bus, the ability to look at your nodes as resources, etc... but all of the scheduling logic is up to the frameworks themselves. I think Mesos has a perception problem because of this. They want to build up hype about what mesos is and can do, so they claim things like Mesos being able to schedule docker images and keep them running, etc... but that's really the job of something like Marathon that runs as a Mesos framework. But if they didn't claim such things, Mesos wouldn't seem very compelling. To me, the biggest benefit of Mesos is what the gain would be if every datacenter scheduler was a Mesos framework (yarn/spark/kubernetes/marathon/fleet/swarm/aurora, etc), and Mesos was only used to maintain multitenancy on the same hardware. That's where the real advantages come from... if you want to try Kubernetes you shouldn't have to dedicate hardware to it, you should just install it on your existing mesos cluster that is already running the rest of your stuff. In this respect, mesos is only useful insofar as all the big cluster managers use it as their underlying substrate.
    - tlrobinson 11y ago
      "X runs on Y" at least implies Y is "lower level" than X. As I understand it, Mesos is analogous to an operating system kernel for your cluster, while Kubernetes is a CaaS (containers-as-a-service) layer on top.
- MehdiEG 11y ago
  To add to discussion, I've found this interview with Malte Schwarzkopf to be quite enlighning to understand how Omega, Kubernetes and Mesos fit together: https://blog.kismatic.com/qa-with-malte-schwarzkopf-on-distributed-systems-orchestration-in-the-modern-data-center/ https://blog.kismatic.com/qa-with-malte-schwarzkopf-on-distr...
- bkeroack 11y ago
  To add to the others, in a nutshell: - Mesos is a generalized, low level framework for distributing workloads over multiple nodes. It provides mechanism not policy. Therefore it requires quite a bit of up-front work to build something usable for any given application. - Kubernetes is an opinionated cluster execution tool. It provides tools and a curated workflow for running distributed containerized applications. It's generally pretty quick and easy to get running. - Mesos has a rich, resource-aware task scheduler. You can specify that your application requires X CPU units and Y RAM units and it will find the optimum node to run the task on. - By contrast, the Kubernetes scheduler currently is rather dumb[1]. There's no way to specify the expected resource utilization for pods, and the scheduler simply tries to spread out replicas as much as possible throughout the available nodes. People are (rightly) excited about things like Mesosphere which could allow the best of both worlds: the ease and API of Kubernetes with a powerful Mesos resource scheduler, not to mention nice-to-haves like a Web UI with pretty visualizations. You can now cut me a check for 50% of the consulting revenue you get from this information. :) 1. The scheduler is intentionally simple and pluggable, to allow improvements easily in the future. My statements only apply to the current state of Kubernetes as deployed today.
  - davidooo 11y ago
    The Kubernetes scheduler also does resource-aware scheduling. You're correct that it tries to spread replicas across nodes, but it only spreads them across the nodes that have enough free resources for the container (more precisely, Pod) that it's scheduling.
    - bkeroack 11y ago
      Nowhere in the pod spec is there a way to specify resource constraints or even hints (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/api/v1beta3/types.go#L864-L897 https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...). So even if the scheduler is vaguely resource-aware (I'm not convinced that's true) it would be entirely static, based on things like container count.
      - 11y ago
kordless 11y ago
Wercker has a post on continous integration for containers to Kubernetes which Micha, their CEO, did back in June: http://blog.wercker.com/2015/06/23/Deploying-minimal-containers-to-kubernetes.html http://blog.wercker.com/2015/06/23/Deploying-minimal-contain..., which uses Google's Container Registry for pushing images. If anyone is interested, I just wrapped up a similar post for deploying containers to Giant Swarm from Wercker, no Docker required: I just got done doing a continuos integration post for containers using Wercker and Giant Swarm: https://github.com/giantswarm/swarm-wercker https://github.com/giantswarm/swarm-wercker.
jcastro 11y ago
If you're using Ubuntu we've got a bundle of Juju charms for those who want to get up and running: - http://insights.ubuntu.com/2015/07/21/introducing-kubernetes-version-1-0/ http://insights.ubuntu.com/2015/07/21/introducing-kubernetes... - https://jujucharms.com/u/kubernetes/kubernetes-cluster https://jujucharms.com/u/kubernetes/kubernetes-cluster PRs and comments welcome!
nickbauman 11y ago
I've found that Kubernetes is a big hammer. If your problem can be backed by a web app, you should start with AppEngine. If you need specialty library support or wider computational space per instance, you can move your AppEngine app to ManagedVMs (which uses Kubernetes under the covers). If you need special, "grid-like" services where you need exquisite control over the entire stack, only then does it make sense to use raw Kubernetes and Docker. And you will spend a lot of time getting it right.
- puja108 11y ago
  ManagedVMs don't use k8s under the covers afaik. However there's GKE (Google Container Engine), which goes on top of (at least 3) MVMs and that one does use k8s. While it is true that going with a PaaS(-like) solution like AppEngine or Heroku is easy in the beginning it can get pretty expensive pretty fast and it limits you in the choice of languages, frameworks, and data stores you can use. This can in some instances bring technical debt with it that will pose a hurdle when growing. Actually, using Docker combined with an orchestration layer like k8s is supposed to give the the ease-of-use of PaaS with the flexibility of IaaS (or MVMs), however, managing sth like k8s by yourself is not that easy and you will need quite a bunch of other tools on top, i.e. for monitoring and stuff, which paves the way for Container-as-a-Service solutions, like RancherOS, tutum, or Giant Swarm (disclaimer, I'm part of the latter company)
  - dlor 11y ago
    Correct, MVMs does not use Kubernetes today. We're looking at rebasing onto GKE now that they've hit 1.0, but nothing is set in stone yet. Disclaimer: I'm a tech lead on Managed VMs/App Engine at Google
    - alooPotato 11y ago
      Since you're already here, any chance you can talk a little bit more about upcoming plans for managed vms? I hear there is a lot of investment in that area, but what will that mean technically going forward? How are things going to change so we can plan better?
arianvanp 11y ago
I'm hoping for ACI support soon so that I can use rkt instead of docker. :-)
- TheIronYuppie 11y ago
  I have good news for you :) https://blog.kismatic.com/running-rkt-on-kubernetes/ https://blog.kismatic.com/running-rkt-on-kubernetes/ Full Disclosure: I work at Google on Kubernetes
  - arianvanp 11y ago
    Yes you can use rkt for docker images, but I want to be able to use ACI's. which doesn't seem possible yet[0]. But awesome that there's at least some support! :D [0] - https://github.com/GoogleCloudPlatform/kubernetes/issues/7203 https://github.com/GoogleCloudPlatform/kubernetes/issues/720...
wereHamster 11y ago
What's with the recent adoption of the .io TLD @ google? cncf.io, gcr.io, ...
- dguaraglia 11y ago
  I guess it's driven by the same motivation everyone else has: it's impossible to get a decent .com domain.
rjusher 11y ago
I might come across as ignorant but what is the relationship between Kubernetes and Docker, because when I was reading the article I tought of it as a Docker competitor, but further down in the comments, there is one that says they do different jobs. And that confused me.
- pat2man 11y ago
  Kubernetes is a scheduler for Docker containers. So lets say you want the following: 1 x MySQL Master 2 x MySQL Slave 5 x PHP Server 1 x Monitoring Script Each of those would be a docker container. Kubernetes would figure out which host to place them on and verify that they are running, rebooting them on another host if one of your hosts goes down.
- TheIronYuppie 11y ago
  Docker and Kubernetes works hand in hand. That is to say, if you choose Docker as your container format, Kubernetes runs Docker on every node to run your container. Kubernetes focuses on _what_ Docker should run, and how to move those container workloads around. Docker also has Docker Swarm, which can be thought of as a competitor in some ways. But Google will be a heavy supporter of their container format for a long time to come. Full Disclosure: I work at Google on Kubernetes
  - rjusher 11y ago
    So Kubernetes compliments Docker, but how it complements it. I had tested Docker just for fun, thinking that maybe I could implement it in the way I work, and sure it is a super tool for developing (far better than Virtual Machines), but deploying was kind of nightmerish, for what I understood Docker wasn't at the time ready for being a deployment tool. Does Kubernetes fixes or extends Docker in this way
    - matthewrudy 11y ago
      I have a docker image I want to run "myapp". So I tell Kubernetes "run me 5 instances of the image 'myapp', and expose port 8080" Kubernetes jobs is to start, monitor, and load balance those docker containers. Docker's job is to run each container.
jacques_chester 11y ago
Interesting to see IBM placing a bet each way by joining the new foundation, seeing as they're already in the Cloud Foundry Foundation. Edit: and I see the Cloud Foundry Foundation logo on the Cloud Native Foundation homepage. It's Foundations all the way down. (Disclaimer: I work for another CFF member, Pivotal).
- justincormack 11y ago
  Its pretty orthogonal to cloud foundry, which is a composite infrastructure, while these are lower level products that can be used to build PaaS.
  - jacques_chester 11y ago
    Right -- the analogous component is Diego. I guess I got caught up by the inside baseball.
geku 11y ago
Getting started with Kubernetes is pretty easy and I wrote a guide to quickly run it on your local machine: https://www.cloudgear.net/blog/2015/5-minutes-kubernetes-setup/ https://www.cloudgear.net/blog/2015/5-minutes-kubernetes-set... It's not up to date with Kubernetes 1.0.0 but I'll update the images as soon as the final version 1 is tagged.
- TheIronYuppie 11y ago
  I definitely don't want this to come off as a sales pitch, but you can get started in one click using Google Container Engine (and $300 in free credit) as well. Full disclosure: I work at Google on Kubernetes
  - teraflop 11y ago
    Except: "Sorry, you aren't eligible for a free trial at this time. The free trial is for new customers only." Apparently, the fact that I've been curious enough to experiment with other Google developer products in the past means I'm not part of the target audience.
    - pestaa 11y ago
      Or they assume your wallet is already open.
      - andybak 11y ago
        Then they are wrong. I actually find this a common issue with a presumed sales pipeline I encounter. They think: 1. He finds us. 2. He's interested and signs up for a trial 3. We hopefully convert before the trial is over What actually tends to happen 1. I find something that looks interesting 2. I sign up 3. Real work intervenes 4. Several months later I have some time to look again but my trial has expired. To be fair most companies respond to a quick email but they could be proactive and do the following: 1. If no activity is detected after the first day pause the trial 2. Some time later send an email saying "We've paused your trial. Please choose either: 1. to reactivate it, 2. be reminded in another x weeks or 3. never hear from us again.
barkingcat 11y ago
how do you pronounce that word?
- packetslave 11y ago
  coo-bur-net-eees
smegel 11y ago
> containers scheduled < 5s on average That seems awfully slow for a fancy chroot. I use KVM to bring up WinXP snapshot VMs in around 2s to a running state...maybe they mean 5ms?
- cjcullen 11y ago
  5s is the right number, but it leaves out that it is 99%ile, on a 100-node, 3000-pod cluster. See https://github.com/GoogleCloudPlatform/kubernetes/issues/3954 https://github.com/GoogleCloudPlatform/kubernetes/issues/395....
- zwischenzug 11y ago
  It's the cadence of the etcd read cycle chosen for kubernetes. They could do it much faster, but I guess there are engineering tradeoffs based on the size of the cluster and the speed of the network.
alanh 11y ago
Is Kubernetes pronounced "koo bur NET ease" or am I way off?
- a-robinson 11y ago
  Yes, that's how it's pronounced.

354 points by Oletros 11y ago

And related to that, CoreOS has launcehd a preview of its Kubernetes comercial platform [0] [0] http://techcrunch.com/2015/07/21/coreos-launches-preview-of-tectonic-its-commercial-kubernetes-platform/?ncid=rss http://techcrunch.com/2015/07/21/coreos-launches-preview-of-...

StevePerkins 11y ago

Can anyone recommend a good "Consultant's", "Solutions Architect's", or "Top-Right Quadrant on a Silly Gardner Chart" overview of how Kubernetes competes or cooperates with Mesos? It feels like there's quite a bit of dense conceptual reading you have to plow through before you can even start to talk about what these things do.

ghshephard 11y ago
Add to that, how does this relate to Docker-Swarm & Core-OS Fleet. StackOverflow has at least one quick briefer, but wow, this field is growing quickly: http://stackoverflow.com/questions/27640633/docker-swarm-kubernetes-mesos-core-os-fleet http://stackoverflow.com/questions/27640633/docker-swarm-kub...
- waffle_ss 11y ago
  A couple things I've found when evaluating these: Docker Swarm seems to be the only one that supports one-off interactive containers that bind a TTY, like a Rails console (i.e. does the cluster support `docker run -it --rm busybox sh`). But its scheduling strategies[1] aren't as sophisticated as the others. Marathon doesn't support linked containers[2], so if you're using Mesos and need linked containers, you probably will want to run Kubernetes on it and use pods. [1]: https://github.com/docker/swarm/tree/b2182d080956040730cc76c31f322e637c85548a/scheduler/strategy https://github.com/docker/swarm/tree/b2182d080956040730cc76c... [2]: https://support.mesosphere.com/hc/en-us/articles/205006415-How-to-manage-applications-with-multiple-components-processes-on-Marathon- https://support.mesosphere.com/hc/en-us/articles/205006415-H...
  - throwaway1979 11y ago
    I've been playing with Swarm and like what I've seen so far. WRT Kubernetes, on a platform other than GCE, is Flannel required for networking?
    - amouat 11y ago
      I think you just need to provide a flat networking infrastructure where all nodes get their own IP and can reach each other. So you can swap flannel for project calico or your own setup if you like.
      - eepyaich 11y ago
        Yep - see http://www.projectcalico.org/calico-networking-for-kubernetes/ http://www.projectcalico.org/calico-networking-for-kubernete.... [Disclosure - I'm part of the Project Calico team.]
      - dward 11y ago
        Weave and Openvswitch are two other options. Many IaaS (gcp, aws) provide the required nobs and dials to configure this "natively" using their API so no extra SDN required if you are already running on a compatible cloud provider.
      - throwaway1979 11y ago
        I thought each host needs an entire /24 subnet, and the pods that run on the host get individual IP addresses. Am I mistaken?
        TheIronYuppie 11y ago
        No, there's a private network that's shared between all the containers running on the pods, but as long as the nodes can see each other, you're good. Full disclosure: I work at Google on Kubernetes
    - TheIronYuppie 11y ago
      Networking on AWS works fine with no additional networking as well. Overlay networking is not required if you're running within a bunch of nodes that can see each other. Only if you get more complex will you require something, and there are quite a few solutions (Flannel, Weave, Calico, etc) Full disclosure: I work at Google on Kubernetes
WestCoastJustin 11y ago
At their most basic level, Kubernetes [1] and Mesos [2] both use a client/server type architecture, where you install a client on many compute nodes, then the server part farms out jobs (in the form of containers) to these client compute nodes. Mesos does not do scheduling, and is pretty low-level in terms of interfaces, so you would typically run some type of software which talks to it via API, like Marathon [3], Aurora [4], or Hadoop [5]. Then Marathon/Aurora/Hadoop/etc tells Mesos to farm out compute jobs to these end client nodes (aka schedules). Complexity can quickly go up depending on your hosting environment, scale, and HA requirements. There is actually a really good high-level overview diagram of what a mesos/hadoop/zookeeper setup looks like here [6]. The stack looks something like this for Kubernetes/Mesos (top down): - Kubernetes and Mesos (client/server packages depending on node type) - Docker (container engine) - OS (Ubuntu/RHEL/etc) What are some use-cases? - you have more work than can fit into one server - need to distribute load across N+ nodes - Google heavily uses containers (not k8s, but that inspired these patterns) - Gmail/Search/etc all run in containers [7] - Apple, Twitter, and Airbnb are running Mesos today [8, 9] There are a bunch of revolving services, like: - distributed key/values stores (etcd/zookeeper) - load balancers - image registries - user interfaces - cli tools - logging/monitoring/alerting - etc But, to answer your question, the main difference between Kubernetes and Mesos, is that Kubernetes offers an opinionated workflow, built-in scheduler, and patterns for how containers are deployed into this cluster of compute notes. The pattern is baked in from the start via Pods, Labels, Services, and Replication Controllers. It also helps to know that Kubernetes comes from Google, where they have been running containers in-house, so much of this workflow (pods, services, replication controllers, etc), comes from their internal use-cases. That's the 10,000 foot view. [1] https://github.com/GoogleCloudPlatform/kubernetes https://github.com/GoogleCloudPlatform/kubernetes [2] http://mesos.apache.org/ http://mesos.apache.org/ [3] https://mesosphere.github.io/marathon/ https://mesosphere.github.io/marathon/ [4] http://aurora.apache.org/ http://aurora.apache.org/ [5] https://hadoop.apache.org/ https://hadoop.apache.org/ [6] http://mesos.apache.org/assets/img/documentation/architecture3.jpg http://mesos.apache.org/assets/img/documentation/architectur... [7] http://www.wired.com/2013/03/google-borg-twitter-mesos/ http://www.wired.com/2013/03/google-borg-twitter-mesos/ [8] http://www.infoq.com/news/2015/05/mesos-powers-apple-siri http://www.infoq.com/news/2015/05/mesos-powers-apple-siri [9] https://www.youtube.com/watch?v=E4lxX6epM_U https://www.youtube.com/watch?v=E4lxX6epM_U
mml 11y ago
Kubernetes can run on Mesos as a Mesos service. Thereby giving you a more opinionated layout for your compute cluster, while running other peer Mesos services, such as Hadoop, Chronos And Marathon. https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/mesos.md https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...
- ninkendo 11y ago
  I'm afraid answers like this actually make the confusion problem worse (nothing against your comment, just an observation in general.) If you're confused about the differences between similar-sounding products X and Y, the fact that "X runs on Y" or "Y supports X" has never made the situation any better, it only makes the line between X and Y even more blurred. I think this is especially true of Mesos, because people have a tendency to attribute qualities to Mesos that are actually qualities of a particular Mesos framework like Marathon or Aurora. As it is, Mesos is more of an SDK than anything, giving you the tools to write an orchestration system. It comes with built-in ways to communicate with nodes over a message bus, the ability to look at your nodes as resources, etc... but all of the scheduling logic is up to the frameworks themselves. I think Mesos has a perception problem because of this. They want to build up hype about what mesos is and can do, so they claim things like Mesos being able to schedule docker images and keep them running, etc... but that's really the job of something like Marathon that runs as a Mesos framework. But if they didn't claim such things, Mesos wouldn't seem very compelling. To me, the biggest benefit of Mesos is what the gain would be if every datacenter scheduler was a Mesos framework (yarn/spark/kubernetes/marathon/fleet/swarm/aurora, etc), and Mesos was only used to maintain multitenancy on the same hardware. That's where the real advantages come from... if you want to try Kubernetes you shouldn't have to dedicate hardware to it, you should just install it on your existing mesos cluster that is already running the rest of your stuff. In this respect, mesos is only useful insofar as all the big cluster managers use it as their underlying substrate.
  - tlrobinson 11y ago
    "X runs on Y" at least implies Y is "lower level" than X. As I understand it, Mesos is analogous to an operating system kernel for your cluster, while Kubernetes is a CaaS (containers-as-a-service) layer on top.
MehdiEG 11y ago
To add to discussion, I've found this interview with Malte Schwarzkopf to be quite enlighning to understand how Omega, Kubernetes and Mesos fit together: https://blog.kismatic.com/qa-with-malte-schwarzkopf-on-distributed-systems-orchestration-in-the-modern-data-center/ https://blog.kismatic.com/qa-with-malte-schwarzkopf-on-distr...
bkeroack 11y ago
To add to the others, in a nutshell: - Mesos is a generalized, low level framework for distributing workloads over multiple nodes. It provides mechanism not policy. Therefore it requires quite a bit of up-front work to build something usable for any given application. - Kubernetes is an opinionated cluster execution tool. It provides tools and a curated workflow for running distributed containerized applications. It's generally pretty quick and easy to get running. - Mesos has a rich, resource-aware task scheduler. You can specify that your application requires X CPU units and Y RAM units and it will find the optimum node to run the task on. - By contrast, the Kubernetes scheduler currently is rather dumb[1]. There's no way to specify the expected resource utilization for pods, and the scheduler simply tries to spread out replicas as much as possible throughout the available nodes. People are (rightly) excited about things like Mesosphere which could allow the best of both worlds: the ease and API of Kubernetes with a powerful Mesos resource scheduler, not to mention nice-to-haves like a Web UI with pretty visualizations. You can now cut me a check for 50% of the consulting revenue you get from this information. :) 1. The scheduler is intentionally simple and pluggable, to allow improvements easily in the future. My statements only apply to the current state of Kubernetes as deployed today.
- davidooo 11y ago
  The Kubernetes scheduler also does resource-aware scheduling. You're correct that it tries to spread replicas across nodes, but it only spreads them across the nodes that have enough free resources for the container (more precisely, Pod) that it's scheduling.
  - bkeroack 11y ago
    Nowhere in the pod spec is there a way to specify resource constraints or even hints (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/api/v1beta3/types.go#L864-L897 https://github.com/GoogleCloudPlatform/kubernetes/blob/maste...). So even if the scheduler is vaguely resource-aware (I'm not convinced that's true) it would be entirely static, based on things like container count.
    - 11y ago

kordless 11y ago

Wercker has a post on continous integration for containers to Kubernetes which Micha, their CEO, did back in June: http://blog.wercker.com/2015/06/23/Deploying-minimal-containers-to-kubernetes.html http://blog.wercker.com/2015/06/23/Deploying-minimal-contain..., which uses Google's Container Registry for pushing images. If anyone is interested, I just wrapped up a similar post for deploying containers to Giant Swarm from Wercker, no Docker required: I just got done doing a continuos integration post for containers using Wercker and Giant Swarm: https://github.com/giantswarm/swarm-wercker https://github.com/giantswarm/swarm-wercker.

jcastro 11y ago

If you're using Ubuntu we've got a bundle of Juju charms for those who want to get up and running: - http://insights.ubuntu.com/2015/07/21/introducing-kubernetes-version-1-0/ http://insights.ubuntu.com/2015/07/21/introducing-kubernetes... - https://jujucharms.com/u/kubernetes/kubernetes-cluster https://jujucharms.com/u/kubernetes/kubernetes-cluster PRs and comments welcome!

nickbauman 11y ago

I've found that Kubernetes is a big hammer. If your problem can be backed by a web app, you should start with AppEngine. If you need specialty library support or wider computational space per instance, you can move your AppEngine app to ManagedVMs (which uses Kubernetes under the covers). If you need special, "grid-like" services where you need exquisite control over the entire stack, only then does it make sense to use raw Kubernetes and Docker. And you will spend a lot of time getting it right.

puja108 11y ago
ManagedVMs don't use k8s under the covers afaik. However there's GKE (Google Container Engine), which goes on top of (at least 3) MVMs and that one does use k8s. While it is true that going with a PaaS(-like) solution like AppEngine or Heroku is easy in the beginning it can get pretty expensive pretty fast and it limits you in the choice of languages, frameworks, and data stores you can use. This can in some instances bring technical debt with it that will pose a hurdle when growing. Actually, using Docker combined with an orchestration layer like k8s is supposed to give the the ease-of-use of PaaS with the flexibility of IaaS (or MVMs), however, managing sth like k8s by yourself is not that easy and you will need quite a bunch of other tools on top, i.e. for monitoring and stuff, which paves the way for Container-as-a-Service solutions, like RancherOS, tutum, or Giant Swarm (disclaimer, I'm part of the latter company)
- dlor 11y ago
  Correct, MVMs does not use Kubernetes today. We're looking at rebasing onto GKE now that they've hit 1.0, but nothing is set in stone yet. Disclaimer: I'm a tech lead on Managed VMs/App Engine at Google
  - alooPotato 11y ago
    Since you're already here, any chance you can talk a little bit more about upcoming plans for managed vms? I hear there is a lot of investment in that area, but what will that mean technically going forward? How are things going to change so we can plan better?

arianvanp 11y ago

I'm hoping for ACI support soon so that I can use rkt instead of docker. :-)

TheIronYuppie 11y ago
I have good news for you :) https://blog.kismatic.com/running-rkt-on-kubernetes/ https://blog.kismatic.com/running-rkt-on-kubernetes/ Full Disclosure: I work at Google on Kubernetes
- arianvanp 11y ago
  Yes you can use rkt for docker images, but I want to be able to use ACI's. which doesn't seem possible yet[0]. But awesome that there's at least some support! :D [0] - https://github.com/GoogleCloudPlatform/kubernetes/issues/7203 https://github.com/GoogleCloudPlatform/kubernetes/issues/720...

wereHamster 11y ago

What's with the recent adoption of the .io TLD @ google? cncf.io, gcr.io, ...

dguaraglia 11y ago
I guess it's driven by the same motivation everyone else has: it's impossible to get a decent .com domain.

rjusher 11y ago

I might come across as ignorant but what is the relationship between Kubernetes and Docker, because when I was reading the article I tought of it as a Docker competitor, but further down in the comments, there is one that says they do different jobs. And that confused me.

pat2man 11y ago
Kubernetes is a scheduler for Docker containers. So lets say you want the following: 1 x MySQL Master 2 x MySQL Slave 5 x PHP Server 1 x Monitoring Script Each of those would be a docker container. Kubernetes would figure out which host to place them on and verify that they are running, rebooting them on another host if one of your hosts goes down.
TheIronYuppie 11y ago
Docker and Kubernetes works hand in hand. That is to say, if you choose Docker as your container format, Kubernetes runs Docker on every node to run your container. Kubernetes focuses on _what_ Docker should run, and how to move those container workloads around. Docker also has Docker Swarm, which can be thought of as a competitor in some ways. But Google will be a heavy supporter of their container format for a long time to come. Full Disclosure: I work at Google on Kubernetes
- rjusher 11y ago
  So Kubernetes compliments Docker, but how it complements it. I had tested Docker just for fun, thinking that maybe I could implement it in the way I work, and sure it is a super tool for developing (far better than Virtual Machines), but deploying was kind of nightmerish, for what I understood Docker wasn't at the time ready for being a deployment tool. Does Kubernetes fixes or extends Docker in this way
  - matthewrudy 11y ago
    I have a docker image I want to run "myapp". So I tell Kubernetes "run me 5 instances of the image 'myapp', and expose port 8080" Kubernetes jobs is to start, monitor, and load balance those docker containers. Docker's job is to run each container.

jacques_chester 11y ago

Interesting to see IBM placing a bet each way by joining the new foundation, seeing as they're already in the Cloud Foundry Foundation. Edit: and I see the Cloud Foundry Foundation logo on the Cloud Native Foundation homepage. It's Foundations all the way down. (Disclaimer: I work for another CFF member, Pivotal).

justincormack 11y ago
Its pretty orthogonal to cloud foundry, which is a composite infrastructure, while these are lower level products that can be used to build PaaS.
- jacques_chester 11y ago
  Right -- the analogous component is Diego. I guess I got caught up by the inside baseball.

geku 11y ago

Getting started with Kubernetes is pretty easy and I wrote a guide to quickly run it on your local machine: https://www.cloudgear.net/blog/2015/5-minutes-kubernetes-setup/ https://www.cloudgear.net/blog/2015/5-minutes-kubernetes-set... It's not up to date with Kubernetes 1.0.0 but I'll update the images as soon as the final version 1 is tagged.

TheIronYuppie 11y ago
I definitely don't want this to come off as a sales pitch, but you can get started in one click using Google Container Engine (and $300 in free credit) as well. Full disclosure: I work at Google on Kubernetes
- teraflop 11y ago
  Except: "Sorry, you aren't eligible for a free trial at this time. The free trial is for new customers only." Apparently, the fact that I've been curious enough to experiment with other Google developer products in the past means I'm not part of the target audience.
  - pestaa 11y ago
    Or they assume your wallet is already open.
    - andybak 11y ago
      Then they are wrong. I actually find this a common issue with a presumed sales pipeline I encounter. They think: 1. He finds us. 2. He's interested and signs up for a trial 3. We hopefully convert before the trial is over What actually tends to happen 1. I find something that looks interesting 2. I sign up 3. Real work intervenes 4. Several months later I have some time to look again but my trial has expired. To be fair most companies respond to a quick email but they could be proactive and do the following: 1. If no activity is detected after the first day pause the trial 2. Some time later send an email saying "We've paused your trial. Please choose either: 1. to reactivate it, 2. be reminded in another x weeks or 3. never hear from us again.

barkingcat 11y ago

how do you pronounce that word?

packetslave 11y ago
coo-bur-net-eees

smegel 11y ago

> containers scheduled < 5s on average That seems awfully slow for a fancy chroot. I use KVM to bring up WinXP snapshot VMs in around 2s to a running state...maybe they mean 5ms?

cjcullen 11y ago
5s is the right number, but it leaves out that it is 99%ile, on a 100-node, 3000-pod cluster. See https://github.com/GoogleCloudPlatform/kubernetes/issues/3954 https://github.com/GoogleCloudPlatform/kubernetes/issues/395....
zwischenzug 11y ago
It's the cadence of the etcd read cycle chosen for kubernetes. They could do it much faster, but I guess there are engineering tradeoffs based on the size of the cluster and the speed of the network.

alanh 11y ago

Is Kubernetes pronounced "koo bur NET ease" or am I way off?

a-robinson 11y ago
Yes, that's how it's pronounced.