8 ms·

Official Kubernetes on CoreOS Guides and Tools

125 points by philips 11y ago

flyt 11y ago
Has Kubernetes grown past its limit of ~100 underlying servers? What's the largest production Kubernetes cluster running today?
- philips 11y ago
  Yes, the community is working hard to make k8s scale to 1000s of machines and beyond. There is a great blog post about the engineering effort from the various communities (k8s, etcd) to make this happen: http://blog.kubernetes.io/2015/09/kubernetes-performance-measurements-and.html http://blog.kubernetes.io/2015/09/kubernetes-performance-mea... In fact someone recently made a video showing a 1000 machine cluster; let me find that. UPDATE: here is the video https://www.youtube.com/watch?v=sgWv9sVTIYQ https://www.youtube.com/watch?v=sgWv9sVTIYQ
  - flyt 11y ago
    The last time that I saw somebody from Google give a presentation about k8 it didn't seem like "larger cluster sizes" was a real priority at this point, and they were more focused on features. Good to hear this is getting some attention, since it was the major issue that made our team dismiss it as useful in production.
    - vruiz 11y ago
      May I ask at which scale do you operate? Keep in mind that is not necessary (and I believe not advised[1]) to keep all your infrastructure in one kubernetes cluster. [1] https://youtu.be/-8aUxpVrD40?t=3105 https://youtu.be/-8aUxpVrD40?t=3105
    - jcastro 11y ago
      I've heard googlers mention that same thing, but now that I think about it I believe they were meaning to say that "for 1.0" they were targeting ~100 nodes, and once they got out there door they would start tackling the large cluster case, which appears to be progressing nicely.
  - merb 11y ago
    why don't they add something like master slaves to make that happen? i mean 1000s machines for a single master is way more than most software will handle anyway, still kubernetes makes it easy to manage it, however even 50 machines are working really fine on kubernetes and that even with a slower master server. currently we run with 20s at production with a master at 4 gigs which works really really well.
    - robszumski 11y ago
      There is an open PR to this project to document an HA control plane. Basically, you add the kube-podmaster onto each master machine, which does a leader election for the API, controller manager, etc so that only one instance is mutating the cluster. If that does down, another leader election happens.
    - bob_wise 11y ago
      The "master" in kubernetes is a collection of components that can be distributed. The kubernetes component that is the most loaded in a 1000 node system is the api server. The 1000 node cluster config in that video is running 10 api servers. I'm the author of that video, BTW, so I know that for sure. :-)
  - bob_wise 11y ago
    Here's an updated video with a voiceover... https://www.youtube.com/watch?v=1bHUsBhPL20 https://www.youtube.com/watch?v=1bHUsBhPL20
- Sym3tri 11y ago
  Support for 250 nodes is scheduled for October. https://github.com/kubernetes/kubernetes/blob/master/docs/roadmap.md#kubernetes-11 https://github.com/kubernetes/kubernetes/blob/master/docs/ro...
philips 11y ago
If you are looking for the fastest possible guide for playing around with Kubernetes check out the vagrant single machine instructions: https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant-single.html https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagr... This will get you to a "single machine" kubernetes cluster and a working `kubectl` tool in a few minutes.
- wstrange 11y ago
  This is awesome. The existing single node solutions have been painful to get up and running, and always seem to be missing some feature or other (DNS, for example).
  - hosh 11y ago
    I just spent 30 hours figuring that out (DNS, flanneld), and that's with using hyperkube. (I'm building a new framework that helps you build a custom dev-environment for your workflow using Kubernetes, called Matsuri). I don't regret it, since I know a lot more about the details of how this works together, and this will help a lot when it comes time to putting this on AWS for production. But I don't want to force this on my team either, so I'll end up putting something together with Vagrant (or finding something that works with it). One thing I realized though: this stuff should be baked in as part of a standard Linux distro. A working, single-node Kubernetes setup works far better than docker-compose / fig / crane / any number of homebrewed orchestration for dev work. No wonder people are using CoreOS and RHEL 7/Atomic, and not Ubuntu. It's seriously making me consider switching to one of those, off of Ubuntu.
    - joshuak 11y ago
      You can always still use ubuntu from within a container on CoreOS if you like ;)
      - hosh 11y ago
        I know. I'm just not as familiar with CoreOS.
bkruse 11y ago
We've been using CoreOS in production with etcd and fleet for over 1 year now on 500+ machines. They're have been some growth pains, specifically with etcd - but now they are mostly gone (with the new raft implementation in 2.x). I really appreciate the CoreOS team and all related contributors. First persistent storage tackled, then networking and now resource-aware orchestration.
- tracker1 11y ago
  How has etcd been running for you more recently? One of the things that kept me away was my test run of etcd failed gloriously (twice), and just wasn't stable/resiliant enough for me.
  - unihorn 11y ago
    We published etcd 2.x earlier this year, and it is super stable in current stage. We run harsh failure injection tests on it all the time, and the cluster could survive well. You could check https://coreos.com/blog/new-functional-testing-in-etcd/ https://coreos.com/blog/new-functional-testing-in-etcd/ for more details.
    - mattkrea 11y ago
      We're about to start running with it but I am definitely a little concerned at the failover table. I'm a lot more comfortable when I can achieve 2/3 failure and still be okay although the cost of adding a few more instances isn't too bad.
      - philips 11y ago
        To avoid split-brain a consensus system like etcd cannot make progress with less than 50% + 1 members operating. This is just a hard constraint of this type of distributed system. You can find resources that explain all of the constraints in depth at the Raft homepage: https://raft.github.io/ https://raft.github.io/
saosebastiao 11y ago
Anybody know of any projects of a similar concept but applying to unikernels instead of containers? I've had way more luck working with OSv than I have had with Docker and I'm kinda burnt out with futzing with it...but I do like the ideas behind kubernetes.
- justincormack 11y ago
  There is no fundamental reason why you shouldnt use the same tools. There is a bit of work required to get it going, around standards and so on that I am looking at.
yuvipanda 11y ago
Nice! Wikimedia Labs is going to start offering Volunteers who want to do any Wikimedia related projects Kubernetes access for free soon (https://lists.wikimedia.org/pipermail/labs-l/2015-September/004033.html https://lists.wikimedia.org/pipermail/labs-l/2015-September/...) although that isn't going to be on CoreOS.
Perceptes 11y ago
How does having the kubelet binary shipped with CoreOS itself work with installing and upgrading Kubernetes? Won't the versions get out of sync and cause problems?
- philips 11y ago
  We rely on the API boundaries between the kubelet and the kubernetes API server remaining well versioned without backwards compatible changes. The k8s project is very conscious of the needs for that and work hard to ensure that there are no breaking changes between releases. This is similar to all pieces of software in a CoreOS release: systemd, docker, cloudinit, Linux, rkt, etc.
  - Perceptes 11y ago
    I see. I suppose the only down side of using the CoreOS version is having to wait for it to be updated when new minor releases come out with new features/improvements.
arunoda 11y ago
Here's another way to jumpstart and learn Kubernetes. Copy and paste this single command into any Ubuntu 64bit VM and it'll give you a single node Kubernetes cluster with all the tools and services(incl. DNS) wget -qO- http://git.io/veKlu | sudo sh Read More Here: https://github.com/meteorhacks/kube-init https://github.com/meteorhacks/kube-init
- ImJasonH 11y ago
  Piping a random shortened link to sudo sh? What could go wrong?

125 points by philips 11y ago

flyt 11y ago

Has Kubernetes grown past its limit of ~100 underlying servers? What's the largest production Kubernetes cluster running today?

philips 11y ago
Yes, the community is working hard to make k8s scale to 1000s of machines and beyond. There is a great blog post about the engineering effort from the various communities (k8s, etcd) to make this happen: http://blog.kubernetes.io/2015/09/kubernetes-performance-measurements-and.html http://blog.kubernetes.io/2015/09/kubernetes-performance-mea... In fact someone recently made a video showing a 1000 machine cluster; let me find that. UPDATE: here is the video https://www.youtube.com/watch?v=sgWv9sVTIYQ https://www.youtube.com/watch?v=sgWv9sVTIYQ
- flyt 11y ago
  The last time that I saw somebody from Google give a presentation about k8 it didn't seem like "larger cluster sizes" was a real priority at this point, and they were more focused on features. Good to hear this is getting some attention, since it was the major issue that made our team dismiss it as useful in production.
  - vruiz 11y ago
    May I ask at which scale do you operate? Keep in mind that is not necessary (and I believe not advised[1]) to keep all your infrastructure in one kubernetes cluster. [1] https://youtu.be/-8aUxpVrD40?t=3105 https://youtu.be/-8aUxpVrD40?t=3105
  - jcastro 11y ago
    I've heard googlers mention that same thing, but now that I think about it I believe they were meaning to say that "for 1.0" they were targeting ~100 nodes, and once they got out there door they would start tackling the large cluster case, which appears to be progressing nicely.
- merb 11y ago
  why don't they add something like master slaves to make that happen? i mean 1000s machines for a single master is way more than most software will handle anyway, still kubernetes makes it easy to manage it, however even 50 machines are working really fine on kubernetes and that even with a slower master server. currently we run with 20s at production with a master at 4 gigs which works really really well.
  - robszumski 11y ago
    There is an open PR to this project to document an HA control plane. Basically, you add the kube-podmaster onto each master machine, which does a leader election for the API, controller manager, etc so that only one instance is mutating the cluster. If that does down, another leader election happens.
  - bob_wise 11y ago
    The "master" in kubernetes is a collection of components that can be distributed. The kubernetes component that is the most loaded in a 1000 node system is the api server. The 1000 node cluster config in that video is running 10 api servers. I'm the author of that video, BTW, so I know that for sure. :-)
- bob_wise 11y ago
  Here's an updated video with a voiceover... https://www.youtube.com/watch?v=1bHUsBhPL20 https://www.youtube.com/watch?v=1bHUsBhPL20
Sym3tri 11y ago
Support for 250 nodes is scheduled for October. https://github.com/kubernetes/kubernetes/blob/master/docs/roadmap.md#kubernetes-11 https://github.com/kubernetes/kubernetes/blob/master/docs/ro...

philips 11y ago

If you are looking for the fastest possible guide for playing around with Kubernetes check out the vagrant single machine instructions: https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant-single.html https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagr... This will get you to a "single machine" kubernetes cluster and a working `kubectl` tool in a few minutes.

wstrange 11y ago
This is awesome. The existing single node solutions have been painful to get up and running, and always seem to be missing some feature or other (DNS, for example).
- hosh 11y ago
  I just spent 30 hours figuring that out (DNS, flanneld), and that's with using hyperkube. (I'm building a new framework that helps you build a custom dev-environment for your workflow using Kubernetes, called Matsuri). I don't regret it, since I know a lot more about the details of how this works together, and this will help a lot when it comes time to putting this on AWS for production. But I don't want to force this on my team either, so I'll end up putting something together with Vagrant (or finding something that works with it). One thing I realized though: this stuff should be baked in as part of a standard Linux distro. A working, single-node Kubernetes setup works far better than docker-compose / fig / crane / any number of homebrewed orchestration for dev work. No wonder people are using CoreOS and RHEL 7/Atomic, and not Ubuntu. It's seriously making me consider switching to one of those, off of Ubuntu.
  - joshuak 11y ago
    You can always still use ubuntu from within a container on CoreOS if you like ;)
    - hosh 11y ago
      I know. I'm just not as familiar with CoreOS.

bkruse 11y ago

We've been using CoreOS in production with etcd and fleet for over 1 year now on 500+ machines. They're have been some growth pains, specifically with etcd - but now they are mostly gone (with the new raft implementation in 2.x). I really appreciate the CoreOS team and all related contributors. First persistent storage tackled, then networking and now resource-aware orchestration.

tracker1 11y ago
How has etcd been running for you more recently? One of the things that kept me away was my test run of etcd failed gloriously (twice), and just wasn't stable/resiliant enough for me.
- unihorn 11y ago
  We published etcd 2.x earlier this year, and it is super stable in current stage. We run harsh failure injection tests on it all the time, and the cluster could survive well. You could check https://coreos.com/blog/new-functional-testing-in-etcd/ https://coreos.com/blog/new-functional-testing-in-etcd/ for more details.
  - mattkrea 11y ago
    We're about to start running with it but I am definitely a little concerned at the failover table. I'm a lot more comfortable when I can achieve 2/3 failure and still be okay although the cost of adding a few more instances isn't too bad.
    - philips 11y ago
      To avoid split-brain a consensus system like etcd cannot make progress with less than 50% + 1 members operating. This is just a hard constraint of this type of distributed system. You can find resources that explain all of the constraints in depth at the Raft homepage: https://raft.github.io/ https://raft.github.io/

saosebastiao 11y ago

Anybody know of any projects of a similar concept but applying to unikernels instead of containers? I've had way more luck working with OSv than I have had with Docker and I'm kinda burnt out with futzing with it...but I do like the ideas behind kubernetes.

justincormack 11y ago
There is no fundamental reason why you shouldnt use the same tools. There is a bit of work required to get it going, around standards and so on that I am looking at.

yuvipanda 11y ago

Nice! Wikimedia Labs is going to start offering Volunteers who want to do any Wikimedia related projects Kubernetes access for free soon (https://lists.wikimedia.org/pipermail/labs-l/2015-September/004033.html https://lists.wikimedia.org/pipermail/labs-l/2015-September/...) although that isn't going to be on CoreOS.

Perceptes 11y ago

How does having the kubelet binary shipped with CoreOS itself work with installing and upgrading Kubernetes? Won't the versions get out of sync and cause problems?

philips 11y ago
We rely on the API boundaries between the kubelet and the kubernetes API server remaining well versioned without backwards compatible changes. The k8s project is very conscious of the needs for that and work hard to ensure that there are no breaking changes between releases. This is similar to all pieces of software in a CoreOS release: systemd, docker, cloudinit, Linux, rkt, etc.
- Perceptes 11y ago
  I see. I suppose the only down side of using the CoreOS version is having to wait for it to be updated when new minor releases come out with new features/improvements.

arunoda 11y ago

Here's another way to jumpstart and learn Kubernetes. Copy and paste this single command into any Ubuntu 64bit VM and it'll give you a single node Kubernetes cluster with all the tools and services(incl. DNS) wget -qO- http://git.io/veKlu | sudo sh Read More Here: https://github.com/meteorhacks/kube-init https://github.com/meteorhacks/kube-init

ImJasonH 11y ago
Piping a random shortened link to sudo sh? What could go wrong?