16 ms·

Kubernetes the Hard Way

240 points by tantalor 10y ago

mattikus 10y ago
Just saw Kelsey give a talk at Abstractions about more advanced patterns in Kubernetes and he mentioned this repo. Looks like a fantastic tutorial and his talk was very informative. Highly recommend watching the video when it's released if you weren't able to attend.
- imglorp 10y ago
  We were in the same room. Yes, good talk.
- mVChr 10y ago
  He gave a great live demo at CodeConf about using Kubernetes for 12-factor apps[1] that I highly recommend as well. The video for the talk isn't up yet, but the code he used is.[2] [1] https://12factor.net/ https://12factor.net/ [2] https://github.com/kelseyhightower/app https://github.com/kelseyhightower/app
- jeffnappi 10y ago
  Same! Actually I asked the question that led him to point me to this repo :) We had a conversation later in the hotel lobby where he made a great analogy: running this stuff yourself is going to be like running your own mail server. Its really nice to know how to do it, but at the end of the day unless you are a very large organization, you're most likely going to use a hosted service. Personally, I'm going to go through with setting up a test kubernetes cluster just so I know what it's made of. Then if I think it's great, maybe .. just maybe I'll give Google's hosted solution a try with a small project to start.
  - mattikus 10y ago
    I feel similarly to Kelsey. I also plan on setting up a test cluster to learn the ins and outs and seeing if it's something that might fit in at work for our needs. For personal stuff GKE looks really nice.
anemic 10y ago
Lots of good information here but it is still not enough for a production setup IMHO. There is a great need of good source of production setups. In open source software this seems to be the secret that no one is willing to reveal. I tried to setup a kubernetes cluster from scratch a while ago and soon I was browsing the source code for answers. Openstack is the same, you need to understand a lot about the inner workings before you can even attempt to setup something for production. There is always a simple "this shell script starts your own <name your tech here> cluster in vagrant" but it is still not a production setup. And even if this article is the "hard way" it describes: > This is being done for ease of use. In production you should strongly consider generating individual TLS certificates for each component." And it does not mention that the crucial part is the common name field in the certificate maps to the user name that is the magic information that I once needed. I sincerely appreciate this article but production setup is still a long learning experience away.
- olalonde 10y ago
  I just used `./cluster/kube-up.sh` to setup my cluster on AWS. I am now wondering what's missing for a production setup. It seems to be working OK so far (though I just have 3 minions and a few pods). One thing I wish I knew how to do is how to safely upgrade the cluster without re-creating it from scratch. Care to elaborate a bit?
  - lobster_johnson 10y ago
    The problem is that it's not declarative. You can't tweak the config and run it again to converge. Kubeup is designed to run once, unlike systems such as Puppet and Terraform that declaratively set up the world to fit your specification. Kubeup also does a lot of mysterious stuff. By using it, you don't have a clear idea of which pieces have been set up and how they slot into each other. It is, in short, opaque and magical. For comparison, I set up Kubernetes with Salt on AWS. It was, by all means, "the hard way", and took me a few days to get running and a couple of weeks to run completely correctly (a lot of stuff, like kubeconfig and TLS behaviour, is still undocumented), but as a byproduct I now have the entire setup in a reproducible, self-documenting, version-controlled config.
bdcravens 10y ago
What's nice about this is it seems to teach you without taking a GKE-first approach, like some of the other tutorials out there.
- SEJeff 10y ago
  Which is doubly awesome coming from a google employee. It shows Kelsey really wants to teach k8s to everyone.
readams 10y ago
It would be nice if all the Kubernetes tutorials didn't assume you're using GCE.
- ibizaman 10y ago
  Exactly. I expected "hard way" to be using your local box without any external cloud provided service.
jondubois 10y ago
The article mentioned Google Container Engine as one of the 'easy ways' but it didn't mention Rancher http://rancher.com/ http://rancher.com/ - This is not quite as easy as GCE but I found it pretty easy.
- nhumrich 10y ago
  I use rancher in prod. I absolutely love it. I don't use kubernetes in prod though. But if you want to go kubernetes through rancher, it is pretty easy.
marcoceppi 10y ago
This is a great starting point. We're been running Kubernetes in production alongside an OpenStack for a while and charm'd up the deployment: https://jujucharms.com/kubernetes-core https://jujucharms.com/kubernetes-core. The majority of the information here (and more) seems to already be encapsulated: `juju deploy kubernetes-core`. Since we need things like logging and monitoring, we bolted the elastic stack on the side and called it observable kubernetes: `juju deploy observable-kubernetes`. While one-liners are typically pretty limited, the charms come with quite a few knobs to help tweak for deployments. http://www.jorgecastro.org/2016/07/29/ubuntu-kubernetes-v1-dot-3-3-ready-for-testing/ http://www.jorgecastro.org/2016/07/29/ubuntu-kubernetes-v1-d... There's still room to improve, but we've been happy with the cluster so far. Considering Juju and charms are open source. Eitherway, great guide for those getting started.
philips 10y ago
For Kubernetes the easy way checkout this blog post on the work going on upstream for "self-hosted" Kubernetes: https://coreos.com/blog/self-hosted-kubernetes.html https://coreos.com/blog/self-hosted-kubernetes.html
mmagin 10y ago
So, I see something like this, assume they mean "from scratch", read a little way down the README and it says "The following labs assume you have a working Google Cloud Platform account and a recent version of the Google Cloud SDK (116.0.0+) installed." The irony.
ldehaan 10y ago
this seems interesting. I have to say I have heard a lot of people talking about kubernetes but few actually using it in production. for your CI woes there is a system that isn't hard to setup and is actually really easy to use: Mesos and Marathon with weave (without the plugin, for now) and docker. Your biggest challenge is learning zookeeper, but really if you're dealing with large scale deployments, you're probably already using it or something like it. there are puppet/chef/ansible modules for installing and configuring mesos and zookeeper. toss in gluster as a storage driver in docker and you're pretty much ready to go for most types of application deployments using docker. heck, there's even kubernetes integration if you're really hung up on it ;)
felixgallo 10y ago
Kelsey is a national treasure. Kubernetes is getting pretty close to being ready, if it can avoid the same fate as OpenStack and the like. Interesting times.
nathan_f77 10y ago
Here is Kubernetes the easy, production-ready way: https://github.com/kz8s/tack https://github.com/kz8s/tack I've spun up a few clusters using this, and I absolutely prefer learning this way. I love to get something running before I dive in and look at all the pieces and individual options. I also really love "convention over configuration". I want to study a production-ready reference implementation that simply works, with sane defaults. I've spent a lot of time banging my head against the wall while I try to follow some complicated tutorial that doesn't work with the latest versions. Maybe this approach works for others, but it's not for me. I like to stand up a cluster, deploy a database and a real application, try to scale it, set up some test DNS records, do some load testing. Figure out the pain points and learn as I go. If something just works perfectly fine behind the scenes, then I probably don't need to learn that much about it (or I have enough experience that I already know what it does and how it works.) This might not be a suitable learning style for a beginner, but I think I have enough experience working and experimenting with AWS, Saltstack, Chef, Puppet, Ansible, OpenStack, Deis, Flynn, Terraform, Docker, CoreOS, etc. etc. So at this point, I just prefer to evaluate new technologies by spinning them up and diving straight in.
user5994461 10y ago
Looked at the documentation and quickly gave up on kubernetes. It's nice and it solves real problem but the barrier to entry is INSANE. And it's lacking wayyy to much documentation deploy in production on own cluster. It's probably gonna take years to improve. I'm wondering. Did anyone tried the kubernetes on GCE? If Google can handle all the annoying setup and makes a mostly managed service, that would be extremely attractive. Actually, that's probably the only way k8s would be achievable in production, i.e. have someone else do it.
- felixgallo 10y ago
  Google did that already: https://cloud.google.com/container-engine/ https://cloud.google.com/container-engine/

240 points by tantalor 10y ago

Just saw Kelsey give a talk at Abstractions about more advanced patterns in Kubernetes and he mentioned this repo. Looks like a fantastic tutorial and his talk was very informative. Highly recommend watching the video when it's released if you weren't able to attend.

imglorp 10y ago
We were in the same room. Yes, good talk.
mVChr 10y ago
He gave a great live demo at CodeConf about using Kubernetes for 12-factor apps[1] that I highly recommend as well. The video for the talk isn't up yet, but the code he used is.[2] [1] https://12factor.net/ https://12factor.net/ [2] https://github.com/kelseyhightower/app https://github.com/kelseyhightower/app
jeffnappi 10y ago
Same! Actually I asked the question that led him to point me to this repo :) We had a conversation later in the hotel lobby where he made a great analogy: running this stuff yourself is going to be like running your own mail server. Its really nice to know how to do it, but at the end of the day unless you are a very large organization, you're most likely going to use a hosted service. Personally, I'm going to go through with setting up a test kubernetes cluster just so I know what it's made of. Then if I think it's great, maybe .. just maybe I'll give Google's hosted solution a try with a small project to start.
- mattikus 10y ago
  I feel similarly to Kelsey. I also plan on setting up a test cluster to learn the ins and outs and seeing if it's something that might fit in at work for our needs. For personal stuff GKE looks really nice.

anemic 10y ago

Lots of good information here but it is still not enough for a production setup IMHO. There is a great need of good source of production setups. In open source software this seems to be the secret that no one is willing to reveal. I tried to setup a kubernetes cluster from scratch a while ago and soon I was browsing the source code for answers. Openstack is the same, you need to understand a lot about the inner workings before you can even attempt to setup something for production. There is always a simple "this shell script starts your own <name your tech here> cluster in vagrant" but it is still not a production setup. And even if this article is the "hard way" it describes: > This is being done for ease of use. In production you should strongly consider generating individual TLS certificates for each component." And it does not mention that the crucial part is the common name field in the certificate maps to the user name that is the magic information that I once needed. I sincerely appreciate this article but production setup is still a long learning experience away.

olalonde 10y ago
I just used `./cluster/kube-up.sh` to setup my cluster on AWS. I am now wondering what's missing for a production setup. It seems to be working OK so far (though I just have 3 minions and a few pods). One thing I wish I knew how to do is how to safely upgrade the cluster without re-creating it from scratch. Care to elaborate a bit?
- lobster_johnson 10y ago
  The problem is that it's not declarative. You can't tweak the config and run it again to converge. Kubeup is designed to run once, unlike systems such as Puppet and Terraform that declaratively set up the world to fit your specification. Kubeup also does a lot of mysterious stuff. By using it, you don't have a clear idea of which pieces have been set up and how they slot into each other. It is, in short, opaque and magical. For comparison, I set up Kubernetes with Salt on AWS. It was, by all means, "the hard way", and took me a few days to get running and a couple of weeks to run completely correctly (a lot of stuff, like kubeconfig and TLS behaviour, is still undocumented), but as a byproduct I now have the entire setup in a reproducible, self-documenting, version-controlled config.

bdcravens 10y ago

What's nice about this is it seems to teach you without taking a GKE-first approach, like some of the other tutorials out there.

SEJeff 10y ago
Which is doubly awesome coming from a google employee. It shows Kelsey really wants to teach k8s to everyone.

readams 10y ago

It would be nice if all the Kubernetes tutorials didn't assume you're using GCE.

ibizaman 10y ago
Exactly. I expected "hard way" to be using your local box without any external cloud provided service.

jondubois 10y ago

The article mentioned Google Container Engine as one of the 'easy ways' but it didn't mention Rancher http://rancher.com/ http://rancher.com/ - This is not quite as easy as GCE but I found it pretty easy.

nhumrich 10y ago
I use rancher in prod. I absolutely love it. I don't use kubernetes in prod though. But if you want to go kubernetes through rancher, it is pretty easy.

marcoceppi 10y ago

This is a great starting point. We're been running Kubernetes in production alongside an OpenStack for a while and charm'd up the deployment: https://jujucharms.com/kubernetes-core https://jujucharms.com/kubernetes-core. The majority of the information here (and more) seems to already be encapsulated: `juju deploy kubernetes-core`. Since we need things like logging and monitoring, we bolted the elastic stack on the side and called it observable kubernetes: `juju deploy observable-kubernetes`. While one-liners are typically pretty limited, the charms come with quite a few knobs to help tweak for deployments. http://www.jorgecastro.org/2016/07/29/ubuntu-kubernetes-v1-dot-3-3-ready-for-testing/ http://www.jorgecastro.org/2016/07/29/ubuntu-kubernetes-v1-d... There's still room to improve, but we've been happy with the cluster so far. Considering Juju and charms are open source. Eitherway, great guide for those getting started.

philips 10y ago

For Kubernetes the easy way checkout this blog post on the work going on upstream for "self-hosted" Kubernetes: https://coreos.com/blog/self-hosted-kubernetes.html https://coreos.com/blog/self-hosted-kubernetes.html

mmagin 10y ago

So, I see something like this, assume they mean "from scratch", read a little way down the README and it says "The following labs assume you have a working Google Cloud Platform account and a recent version of the Google Cloud SDK (116.0.0+) installed." The irony.

ldehaan 10y ago

this seems interesting. I have to say I have heard a lot of people talking about kubernetes but few actually using it in production. for your CI woes there is a system that isn't hard to setup and is actually really easy to use: Mesos and Marathon with weave (without the plugin, for now) and docker. Your biggest challenge is learning zookeeper, but really if you're dealing with large scale deployments, you're probably already using it or something like it. there are puppet/chef/ansible modules for installing and configuring mesos and zookeeper. toss in gluster as a storage driver in docker and you're pretty much ready to go for most types of application deployments using docker. heck, there's even kubernetes integration if you're really hung up on it ;)

felixgallo 10y ago

Kelsey is a national treasure. Kubernetes is getting pretty close to being ready, if it can avoid the same fate as OpenStack and the like. Interesting times.

nathan_f77 10y ago

Here is Kubernetes the easy, production-ready way: https://github.com/kz8s/tack https://github.com/kz8s/tack I've spun up a few clusters using this, and I absolutely prefer learning this way. I love to get something running before I dive in and look at all the pieces and individual options. I also really love "convention over configuration". I want to study a production-ready reference implementation that simply works, with sane defaults. I've spent a lot of time banging my head against the wall while I try to follow some complicated tutorial that doesn't work with the latest versions. Maybe this approach works for others, but it's not for me. I like to stand up a cluster, deploy a database and a real application, try to scale it, set up some test DNS records, do some load testing. Figure out the pain points and learn as I go. If something just works perfectly fine behind the scenes, then I probably don't need to learn that much about it (or I have enough experience that I already know what it does and how it works.) This might not be a suitable learning style for a beginner, but I think I have enough experience working and experimenting with AWS, Saltstack, Chef, Puppet, Ansible, OpenStack, Deis, Flynn, Terraform, Docker, CoreOS, etc. etc. So at this point, I just prefer to evaluate new technologies by spinning them up and diving straight in.

user5994461 10y ago

Looked at the documentation and quickly gave up on kubernetes. It's nice and it solves real problem but the barrier to entry is INSANE. And it's lacking wayyy to much documentation deploy in production on own cluster. It's probably gonna take years to improve. I'm wondering. Did anyone tried the kubernetes on GCE? If Google can handle all the annoying setup and makes a mostly managed service, that would be extremely attractive. Actually, that's probably the only way k8s would be achievable in production, i.e. have someone else do it.

felixgallo 10y ago
Google did that already: https://cloud.google.com/container-engine/ https://cloud.google.com/container-engine/