5 ms·

I'm really liking Kubernetes — we're in the process of migrating to it. If there's on area that is in dire need of improvement, though, it's the documentation.

I'm really liking Kubernetes — we're in the process of migrating to it. If there's on area that is in dire need of improvement, though, it's the documentation. If you look around, there is essentially no documentation that starts from first principles, going through the different components (and their lifecycle, dependencies, requirements and so on) one by one, irrrespectively of the cloud environment. There is a "Kubernetes from scratch" [1] document, but it's just a bunch of loose fragments that lacks almost all the necessary detail, and has too many dependencies. (Tip: ask the user to install from source, and leave out how to use images, cloud providers and other things that obscure the workings of everything.) Almost all of the documentation assumes you're running kube-up or some other automated setup, which is of course convention, but hides a huge amount of magic in a bunch of shell scripts, Salt config and so on that prevents true understanding. If you run it for, say, AWS, then you'll end up with a configuration that you don't understand. It doesn't help that much of the official documentation is heavily skewed towards GCE/GKE, where certain things have a level of automatic magic that you won't benefit from when you run on bare metal, for example. kube-up will help someone get it up and running fast, but does not help someone who needs to maintain it in a careful, controller manner. Right now, I have a working cluster, but getting there involved a bunch of trial and error, a lot of open browser tabs, source code reading, and so on. (Quick, what version of Docker does Kubernetes want? Kubernetes doesn't seem to tell us, and it doesn't even verify it on startup. One of the reefs that I ran aground on was when 1.11 didn't work, and had to revert to 1.9, based on a random Github issue I found.) [1] http://kubernetes.io/docs/getting-started-guides/scratch/ http://kubernetes.io/docs/getting-started-guides/scratch/

Rapzid 10y ago
I can't agree with this enough. We are all on AWS and the level of effort it would take to migrate to Kubernetes while maintaining our ability to spin up complete ad-hoc environments on the fly(which also serves as continual DR testing) seems too much to justify at this point. Also, I can't come out the other side with just one or two people understanding, or having any hope of understanding, how everything works :| Likely if I had to choose today or this quarter we would go the empire route and build on top of ECS. Though, our model and requirements are a bit different so we'd have to heavily modify or roll our own.
- lobster_johnson 10y ago
  One thing I would say is that — because of the aforementioned documentation mess — it seems more daunting than it actually is. And the documentation does make it seem like a lot of work. All you need to do, in broad strokes, is: * Set up a VPC. Defaults work. * Create an AWS instance. Make sure it has a dedicated IAM role that has a policy like this [1], so that it can do things like create ELBs. * Install Kubernetes from binary packages. I've been using Kismatic's Debian/Ubuntu packages [2], which are nice. * Install Docker >= 1.9 < 1.10 (apparently). * Install etcd. * Make sure your AWS instance has a sane MTU ("sudo ifconfig eth0 mtu 1500"). AWS uses jumbo frames by default [3], which I found does not work with Docker Hub (even though it's also on AWS). * Edit /etc/default/docker to disable its iptables magic and use the Kubernetes bridge, which Kubelet will eventually create for you on startup: DOCKER_OPTS="--iptables=false --ip-masq=false --bridge=cbr0" * Decide which CIDR ranges to use for pods and services. You can carve a /24 from your VPC subnet for each. They have to be non-overlapping ranges. * Edit the /etc/default/kube* configs to set DAEMON_ARGS in each. Read the help page for each daemon to see what flags they take. Most have sane defaults or are ignorable, but you'll need some specific ones [4]. * Start etcd, Docker and all the Kubernetes daemons. * Verify it's working with something like: kubectl run test --image=dockercloud/hello-world Unless I'm forgetting something, that's basically it for one master node. For multiple nodes, you'll have to run Kubelet on each. You can run as many masters (kube-apiserver) as you want, and they'll use etcd leases to ensure that only one is active. [1] https://gist.github.com/atombender/3f9ba857590ea98d18163e9832965166 https://gist.github.com/atombender/3f9ba857590ea98d18163e983... [2] http://repos.kismatic.com/debian/ http://repos.kismatic.com/debian/ [3] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_m... [4] https://gist.github.com/atombender/e72c2acc2d30b0965543273a22e4a7d0 https://gist.github.com/atombender/e72c2acc2d30b0965543273a2...
  - TheIronYuppie 10y ago
    Quick question - if you're using AWS (or GCP or Azure), was there a reason that: ./kube-up.sh Didn't work for you? Disclosure: I work at Google on Kubernetes

Rapzid 10y ago

I can't agree with this enough. We are all on AWS and the level of effort it would take to migrate to Kubernetes while maintaining our ability to spin up complete ad-hoc environments on the fly(which also serves as continual DR testing) seems too much to justify at this point. Also, I can't come out the other side with just one or two people understanding, or having any hope of understanding, how everything works :| Likely if I had to choose today or this quarter we would go the empire route and build on top of ECS. Though, our model and requirements are a bit different so we'd have to heavily modify or roll our own.

lobster_johnson 10y ago
One thing I would say is that — because of the aforementioned documentation mess — it seems more daunting than it actually is. And the documentation does make it seem like a lot of work. All you need to do, in broad strokes, is: * Set up a VPC. Defaults work. * Create an AWS instance. Make sure it has a dedicated IAM role that has a policy like this [1], so that it can do things like create ELBs. * Install Kubernetes from binary packages. I've been using Kismatic's Debian/Ubuntu packages [2], which are nice. * Install Docker >= 1.9 < 1.10 (apparently). * Install etcd. * Make sure your AWS instance has a sane MTU ("sudo ifconfig eth0 mtu 1500"). AWS uses jumbo frames by default [3], which I found does not work with Docker Hub (even though it's also on AWS). * Edit /etc/default/docker to disable its iptables magic and use the Kubernetes bridge, which Kubelet will eventually create for you on startup: DOCKER_OPTS="--iptables=false --ip-masq=false --bridge=cbr0" * Decide which CIDR ranges to use for pods and services. You can carve a /24 from your VPC subnet for each. They have to be non-overlapping ranges. * Edit the /etc/default/kube* configs to set DAEMON_ARGS in each. Read the help page for each daemon to see what flags they take. Most have sane defaults or are ignorable, but you'll need some specific ones [4]. * Start etcd, Docker and all the Kubernetes daemons. * Verify it's working with something like: kubectl run test --image=dockercloud/hello-world Unless I'm forgetting something, that's basically it for one master node. For multiple nodes, you'll have to run Kubelet on each. You can run as many masters (kube-apiserver) as you want, and they'll use etcd leases to ensure that only one is active. [1] https://gist.github.com/atombender/3f9ba857590ea98d18163e9832965166 https://gist.github.com/atombender/3f9ba857590ea98d18163e983... [2] http://repos.kismatic.com/debian/ http://repos.kismatic.com/debian/ [3] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_m... [4] https://gist.github.com/atombender/e72c2acc2d30b0965543273a22e4a7d0 https://gist.github.com/atombender/e72c2acc2d30b0965543273a2...
- TheIronYuppie 10y ago
  Quick question - if you're using AWS (or GCP or Azure), was there a reason that: ./kube-up.sh Didn't work for you? Disclosure: I work at Google on Kubernetes