4 ms·

I can't agree with this enough. We are all on AWS and the level of effort it would take to migrate to Kubernetes while maintaining our ability to spin up comple

by Rapzid 10y ago

I can't agree with this enough. We are all on AWS and the level of effort it would take to migrate to Kubernetes while maintaining our ability to spin up complete ad-hoc environments on the fly(which also serves as continual DR testing) seems too much to justify at this point. Also, I can't come out the other side with just one or two people understanding, or having any hope of understanding, how everything works :| Likely if I had to choose today or this quarter we would go the empire route and build on top of ECS. Though, our model and requirements are a bit different so we'd have to heavily modify or roll our own.

lobster_johnson 10y ago
One thing I would say is that — because of the aforementioned documentation mess — it seems more daunting than it actually is. And the documentation does make it seem like a lot of work. All you need to do, in broad strokes, is: * Set up a VPC. Defaults work. * Create an AWS instance. Make sure it has a dedicated IAM role that has a policy like this [1], so that it can do things like create ELBs. * Install Kubernetes from binary packages. I've been using Kismatic's Debian/Ubuntu packages [2], which are nice. * Install Docker >= 1.9 < 1.10 (apparently). * Install etcd. * Make sure your AWS instance has a sane MTU ("sudo ifconfig eth0 mtu 1500"). AWS uses jumbo frames by default [3], which I found does not work with Docker Hub (even though it's also on AWS). * Edit /etc/default/docker to disable its iptables magic and use the Kubernetes bridge, which Kubelet will eventually create for you on startup: DOCKER_OPTS="--iptables=false --ip-masq=false --bridge=cbr0" * Decide which CIDR ranges to use for pods and services. You can carve a /24 from your VPC subnet for each. They have to be non-overlapping ranges. * Edit the /etc/default/kube* configs to set DAEMON_ARGS in each. Read the help page for each daemon to see what flags they take. Most have sane defaults or are ignorable, but you'll need some specific ones [4]. * Start etcd, Docker and all the Kubernetes daemons. * Verify it's working with something like: kubectl run test --image=dockercloud/hello-world Unless I'm forgetting something, that's basically it for one master node. For multiple nodes, you'll have to run Kubelet on each. You can run as many masters (kube-apiserver) as you want, and they'll use etcd leases to ensure that only one is active. [1] https://gist.github.com/atombender/3f9ba857590ea98d18163e9832965166 https://gist.github.com/atombender/3f9ba857590ea98d18163e983... [2] http://repos.kismatic.com/debian/ http://repos.kismatic.com/debian/ [3] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_m... [4] https://gist.github.com/atombender/e72c2acc2d30b0965543273a22e4a7d0 https://gist.github.com/atombender/e72c2acc2d30b0965543273a2...
- TheIronYuppie 10y ago
  Quick question - if you're using AWS (or GCP or Azure), was there a reason that: ./kube-up.sh Didn't work for you? Disclosure: I work at Google on Kubernetes
  - lobster_johnson 10y ago
    Did you read my earlier comment (https://news.ycombinator.com/item?id=12024148 https://news.ycombinator.com/item?id=12024148)? In short, I want and need to understand how it's put together so that I can use it. There was someone on the #kubernetes-novices slack today [1] who rightly pointed out who described his approach as: Run kube-up, then try to deconstruct everything that kube-up did into a repeatable recipe. I went the other route, by trying to understand what kube-up did and replicating it. I'm still working through things I missed or did wrong. To be honest, I think Google's approach here is wrong. Kubernetes is being developed at a frenetic pace, but documentation is not being maintained (it's pretty lacking even if you're on GCP!), and users are understandably frustrated with the obscurity of the whole thing. It works, but it takes weeks to gather enough of an understanding of the system, and that's entirely due to lack of documentation. The documentaton is lacking both a high and low level. At no point does the documentation offer a big-picture view of how everything works together, nor does it offer low-level descriptions of the stack. I also think the strong focus on kube-up is a mistake, given the lack of docs. I'm sure it works great, but it's not an option for production use, in my opinion. Terraform would have been better here. You're also using Salt — honestly, it would have been so much cooler if kube-up could just take a few inputs ("what cloud?", "what are your credentials?" etc.) and generate a finished Salt config for you, with a separate salt-cloud orchestration config for the provisioning. The current Salt config is a bit of a mess, and not really something you can build on. Feel free to reach out to me (@atombender) on the Kubernetes Slack if you want to chat. [1] https://kubernetes.slack.com/archives/kubernetes-novice/p1467793569000911 https://kubernetes.slack.com/archives/kubernetes-novice/p146...
    - TheIronYuppie 10y ago
      Great feedback! Both Kubernetes Anywhere (https://github.com/kubernetes/kubernetes-anywhere https://github.com/kubernetes/kubernetes-anywhere) and our documentation efforts (https://github.com/kubernetes/kubernetes.github.io https://github.com/kubernetes/kubernetes.github.io) are very much in flight - and they're both coming by 1.4 (~90 days).

by Rapzid 10y ago

lobster_johnson 10y ago

One thing I would say is that — because of the aforementioned documentation mess — it seems more daunting than it actually is. And the documentation does make it seem like a lot of work. All you need to do, in broad strokes, is: * Set up a VPC. Defaults work. * Create an AWS instance. Make sure it has a dedicated IAM role that has a policy like this [1], so that it can do things like create ELBs. * Install Kubernetes from binary packages. I've been using Kismatic's Debian/Ubuntu packages [2], which are nice. * Install Docker >= 1.9 < 1.10 (apparently). * Install etcd. * Make sure your AWS instance has a sane MTU ("sudo ifconfig eth0 mtu 1500"). AWS uses jumbo frames by default [3], which I found does not work with Docker Hub (even though it's also on AWS). * Edit /etc/default/docker to disable its iptables magic and use the Kubernetes bridge, which Kubelet will eventually create for you on startup: DOCKER_OPTS="--iptables=false --ip-masq=false --bridge=cbr0" * Decide which CIDR ranges to use for pods and services. You can carve a /24 from your VPC subnet for each. They have to be non-overlapping ranges. * Edit the /etc/default/kube* configs to set DAEMON_ARGS in each. Read the help page for each daemon to see what flags they take. Most have sane defaults or are ignorable, but you'll need some specific ones [4]. * Start etcd, Docker and all the Kubernetes daemons. * Verify it's working with something like: kubectl run test --image=dockercloud/hello-world Unless I'm forgetting something, that's basically it for one master node. For multiple nodes, you'll have to run Kubelet on each. You can run as many masters (kube-apiserver) as you want, and they'll use etcd leases to ensure that only one is active. [1] https://gist.github.com/atombender/3f9ba857590ea98d18163e9832965166 https://gist.github.com/atombender/3f9ba857590ea98d18163e983... [2] http://repos.kismatic.com/debian/ http://repos.kismatic.com/debian/ [3] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_mtu.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/network_m... [4] https://gist.github.com/atombender/e72c2acc2d30b0965543273a22e4a7d0 https://gist.github.com/atombender/e72c2acc2d30b0965543273a2...

TheIronYuppie 10y ago
Quick question - if you're using AWS (or GCP or Azure), was there a reason that: ./kube-up.sh Didn't work for you? Disclosure: I work at Google on Kubernetes
- lobster_johnson 10y ago
  Did you read my earlier comment (https://news.ycombinator.com/item?id=12024148 https://news.ycombinator.com/item?id=12024148)? In short, I want and need to understand how it's put together so that I can use it. There was someone on the #kubernetes-novices slack today [1] who rightly pointed out who described his approach as: Run kube-up, then try to deconstruct everything that kube-up did into a repeatable recipe. I went the other route, by trying to understand what kube-up did and replicating it. I'm still working through things I missed or did wrong. To be honest, I think Google's approach here is wrong. Kubernetes is being developed at a frenetic pace, but documentation is not being maintained (it's pretty lacking even if you're on GCP!), and users are understandably frustrated with the obscurity of the whole thing. It works, but it takes weeks to gather enough of an understanding of the system, and that's entirely due to lack of documentation. The documentaton is lacking both a high and low level. At no point does the documentation offer a big-picture view of how everything works together, nor does it offer low-level descriptions of the stack. I also think the strong focus on kube-up is a mistake, given the lack of docs. I'm sure it works great, but it's not an option for production use, in my opinion. Terraform would have been better here. You're also using Salt — honestly, it would have been so much cooler if kube-up could just take a few inputs ("what cloud?", "what are your credentials?" etc.) and generate a finished Salt config for you, with a separate salt-cloud orchestration config for the provisioning. The current Salt config is a bit of a mess, and not really something you can build on. Feel free to reach out to me (@atombender) on the Kubernetes Slack if you want to chat. [1] https://kubernetes.slack.com/archives/kubernetes-novice/p1467793569000911 https://kubernetes.slack.com/archives/kubernetes-novice/p146...
  - TheIronYuppie 10y ago
    Great feedback! Both Kubernetes Anywhere (https://github.com/kubernetes/kubernetes-anywhere https://github.com/kubernetes/kubernetes-anywhere) and our documentation efforts (https://github.com/kubernetes/kubernetes.github.io https://github.com/kubernetes/kubernetes.github.io) are very much in flight - and they're both coming by 1.4 (~90 days).