4 ms·

I really wanted to like Kubernetes, but I couldn't figure it out. I'm a pretty experienced Linux admin, and despite days of reading all the sources of document

I really wanted to like Kubernetes, but I couldn't figure it out. I'm a pretty experienced Linux admin, and despite days of reading all the sources of documentation I could find I couldn't get the networking setup to where I thought it should be. I don't know if that was a problem of my mental model being wrong and the documentation not educating me on the correct model, or the documentation of the CNI, Flannel/whatever just being too sparse, or what. I started with the Google Ubuntu packages, and was able to get containers up fairly quickly, but being able to access them from anything other than the host machine I just couldn't figure out. Asking for help on IRC wasn't useful (aside: what's the deal with IRC channels with tons of users but no messages for days?) I've since wiped that cluster and tried the Ubuntu Kubernetes installation using conjure-up, it seems like it did the right thing but I'm not sure yet how to get to a cluster from a single machine. Trying to decide if I pursue Kubernetes more, try Dokku/Flynn/Deis (which I just saw referenced in a previous HN discussion) because they look really promising... I had hoped it would be the ganeti of containerization, but I had far fewer problems with ganeti.

user5994461 9y ago
No worries, we understand the pain, we've all been there. The abstracted network is a constant source of headache when dealing with kubernetes and containers.
slap_shot 9y ago
FWIW, if you create a GKE cluster and create an nginx deployment: kubectl apply -f https://k8s.io/docs/tasks/run-application/deployment.yaml https://k8s.io/docs/tasks/run-application/deployment.yaml And simply create a Load Balancer: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/ https://kubernetes.io/docs/tasks/access-application-cluster/... You have nginx exposed via an external IP address in about two minutes. Once you're comfortable with that, you can start working in Ingress and such to get more sophisticated. I agree the IRC/Slack channel communication isn't the best. I prefer the Kubernetes User Group. You can usually get a great response in less than a day. I've used it a lot.
- linkmotif 9y ago
  I was going to suggest this. Create a GKE cluster and play with it for a while to see what you’re supposed to have. Then you can create one yourself from scratch once you understand the target.
  - merb 9y ago
    well that's what I tried. and basically I get: > gcloud container clusters get-credentials cluster-1 ...trimmed > Fetching cluster endpoint and auth data. > ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=404, message=The resource "projects/infinite-alcove-164114/zones/europe-west3-c/clusters/cluster-1" was not found. > Could not find [cluster-1] in [europe-west3-c]. > Did you mean [cluster-1] in [europe-west3]? note that neither gcloud alpha, nor glcoud beta nor europe-west3-(a|b|c) worked. prolly a bug when using `Regional (Beta)`.
    - thesandlord 9y ago
      Try: gcloud beta clusters get-credentials cluster-1 --region europe-west3 You need the beta command, and you need to set the region flag instead of the zone flag. Looks like there are some documentation and UX gaps, I'm opening tickets to get them fixed (I work for Google).
      - merb 9y ago
        hm... thanks. I also needed to set `gcloud beta config set container/use_v1_api_client false`. well I also was to lazy to just copy & past from the connect window.
    - swozey 9y ago
      I was in the Regional Alpha so I'm familiar with this but it's been awhile since I had the issue. The fix I believe was;; export CLOUDSDK_CONTAINER_USE_V1_API_CLIENT=false Note; It's in beta now and I believe --region was added to the recent gcloud command but I could be incorrect. If you have problems msg me on k8s slack, mikej, I can run through it with you. I don't believe you need this at all now that it's beta, but I set up my beta regional cluster a few weeks ago. Also, re: IRC, there are 27k people on the k8s slack. I've used k8s for years and never gone into irc. I just don't use irc anymore.
sercand 9y ago
I am not a Linux expert, but I deployed clusters to 5 Raspberry Pi and Azure with custom terraform scripts. With the kubeadm things are much more straightforward now. My suggestion is to start with minikube with kubeadm bootstrapper than use more complicated solutions. https://kubernetes.io/docs/setup/pick-right-solution https://kubernetes.io/docs/setup/pick-right-solution
nickstinemates 9y ago
You should take a look at Rancher[1]. I think it has the right mix of raw access to Kubernetes and associated tools, but also addresses some of the requirements you have. 1: https://rancher.com https://rancher.com Disclaimer: I've been working in the container ecosystem for a number of years, currently at Rancher.
lobster_johnson 9y ago
I don't know what IRC channels you're referring to, but the Kubernetes Slack is very active. I've gotten a lot of help there. It also has channels where the devs are active.
gnufied 9y ago
Most of kube-users hang out on slack. I am not saying you will get definite answers but as a k8s core contributor - I can definitely say that most developers are on slack rather than IRC.
tetraodonpuffer 9y ago
not sure if it helps, but I recently did a series of posts on my blog on installing Kubernetes from scratch on a set of CoreOS guests on Xen, with flannel, RBAC, TLS and so on and it does give you a cluster that you should be able to access from the outside no problem. Link in my profile, I tried to make as low in magic as possible, I just ended up writing a couple of bash functions and a python script to make it easier to assemble the Ignition files for the deployment, but besides that it should all be step by step.
org3432 9y ago
Watch tutorials by Kelsey Hightower, he's been helping folks understand the differences and the models. e.g. https://www.youtube.com/watch?v=HlAXp0-M6SY https://www.youtube.com/watch?v=HlAXp0-M6SY
btmiller 9y ago
Your experience is a real shame, truly. If Kubernetes could fix their horrendous documentation and onboarding story, I think you'd have a much better experience. After bruising past that battle, I found Kubernetes to be the perfect mix of sanity and opinionated architecture. I think part of it is also due to how rapidly it's evolved over the past 2 years, so hopefully you get a chance to revisit it once some of those rough edges are smoothed over!
linsomniac 9y ago
I've found one way past it: kubespray I spent the morning listening to presentations from Kubecon last week, and a guy mentioned it in passing in one of the presentations. Went and looked at it and it's an Ansible playbook for deploying Kubernetes clusters. Respun my test nodes with a bare install and ran kubespray and I was up and running with Calico networking all set up, I'm able to get to the test pod I set up from anywhere. So pro tip: Kubespray, especially if you are familiar with ansible!
ksajadi 9y ago
I feel what you are going through. I wrote this last week trying to help: http://blog.cloud66.com/8-components-you-need-to-run-containers-in-production/ http://blog.cloud66.com/8-components-you-need-to-run-contain...

user5994461 9y ago

No worries, we understand the pain, we've all been there. The abstracted network is a constant source of headache when dealing with kubernetes and containers.

slap_shot 9y ago

FWIW, if you create a GKE cluster and create an nginx deployment: kubectl apply -f https://k8s.io/docs/tasks/run-application/deployment.yaml https://k8s.io/docs/tasks/run-application/deployment.yaml And simply create a Load Balancer: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/ https://kubernetes.io/docs/tasks/access-application-cluster/... You have nginx exposed via an external IP address in about two minutes. Once you're comfortable with that, you can start working in Ingress and such to get more sophisticated. I agree the IRC/Slack channel communication isn't the best. I prefer the Kubernetes User Group. You can usually get a great response in less than a day. I've used it a lot.

linkmotif 9y ago
I was going to suggest this. Create a GKE cluster and play with it for a while to see what you’re supposed to have. Then you can create one yourself from scratch once you understand the target.
- merb 9y ago
  well that's what I tried. and basically I get: > gcloud container clusters get-credentials cluster-1 ...trimmed > Fetching cluster endpoint and auth data. > ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=404, message=The resource "projects/infinite-alcove-164114/zones/europe-west3-c/clusters/cluster-1" was not found. > Could not find [cluster-1] in [europe-west3-c]. > Did you mean [cluster-1] in [europe-west3]? note that neither gcloud alpha, nor glcoud beta nor europe-west3-(a|b|c) worked. prolly a bug when using `Regional (Beta)`.
  - thesandlord 9y ago
    Try: gcloud beta clusters get-credentials cluster-1 --region europe-west3 You need the beta command, and you need to set the region flag instead of the zone flag. Looks like there are some documentation and UX gaps, I'm opening tickets to get them fixed (I work for Google).
    - merb 9y ago
      hm... thanks. I also needed to set `gcloud beta config set container/use_v1_api_client false`. well I also was to lazy to just copy & past from the connect window.
  - swozey 9y ago
    I was in the Regional Alpha so I'm familiar with this but it's been awhile since I had the issue. The fix I believe was;; export CLOUDSDK_CONTAINER_USE_V1_API_CLIENT=false Note; It's in beta now and I believe --region was added to the recent gcloud command but I could be incorrect. If you have problems msg me on k8s slack, mikej, I can run through it with you. I don't believe you need this at all now that it's beta, but I set up my beta regional cluster a few weeks ago. Also, re: IRC, there are 27k people on the k8s slack. I've used k8s for years and never gone into irc. I just don't use irc anymore.

sercand 9y ago

I am not a Linux expert, but I deployed clusters to 5 Raspberry Pi and Azure with custom terraform scripts. With the kubeadm things are much more straightforward now. My suggestion is to start with minikube with kubeadm bootstrapper than use more complicated solutions. https://kubernetes.io/docs/setup/pick-right-solution https://kubernetes.io/docs/setup/pick-right-solution

nickstinemates 9y ago

You should take a look at Rancher[1]. I think it has the right mix of raw access to Kubernetes and associated tools, but also addresses some of the requirements you have. 1: https://rancher.com https://rancher.com Disclaimer: I've been working in the container ecosystem for a number of years, currently at Rancher.

lobster_johnson 9y ago

I don't know what IRC channels you're referring to, but the Kubernetes Slack is very active. I've gotten a lot of help there. It also has channels where the devs are active.

gnufied 9y ago

Most of kube-users hang out on slack. I am not saying you will get definite answers but as a k8s core contributor - I can definitely say that most developers are on slack rather than IRC.

tetraodonpuffer 9y ago

not sure if it helps, but I recently did a series of posts on my blog on installing Kubernetes from scratch on a set of CoreOS guests on Xen, with flannel, RBAC, TLS and so on and it does give you a cluster that you should be able to access from the outside no problem. Link in my profile, I tried to make as low in magic as possible, I just ended up writing a couple of bash functions and a python script to make it easier to assemble the Ignition files for the deployment, but besides that it should all be step by step.

org3432 9y ago

Watch tutorials by Kelsey Hightower, he's been helping folks understand the differences and the models. e.g. https://www.youtube.com/watch?v=HlAXp0-M6SY https://www.youtube.com/watch?v=HlAXp0-M6SY

btmiller 9y ago

Your experience is a real shame, truly. If Kubernetes could fix their horrendous documentation and onboarding story, I think you'd have a much better experience. After bruising past that battle, I found Kubernetes to be the perfect mix of sanity and opinionated architecture. I think part of it is also due to how rapidly it's evolved over the past 2 years, so hopefully you get a chance to revisit it once some of those rough edges are smoothed over!

linsomniac 9y ago

I've found one way past it: kubespray I spent the morning listening to presentations from Kubecon last week, and a guy mentioned it in passing in one of the presentations. Went and looked at it and it's an Ansible playbook for deploying Kubernetes clusters. Respun my test nodes with a bare install and ran kubespray and I was up and running with Calico networking all set up, I'm able to get to the test pod I set up from anywhere. So pro tip: Kubespray, especially if you are familiar with ansible!

ksajadi 9y ago

I feel what you are going through. I wrote this last week trying to help: http://blog.cloud66.com/8-components-you-need-to-run-containers-in-production/ http://blog.cloud66.com/8-components-you-need-to-run-contain...