4 ms·

Those who have used kubernetes so far, what is your opinion on the kubernetes ecosystem? Is this the holy grail? Is there anything missing that you would like t

Those who have used kubernetes so far, what is your opinion on the kubernetes ecosystem? Is this the holy grail? Is there anything missing that you would like to have? I have found kubernetes hard to learn. There are a lot of concepts. That in itself isn't bad, but I have found that the quality of the documentation is variable. Once I get past the basics tutorial, I have the feeling that I have landed in a big information dump that require expert knowledge ala the git man pages.

newsat13 10y ago
Yeah, I feel you. I found kubernetes very hard to learn as well. I wanted was to find a good tool to run a SaaS (to handle 10k users). Only after reading up a lot did I even realize that this was not for me. I had fallen into the Hackernews hype trap and thought kubernetes is the right tool for me. I am back to my simple docker based deploys in 5 EC2 instances. It's very much by hand but works with some simple instructions to follow on a wiki page. I have also found kubernetes labor hard to find. The ones I found were way too expensive for me (upwards of 150k USD).
- rodgerd 10y ago
  > I found kubernetes very hard to learn as well. This is why we went down the OpenShift root, but there are not shortage of people wrapping Kube in other things, which seems sensible for a lot of use cases.
  - stephane-klein 10y ago
    I have deployed and I use OpenShift since more that one year but I think that Ansible Playbook is crappy. What is you feeling about that?
- Kubuxu 10y ago
  Check out Nomad, I have no personal experience with it yet but it seems the best candidate so far for what we are planning in terms of docker deployment and so on.
- FooBarWidget 10y ago
  Have you tried kontena? They aim to be a container orchestrator designed for developer happiness and ux. Certainly looks a lot simpler.
- kt9 10y ago
  Checkout Distelli (https://www.distelli.com https://www.distelli.com). UI and pipelines that make using Kubernetes easier. Tons of features coming soon. I can also help you with Docker / Kubernetes even if you decide not to buy anything from us - happy to help. Email is in my profile. Disclaimer: I'm the founder at Distelli.
  - stephane-klein 10y ago
    I think that Distelli is like Build in OpenShift (https://docs.openshift.org/latest/architecture/core_concepts/builds_and_image_streams.html#builds https://docs.openshift.org/latest/architecture/core_concepts...), isn't it ?
- jdc0589 10y ago
  there's definitely a lot to it...I'm still in the phase where I'm a bit overwhelmed with trying to learn everything. As far as setup/ops goes, definitely check out https://github.com/kubernetes/kops https://github.com/kubernetes/kops if you are on AWS; it's not perfect, but it made some stuff way easier in the early stages for me.
dluc 10y ago
Having worked mostly with mesos and k8s, I found k8s configuration superior, e.g. how one can import secrets, config files, and more importantly set up the network without address translation or port forwarding. Tooling seems OK with both, and I agree one needs to spend some time to get familiar with the CLI and nomenclature, IMHO because both are quite flexible and powerful.
Agrosis 10y ago
There are definitely a lot of moving parts in Kubernetes just to get it running and it was a big hurdle to get everything set up (I'm running a vanilla cluster in Vagrant). The documentation definitely covers a lot, but I found myself hunting issues down for trying to get something to work like getting k8s to schedule pods across a flannel overlay network and making sure traffic is being routed properly. Of course, you don't have to worry about any of this if you use GKE or another cloud provider. Once I got that set up, everything worked beautifully. I can just write a config and apply it to the cluster and services are deployed and scaled and whatnot. Treating your cluster as a pool of resources is pretty powerful. There are a lot of Kubernetes API objects (Deployments, Stateful Sets, Daemon Sets, ConfigMaps, Secrets), but everything has a purpose and Kubernetes makes everything come together nicely. The Kubernetes Slack channel was really helpful for me as well, there's a lot of nice and friendly people there willing to answer your questions and help you debug your setup.
roque 10y ago
Yes, it is the holy grail. Yes, the documentation is a bit spotty in some cases. We've been using kubernetes in GCE for about 1 year. We run both interactive services and batch jobs in 2 clusters: dev and production. The production cluster runs fully with declarative configs (we have a jenkins instance that performs the deployments of committed k8s configs). There are a couple of features that we are missing: (pod anti-affinity now in 1.6) and better support for storage. But in general kubernetes has the right concepts for running services, imho. We have 0 ops people. Our clusters run on large VMs with 10s of PODs in the same VM. We can get high cpu/mem utilization; resilience to failures; logging; monitoring (via prometheus - GCE /stackdriver is awful). I don't think there is anything out there that comes close in terms of supporting the right abstractions and being ready to use in production.
- jdc0589 10y ago
  we aren't anywhere close to using it in production yet, but my impression so far is to agree with you. Every single use case or weird requirement I've made up so far has had an answer, or had a solution in at least the proposal phase.
- thockingoog 10y ago
  Is your jenkins code public?
caio1982 10y ago
I don't know if it's the holy grail but it's surely one of the nicest and most impressive piece of tech I've seen since I tried Asterisk 10 years ago and had a long OMG moment myself. The ecosystem is growing by the day and Kubernetes itself changes a lot very quickly. The pace of its development is sometimes crazy. You might find yourself trying to stay on top of the latest release just to notice there's a new one out of the door and that you will surely want to move to. In this regard I would say to wait a bit, hold your horses and come back to Kubernetes once its release cycle is a bit saner for production environments (my personal estimation is by the end of this year, or earlier than that). What is missing in my opinion is a better container backend other than Docker (I just find it not enough for more complex scenarios, too bloated, there is the PID-1-hell, etc). With the new container runtime interface I am sure this won't be a problem in the long term. Also, storage. I mean, the whole user side of setting it up is a breeze. The new release makes it even more simple apparently. However I'm still not very sure how reliable things are here. Maybe it's just me not understanding how storage works in a Kubernetes world... whatever. I know for a fact I am not the only one with this concern though. What I can tell you is that after seeing a full deployment pipeline in Kubernetes actually working, and seeing all its load balancing and HA and rollout/rollback features, I am not going to stop using it. My recommendation for starters: the docs are good, read about every Object type and things will eventually "click" in your head. Make sure you understand every bit that is inside a spec file, it will make things easier to understand. Get how ingress/egress work, they can be a pain in Kubernetes sometimes. Try minikube, for Darwin's sake.
- caio1982 10y ago
  Just a clarification: I am not saying Kubernetes will only be good for production envs by the end of the year, what I tried to say is that this year is going to be THE year for Kubernetes and it will see lots of changes that makes it worth to wait a little longer and start with the right release that contains all the nice bits you would expect it to have, say, today.
- marccarre 10y ago
  > What is missing in my opinion is a better container backend other than Docker (I just find it not enough for more complex scenarios, too bloated, there is the PID-1-hell, etc). With the new container runtime interface I am sure this won't be a problem in the long term. I have only been experimenting a bit with rkt, and I am very new to it, but as far as I understand it should help to address some of the points regarding the container backend, since: 1) it is just a container runtime, 2) doesn't have the client/server architecture that Docker has, 3) has a lightweight systemd starting as PID1 and reaping/adopting children, and 4) works with Kubernetes. See also: - https://coreos.com/rkt/docs/latest/devel/architecture.html https://coreos.com/rkt/docs/latest/devel/architecture.html - https://kubernetes.io/docs/getting-started-guides/rkt/ https://kubernetes.io/docs/getting-started-guides/rkt/ - https://coreos.com/rkt/docs/latest/using-rkt-with-kubernetes.html https://coreos.com/rkt/docs/latest/using-rkt-with-kubernetes... Also note that `docker run` has an `--init` option based on `tini` which can help with the PID-1 hell. See: https://docs.docker.com/engine/reference/run/#specify-an-init-process https://docs.docker.com/engine/reference/run/#specify-an-ini...
jondubois 10y ago
I think it's the holy grail for stateless services. For stateful systems though, (that need to persist data) things can be tricky. Maybe the improved node affinity features in this release will improve things but I still feel limited because I can't seem to find a way to permanently bind a Pod to a specific host's file system. They keep pushing for NFS or other complex network file systems that add huge network I/O overheads instead of letting you deal with distributed storage across multiple hosts yourself (which would work better with most new database engines that have built-in cluster support). Last time I checked, you COULD bind a Pod to the underlying host's file system but the documentation says that this is only for debugging purposes and not suitable for production.
- vishh 10y ago
  FYI: Local Storage for stateful services will be tackled in the upcoming releases - https://github.com/kubernetes/community/pull/306 https://github.com/kubernetes/community/pull/306
thockingoog 10y ago
ACK. We recognize this aspect of the docs (engineers writing docs == encyclopedia articles). We have real, honest to goodness tech writers now, and they are working on it. One piece at a time.

newsat13 10y ago

Yeah, I feel you. I found kubernetes very hard to learn as well. I wanted was to find a good tool to run a SaaS (to handle 10k users). Only after reading up a lot did I even realize that this was not for me. I had fallen into the Hackernews hype trap and thought kubernetes is the right tool for me. I am back to my simple docker based deploys in 5 EC2 instances. It's very much by hand but works with some simple instructions to follow on a wiki page. I have also found kubernetes labor hard to find. The ones I found were way too expensive for me (upwards of 150k USD).

rodgerd 10y ago
> I found kubernetes very hard to learn as well. This is why we went down the OpenShift root, but there are not shortage of people wrapping Kube in other things, which seems sensible for a lot of use cases.
- stephane-klein 10y ago
  I have deployed and I use OpenShift since more that one year but I think that Ansible Playbook is crappy. What is you feeling about that?
Kubuxu 10y ago
Check out Nomad, I have no personal experience with it yet but it seems the best candidate so far for what we are planning in terms of docker deployment and so on.
FooBarWidget 10y ago
Have you tried kontena? They aim to be a container orchestrator designed for developer happiness and ux. Certainly looks a lot simpler.
kt9 10y ago
Checkout Distelli (https://www.distelli.com https://www.distelli.com). UI and pipelines that make using Kubernetes easier. Tons of features coming soon. I can also help you with Docker / Kubernetes even if you decide not to buy anything from us - happy to help. Email is in my profile. Disclaimer: I'm the founder at Distelli.
- stephane-klein 10y ago
  I think that Distelli is like Build in OpenShift (https://docs.openshift.org/latest/architecture/core_concepts/builds_and_image_streams.html#builds https://docs.openshift.org/latest/architecture/core_concepts...), isn't it ?
jdc0589 10y ago
there's definitely a lot to it...I'm still in the phase where I'm a bit overwhelmed with trying to learn everything. As far as setup/ops goes, definitely check out https://github.com/kubernetes/kops https://github.com/kubernetes/kops if you are on AWS; it's not perfect, but it made some stuff way easier in the early stages for me.

dluc 10y ago

Having worked mostly with mesos and k8s, I found k8s configuration superior, e.g. how one can import secrets, config files, and more importantly set up the network without address translation or port forwarding. Tooling seems OK with both, and I agree one needs to spend some time to get familiar with the CLI and nomenclature, IMHO because both are quite flexible and powerful.

Agrosis 10y ago

There are definitely a lot of moving parts in Kubernetes just to get it running and it was a big hurdle to get everything set up (I'm running a vanilla cluster in Vagrant). The documentation definitely covers a lot, but I found myself hunting issues down for trying to get something to work like getting k8s to schedule pods across a flannel overlay network and making sure traffic is being routed properly. Of course, you don't have to worry about any of this if you use GKE or another cloud provider. Once I got that set up, everything worked beautifully. I can just write a config and apply it to the cluster and services are deployed and scaled and whatnot. Treating your cluster as a pool of resources is pretty powerful. There are a lot of Kubernetes API objects (Deployments, Stateful Sets, Daemon Sets, ConfigMaps, Secrets), but everything has a purpose and Kubernetes makes everything come together nicely. The Kubernetes Slack channel was really helpful for me as well, there's a lot of nice and friendly people there willing to answer your questions and help you debug your setup.

roque 10y ago

Yes, it is the holy grail. Yes, the documentation is a bit spotty in some cases. We've been using kubernetes in GCE for about 1 year. We run both interactive services and batch jobs in 2 clusters: dev and production. The production cluster runs fully with declarative configs (we have a jenkins instance that performs the deployments of committed k8s configs). There are a couple of features that we are missing: (pod anti-affinity now in 1.6) and better support for storage. But in general kubernetes has the right concepts for running services, imho. We have 0 ops people. Our clusters run on large VMs with 10s of PODs in the same VM. We can get high cpu/mem utilization; resilience to failures; logging; monitoring (via prometheus - GCE /stackdriver is awful). I don't think there is anything out there that comes close in terms of supporting the right abstractions and being ready to use in production.

jdc0589 10y ago
we aren't anywhere close to using it in production yet, but my impression so far is to agree with you. Every single use case or weird requirement I've made up so far has had an answer, or had a solution in at least the proposal phase.
thockingoog 10y ago
Is your jenkins code public?

caio1982 10y ago

I don't know if it's the holy grail but it's surely one of the nicest and most impressive piece of tech I've seen since I tried Asterisk 10 years ago and had a long OMG moment myself. The ecosystem is growing by the day and Kubernetes itself changes a lot very quickly. The pace of its development is sometimes crazy. You might find yourself trying to stay on top of the latest release just to notice there's a new one out of the door and that you will surely want to move to. In this regard I would say to wait a bit, hold your horses and come back to Kubernetes once its release cycle is a bit saner for production environments (my personal estimation is by the end of this year, or earlier than that). What is missing in my opinion is a better container backend other than Docker (I just find it not enough for more complex scenarios, too bloated, there is the PID-1-hell, etc). With the new container runtime interface I am sure this won't be a problem in the long term. Also, storage. I mean, the whole user side of setting it up is a breeze. The new release makes it even more simple apparently. However I'm still not very sure how reliable things are here. Maybe it's just me not understanding how storage works in a Kubernetes world... whatever. I know for a fact I am not the only one with this concern though. What I can tell you is that after seeing a full deployment pipeline in Kubernetes actually working, and seeing all its load balancing and HA and rollout/rollback features, I am not going to stop using it. My recommendation for starters: the docs are good, read about every Object type and things will eventually "click" in your head. Make sure you understand every bit that is inside a spec file, it will make things easier to understand. Get how ingress/egress work, they can be a pain in Kubernetes sometimes. Try minikube, for Darwin's sake.

caio1982 10y ago
Just a clarification: I am not saying Kubernetes will only be good for production envs by the end of the year, what I tried to say is that this year is going to be THE year for Kubernetes and it will see lots of changes that makes it worth to wait a little longer and start with the right release that contains all the nice bits you would expect it to have, say, today.
marccarre 10y ago
> What is missing in my opinion is a better container backend other than Docker (I just find it not enough for more complex scenarios, too bloated, there is the PID-1-hell, etc). With the new container runtime interface I am sure this won't be a problem in the long term. I have only been experimenting a bit with rkt, and I am very new to it, but as far as I understand it should help to address some of the points regarding the container backend, since: 1) it is just a container runtime, 2) doesn't have the client/server architecture that Docker has, 3) has a lightweight systemd starting as PID1 and reaping/adopting children, and 4) works with Kubernetes. See also: - https://coreos.com/rkt/docs/latest/devel/architecture.html https://coreos.com/rkt/docs/latest/devel/architecture.html - https://kubernetes.io/docs/getting-started-guides/rkt/ https://kubernetes.io/docs/getting-started-guides/rkt/ - https://coreos.com/rkt/docs/latest/using-rkt-with-kubernetes.html https://coreos.com/rkt/docs/latest/using-rkt-with-kubernetes... Also note that `docker run` has an `--init` option based on `tini` which can help with the PID-1 hell. See: https://docs.docker.com/engine/reference/run/#specify-an-init-process https://docs.docker.com/engine/reference/run/#specify-an-ini...

jondubois 10y ago

I think it's the holy grail for stateless services. For stateful systems though, (that need to persist data) things can be tricky. Maybe the improved node affinity features in this release will improve things but I still feel limited because I can't seem to find a way to permanently bind a Pod to a specific host's file system. They keep pushing for NFS or other complex network file systems that add huge network I/O overheads instead of letting you deal with distributed storage across multiple hosts yourself (which would work better with most new database engines that have built-in cluster support). Last time I checked, you COULD bind a Pod to the underlying host's file system but the documentation says that this is only for debugging purposes and not suitable for production.

vishh 10y ago
FYI: Local Storage for stateful services will be tackled in the upcoming releases - https://github.com/kubernetes/community/pull/306 https://github.com/kubernetes/community/pull/306

thockingoog 10y ago

ACK. We recognize this aspect of the docs (engineers writing docs == encyclopedia articles). We have real, honest to goodness tech writers now, and they are working on it. One piece at a time.