9 ms·

Kubernetes 1.9: Apps Workloads GA and Expanded Ecosystem

151 points by el_duderino 9y ago

slap_shot 9y ago
I cannot give enough thanks to everyone that works so hard on Kubernetes. The speed of these releases, community support, industry adoption, feature set, and ease of use are beyond anything I could have asked for. I started a startup about 18 months ago and our product requires a massive amount data pipelining. The infrastructure we have running on Kubernetes is beyond what a small group should be able to deploy and maintain by ourselves, but Kubernetes makes it not only possible, but enjoyable (really). To everyone involved (and there are a lot): Thank you SO much. I can't wait to see what 2018 looks like for Kubernetes.
- ec109685 9y ago
  What type of infrastructures do you run kubernetes on?
  - slap_shot 9y ago
    We run Kubernetes on all the three major clouds: AWS, GCP and Azure.
- cirowrc 9y ago
  That's so great to hear! I love to see how it empowers small teams to deliver big stuff. What was the trickiest part for you when it comes to real production usage?
  - linkmotif 9y ago
    Using it on GKE. Literally no tricky parts except figuring out how to keep my images on GCR in sync with my manifests. For that this tool is quite helpful: https://github.com/dminkovsky/kube-cloud-build https://github.com/dminkovsky/kube-cloud-build. Otherwise it's been such a pleasure. PVs and StatefulSets are a total game changer. Looking forward to 2018. EDIT: I should say RBAC has been a little frustrating. But obviously it's for the better.
- linkmotif 9y ago
  Couldn’t have said it better myself. I am so grateful.
- ericand 9y ago
  Great post. Could you describe further what you mean by data pipelining? Is this something that you could also use Spark, Kafka for? I'd be interested to know if Kubernetes would work for me as well.
  - deleted 9y ago
    [deleted]
  - lobster_johnson 9y ago
    One promising piece of tech is Pachyderm [1], which runs natively on Kubernetes. It allows you to set up pipelines of batch jobs (no streaming yet, as far as I know) that operate on immutable, versioned data. There are some gotchas, though. For one [2], it requires read-write access to your storage buckets (!), and so on GKE you have to modify your nodepool's access scopes to give the whole node read-write access to all your buckets. Not is this terrible security practice, but access scopes are deprecated on Google Cloud. You should always grant access through service accounts, not scopes. We spent some time fighting this, as there's literally no way to make Pachyderm use a service account. There are some other niggling issues, and the Pachyderm team has been less than enthuastic in addressing them (e.g. see [3], which has zero activity despite being absolutely crucial to run in production), but it does seem quite promising. The architecture, at least, is sound. [1] http://pachyderm.io http://pachyderm.io [2] https://github.com/pachyderm/pachyderm/issues/2538 https://github.com/pachyderm/pachyderm/issues/2538 [3] https://github.com/pachyderm/pachyderm/issues/2537 https://github.com/pachyderm/pachyderm/issues/2537
    - jdoliner 9y ago
      Hey lobster_johnson, Pachyderm CEO here. Thanks for the shout out and we're glad you feel Pachyderm is promising for these use cases. I just wanted to clarify / respond to a few things. > no streaming yet, as far as I know Pachyderm actually does do streaming, there's nothing in the API about it because it all happens automatically. This is one of the huge advantages of immutable data, the system can automatically detect if it's already done a computation by looking at a hash of the input and the code. The system will never do the same computation twice, all pipelines are streaming by default. Sorry we've been less than enthusiastic in addressing these niggling issues. The truth is it's pretty hard to be enthusiastic in addressing issues for specific deployment scenarios. AFAIK the Google Cloud Storage library we're using doesn't support service accounts, the whole library has been deprecated, the second GCS library we've had deprecated underneath us. So upgrading that is a bit of a pain but the bigger pain is that now users with existing Google clusters will have their clusters break under them unless we can figure out a way to get scopes and service accounts to both work. And we have every reason to believe that all of this will be deprecated within the next 6 months and we'll be on to our 4th GCS library. And that's only one deployment scenario, we're doing the same thing for AWS, Azure and a host of local deployment options. Anyways, these are our problems to solve, and we will solve them. We're thrilled that you want to use Pachyderm and we want to make that experience as smooth as possible. We just need to ask for a bit of patience regarding these deployment scenarios.
- jaz46 9y ago
  Have you checked out Pachyderm (disclaimer: my company) for data pipelining on Kubernetes? Our product has been available since the first Kubernetes release and is specifically designed for data analysis and pipelining tasks on Kubernetes. github.com/pachyderm/pachyderm pachyderm.io
- lemoncucumber 9y ago
  As much as I like kubernetes, I find the whole ecosystem to be frustratingly immature and unreliable. I'm sure it's trending in the right direction with respect to reliability, but it just doesn't feel like it's there yet. Minikube seems especially buggy and is missing some obvious features.
  - akvadrako 9y ago
    Well minikube is not for production, so maybe it's buginess is evidence they are focusing their time on the more important components.
eduren 9y ago
Its really fascinating to see Windows support be added throughout the container ecosystem. Does anybody have experience actually moving legacy Windows applications into a containerized infrastructure?
jcastro 9y ago
Kubecon just wrapped up and all the videos of the talk are up, lots of good content here: https://www.youtube.com/channel/UCvqbFHwN-nwalWPjPUKpvTA/videos https://www.youtube.com/channel/UCvqbFHwN-nwalWPjPUKpvTA/vid...
linsomniac 9y ago
I really wanted to like Kubernetes, but I couldn't figure it out. I'm a pretty experienced Linux admin, and despite days of reading all the sources of documentation I could find I couldn't get the networking setup to where I thought it should be. I don't know if that was a problem of my mental model being wrong and the documentation not educating me on the correct model, or the documentation of the CNI, Flannel/whatever just being too sparse, or what. I started with the Google Ubuntu packages, and was able to get containers up fairly quickly, but being able to access them from anything other than the host machine I just couldn't figure out. Asking for help on IRC wasn't useful (aside: what's the deal with IRC channels with tons of users but no messages for days?) I've since wiped that cluster and tried the Ubuntu Kubernetes installation using conjure-up, it seems like it did the right thing but I'm not sure yet how to get to a cluster from a single machine. Trying to decide if I pursue Kubernetes more, try Dokku/Flynn/Deis (which I just saw referenced in a previous HN discussion) because they look really promising... I had hoped it would be the ganeti of containerization, but I had far fewer problems with ganeti.
- user5994461 9y ago
  No worries, we understand the pain, we've all been there. The abstracted network is a constant source of headache when dealing with kubernetes and containers.
- slap_shot 9y ago
  FWIW, if you create a GKE cluster and create an nginx deployment: kubectl apply -f https://k8s.io/docs/tasks/run-application/deployment.yaml https://k8s.io/docs/tasks/run-application/deployment.yaml And simply create a Load Balancer: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/ https://kubernetes.io/docs/tasks/access-application-cluster/... You have nginx exposed via an external IP address in about two minutes. Once you're comfortable with that, you can start working in Ingress and such to get more sophisticated. I agree the IRC/Slack channel communication isn't the best. I prefer the Kubernetes User Group. You can usually get a great response in less than a day. I've used it a lot.
  - 9y ago
cirowrc 9y ago
I'm particularly excited about the advance of the Container Storage Interface (CSI). Having developed two custom docker volume plugins I wonder whether Docker is going to adopt it for its cluster offering (swarm mode) or leave it aside as it did with CNI (real question). Anybody knows about it?
- dward 9y ago
  It's to be seen but owners of the spec represent mesos, kubernetes, docker and cloud foundry so it seems like Docker is open to the idea. source: https://github.com/container-storage-interface/spec/blob/master/OWNERS https://github.com/container-storage-interface/spec/blob/mas...
  - cirowrc 9y ago
    oh, I didn't look at that. Thanks!
thomasahle 9y ago
I have a simple service I would like to put in production. It is a simple ML code + Rest API + testing sandwich. I only have one (virtual) server to run this on, but I still think Kubernetes looks really helpful, allowing me to throw new versions in test and update the production server without downtime. Minikube is really nice to use, and everything seems to work well, however everywhere I read, I am told minikube shouldn't be used in production. Can anybody tell me if there's a reason for this? And if so what is the recommended setup for a single-node Kubernetes? (Side question: Why does minikube run everything in a virtual machine? Doesn't that hurt performance?)
- iamwil 9y ago
  Huh, I've been wrapping my ML models into a flask app and deploying it as well. If you don't mind me asking, what are you using the ML model for, and what are the current suck-ass parts of deploying it?
- dosethree 9y ago
  You can get what you want with capistrano and a something like unicorn taht supports no downtime deploys
- snug 9y ago
  Kuberenetes is meant to be a self-healing always up infrastructure, if your minikube goes down, it cannot heal fast enough, and your services will go down. It is recommended to have at least 3 nodes.
- rckrd 9y ago
  Minikube runs in a virtual machine because its meant to be a cross platform development environment, and linux containers aren't natively supported on macOS or windows. Minikube does have a --none driver, which run on bare metal and can be used to provision a cluster on virtualized hardware, such as a CI system or a GCE instance. Why not minikube in production? Upgrades, downgrades, clean-up and maintenance. We don't offer any real guarantees on these lifecycle operations unlike a more robust setup like GKE. Disclaimer: I work on minikube
odammit 9y ago
Keep up the good work. You guys/gals make my job easy. Infinity thanks, I owe you a drink (or three).
reacharavindh 9y ago
Acknowledging the risk of sounding like "why can't I use rsync scripts instead of this Dropbox?", Can someone explain to me like I just got out of high school, why do I need to complicate my setup with Kubernetes/Docker Swarm/Deis/Dolly/Flynn instead of spinning up&down containers using scripted LXC commands? Someone was talking about using Kubernetes for data pipelines. For such a case, why is Kubernetes better than a script that spins up pre-made container images? Not being snark at all. Genuinely asking a "why?". Appreciate any educating responses!
- drdaeman 9y ago
  Maybe you don't. I wouldn't recommend anyone to use something extremely complex for mission critical architecture, unless unavoidable. If you feel the same, it's probably a perfectly sound reason to avoid Kubernetes if there's no necessity. When I wrote "extremely complex" I did't mean all that models and concepts from user documentation - they're the easy part (although I haven't used K8s much and probably missing a lot about its user-facing parts). I mean the internals. Because when things break and cluster goes down it's the internals that matter. I use Docker instead of plain LXC because it adds convenience of layers and versioning while not introducing any significant complexity. I had some issues with libnetwork (which sometimes behaved oddly when containers crash under high load or docker-proxy gets somehow stuck) but I can live with it. I also prefer Swarm to manual networking. It's trivial to set up and just works under normal circumstances. And I feel confident that I can fix issues reasonably fast. I had experimented with it under some odd scenarios (like a heterogeneous cluster with a mix of x86_64 and armv7h nodes) and when it had (obviously) failed, I was able to quickly find the relevant source code, read it, understand the details and correct the problem (which was trivial but not documented). And if I wouldn't be able to, I'm absolutely certain I would be able to scrap the cluster and script reprovisioning with standalone nodes, Compose and ad-hoc VPNs - I did this when Rancher (old version) had failed me badly. Now, maybe I'm just too stupid but I can't say this for Kubernetes. I've tried to experiment with it - also under explicitly unsupported conditions - and tried to dig into its documentation and code when it had failed (obviously it did, I was asking for it). However, I was literally overwhelmed by how much there is. What I've learned is a little tiny bit about CNI internals but the primary result was a conclusion "sorry but nope, that's not something I feel comfortable dealing with if I'm to support it". And I don't mean that it fail-prone or anything - just that every software project has its bugs, and sometimes they manifest themselves at really inconvenient times. On the other hand, Kubernetes is very nice when it's not you who's responsible for its operation but you're just an user. Spin a cluster on GKE and you know Google would take care of it - you'd just have to write some YAML to describe your project and enjoy. In this scenario, I think I can recommend it - even though I haven't ran anything serious on K8s but some toy projects worked without issues and were easy to deploy and maintain. And Kubernetes feels more mature than other options, having lots of nice functionality, supposedly covering a lot of different use cases and scenarios. E.g. Swarm feels like it lacks a lot of stuff - I've subscribed to a lot of tickets for features that I wanted but that aren't yet there. Just a personal opinion.

151 points by el_duderino 9y ago

I cannot give enough thanks to everyone that works so hard on Kubernetes. The speed of these releases, community support, industry adoption, feature set, and ease of use are beyond anything I could have asked for. I started a startup about 18 months ago and our product requires a massive amount data pipelining. The infrastructure we have running on Kubernetes is beyond what a small group should be able to deploy and maintain by ourselves, but Kubernetes makes it not only possible, but enjoyable (really). To everyone involved (and there are a lot): Thank you SO much. I can't wait to see what 2018 looks like for Kubernetes.

ec109685 9y ago
What type of infrastructures do you run kubernetes on?
- slap_shot 9y ago
  We run Kubernetes on all the three major clouds: AWS, GCP and Azure.
cirowrc 9y ago
That's so great to hear! I love to see how it empowers small teams to deliver big stuff. What was the trickiest part for you when it comes to real production usage?
- linkmotif 9y ago
  Using it on GKE. Literally no tricky parts except figuring out how to keep my images on GCR in sync with my manifests. For that this tool is quite helpful: https://github.com/dminkovsky/kube-cloud-build https://github.com/dminkovsky/kube-cloud-build. Otherwise it's been such a pleasure. PVs and StatefulSets are a total game changer. Looking forward to 2018. EDIT: I should say RBAC has been a little frustrating. But obviously it's for the better.
linkmotif 9y ago
Couldn’t have said it better myself. I am so grateful.
ericand 9y ago
Great post. Could you describe further what you mean by data pipelining? Is this something that you could also use Spark, Kafka for? I'd be interested to know if Kubernetes would work for me as well.
- deleted 9y ago
  [deleted]
- lobster_johnson 9y ago
  One promising piece of tech is Pachyderm [1], which runs natively on Kubernetes. It allows you to set up pipelines of batch jobs (no streaming yet, as far as I know) that operate on immutable, versioned data. There are some gotchas, though. For one [2], it requires read-write access to your storage buckets (!), and so on GKE you have to modify your nodepool's access scopes to give the whole node read-write access to all your buckets. Not is this terrible security practice, but access scopes are deprecated on Google Cloud. You should always grant access through service accounts, not scopes. We spent some time fighting this, as there's literally no way to make Pachyderm use a service account. There are some other niggling issues, and the Pachyderm team has been less than enthuastic in addressing them (e.g. see [3], which has zero activity despite being absolutely crucial to run in production), but it does seem quite promising. The architecture, at least, is sound. [1] http://pachyderm.io http://pachyderm.io [2] https://github.com/pachyderm/pachyderm/issues/2538 https://github.com/pachyderm/pachyderm/issues/2538 [3] https://github.com/pachyderm/pachyderm/issues/2537 https://github.com/pachyderm/pachyderm/issues/2537
  - jdoliner 9y ago
    Hey lobster_johnson, Pachyderm CEO here. Thanks for the shout out and we're glad you feel Pachyderm is promising for these use cases. I just wanted to clarify / respond to a few things. > no streaming yet, as far as I know Pachyderm actually does do streaming, there's nothing in the API about it because it all happens automatically. This is one of the huge advantages of immutable data, the system can automatically detect if it's already done a computation by looking at a hash of the input and the code. The system will never do the same computation twice, all pipelines are streaming by default. Sorry we've been less than enthusiastic in addressing these niggling issues. The truth is it's pretty hard to be enthusiastic in addressing issues for specific deployment scenarios. AFAIK the Google Cloud Storage library we're using doesn't support service accounts, the whole library has been deprecated, the second GCS library we've had deprecated underneath us. So upgrading that is a bit of a pain but the bigger pain is that now users with existing Google clusters will have their clusters break under them unless we can figure out a way to get scopes and service accounts to both work. And we have every reason to believe that all of this will be deprecated within the next 6 months and we'll be on to our 4th GCS library. And that's only one deployment scenario, we're doing the same thing for AWS, Azure and a host of local deployment options. Anyways, these are our problems to solve, and we will solve them. We're thrilled that you want to use Pachyderm and we want to make that experience as smooth as possible. We just need to ask for a bit of patience regarding these deployment scenarios.
jaz46 9y ago
Have you checked out Pachyderm (disclaimer: my company) for data pipelining on Kubernetes? Our product has been available since the first Kubernetes release and is specifically designed for data analysis and pipelining tasks on Kubernetes. github.com/pachyderm/pachyderm pachyderm.io
lemoncucumber 9y ago
As much as I like kubernetes, I find the whole ecosystem to be frustratingly immature and unreliable. I'm sure it's trending in the right direction with respect to reliability, but it just doesn't feel like it's there yet. Minikube seems especially buggy and is missing some obvious features.
- akvadrako 9y ago
  Well minikube is not for production, so maybe it's buginess is evidence they are focusing their time on the more important components.

eduren 9y ago

Its really fascinating to see Windows support be added throughout the container ecosystem. Does anybody have experience actually moving legacy Windows applications into a containerized infrastructure?

jcastro 9y ago

Kubecon just wrapped up and all the videos of the talk are up, lots of good content here: https://www.youtube.com/channel/UCvqbFHwN-nwalWPjPUKpvTA/videos https://www.youtube.com/channel/UCvqbFHwN-nwalWPjPUKpvTA/vid...

linsomniac 9y ago

I really wanted to like Kubernetes, but I couldn't figure it out. I'm a pretty experienced Linux admin, and despite days of reading all the sources of documentation I could find I couldn't get the networking setup to where I thought it should be. I don't know if that was a problem of my mental model being wrong and the documentation not educating me on the correct model, or the documentation of the CNI, Flannel/whatever just being too sparse, or what. I started with the Google Ubuntu packages, and was able to get containers up fairly quickly, but being able to access them from anything other than the host machine I just couldn't figure out. Asking for help on IRC wasn't useful (aside: what's the deal with IRC channels with tons of users but no messages for days?) I've since wiped that cluster and tried the Ubuntu Kubernetes installation using conjure-up, it seems like it did the right thing but I'm not sure yet how to get to a cluster from a single machine. Trying to decide if I pursue Kubernetes more, try Dokku/Flynn/Deis (which I just saw referenced in a previous HN discussion) because they look really promising... I had hoped it would be the ganeti of containerization, but I had far fewer problems with ganeti.

user5994461 9y ago
No worries, we understand the pain, we've all been there. The abstracted network is a constant source of headache when dealing with kubernetes and containers.
slap_shot 9y ago
FWIW, if you create a GKE cluster and create an nginx deployment: kubectl apply -f https://k8s.io/docs/tasks/run-application/deployment.yaml https://k8s.io/docs/tasks/run-application/deployment.yaml And simply create a Load Balancer: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/ https://kubernetes.io/docs/tasks/access-application-cluster/... You have nginx exposed via an external IP address in about two minutes. Once you're comfortable with that, you can start working in Ingress and such to get more sophisticated. I agree the IRC/Slack channel communication isn't the best. I prefer the Kubernetes User Group. You can usually get a great response in less than a day. I've used it a lot.
- 9y ago

cirowrc 9y ago

I'm particularly excited about the advance of the Container Storage Interface (CSI). Having developed two custom docker volume plugins I wonder whether Docker is going to adopt it for its cluster offering (swarm mode) or leave it aside as it did with CNI (real question). Anybody knows about it?

dward 9y ago
It's to be seen but owners of the spec represent mesos, kubernetes, docker and cloud foundry so it seems like Docker is open to the idea. source: https://github.com/container-storage-interface/spec/blob/master/OWNERS https://github.com/container-storage-interface/spec/blob/mas...
- cirowrc 9y ago
  oh, I didn't look at that. Thanks!

thomasahle 9y ago

I have a simple service I would like to put in production. It is a simple ML code + Rest API + testing sandwich. I only have one (virtual) server to run this on, but I still think Kubernetes looks really helpful, allowing me to throw new versions in test and update the production server without downtime. Minikube is really nice to use, and everything seems to work well, however everywhere I read, I am told minikube shouldn't be used in production. Can anybody tell me if there's a reason for this? And if so what is the recommended setup for a single-node Kubernetes? (Side question: Why does minikube run everything in a virtual machine? Doesn't that hurt performance?)

iamwil 9y ago
Huh, I've been wrapping my ML models into a flask app and deploying it as well. If you don't mind me asking, what are you using the ML model for, and what are the current suck-ass parts of deploying it?
dosethree 9y ago
You can get what you want with capistrano and a something like unicorn taht supports no downtime deploys
snug 9y ago
Kuberenetes is meant to be a self-healing always up infrastructure, if your minikube goes down, it cannot heal fast enough, and your services will go down. It is recommended to have at least 3 nodes.
rckrd 9y ago
Minikube runs in a virtual machine because its meant to be a cross platform development environment, and linux containers aren't natively supported on macOS or windows. Minikube does have a --none driver, which run on bare metal and can be used to provision a cluster on virtualized hardware, such as a CI system or a GCE instance. Why not minikube in production? Upgrades, downgrades, clean-up and maintenance. We don't offer any real guarantees on these lifecycle operations unlike a more robust setup like GKE. Disclaimer: I work on minikube

odammit 9y ago

Keep up the good work. You guys/gals make my job easy. Infinity thanks, I owe you a drink (or three).

reacharavindh 9y ago

Acknowledging the risk of sounding like "why can't I use rsync scripts instead of this Dropbox?", Can someone explain to me like I just got out of high school, why do I need to complicate my setup with Kubernetes/Docker Swarm/Deis/Dolly/Flynn instead of spinning up&down containers using scripted LXC commands? Someone was talking about using Kubernetes for data pipelines. For such a case, why is Kubernetes better than a script that spins up pre-made container images? Not being snark at all. Genuinely asking a "why?". Appreciate any educating responses!

drdaeman 9y ago
Maybe you don't. I wouldn't recommend anyone to use something extremely complex for mission critical architecture, unless unavoidable. If you feel the same, it's probably a perfectly sound reason to avoid Kubernetes if there's no necessity. When I wrote "extremely complex" I did't mean all that models and concepts from user documentation - they're the easy part (although I haven't used K8s much and probably missing a lot about its user-facing parts). I mean the internals. Because when things break and cluster goes down it's the internals that matter. I use Docker instead of plain LXC because it adds convenience of layers and versioning while not introducing any significant complexity. I had some issues with libnetwork (which sometimes behaved oddly when containers crash under high load or docker-proxy gets somehow stuck) but I can live with it. I also prefer Swarm to manual networking. It's trivial to set up and just works under normal circumstances. And I feel confident that I can fix issues reasonably fast. I had experimented with it under some odd scenarios (like a heterogeneous cluster with a mix of x86_64 and armv7h nodes) and when it had (obviously) failed, I was able to quickly find the relevant source code, read it, understand the details and correct the problem (which was trivial but not documented). And if I wouldn't be able to, I'm absolutely certain I would be able to scrap the cluster and script reprovisioning with standalone nodes, Compose and ad-hoc VPNs - I did this when Rancher (old version) had failed me badly. Now, maybe I'm just too stupid but I can't say this for Kubernetes. I've tried to experiment with it - also under explicitly unsupported conditions - and tried to dig into its documentation and code when it had failed (obviously it did, I was asking for it). However, I was literally overwhelmed by how much there is. What I've learned is a little tiny bit about CNI internals but the primary result was a conclusion "sorry but nope, that's not something I feel comfortable dealing with if I'm to support it". And I don't mean that it fail-prone or anything - just that every software project has its bugs, and sometimes they manifest themselves at really inconvenient times. On the other hand, Kubernetes is very nice when it's not you who's responsible for its operation but you're just an user. Spin a cluster on GKE and you know Google would take care of it - you'd just have to write some YAML to describe your project and enjoy. In this scenario, I think I can recommend it - even though I haven't ran anything serious on K8s but some toy projects worked without issues and were easy to deploy and maintain. And Kubernetes feels more mature than other options, having lots of nice functionality, supposedly covering a lot of different use cases and scenarios. E.g. Swarm feels like it lacks a lot of stuff - I've subscribed to a lot of tickets for features that I wanted but that aren't yet there. Just a personal opinion.