7 ms·

Kubernetes 1.14 released

173 points by gtirloni 8y ago

grantlmiller 8y ago
The inclusion of Kustomize[1] into kubectl is a big step forward for the K8s ecosystem as it provides a native solution for application configuration. Once you really grok the pattern of using overlays and patches, it starts to feel like a pattern that you'll want to use everywhere (even outside of the k8s ecosystem). I'm excited to see that the kubectl docs[2] are actually recommending -k as the default solution (vs -f). Though Apply can be run directly against Resource Config files or directories using -f, it is recommended to run Apply against a kustomization.yaml using -k. The kustomization.yaml allows users to define configuration that cuts across many Resources (e.g. namespace). Really amazing work from everyone on this release. [1]https://kustomize.io https://kustomize.io [2]https://kubectl.docs.kubernetes.io/pages/app_management/apply.html https://kubectl.docs.kubernetes.io/pages/app_management/appl...
- jsmeaton 8y ago
  In which situations would you consider using Kustomize over helm? Helm seems like a decent option for defining your app, especially if you’re using other helm dependencies. I’m yet to deploy anything to a production k8s cluster, so I’m looking for opinions of people that know what they’re talking about.
  - chupasaurus 8y ago
    > Helm seems like a decent option If you want to: have PITA with RBAC, forget about manual changes because there are no 3-way-merge in Helm2 (that's what kubectl apply does, only hope it will be in Helm 3) and after this change helm cannot apply desired state, some of the biggest pains lasting for years had been resolved only in 2.13 with --atomic parameter. Aside from dependency resolution, everything that Helm does could be made by any template engine + API caller.
  - littlekosh 8y ago
    Helm is great right up until you run into a chart that doesn’t support what you need, i.e. if you want to run on tainted nodes, it needs to support tolerations. Want to add a sidecar to a pod? No. It also lies about the actual status of the installation. A deployment that applies successfully but has issues with the new pods coming up can still show successful. The larger charts are a nightmare which isn’t helm’s fault but it doesn’t make things appreciably easier either. Kustomize has a lot of warts too. If the type definition in k8s source doesn’t have a merge key defined, it’s going to overwrite the entire section of the resource. You can use JsonPatches to get around that problem but that is super gross too. The entire ecosystem is trying to solve the same problem and no one has come up with the definitive way of doing it (because it’s a HARD problem) It seems like Helm templating and then passing to kustomize to add in the one-off changes is the direction a lot of people are heading but it’s more of a least-bad solution than a good one. I will say, if you’re using helm templates for your own stuff, you very likely going to be happy with it.
    - jaxxstorm 8y ago
      This is a problem we got sick and tired of at $work, so we made kr8: https://github.com/apptio/kr8 https://github.com/apptio/kr8 You can define a component which you install into many clusters and then slightly differentiate them based on cluster parameters, kind of like Puppet or Chef (without the application stage) Alongside this, you can actually patch helm charts. An example component can be found here https://github.com/jaxxstorm/kr8-cfgmgmt-example/blob/master/components/external_dns https://github.com/jaxxstorm/kr8-cfgmgmt-example/blob/master... The patches.jsonnet allows you to add a commandline flag that wasn't in the helm chart at one time.
      - littlekosh 8y ago
        Oh no, jsonnet! Seriously though, gonna look closer at this tomorrow at the office. Thanks for the heads up!
    - grantlmiller 8y ago
      The pattern of Helm template + Kustomize is operationalized with Ship: https://github.com/replicatedhq/ship https://github.com/replicatedhq/ship plus some other functionality to automate pull requests from upstream updates.
  - halbritt 8y ago
    My team uses Helm extensively and will probably stick with it for the time being as it seems to have the most traction at the moment. As an example, MSSQL isn't a part of my core stack, but occasionally data engineers get databases from folks that they need to extract data from. Helm allows me to get one running in approximately 2 minutes or less. For many of the core components of our infrastructure we can deploy similarly quickly. We do our best to track stable for those charts. This is the tremendous advantage of helm. Unfortunately, many of those stable charts for core infrastructure are barely POC-worthy and take a great deal of effort to get into production state. For example, there is no helm chart for Kafka with TLS support. Maybe not necessary for every production implementation, but dammit, this is 2019. That said, helm is not safe. A helm "release" is not an artifact. It's a deployment of a helm chart of a certain version. That helm chart version may or may not correlate to the underlying version of the software you're running as that artifact is presumably defined with an "image" in values. For example, I've run the same aforementioned MSSQL helm chart with 2017-CU7-ubuntu, 2017-CU8-ubuntu, and 2017-CU9-ubuntu tags. A quick "helm list" might show you the same chart version across three different environments but they're still very different. Helm is not idempotent. Helm upgrade is a nightmare. One has to perform a helm upgrade to set values. Helm upgrades often fail. Even though they report failure, a look under the hood reveals that it actually succeeded. Occasionally a helm upgrade blows away the persistent volume attached to a stateful thing. That's always the best. tl;dr helm is the most popular, but has some massive shortcomings so we're all looking for something to replace it and kustomize is exciting because it's now for better or worse part of the core.
- bogomipz 8y ago
  Is Kustomize a native alternative to Helm then?
  - shaklee3 8y ago
    Helm provides different capabilities. The chart functionally allows you to install and manage deployments with the intent that it's little work. It also has a templating engine that can work with or without the charts. Kustomize is purely hierarchical templates.
  - nouney 8y ago
    Yes it is. You can do more with Kustomize
    - bogomipz 8y ago
      Interesting these two responses seem to contradict each other no?
- jpgvm 8y ago
  I really wish they didn't include this. Whilst kustomize is infinitely better than helm neither are good tools. That said if there was a choice between the 2 kustomize wins hands down I guess.
  - ses1984 8y ago
    What's a good tool then?
    - SirensOfTitan 8y ago
      I really like kubecfg--it's an unopinionated powerful tool that uses jsonnet to describe k8s resources. I really tried to make kustomize work but I found it really, really rigid. I am, however, not sure what kubecfg's future is: ksonnet (its way more complicated cousin) is being deprecated.
      - dijital 8y ago
        Re: ksonnet deprecation: I heard from some of the CoreOS team at a Prometheus Meetup recently that ksonnet is not exactly being deprecated. Heptio (the owners of ksonnet) were bought by VMWare and will no longer maintain it. But it's possible the CoreOS guys will take ownership since they use it heavily in some of their tools.
- moltar 8y ago
  I’m liking two other projects in the same space: - kosko - pulumi
shereadsthenews 8y ago
Just think about how much developer effort went into creating a web publishing framework so these release notes would be utterly unreadable on a mobile browser. The mind reels.
- sheeshkebab 8y ago
  Created by developers for developers (on 27” monitors)
  - kevinsimper 8y ago
    I submitted a PR to fix it :D https://github.com/kubernetes/website/pull/13417 https://github.com/kubernetes/website/pull/13417
    - wheelsinwheels 8y ago
      Approved. Thank you!
- altmind 8y ago
  they used bootstrap, which is responsive by default, but they disabled the responsiveness with body{min-width: 1200px;}, they are actually fighting their framework, disabling the acceessibility features that are already there.
  - daveFNbuck 8y ago
    This is super weird because that min-width has no positive effects. It just makes things worse on smaller screens. Removing it makes the page fully responsive and usable on all screen sizes.
  - shaklee3 8y ago
    Can you submit a PR?
- wheelsinwheels 8y ago
  I'm one of the leads for docs. We've got an issue open for the blog CSS: https://github.com/kubernetes/website/issues/13412 https://github.com/kubernetes/website/issues/13412
jasonhansel 8y ago
And right when you thought proprietary, non-standard OSes were finally going away...
- zufallsheld 8y ago
  What do you mean? Kubernetes is open source and the de-facto standard in container orchestration.
  - pgeorgi 8y ago
    I guess that was a reference to Windows nodes
  - lsh 8y ago
    might be referring to this news item from a few days ago https://news.ycombinator.com/item?id=19435854 https://news.ycombinator.com/item?id=19435854 "The Cloud is Just Another Sun"
  - jasonhansel 8y ago
    I was referring to Windows support, not k8s itself--sorry that my language was confusing.
joseph 8y ago
Keights[1] has a snapshot version available for running 1.14.0 in AWS, in case you can't wait for EKS :). 1. https://github.com/cloudboss/keights https://github.com/cloudboss/keights
- conradk 8y ago
  How does Keights compare to Kops ? https://github.com/kubernetes/kops https://github.com/kubernetes/kops
  - Nullabillity 8y ago
    Kops is (intentionally) very out-of-date. This is a blog post about K8s 1.14 being released, but the latest stable Kops is still based on K8s 1.11.x.
    - conradk 8y ago
      Ah yes, OK. What about Kubespray ? https://github.com/kubernetes-sigs/kubespray https://github.com/kubernetes-sigs/kubespray
      - Nullabillity 8y ago
        Kubespray SSH:s in and does what you'd do manually, while Kops deploys a prebuilt image. That means Kubespray is pretty much cloud-agnostic, and much easier to customize, but it's also not really compatible with auto-scaling groups and the like.
    - slyall 8y ago
      I don't think they intend to be as out-of-date as they are. They are supposed to be around a version behind. So they should really be releasing 1.13 soon rather than 1.12. They have had a few big tech changes in the 1.9/1.10/1.11 timeline so hopefully they will be able to catchup a little.
      - justinsb 8y ago
solatic 8y ago
The big news here is Local Persistent Volumes hitting GA, which is the key prerequisite for running production databases on Kubernetes and envisioning Kubernetes-only production environments. Big kudos to SIG Storage!
- dkhenry 8y ago
  The Local Persistent Volumes are not quite ready for things like DB workloads. They still require manual work to be done before they can be used. So if you plan on using them you need to configure your servers and manually create the PV objects, and you need to do some configuration to make sure disks are the correct sizes since there is not dynamic packing for smaller DB's. There was talk earlier about using LVM to be able to really provide a Local PV that is like the network attached PV's, but it doesn't look like that landed yet
joemccall86 8y ago
Is the conventional wisdom to still avoid self-hosting? I'd like to adopt k8s but managed hosting isn't feasible in our aws region or on-premise.
- favadi 8y ago
  Yes, you will spend more time maintaining Kubernetes than running your infrastructure if you are a small team.
  - jazoom 8y ago
    This is not necessarily true. I learned Kubernetes and converted a complex network of services over from Docker about a year ago. It took me about a month. I barely have to do anything for ongoing maintenance. It's taking up less of my time than Docker Swarm did, and Swarm took up less of my time than the Ansible mess I had before that. More stuff is automated (e.g. by using k8s cron jobs). The scheduler is pretty good now and handles almost all situations by itself. As a sole developer, I couldn't run what I do WITHOUT such an orchestration platform. And yes, I administer my own cluster on a bunch of Vultr VMs. I've had fewer problems with this over the last 3 years than it seems people have had on GCS (recent news of outage fresh in memory).
    - fredsir 8y ago
      What are you using to orchestrate the VMs? (provisioning new VMs, taking down VMs and so on)
      - jazoom 8y ago
        Nothing. Occasionally I upgrade the specs on one of the machines (vertical scaling) with one click. I'm not so bothered about having a bit of extra capacity since the VMs are so much cheaper than on GCS/AWS/Azure. Eventually I might want to add more machines to the cluster. That would take me a few minutes. I use sup scripts to do the setup. I like it much better than Ansible. https://github.com/pressly/sup https://github.com/pressly/sup

173 points by gtirloni 8y ago

The inclusion of Kustomize[1] into kubectl is a big step forward for the K8s ecosystem as it provides a native solution for application configuration. Once you really grok the pattern of using overlays and patches, it starts to feel like a pattern that you'll want to use everywhere (even outside of the k8s ecosystem). I'm excited to see that the kubectl docs[2] are actually recommending -k as the default solution (vs -f). Though Apply can be run directly against Resource Config files or directories using -f, it is recommended to run Apply against a kustomization.yaml using -k. The kustomization.yaml allows users to define configuration that cuts across many Resources (e.g. namespace). Really amazing work from everyone on this release. [1]https://kustomize.io https://kustomize.io [2]https://kubectl.docs.kubernetes.io/pages/app_management/apply.html https://kubectl.docs.kubernetes.io/pages/app_management/appl...

jsmeaton 8y ago
In which situations would you consider using Kustomize over helm? Helm seems like a decent option for defining your app, especially if you’re using other helm dependencies. I’m yet to deploy anything to a production k8s cluster, so I’m looking for opinions of people that know what they’re talking about.
- chupasaurus 8y ago
  > Helm seems like a decent option If you want to: have PITA with RBAC, forget about manual changes because there are no 3-way-merge in Helm2 (that's what kubectl apply does, only hope it will be in Helm 3) and after this change helm cannot apply desired state, some of the biggest pains lasting for years had been resolved only in 2.13 with --atomic parameter. Aside from dependency resolution, everything that Helm does could be made by any template engine + API caller.
- littlekosh 8y ago
  Helm is great right up until you run into a chart that doesn’t support what you need, i.e. if you want to run on tainted nodes, it needs to support tolerations. Want to add a sidecar to a pod? No. It also lies about the actual status of the installation. A deployment that applies successfully but has issues with the new pods coming up can still show successful. The larger charts are a nightmare which isn’t helm’s fault but it doesn’t make things appreciably easier either. Kustomize has a lot of warts too. If the type definition in k8s source doesn’t have a merge key defined, it’s going to overwrite the entire section of the resource. You can use JsonPatches to get around that problem but that is super gross too. The entire ecosystem is trying to solve the same problem and no one has come up with the definitive way of doing it (because it’s a HARD problem) It seems like Helm templating and then passing to kustomize to add in the one-off changes is the direction a lot of people are heading but it’s more of a least-bad solution than a good one. I will say, if you’re using helm templates for your own stuff, you very likely going to be happy with it.
  - jaxxstorm 8y ago
    This is a problem we got sick and tired of at $work, so we made kr8: https://github.com/apptio/kr8 https://github.com/apptio/kr8 You can define a component which you install into many clusters and then slightly differentiate them based on cluster parameters, kind of like Puppet or Chef (without the application stage) Alongside this, you can actually patch helm charts. An example component can be found here https://github.com/jaxxstorm/kr8-cfgmgmt-example/blob/master/components/external_dns https://github.com/jaxxstorm/kr8-cfgmgmt-example/blob/master... The patches.jsonnet allows you to add a commandline flag that wasn't in the helm chart at one time.
    - littlekosh 8y ago
      Oh no, jsonnet! Seriously though, gonna look closer at this tomorrow at the office. Thanks for the heads up!
  - grantlmiller 8y ago
    The pattern of Helm template + Kustomize is operationalized with Ship: https://github.com/replicatedhq/ship https://github.com/replicatedhq/ship plus some other functionality to automate pull requests from upstream updates.
- halbritt 8y ago
  My team uses Helm extensively and will probably stick with it for the time being as it seems to have the most traction at the moment. As an example, MSSQL isn't a part of my core stack, but occasionally data engineers get databases from folks that they need to extract data from. Helm allows me to get one running in approximately 2 minutes or less. For many of the core components of our infrastructure we can deploy similarly quickly. We do our best to track stable for those charts. This is the tremendous advantage of helm. Unfortunately, many of those stable charts for core infrastructure are barely POC-worthy and take a great deal of effort to get into production state. For example, there is no helm chart for Kafka with TLS support. Maybe not necessary for every production implementation, but dammit, this is 2019. That said, helm is not safe. A helm "release" is not an artifact. It's a deployment of a helm chart of a certain version. That helm chart version may or may not correlate to the underlying version of the software you're running as that artifact is presumably defined with an "image" in values. For example, I've run the same aforementioned MSSQL helm chart with 2017-CU7-ubuntu, 2017-CU8-ubuntu, and 2017-CU9-ubuntu tags. A quick "helm list" might show you the same chart version across three different environments but they're still very different. Helm is not idempotent. Helm upgrade is a nightmare. One has to perform a helm upgrade to set values. Helm upgrades often fail. Even though they report failure, a look under the hood reveals that it actually succeeded. Occasionally a helm upgrade blows away the persistent volume attached to a stateful thing. That's always the best. tl;dr helm is the most popular, but has some massive shortcomings so we're all looking for something to replace it and kustomize is exciting because it's now for better or worse part of the core.
bogomipz 8y ago
Is Kustomize a native alternative to Helm then?
- shaklee3 8y ago
  Helm provides different capabilities. The chart functionally allows you to install and manage deployments with the intent that it's little work. It also has a templating engine that can work with or without the charts. Kustomize is purely hierarchical templates.
- nouney 8y ago
  Yes it is. You can do more with Kustomize
  - bogomipz 8y ago
    Interesting these two responses seem to contradict each other no?
jpgvm 8y ago
I really wish they didn't include this. Whilst kustomize is infinitely better than helm neither are good tools. That said if there was a choice between the 2 kustomize wins hands down I guess.
- ses1984 8y ago
  What's a good tool then?
  - SirensOfTitan 8y ago
    I really like kubecfg--it's an unopinionated powerful tool that uses jsonnet to describe k8s resources. I really tried to make kustomize work but I found it really, really rigid. I am, however, not sure what kubecfg's future is: ksonnet (its way more complicated cousin) is being deprecated.
    - dijital 8y ago
      Re: ksonnet deprecation: I heard from some of the CoreOS team at a Prometheus Meetup recently that ksonnet is not exactly being deprecated. Heptio (the owners of ksonnet) were bought by VMWare and will no longer maintain it. But it's possible the CoreOS guys will take ownership since they use it heavily in some of their tools.
moltar 8y ago
I’m liking two other projects in the same space: - kosko - pulumi

shereadsthenews 8y ago

Just think about how much developer effort went into creating a web publishing framework so these release notes would be utterly unreadable on a mobile browser. The mind reels.

sheeshkebab 8y ago
Created by developers for developers (on 27” monitors)
- kevinsimper 8y ago
  I submitted a PR to fix it :D https://github.com/kubernetes/website/pull/13417 https://github.com/kubernetes/website/pull/13417
  - wheelsinwheels 8y ago
    Approved. Thank you!
altmind 8y ago
they used bootstrap, which is responsive by default, but they disabled the responsiveness with body{min-width: 1200px;}, they are actually fighting their framework, disabling the acceessibility features that are already there.
- daveFNbuck 8y ago
  This is super weird because that min-width has no positive effects. It just makes things worse on smaller screens. Removing it makes the page fully responsive and usable on all screen sizes.
- shaklee3 8y ago
  Can you submit a PR?
wheelsinwheels 8y ago
I'm one of the leads for docs. We've got an issue open for the blog CSS: https://github.com/kubernetes/website/issues/13412 https://github.com/kubernetes/website/issues/13412

jasonhansel 8y ago

And right when you thought proprietary, non-standard OSes were finally going away...

zufallsheld 8y ago
What do you mean? Kubernetes is open source and the de-facto standard in container orchestration.
- pgeorgi 8y ago
  I guess that was a reference to Windows nodes
- lsh 8y ago
  might be referring to this news item from a few days ago https://news.ycombinator.com/item?id=19435854 https://news.ycombinator.com/item?id=19435854 "The Cloud is Just Another Sun"
- jasonhansel 8y ago
  I was referring to Windows support, not k8s itself--sorry that my language was confusing.

joseph 8y ago

Keights[1] has a snapshot version available for running 1.14.0 in AWS, in case you can't wait for EKS :). 1. https://github.com/cloudboss/keights https://github.com/cloudboss/keights

conradk 8y ago
How does Keights compare to Kops ? https://github.com/kubernetes/kops https://github.com/kubernetes/kops
- Nullabillity 8y ago
  Kops is (intentionally) very out-of-date. This is a blog post about K8s 1.14 being released, but the latest stable Kops is still based on K8s 1.11.x.
  - conradk 8y ago
    Ah yes, OK. What about Kubespray ? https://github.com/kubernetes-sigs/kubespray https://github.com/kubernetes-sigs/kubespray
    - Nullabillity 8y ago
      Kubespray SSH:s in and does what you'd do manually, while Kops deploys a prebuilt image. That means Kubespray is pretty much cloud-agnostic, and much easier to customize, but it's also not really compatible with auto-scaling groups and the like.
  - slyall 8y ago
    I don't think they intend to be as out-of-date as they are. They are supposed to be around a version behind. So they should really be releasing 1.13 soon rather than 1.12. They have had a few big tech changes in the 1.9/1.10/1.11 timeline so hopefully they will be able to catchup a little.
    - justinsb 8y ago

solatic 8y ago

The big news here is Local Persistent Volumes hitting GA, which is the key prerequisite for running production databases on Kubernetes and envisioning Kubernetes-only production environments. Big kudos to SIG Storage!

dkhenry 8y ago
The Local Persistent Volumes are not quite ready for things like DB workloads. They still require manual work to be done before they can be used. So if you plan on using them you need to configure your servers and manually create the PV objects, and you need to do some configuration to make sure disks are the correct sizes since there is not dynamic packing for smaller DB's. There was talk earlier about using LVM to be able to really provide a Local PV that is like the network attached PV's, but it doesn't look like that landed yet

joemccall86 8y ago

Is the conventional wisdom to still avoid self-hosting? I'd like to adopt k8s but managed hosting isn't feasible in our aws region or on-premise.

favadi 8y ago
Yes, you will spend more time maintaining Kubernetes than running your infrastructure if you are a small team.
- jazoom 8y ago
  This is not necessarily true. I learned Kubernetes and converted a complex network of services over from Docker about a year ago. It took me about a month. I barely have to do anything for ongoing maintenance. It's taking up less of my time than Docker Swarm did, and Swarm took up less of my time than the Ansible mess I had before that. More stuff is automated (e.g. by using k8s cron jobs). The scheduler is pretty good now and handles almost all situations by itself. As a sole developer, I couldn't run what I do WITHOUT such an orchestration platform. And yes, I administer my own cluster on a bunch of Vultr VMs. I've had fewer problems with this over the last 3 years than it seems people have had on GCS (recent news of outage fresh in memory).
  - fredsir 8y ago
    What are you using to orchestrate the VMs? (provisioning new VMs, taking down VMs and so on)
    - jazoom 8y ago
      Nothing. Occasionally I upgrade the specs on one of the machines (vertical scaling) with one click. I'm not so bothered about having a bit of extra capacity since the VMs are so much cheaper than on GCS/AWS/Azure. Eventually I might want to add more machines to the cluster. That would take me a few minutes. I use sup scripts to do the setup. I like it much better than Ansible. https://github.com/pressly/sup https://github.com/pressly/sup