3 ms·

I definitely have not found updates on GKE to be easy, even though all it takes to trigger it is clicking a button. Each one I’ve done on our cluster over the

I definitely have not found updates on GKE to be easy, even though all it takes to trigger it is clicking a button. Each one I’ve done on our cluster over the past couple of years has caused something to break, often in a strange and obscure manner. One time an automatic master upgrade happened at 9pm and effectively sent all of our logs into a black hole for reasons unknown. As with many of the other upgrade hiccups, I just had to poke, prod, delete and recreate entities and things magically started working again. There is so much going on under the hood it makes sense that there would be issues with upgrades, so the frequency of the releases is frustrating. Dealing with after hours alarms just because the k8s release cycle is so aggressive really grinds my gears. I would fully support an LTS version.

cagenut 8y ago
this is the hardest thing for people to get sometimes, it was the original core "concept" behind devops/ci-cd: upgrades don't get easier when you delay them and make them bigger, they get harder. if you think k8s upgrades are bad now, having them change twice as much half as often would be worse.
- macNchz 8y ago
  We've certainly embraced the devops approach, releasing many times per day with lots of automation on k8s, but I don't see this as being incompatible with using LTS releases. Having an LTS release available offers the flexibility to choose when to upgrade based on any number of factors affecting the business and engineering cycles, the same way we pin software libraries to specific versions so our own stability and build consistency isn't subject to the varying release cycles of the package authors. There's definitely a balance to be found between waiting so long to upgrade that it becomes insurmountable, and having to frequently run disruptive, mystery-downtime-causing upgrades for unneeded new features.
  - solatic 8y ago
    If you have an automated test suite for your applications, and you use it to ensure that candidates pass before deployment, why can't you test new Kubernetes releases by ensuring that your apps' test suites (which are known to be good) pass on new Kubernetes versions? If the test suite fails on a new Kubernetes version, then either improve the test suite to get rid of false positives or fix the software. In proper CI/CD practices, build "stability" is an anachronism. Master is always green. Test changes before merging. If the changes fail the test suite then don't merge and don't deploy until it's fixed. Master stays green. Why is dynamic infrastructure any different? LTS only makes sense as a strategy if testing and carrying out the upgrade involve significant amounts of work which needs to be planned out in advance. Ideally, this isn't the case.
manigandham 8y ago
That's mostly on the GKE team and not K8S itself. They run plenty of other extra components that arent well tested and often have issues during upgrades, along with the recent infrastructure updates for regional clusters, network endpoint groups and the rest that complicate things further. Errors have never been major for us though, and they do seem to be greatly reduced recently with no noticeable issues in the 1.10x line.

cagenut 8y ago

this is the hardest thing for people to get sometimes, it was the original core "concept" behind devops/ci-cd: upgrades don't get easier when you delay them and make them bigger, they get harder. if you think k8s upgrades are bad now, having them change twice as much half as often would be worse.

macNchz 8y ago
We've certainly embraced the devops approach, releasing many times per day with lots of automation on k8s, but I don't see this as being incompatible with using LTS releases. Having an LTS release available offers the flexibility to choose when to upgrade based on any number of factors affecting the business and engineering cycles, the same way we pin software libraries to specific versions so our own stability and build consistency isn't subject to the varying release cycles of the package authors. There's definitely a balance to be found between waiting so long to upgrade that it becomes insurmountable, and having to frequently run disruptive, mystery-downtime-causing upgrades for unneeded new features.
- solatic 8y ago
  If you have an automated test suite for your applications, and you use it to ensure that candidates pass before deployment, why can't you test new Kubernetes releases by ensuring that your apps' test suites (which are known to be good) pass on new Kubernetes versions? If the test suite fails on a new Kubernetes version, then either improve the test suite to get rid of false positives or fix the software. In proper CI/CD practices, build "stability" is an anachronism. Master is always green. Test changes before merging. If the changes fail the test suite then don't merge and don't deploy until it's fixed. Master stays green. Why is dynamic infrastructure any different? LTS only makes sense as a strategy if testing and carrying out the upgrade involve significant amounts of work which needs to be planned out in advance. Ideally, this isn't the case.

manigandham 8y ago

That's mostly on the GKE team and not K8S itself. They run plenty of other extra components that arent well tested and often have issues during upgrades, along with the recent infrastructure updates for regional clusters, network endpoint groups and the rest that complicate things further. Errors have never been major for us though, and they do seem to be greatly reduced recently with no noticeable issues in the 1.10x line.