4 ms·

Is upgrading a kernel of a docker host that straight forward? I would worry to keep everything compatible, with a lot of testing before any upgrade of this kind

Is upgrading a kernel of a docker host that straight forward? I would worry to keep everything compatible, with a lot of testing before any upgrade of this kind. It looks like they're running k8s with kops, and the fix was merged just a few weeks ago.

YawningAngel 6y ago
I'm not in a good position to say for sure, as I've only used managed Kubernetes distributions, but I think it probably works out to less work than removing CPU limits. Kernel upgrades are at least semi-routine, so most shops that run Kubernetes themselves are going to have a process for them. Conversely, removing CPU limits and migrating the critical path to a different set of tainted nodes is a substantial one-off change with a long tail of failure scenarios that need to be tested. Thus, I would expect that a kernel upgrade would be easier than doing what Buffer did.
op00to 6y ago
It should be. Kernel updates are a critical path for bug fixes and security patches. If you can’t upgrade your kernel using a clear, premeditated plan, you are not doing your job right.
metafunctor 6y ago
By and large, you can just upgrade the kernel and nothing will break. It's one of the biggest things kernel developers worry over, in fact. The reason, I think, K8s people might do this is that they often do not have the access, experience, or skills to do platform upgrades. It's easier to just futz around on the layer they feel they understand well; the world of containers.
AmazingTurtle 6y ago
You can test out your shit by creating a new node with the recent kernel version installed. Then schedule some apps for testing on that node and see if everything works. Swap it out when you're done testing.
smcleod 6y ago
Can’t remember the last time I had a centos kernel update (kernel-ml) break anything over the past 7 odd years I’ve been running it across hundreds of servers.
q3k 6y ago
Not less straightforward than any other system upgrade, which you need to be able to do confidently, anyway.
Thaxll 6y ago
It's a single command that is safe and straight forward especially if you use a LTS distro.
bogomipz 6y ago
Yes, you just install the kernel image, kernel headers and kernel modules and reboot. You are not upgrading the distro. With Kubernetes it would be quite easy to test in isolation by simply tainting the worker node(s) that have the upgraded kernel.

YawningAngel 6y ago

I'm not in a good position to say for sure, as I've only used managed Kubernetes distributions, but I think it probably works out to less work than removing CPU limits. Kernel upgrades are at least semi-routine, so most shops that run Kubernetes themselves are going to have a process for them. Conversely, removing CPU limits and migrating the critical path to a different set of tainted nodes is a substantial one-off change with a long tail of failure scenarios that need to be tested. Thus, I would expect that a kernel upgrade would be easier than doing what Buffer did.

op00to 6y ago

It should be. Kernel updates are a critical path for bug fixes and security patches. If you can’t upgrade your kernel using a clear, premeditated plan, you are not doing your job right.

metafunctor 6y ago

By and large, you can just upgrade the kernel and nothing will break. It's one of the biggest things kernel developers worry over, in fact. The reason, I think, K8s people might do this is that they often do not have the access, experience, or skills to do platform upgrades. It's easier to just futz around on the layer they feel they understand well; the world of containers.

AmazingTurtle 6y ago

You can test out your shit by creating a new node with the recent kernel version installed. Then schedule some apps for testing on that node and see if everything works. Swap it out when you're done testing.

smcleod 6y ago

Can’t remember the last time I had a centos kernel update (kernel-ml) break anything over the past 7 odd years I’ve been running it across hundreds of servers.

q3k 6y ago

Not less straightforward than any other system upgrade, which you need to be able to do confidently, anyway.

Thaxll 6y ago

It's a single command that is safe and straight forward especially if you use a LTS distro.

bogomipz 6y ago

Yes, you just install the kernel image, kernel headers and kernel modules and reboot. You are not upgrading the distro. With Kubernetes it would be quite easy to test in isolation by simply tainting the worker node(s) that have the upgraded kernel.