6 ms·

How Meta patches Linux at hyperscale

135 points by elorant 3y ago

fragmede 3y ago
Ksplice is the original live patching technology that got bought by Oracle and was later extended to user space programs while I worked there. It's a really neat technology that isn't made obsolete by the move to cloud, since you still don't want to have to restart the whole fleet at scale.
- metadat 3y ago
  What's the live app patcher called?
  - fragmede 3y ago
    I have objections to the name, but it goes under Ksplice for User Space. https://blogs.oracle.com/virtualization/post/ksplice-zero-downtime-patching-for-user-space-packages https://blogs.oracle.com/virtualization/post/ksplice-zero-do...
    - Sarkie 3y ago
      How was Oracle to work at?
      - grepfru_it 3y ago
        Not OP but I loved oracle in the 90s/early 2000s. After the Sun acquisition it became cutthroat and I left. IBM acquiring redhat seems very similar
      - fragmede 3y ago
        Post acquisition, we moved offices, and used their HR department, and certain product decisions were made for us, but we largely did our own thing, for better or worse, so I don't know how much I can speak about how it was to work at Oracle. We were under Wim Coekaerts who is a big open source guy. Oracle's reputation as a cutthroat legal entity is well deserved, but working at that side of the company it felt unfair because, Oracle's open source contributions with VirtualBox, the UEK, and others are lost in the grar over MySQL. I think the biggest "big company" blunder while I was there was a public blog post by someone high up at the company decrying open source as bad/wrong while at the same time, Oracle was doing all this other open source stuff.
        latchkey 3y ago
        Funny given that they have been selling open source software for ages... https://docs.oracle.com/cd/B10463_01/web.904/b10320/apjsvsup.htm https://docs.oracle.com/cd/B10463_01/web.904/b10320/apjsvsup... I believe this was circa 2000, but going on longer than that.
        yjftsjthsd-h 3y ago
        > but working at that side of the company it felt unfair because, Oracle's open source contributions with VirtualBox, the UEK, and others are lost in the grar over MySQL. They are also remembered for closing OpenSolaris and shaking people down over the VirtualBox extension pack ( https://www.theregister.com/2019/10/04/oracle_virtualbox_merula/ https://www.theregister.com/2019/10/04/oracle_virtualbox_mer... ).
        andyjohnson0 3y ago
        > grar Useful word.
civilitty 3y ago
[flagged]
ravenstine 3y ago
Never heard of this "hyperscale" concept before. How is this any different from... scaling?
- mc32 3y ago
  Supermarkets and super cars aren't enough. We need hypermarkets and hyper cars to be current. So now we can’t just scale, I guess that’s for trucks, so to be current you need hyperscale.
  - froh 3y ago
    Cybertruck and cyberscale? (I'm joking)
  - phyrex 3y ago
    https://en.m.wikipedia.org/wiki/Hypermarket https://en.m.wikipedia.org/wiki/Hypermarket ¯\_(ツ)_/¯
  - kevindamm 3y ago
    give it a couple more years, we'll have ultrascale
    - mnd999 3y ago
      And then the wiiscale?
- wmf 3y ago
  In general every order of magnitude brings new challenges. Companies running over a million servers have a lot of problems that smaller ones don't. Also they just have more room to amortize R&D.
- revskill 3y ago
  Universe scale i guess.
  - adaml_623 3y ago
hellozomo 3y ago
"Draining and un-draining hosts is hard." I'd stop right there and fix that, because that's a bullshit reason. Cycling hosts in and out of service is easy unless you're not doing things properly. The Linux kernel is simply not designed to be live patched and it's a total hack to try to do it, it will never work 100% of the time, always be a source of uncertainty, and always be expensive in terms of engineering work. Disaster will always be looming. By contrast, fixing their system for taking hosts in and out of service, so that it's extremely robust and reliable would likely pay big dividends in reliability. My guess would be that this approach is papering over organizational dysfunction. One team can patch all the kernels but one team can't make all the hosts support proper cycling in and out of service. And no one cares to fix it because there's no real incentive to do so. Only cool hacks and new projects are properly rewarded.
- deleted 3y ago
  [deleted]
- extr 3y ago
  > My guess would be that this approach is papering over organizational dysfunction. So what? At large enough scale organization problems are harder than technical ones: if you can fix the former with the latter, that's still a win.
  - hellozomo 3y ago
    But it's not "fixing" the problem, it's papering over the problem by piling tech debt on top of tech debt. Framing this like it's a good thing is my only objection.
- tysonfurytoo 3y ago
  It may be easy to cycle hosts in and out, but it can also be time consuming, apparently. In the article it mentions taking 45 days to patch all hosts. The article also points out that this is too long for security updates. Nothing will work 100% of the time. If their patching mechanism is thoroughly tested and battle hardened, I think the risk would be acceptable. Once you do the initial kpatch security upgrade, you could even schedule the machine for serivce so that it's not relying on that, limiting your exposure to bugs.
faitswulff 3y ago
> So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Most orgs don’t need and won’t benefit from emulating Meta for the sake of emulating Meta.
- baby_souffle 3y ago
  > Most orgs don’t need and won’t benefit from emulating Meta for the sake of emulating Meta. SO. Much. This. Worked at more than a few places where the stack had almost as many layers as it had engineers "because this is how we did it at FAANG..." Right, and those places also had a few orders of magnitude more engineers on staff to support it all. We do one hundredth of the things FAANG does and we have less than 50 _total_ people in the company; the simpler the stack, the better.
- influx 3y ago
  Likewise at Meta there's a bunch of cargo-culting from Google.
  - phyrex 3y ago
    Could you expand please?
- nickysielicki 3y ago
  This sort of criticism gets repeated all the time ("Google designs for Google scale, but you're not Google, so don't use kubernetes!"), and sometimes it's fair, but this doesn't really make sense to me on this particular article. If the infrastructure exists within your org's distribution of choice to do this, it's basically all upside. On AL2023, you just do: `sudo dnf install -y kpatch-dnf kpatch-runtime` `sudo dnf kernel-livepatch -y auto` `sudo systemctl enable --now kpatch.service` Super simple, one less thing to worry about.
  - ies7 3y ago
    > Super simple, one less thing to worry about. We got bitten by kpatch a few times before decided to abandon it around 2017. From kpatch github (2023): WARNING: Use with caution! Kernel crashes, spontaneous reboots, and data loss may occur!
TheAceOfHearts 3y ago
I wish they mentioned how long a full deployment takes Meta using this method, that seems like an important detail to omit. > So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by just fine with a full reboot of any single server. I can't really fathom the complexity of managing millions of servers though.
- ChatGTP 3y ago
  [flagged]
  - phyrex 3y ago
    It’s a very reductive take. Meta is helping a lot of people get and keep in touch for free (like my family is all over the place and we use WhatsApp extensively to communicate). There’s clearly a demand for this, and doing it for the price of a couple of ads doesn’t seem sad to me
    - ChatGTP 3y ago
      No, the users are the products. It’s not free.
      - phyrex 3y ago
        You’re not paying them out of your own pocket. I’ll happily use a better word for that if you have one.
        ChatGTP 3y ago
        It’s not free.
        white_dragon88 3y ago
iseletsk 3y ago
There is also KernelCare from tuxcare https://tuxcare.com/enterprise-live-patching-services/kernelcare-enterprise/ https://tuxcare.com/enterprise-live-patching-services/kernel... for kernel livepatching. It supports most linux distros.
cloudengineer94 3y ago
Been using Kpatch with tons of my VMs, works quite well
1letterunixname 3y ago
Roughly 13 million servers at present, with $20B spent on new datacenter gear in 2023 and another $20B in 2024. Running CentOS 8 Stream currently, going to 9 soonish.

135 points by elorant 3y ago

Ksplice is the original live patching technology that got bought by Oracle and was later extended to user space programs while I worked there. It's a really neat technology that isn't made obsolete by the move to cloud, since you still don't want to have to restart the whole fleet at scale.

metadat 3y ago
What's the live app patcher called?
- fragmede 3y ago
  I have objections to the name, but it goes under Ksplice for User Space. https://blogs.oracle.com/virtualization/post/ksplice-zero-downtime-patching-for-user-space-packages https://blogs.oracle.com/virtualization/post/ksplice-zero-do...
  - Sarkie 3y ago
    How was Oracle to work at?
    - grepfru_it 3y ago
      Not OP but I loved oracle in the 90s/early 2000s. After the Sun acquisition it became cutthroat and I left. IBM acquiring redhat seems very similar
    - fragmede 3y ago
      Post acquisition, we moved offices, and used their HR department, and certain product decisions were made for us, but we largely did our own thing, for better or worse, so I don't know how much I can speak about how it was to work at Oracle. We were under Wim Coekaerts who is a big open source guy. Oracle's reputation as a cutthroat legal entity is well deserved, but working at that side of the company it felt unfair because, Oracle's open source contributions with VirtualBox, the UEK, and others are lost in the grar over MySQL. I think the biggest "big company" blunder while I was there was a public blog post by someone high up at the company decrying open source as bad/wrong while at the same time, Oracle was doing all this other open source stuff.
      - latchkey 3y ago
        Funny given that they have been selling open source software for ages... https://docs.oracle.com/cd/B10463_01/web.904/b10320/apjsvsup.htm https://docs.oracle.com/cd/B10463_01/web.904/b10320/apjsvsup... I believe this was circa 2000, but going on longer than that.
      - yjftsjthsd-h 3y ago
        > but working at that side of the company it felt unfair because, Oracle's open source contributions with VirtualBox, the UEK, and others are lost in the grar over MySQL. They are also remembered for closing OpenSolaris and shaking people down over the VirtualBox extension pack ( https://www.theregister.com/2019/10/04/oracle_virtualbox_merula/ https://www.theregister.com/2019/10/04/oracle_virtualbox_mer... ).
        andyjohnson0 3y ago
        > grar Useful word.

civilitty 3y ago

[flagged]

ravenstine 3y ago

Never heard of this "hyperscale" concept before. How is this any different from... scaling?

mc32 3y ago
Supermarkets and super cars aren't enough. We need hypermarkets and hyper cars to be current. So now we can’t just scale, I guess that’s for trucks, so to be current you need hyperscale.
- froh 3y ago
  Cybertruck and cyberscale? (I'm joking)
- phyrex 3y ago
  https://en.m.wikipedia.org/wiki/Hypermarket https://en.m.wikipedia.org/wiki/Hypermarket ¯\_(ツ)_/¯
- kevindamm 3y ago
  give it a couple more years, we'll have ultrascale
  - mnd999 3y ago
    And then the wiiscale?
wmf 3y ago
In general every order of magnitude brings new challenges. Companies running over a million servers have a lot of problems that smaller ones don't. Also they just have more room to amortize R&D.
revskill 3y ago
Universe scale i guess.
- adaml_623 3y ago

hellozomo 3y ago

"Draining and un-draining hosts is hard." I'd stop right there and fix that, because that's a bullshit reason. Cycling hosts in and out of service is easy unless you're not doing things properly. The Linux kernel is simply not designed to be live patched and it's a total hack to try to do it, it will never work 100% of the time, always be a source of uncertainty, and always be expensive in terms of engineering work. Disaster will always be looming. By contrast, fixing their system for taking hosts in and out of service, so that it's extremely robust and reliable would likely pay big dividends in reliability. My guess would be that this approach is papering over organizational dysfunction. One team can patch all the kernels but one team can't make all the hosts support proper cycling in and out of service. And no one cares to fix it because there's no real incentive to do so. Only cool hacks and new projects are properly rewarded.

deleted 3y ago
[deleted]
extr 3y ago
> My guess would be that this approach is papering over organizational dysfunction. So what? At large enough scale organization problems are harder than technical ones: if you can fix the former with the latter, that's still a win.
- hellozomo 3y ago
  But it's not "fixing" the problem, it's papering over the problem by piling tech debt on top of tech debt. Framing this like it's a good thing is my only objection.
tysonfurytoo 3y ago
It may be easy to cycle hosts in and out, but it can also be time consuming, apparently. In the article it mentions taking 45 days to patch all hosts. The article also points out that this is too long for security updates. Nothing will work 100% of the time. If their patching mechanism is thoroughly tested and battle hardened, I think the risk would be acceptable. Once you do the initial kpatch security upgrade, you could even schedule the machine for serivce so that it's not relying on that, limiting your exposure to bugs.

faitswulff 3y ago

> So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Most orgs don’t need and won’t benefit from emulating Meta for the sake of emulating Meta.

baby_souffle 3y ago
> Most orgs don’t need and won’t benefit from emulating Meta for the sake of emulating Meta. SO. Much. This. Worked at more than a few places where the stack had almost as many layers as it had engineers "because this is how we did it at FAANG..." Right, and those places also had a few orders of magnitude more engineers on staff to support it all. We do one hundredth of the things FAANG does and we have less than 50 _total_ people in the company; the simpler the stack, the better.
influx 3y ago
Likewise at Meta there's a bunch of cargo-culting from Google.
- phyrex 3y ago
  Could you expand please?
nickysielicki 3y ago
This sort of criticism gets repeated all the time ("Google designs for Google scale, but you're not Google, so don't use kubernetes!"), and sometimes it's fair, but this doesn't really make sense to me on this particular article. If the infrastructure exists within your org's distribution of choice to do this, it's basically all upside. On AL2023, you just do: `sudo dnf install -y kpatch-dnf kpatch-runtime` `sudo dnf kernel-livepatch -y auto` `sudo systemctl enable --now kpatch.service` Super simple, one less thing to worry about.
- ies7 3y ago
  > Super simple, one less thing to worry about. We got bitten by kpatch a few times before decided to abandon it around 2017. From kpatch github (2023): WARNING: Use with caution! Kernel crashes, spontaneous reboots, and data loss may occur!

TheAceOfHearts 3y ago

I wish they mentioned how long a full deployment takes Meta using this method, that seems like an important detail to omit. > So, if you’d rather not have downtime with your servers, data centers, and clouds, follow Meta’s example and use live patching. You’ll be glad you did. Maybe if you're working at Meta's scale it makes sense... But I think most well designed services and applications should be able to get by just fine with a full reboot of any single server. I can't really fathom the complexity of managing millions of servers though.

ChatGTP 3y ago
[flagged]
- phyrex 3y ago
  It’s a very reductive take. Meta is helping a lot of people get and keep in touch for free (like my family is all over the place and we use WhatsApp extensively to communicate). There’s clearly a demand for this, and doing it for the price of a couple of ads doesn’t seem sad to me
  - ChatGTP 3y ago
    No, the users are the products. It’s not free.
    - phyrex 3y ago
      You’re not paying them out of your own pocket. I’ll happily use a better word for that if you have one.
      - ChatGTP 3y ago
        It’s not free.
        white_dragon88 3y ago

iseletsk 3y ago

There is also KernelCare from tuxcare https://tuxcare.com/enterprise-live-patching-services/kernelcare-enterprise/ https://tuxcare.com/enterprise-live-patching-services/kernel... for kernel livepatching. It supports most linux distros.

cloudengineer94 3y ago

Been using Kpatch with tons of my VMs, works quite well

1letterunixname 3y ago

Roughly 13 million servers at present, with $20B spent on new datacenter gear in 2023 and another $20B in 2024. Running CentOS 8 Stream currently, going to 9 soonish.