4 ms·

KernelCare – Kernel updates without rebooting the system

45 points by yiedyie 12y ago

rwmj 12y ago
How does this relate to ksplice/kpatch/kgraft?
- dsr_ 12y ago
  It's in the FAQ. >>Q: Are you using same technology as the one from Oracle, Red Hat or Suse? >>No, Our technology was fully developed in house, and uses different methods to generate the patche as well as to apply the patch. We believe that our method of generating patchines is significantly more efficent that what we have seen or read up to date from other vendors. But here's the concerning thing: >>Q: Do you apply all the patches from the newest kernels? >>We apply only security patches. Sometimes we might decide to apply patches for critical bugs. That sounds like the kernel you run will slowly diverge from upstream. Different security teams make different decisions. I would value this service more if it came with the blessing and cooperation of my distro's security and kernel teams.
  - yiedyie 12y ago
    Unfortunately critical bugs can also be major security issues in many ways.
    - michaelmior 12y ago
      Agreed. Although the policy implies that if that is the case, then a patch would be made available for those bugs.
Already__Taken 12y ago
Not that this isn't a cool product, isn't it important to know your server can reboot?
- Gonzih 12y ago
  Of course it can. It's just not necessary to reboot with solution like that.
  - bananas 12y ago
    Until you reboot and it doesn't :)
    - ithkuil 12y ago
      And if you do reboot frequently but then at one point it doesn't work ? You either have a long downtime or have a secondary ready to take over. I understand you're talking about avoiding to rely on a false sense of confidence; that's a good rule of thumb, but shouldn't get in the way of better solutions: Test the secondary more often, and automate the recovery, and you'll cover more failure cases. You don't have the resources for a secondary ? This means you're to small; neither this tool is for you; keep it simple, and just reboot. Moreover, in some context is not desirable to perform a maintenance window until some designated moments like weekends. This means that you could leave your system vulnerable for up to a whole week.
      - jdsnape 12y ago
        I agree with what you're saying about having better recovery, but I think you need some balance between the two. I'd rather look through a week's worth of changes to work out what borked the restart than two years worth!
      - zyx321 12y ago
        >And if you do reboot frequently but then at one point it doesn't work ? Then you restore from yesterday's backup. It's probably still on-site. If a kernel live patch fails, you might not notice for a long time.
        ithkuil 12y ago
        My point is you should always be prepared that a reboot doesn't work, since it could happen at any time between your maintenance windows. Frequently exercising a feature is a good idea, especially when you depend on that feature. That's why exercising backup restores is important; you want that to work when you need it. Do you really need a production machine to boot when you need it? I think is a wrong target to optimize for, given that that failure more is already covered by backups cold/hot secondary as long as you do exercise them frequently. That said, you do have a point about whether a live patch is reliable, but this doesn't have anything to do with whether increasing the average uptime of a host is a good or bad idea w.r.t to ensuring that the machine can start or not. TBH I wouldn't personally feel much comfortable using this live update thing, but I have no experience with it. I'd be curious to know more though.
Fizzadar 12y ago
The site is pretty light on details; I'd want to know more how it works before moving from kSplice...
fleitz 12y ago
Shouldn't one design their systems so that a single machine rebooting is not fatal? Generally I make my servers reboot at least once every 24 hours.
- chris_wot 12y ago
  Depends on the environment. If you are a big enough business, then yes this would be a good idea. Otherwise, if you are small and don't have much money for infrastructure then perhaps not. Though I have to ask: why do you reboot your servers every 24 hours?
  - valarauca1 12y ago
    I was gonna say 24 hours is short. I think a good time is about ~1 month. I know HP-UX suggests 2 weeks, and that OS has a history of 10-15 years up time, stupid y2K bug killing server up times.
- laumars 12y ago
  > "Shouldn't one design their systems so that a single machine rebooting is not fatal?" In an ideal world yes. Not everybody has that opportunity though. > "Generally I make my servers reboot at least once every 24 hours." That can't be true. There's Belkin routers out there with greater uptimes. Christ, I think even my old Windows ME PC had longer uptimes than that! No sane sysadmin would reboot a server multiple times a day. If nothing else, it's just a needless waste of electricity and any other resources it might drain during the power cycle (eg network IO if a SAN hosted VM). I'm amazed at the just how much some people will exaggerate figures to make a point....
  - fleitz 12y ago
    What if reproducibility is more important than uptime? How does memory fragmentation affect performance after more than 24 hours? What happens when your server has to restart from a cold cache scenario? What happens when your server is down? What about that setting that a sys admin manually applied to the server 6 months ago to fix some issue and isn't in saved in the server config? By forcing a condition people try to avoid you get good at dealing with those situations. How often do your database servers actually fail over? How confident are you in your code and systems to actually fail over properly? How do you prime the caches when your memcached servers reboot?
akx 12y ago
The drawings are pretty shoddy and amateurish. Doesn't inspire too much confidence in me...
- LeonM 12y ago
  It does to me, because real programmers cannot draw ;) But jokes aside, they just didn't spend their money on an artist, that does not imply that the product is of the same quality as the drawings.
- huhtenberg 12y ago
  I don't know why you are in gray, but this is a valid argument. If I am about to entrust some company with an access to the kernel on my live boxes, you bet I'd be looking very closely at who the hell they are and be expecting them to be a reputable and stable entity. These silly illustrations would've been OK if they were on RedHat's site, but if you are just starting up, I would strongly suggest to put (much) more effort into establishing your credibility.
  - teacup50 12y ago
    I hope you can see the irony of your position when you're talking about the Linux kernel.
    - huhtenberg 12y ago
      If you are referring to the Tux, then you are missing the point. Submitted project has very ambitious goals but comes from an unknown source and no history. It also makes a deliberate effort to look silly, which is the exact opposite of what they should be doing if they are aiming at the actual production use in live environments.
pxsta 12y ago
We can download the kernel module. It seems so simple. http://patches.kernelcare.com/kmod_kcare.tar.gz http://patches.kernelcare.com/kmod_kcare.tar.gz
- yiedyie 12y ago
  Yes but we still need to make sure that on the next restart the kernel is patched.
deleted 12y ago
[deleted]
orf 12y ago
looks good, our servers stay up for years at a time with KSplice, but now oracle owns it and it's not available to new customers means I think we might have to find something new soon.
sciurus 12y ago
LWN has recently covered different methods of doing this that SuSE and Red Hat have developed. The initial kGraft submission: https://lwn.net/Articles/596854/ https://lwn.net/Articles/596854/ The first kpatch submission: http://lwn.net/SubscriberLink/597407/cd7861dacf2be0e9/ http://lwn.net/SubscriberLink/597407/cd7861dacf2be0e9/ Bask in 2008 they covered how ksplice works: https://lwn.net/Articles/280058/ https://lwn.net/Articles/280058/

45 points by yiedyie 12y ago

rwmj 12y ago

How does this relate to ksplice/kpatch/kgraft?

dsr_ 12y ago
It's in the FAQ. >>Q: Are you using same technology as the one from Oracle, Red Hat or Suse? >>No, Our technology was fully developed in house, and uses different methods to generate the patche as well as to apply the patch. We believe that our method of generating patchines is significantly more efficent that what we have seen or read up to date from other vendors. But here's the concerning thing: >>Q: Do you apply all the patches from the newest kernels? >>We apply only security patches. Sometimes we might decide to apply patches for critical bugs. That sounds like the kernel you run will slowly diverge from upstream. Different security teams make different decisions. I would value this service more if it came with the blessing and cooperation of my distro's security and kernel teams.
- yiedyie 12y ago
  Unfortunately critical bugs can also be major security issues in many ways.
  - michaelmior 12y ago
    Agreed. Although the policy implies that if that is the case, then a patch would be made available for those bugs.

Already__Taken 12y ago

Not that this isn't a cool product, isn't it important to know your server can reboot?

Gonzih 12y ago
Of course it can. It's just not necessary to reboot with solution like that.
- bananas 12y ago
  Until you reboot and it doesn't :)
  - ithkuil 12y ago
    And if you do reboot frequently but then at one point it doesn't work ? You either have a long downtime or have a secondary ready to take over. I understand you're talking about avoiding to rely on a false sense of confidence; that's a good rule of thumb, but shouldn't get in the way of better solutions: Test the secondary more often, and automate the recovery, and you'll cover more failure cases. You don't have the resources for a secondary ? This means you're to small; neither this tool is for you; keep it simple, and just reboot. Moreover, in some context is not desirable to perform a maintenance window until some designated moments like weekends. This means that you could leave your system vulnerable for up to a whole week.
    - jdsnape 12y ago
      I agree with what you're saying about having better recovery, but I think you need some balance between the two. I'd rather look through a week's worth of changes to work out what borked the restart than two years worth!
    - zyx321 12y ago
      >And if you do reboot frequently but then at one point it doesn't work ? Then you restore from yesterday's backup. It's probably still on-site. If a kernel live patch fails, you might not notice for a long time.
      - ithkuil 12y ago
        My point is you should always be prepared that a reboot doesn't work, since it could happen at any time between your maintenance windows. Frequently exercising a feature is a good idea, especially when you depend on that feature. That's why exercising backup restores is important; you want that to work when you need it. Do you really need a production machine to boot when you need it? I think is a wrong target to optimize for, given that that failure more is already covered by backups cold/hot secondary as long as you do exercise them frequently. That said, you do have a point about whether a live patch is reliable, but this doesn't have anything to do with whether increasing the average uptime of a host is a good or bad idea w.r.t to ensuring that the machine can start or not. TBH I wouldn't personally feel much comfortable using this live update thing, but I have no experience with it. I'd be curious to know more though.

Fizzadar 12y ago

The site is pretty light on details; I'd want to know more how it works before moving from kSplice...

fleitz 12y ago

Shouldn't one design their systems so that a single machine rebooting is not fatal? Generally I make my servers reboot at least once every 24 hours.

chris_wot 12y ago
Depends on the environment. If you are a big enough business, then yes this would be a good idea. Otherwise, if you are small and don't have much money for infrastructure then perhaps not. Though I have to ask: why do you reboot your servers every 24 hours?
- valarauca1 12y ago
  I was gonna say 24 hours is short. I think a good time is about ~1 month. I know HP-UX suggests 2 weeks, and that OS has a history of 10-15 years up time, stupid y2K bug killing server up times.
laumars 12y ago
> "Shouldn't one design their systems so that a single machine rebooting is not fatal?" In an ideal world yes. Not everybody has that opportunity though. > "Generally I make my servers reboot at least once every 24 hours." That can't be true. There's Belkin routers out there with greater uptimes. Christ, I think even my old Windows ME PC had longer uptimes than that! No sane sysadmin would reboot a server multiple times a day. If nothing else, it's just a needless waste of electricity and any other resources it might drain during the power cycle (eg network IO if a SAN hosted VM). I'm amazed at the just how much some people will exaggerate figures to make a point....
- fleitz 12y ago
  What if reproducibility is more important than uptime? How does memory fragmentation affect performance after more than 24 hours? What happens when your server has to restart from a cold cache scenario? What happens when your server is down? What about that setting that a sys admin manually applied to the server 6 months ago to fix some issue and isn't in saved in the server config? By forcing a condition people try to avoid you get good at dealing with those situations. How often do your database servers actually fail over? How confident are you in your code and systems to actually fail over properly? How do you prime the caches when your memcached servers reboot?

akx 12y ago

The drawings are pretty shoddy and amateurish. Doesn't inspire too much confidence in me...

LeonM 12y ago
It does to me, because real programmers cannot draw ;) But jokes aside, they just didn't spend their money on an artist, that does not imply that the product is of the same quality as the drawings.
huhtenberg 12y ago
I don't know why you are in gray, but this is a valid argument. If I am about to entrust some company with an access to the kernel on my live boxes, you bet I'd be looking very closely at who the hell they are and be expecting them to be a reputable and stable entity. These silly illustrations would've been OK if they were on RedHat's site, but if you are just starting up, I would strongly suggest to put (much) more effort into establishing your credibility.
- teacup50 12y ago
  I hope you can see the irony of your position when you're talking about the Linux kernel.
  - huhtenberg 12y ago
    If you are referring to the Tux, then you are missing the point. Submitted project has very ambitious goals but comes from an unknown source and no history. It also makes a deliberate effort to look silly, which is the exact opposite of what they should be doing if they are aiming at the actual production use in live environments.

pxsta 12y ago

We can download the kernel module. It seems so simple. http://patches.kernelcare.com/kmod_kcare.tar.gz http://patches.kernelcare.com/kmod_kcare.tar.gz

yiedyie 12y ago
Yes but we still need to make sure that on the next restart the kernel is patched.

deleted 12y ago

[deleted]

orf 12y ago

looks good, our servers stay up for years at a time with KSplice, but now oracle owns it and it's not available to new customers means I think we might have to find something new soon.

sciurus 12y ago

LWN has recently covered different methods of doing this that SuSE and Red Hat have developed. The initial kGraft submission: https://lwn.net/Articles/596854/ https://lwn.net/Articles/596854/ The first kpatch submission: http://lwn.net/SubscriberLink/597407/cd7861dacf2be0e9/ http://lwn.net/SubscriberLink/597407/cd7861dacf2be0e9/ Bask in 2008 they covered how ksplice works: https://lwn.net/Articles/280058/ https://lwn.net/Articles/280058/