43 ms·

Ksplice: Automatic Rebootless Kernel Updates (2008) [pdf]

64 points by chicken_lady 12y ago

voidlogic 12y ago
The tragedy (from wikipedia): On 21 July 2011, Oracle announced they acquired Ksplice, Inc. At the time the company was acquired, Ksplice, Inc. claimed to have over 700 companies using the service to protect over 100,000 servers. While the service had been available for multiple Linux distributions, it was stated at the time of acquisition that "Oracle believes it will be the only enterprise Linux provider that can offer zero downtime updates." More explicitly, "Oracle does not plan to support the use of Ksplice technology with Red Hat Enterprise Linux."[5] Existing legacy customers continue to be supported by Ksplice, but no new customers are being accepted for other platforms.[11] https://en.wikipedia.org/wiki/Ksplice https://en.wikipedia.org/wiki/Ksplice
- tedks 12y ago
  The Uptrack service was obviously proprietary, but the Ksplice software itself (everything to actually make and install the patches) was open source. There are plenty of mirrors on github: https://github.com/linux-kernel-live-patching/ksplice https://github.com/linux-kernel-live-patching/ksplice But there's also kgraft and kpatch.
- amalag 12y ago
  I thought they merged this in with their Oracle Linux http://www.oracle.com/us/technologies/linux/ksplice-datasheet-487388.pdf http://www.oracle.com/us/technologies/linux/ksplice-datashee... http://www.ksplice.com/uptrack/using http://www.ksplice.com/uptrack/using
- tshtf 12y ago
  Not exactly the case: Ubuntu, Fedora and Debian are supported for free, and RHEL for a fee: Ksplice Uptrack is available for Oracle Linux, free of charge, for Oracle Linux customers with a Premier support subscription. Additionally, anyone can use Ksplice Uptrack for free on Ubuntu Desktop and Fedora. http://www.ksplice.com/pricing http://www.ksplice.com/pricing
  - vertex-four 12y ago
    Thing is, while Oracle Linux is pretty much rebranded RHEL last time I checked, the KSplice commercial license won't let you use it with RHEL, and that won't be supported.
    - fragmede 12y ago
      It is, you just have to buy an Oracle Premier Support License for your machine(s) and switch them from Red Hat to Oracle Linux (which requires switching yum repos, removing some packages, and installing a few others), but does not require a reboot - which means you can keep using the official Red Hat kernel (until you do reboot). Disclaimer: I work there.
      - ryan-c 12y ago
        You forgot the part where you give Oracle your balls on a silver platter so they can hold them for you. We've all seen what Oracle likes to do with nice things.
      - deleted 12y ago
        [deleted]
    - dmourati 12y ago
      Fairly sure Oracle Linux was at least in its initial incarnation a rebranded CentOS, which itself is of course a rebranded RHEL.
  - e40 12y ago
    That "pricing" page is strangely devoid of pricing information. EDIT: so, how much does it cost?
    - SoftwareMaven 12y ago
      For Ubuntu, Fedora and Debian, nothing. For Oracle Linux, you need an Oracle support contract. Those don't come with a standard price tag. Unfortunately, the website hasn't been fully updated since the acquisition, so there is some confusing verbiage. It's on my list (near the top, no less; sorry about the confusion).
- fragmede 12y ago
  I won't mince words, the cost of Ksplice these days is much higher than pre-acquisition, however you can get Ksplice for Enterprise Linux 5, 6 and 7, with the Redhat kernel without rebooting, as well as the Red Hat Compatible Kernel that Oracle recompiles, as well as Unbreakable Enterprise Kernel (UEK) r1, r2 and r3. https://www.ksplice.com/uptrack/supported-kernels https://www.ksplice.com/uptrack/supported-kernels
- lambda 12y ago
  Note that since Oracle basically closed ksplice, RedHat has developed kpatch and SuSE has developed kgraft, which are both similar ways of doing the same kind of thing, and there is some discussion going on about whether they can be merged or a better general purpose kernel patching method developed: http://wiki.linuxplumbersconf.org/2014:live_kernel_patching http://wiki.linuxplumbersconf.org/2014:live_kernel_patching
- rasz_pl 12y ago
  dont worry, no reboot updates are thing of the past thanks to systemd
  - xorcist 12y ago
    Care to elaborate?
- fafner 12y ago
  Red Hat is working on ksplice and SuSE on kGraft. And unlike Ksplice they try to merge it into the mainline kernel.
TallGuyShort 12y ago
Anyone interested in this should also look up kGraft and kpatch, developed by SUSE and RedHat, respectively. Each has been submitted for inclusion in the mainline kernel.
- fragmede 12y ago
  Also kernelcare by the cloudlinux folk. Although that has a closed-source blob that it uses, it's available in production now, as opposed to kGraph and kpatch.
tzs 12y ago
There are at least two reasons to restart something after you update it. (1) the update could not be applied to the running instance, so you have to restart to put the update into service, and (2) to demonstrate that the update did not break the ability of the thing to start. This address #1. It does not address #2. It is extremely annoying if something has run for a year with a couple dozen live updates applied without restarting and then you have to restart for some reason (power failure, perhaps), and then you discover that somewhere in the last year it lost the ability to start. I'd much rather discover that an update has introduced a problem with startup immediately after applying the update. Restarting things should be a regular part of maintenance.
- tedks 12y ago
  Even with a dynamic software updating system in place, you shouldn't be deploying dynamic updates directly to production without testing them. It'd be pretty stupid to deploy an update by any mechanism that would render a service unstartable.
- SoftwareMaven 12y ago
  When you restart, you will restart into whichever kernel you have installed, which may be whatever you booted into previously or a new distro kernel with the updates compiled in. There are no Ksplice updates loaded until you tell them to load and Ksplice does not make any modifications to the actual Kernel binary, so #2 isn't really an issue. The real problem is there may be limitations on when you can reboot, leaving you in a vulnerable state. Sure, an ideal environment would let you reboot any system at any time, but those places are like unicorn ranches. I completely agree with your last comment, though: systems suffer from weird bit-rot diseases. If you never reboot, you may find things challenging when you finally have to. Disclosure: I work on the Ksplice team.
  - jdhendrickson 12y ago
    How do you feel about Oracles handling of the code?
    - 12y ago
sciurus 12y ago
There are a lot of competing implementations of live kernel patching, but work has started to agree on a common approach. As explained at http://www.linuxplumbersconf.org/2014/an-in-depth-look-live-kernel-patching-microconference/ http://www.linuxplumbersconf.org/2014/an-in-depth-look-live-... "There has been a great deal of interest in live kernel patching (see http://lwn.net/Articles/597407/ http://lwn.net/Articles/597407/) over the past few months, with several different approaches proposed, including CRIU+kexec, kGraft, and kpatch, all in addition to ksplice. This microconference will host discussions on required infrastructure (including tracing, checkpoint/restart, kexec, and live patching), along with expositions and comparisons of the various approaches. The purpose, believe it or not, is to work towards a common implementation that everyone can live with. It should be a spirited discussion! For more details, please see http://wiki.linuxplumbersconf.org/2014:live_kernel_patching" http://wiki.linuxplumbersconf.org/2014:live_kernel_patching"
codehero 12y ago
Reminds me of Multics; it had online updates.
ahupp 12y ago
This seems like amazing technology, but I'm having a hard time seeing the use case for it. If you are sensitive to a single machine going down for a reboot, the occasional kernel upgrade is the least of your concerns. Those are at least predictable and rare. I'm sure there are environments where (say) the time to failover to a replica is business-critical, but what are they?
- ryan-c 12y ago
  Full disk encryption or other situations where manual intervention is a hard requirement to get the system fully back online is one situation. Any instance where you're running untrusted binaries on your server you'd want to update ASAP and not worry about getting pwned while waiting for scheduled downtime. Before Oracle closed it to new customers, Ksplice was fairly cheap, ~$3-4/mo/server.
- Intermernet 12y ago
  Most SME's usually don't have redundant everything (unlike large enterprises, and IT specific companies) in their environments. They will usually do a cost / benefit analysis and choose to skimp on certain redundancies (usually somewhat "stateless" stuff like routers, VPN endpoints and other network gear). These companies are always hindered by the downtime incurred when one of these non-redundant devices needs restarting. In a couple of previous positions over the years I would have loved the ability to apply patches with no downtime. There are environments where the time to failover to redundant infrastructure is critical (HVT, particle physics etc.) but the more common use case is probably going to be the companies that have no failover capability for certain parts of their infrastructure.
Nux 12y ago
Ksplice is now owned by Oracle, meaning it's dead and buried (unless you give Oracle your moneyz). Look at KPatch (included in RHEL7 as tech preview, will probably mature in subsequent minor version updates): http://rhelblog.redhat.com/2014/02/26/kpatch/ http://rhelblog.redhat.com/2014/02/26/kpatch/
nextweek2 12y ago
I'd love to see online patching on the free tier of Linux distros (Fedora, ubuntu, etc) as its the ideal test location. To me this is the first step, the second is doing the same with system libraries. If zlib gets updated what do I have to restart to ensure running applications are using the right version? Right now the answer is a reboot.

64 points by chicken_lady 12y ago

The tragedy (from wikipedia): On 21 July 2011, Oracle announced they acquired Ksplice, Inc. At the time the company was acquired, Ksplice, Inc. claimed to have over 700 companies using the service to protect over 100,000 servers. While the service had been available for multiple Linux distributions, it was stated at the time of acquisition that "Oracle believes it will be the only enterprise Linux provider that can offer zero downtime updates." More explicitly, "Oracle does not plan to support the use of Ksplice technology with Red Hat Enterprise Linux."[5] Existing legacy customers continue to be supported by Ksplice, but no new customers are being accepted for other platforms.[11] https://en.wikipedia.org/wiki/Ksplice https://en.wikipedia.org/wiki/Ksplice

tedks 12y ago
The Uptrack service was obviously proprietary, but the Ksplice software itself (everything to actually make and install the patches) was open source. There are plenty of mirrors on github: https://github.com/linux-kernel-live-patching/ksplice https://github.com/linux-kernel-live-patching/ksplice But there's also kgraft and kpatch.
amalag 12y ago
I thought they merged this in with their Oracle Linux http://www.oracle.com/us/technologies/linux/ksplice-datasheet-487388.pdf http://www.oracle.com/us/technologies/linux/ksplice-datashee... http://www.ksplice.com/uptrack/using http://www.ksplice.com/uptrack/using
tshtf 12y ago
Not exactly the case: Ubuntu, Fedora and Debian are supported for free, and RHEL for a fee: Ksplice Uptrack is available for Oracle Linux, free of charge, for Oracle Linux customers with a Premier support subscription. Additionally, anyone can use Ksplice Uptrack for free on Ubuntu Desktop and Fedora. http://www.ksplice.com/pricing http://www.ksplice.com/pricing
- vertex-four 12y ago
  Thing is, while Oracle Linux is pretty much rebranded RHEL last time I checked, the KSplice commercial license won't let you use it with RHEL, and that won't be supported.
  - fragmede 12y ago
    It is, you just have to buy an Oracle Premier Support License for your machine(s) and switch them from Red Hat to Oracle Linux (which requires switching yum repos, removing some packages, and installing a few others), but does not require a reboot - which means you can keep using the official Red Hat kernel (until you do reboot). Disclaimer: I work there.
    - ryan-c 12y ago
      You forgot the part where you give Oracle your balls on a silver platter so they can hold them for you. We've all seen what Oracle likes to do with nice things.
    - deleted 12y ago
      [deleted]
  - dmourati 12y ago
    Fairly sure Oracle Linux was at least in its initial incarnation a rebranded CentOS, which itself is of course a rebranded RHEL.
- e40 12y ago
  That "pricing" page is strangely devoid of pricing information. EDIT: so, how much does it cost?
  - SoftwareMaven 12y ago
    For Ubuntu, Fedora and Debian, nothing. For Oracle Linux, you need an Oracle support contract. Those don't come with a standard price tag. Unfortunately, the website hasn't been fully updated since the acquisition, so there is some confusing verbiage. It's on my list (near the top, no less; sorry about the confusion).
fragmede 12y ago
I won't mince words, the cost of Ksplice these days is much higher than pre-acquisition, however you can get Ksplice for Enterprise Linux 5, 6 and 7, with the Redhat kernel without rebooting, as well as the Red Hat Compatible Kernel that Oracle recompiles, as well as Unbreakable Enterprise Kernel (UEK) r1, r2 and r3. https://www.ksplice.com/uptrack/supported-kernels https://www.ksplice.com/uptrack/supported-kernels
lambda 12y ago
Note that since Oracle basically closed ksplice, RedHat has developed kpatch and SuSE has developed kgraft, which are both similar ways of doing the same kind of thing, and there is some discussion going on about whether they can be merged or a better general purpose kernel patching method developed: http://wiki.linuxplumbersconf.org/2014:live_kernel_patching http://wiki.linuxplumbersconf.org/2014:live_kernel_patching
rasz_pl 12y ago
dont worry, no reboot updates are thing of the past thanks to systemd
- xorcist 12y ago
  Care to elaborate?
fafner 12y ago
Red Hat is working on ksplice and SuSE on kGraft. And unlike Ksplice they try to merge it into the mainline kernel.

TallGuyShort 12y ago

Anyone interested in this should also look up kGraft and kpatch, developed by SUSE and RedHat, respectively. Each has been submitted for inclusion in the mainline kernel.

fragmede 12y ago
Also kernelcare by the cloudlinux folk. Although that has a closed-source blob that it uses, it's available in production now, as opposed to kGraph and kpatch.

tzs 12y ago

There are at least two reasons to restart something after you update it. (1) the update could not be applied to the running instance, so you have to restart to put the update into service, and (2) to demonstrate that the update did not break the ability of the thing to start. This address #1. It does not address #2. It is extremely annoying if something has run for a year with a couple dozen live updates applied without restarting and then you have to restart for some reason (power failure, perhaps), and then you discover that somewhere in the last year it lost the ability to start. I'd much rather discover that an update has introduced a problem with startup immediately after applying the update. Restarting things should be a regular part of maintenance.

tedks 12y ago
Even with a dynamic software updating system in place, you shouldn't be deploying dynamic updates directly to production without testing them. It'd be pretty stupid to deploy an update by any mechanism that would render a service unstartable.
SoftwareMaven 12y ago
When you restart, you will restart into whichever kernel you have installed, which may be whatever you booted into previously or a new distro kernel with the updates compiled in. There are no Ksplice updates loaded until you tell them to load and Ksplice does not make any modifications to the actual Kernel binary, so #2 isn't really an issue. The real problem is there may be limitations on when you can reboot, leaving you in a vulnerable state. Sure, an ideal environment would let you reboot any system at any time, but those places are like unicorn ranches. I completely agree with your last comment, though: systems suffer from weird bit-rot diseases. If you never reboot, you may find things challenging when you finally have to. Disclosure: I work on the Ksplice team.
- jdhendrickson 12y ago
  How do you feel about Oracles handling of the code?
  - 12y ago

sciurus 12y ago

There are a lot of competing implementations of live kernel patching, but work has started to agree on a common approach. As explained at http://www.linuxplumbersconf.org/2014/an-in-depth-look-live-kernel-patching-microconference/ http://www.linuxplumbersconf.org/2014/an-in-depth-look-live-... "There has been a great deal of interest in live kernel patching (see http://lwn.net/Articles/597407/ http://lwn.net/Articles/597407/) over the past few months, with several different approaches proposed, including CRIU+kexec, kGraft, and kpatch, all in addition to ksplice. This microconference will host discussions on required infrastructure (including tracing, checkpoint/restart, kexec, and live patching), along with expositions and comparisons of the various approaches. The purpose, believe it or not, is to work towards a common implementation that everyone can live with. It should be a spirited discussion! For more details, please see http://wiki.linuxplumbersconf.org/2014:live_kernel_patching" http://wiki.linuxplumbersconf.org/2014:live_kernel_patching"

codehero 12y ago

Reminds me of Multics; it had online updates.

ahupp 12y ago

This seems like amazing technology, but I'm having a hard time seeing the use case for it. If you are sensitive to a single machine going down for a reboot, the occasional kernel upgrade is the least of your concerns. Those are at least predictable and rare. I'm sure there are environments where (say) the time to failover to a replica is business-critical, but what are they?

ryan-c 12y ago
Full disk encryption or other situations where manual intervention is a hard requirement to get the system fully back online is one situation. Any instance where you're running untrusted binaries on your server you'd want to update ASAP and not worry about getting pwned while waiting for scheduled downtime. Before Oracle closed it to new customers, Ksplice was fairly cheap, ~$3-4/mo/server.
Intermernet 12y ago
Most SME's usually don't have redundant everything (unlike large enterprises, and IT specific companies) in their environments. They will usually do a cost / benefit analysis and choose to skimp on certain redundancies (usually somewhat "stateless" stuff like routers, VPN endpoints and other network gear). These companies are always hindered by the downtime incurred when one of these non-redundant devices needs restarting. In a couple of previous positions over the years I would have loved the ability to apply patches with no downtime. There are environments where the time to failover to redundant infrastructure is critical (HVT, particle physics etc.) but the more common use case is probably going to be the companies that have no failover capability for certain parts of their infrastructure.

Nux 12y ago

Ksplice is now owned by Oracle, meaning it's dead and buried (unless you give Oracle your moneyz). Look at KPatch (included in RHEL7 as tech preview, will probably mature in subsequent minor version updates): http://rhelblog.redhat.com/2014/02/26/kpatch/ http://rhelblog.redhat.com/2014/02/26/kpatch/

nextweek2 12y ago

I'd love to see online patching on the free tier of Linux distros (Fedora, ubuntu, etc) as its the ideal test location. To me this is the first step, the second is doing the same with system libraries. If zlib gets updated what do I have to restart to ensure running applications are using the right version? Right now the answer is a reboot.