9 ms·

This article[0] on LWN seems to suggest that Linux has no kernel support for checkpoint/restore because it's hard to implement (no arguments there). But hypervi

This article[0] on LWN seems to suggest that Linux has no kernel support for checkpoint/restore because it's hard to implement (no arguments there). But hypervisors support checkpoint/restore for virtual machines, e.g. ESXi VMotion and KVM live migration, so it seems like these technical problems are solvable. Indeed all the benefits of VM migration seem to also apply to process migration (load balancing, service uptime, etc). 0. https://lwn.net/Articles/293575/ https://lwn.net/Articles/293575/

justapassenger 6y ago
IMO, big reason for why Linux doesn’t support it, is that big tech companies, who drive a lot of development and funding, found that it’s just not worth it. Tech moved away from mainframe model of keeping single processes up for as long as possible and isolating software from the failures. Instead they embrace failure at the software layer, which gives you (when executed correctly) both same high level of availability, but also layers of protection against other issues (as node going down in expected way is the same, no matter what’s the root cause) and makes overall maintainability and upgradability systems higher. Basically, why deal with complicated checkpoints at kernel level, that doesn’t understand your software, when you ca instead deal with that at the software layer, with full control over it.
- spullara 6y ago
  Swap performance has been similarly underinvested.
  - snazz 6y ago
    How commonly do people need to use swap in the first place? On my current computer, I don't even have a swap partition because I've never gotten close to filling my 16GB, even with a big virtual machine and hundreds of browser tabs. In my experience on desktop systems, it just makes your computer painfully slow when you write a program that allocates far too much memory. Without swap, the OOM killer gets involved much earlier and you don't have to do a hard power cut to stop the disk from thrashing.
    - cesarb 6y ago
      I just opened the site of the biggest retailer in my country, and looked at one entry-level laptop: it has 2GB of RAM. Not everybody has a high-end computer.
      - sjwright 6y ago
        A Chromebook?
        saagarjha 6y ago
        My $150 Chromebook has twice that; most computers these days tend to ship with more than than I think.
        cesarb 6y ago
        No, a normal laptop from a local brand.
    - haneefmubarak 6y ago
      When done well, swap can be pretty useful. I have a Fall 2019 MacBook Pro maxed out on CPU (8 cores) and RAM (64 GB) - MacOS still regularly compresses (highest I've seen so far: 12 GB) and/or swaps (highest I've seen so far: 16 GB) relatively unused memory to make more space for other applications and hot caches (which are still useful even with an PCIe SSD). Keep in mind that everything feels fully snappy and responsive while this is all happening.
    - barrkel 6y ago
      If you don't have enough memory for a single app, then your life is completely miserable - in that case, swapping is almost indistinguishable from crashed. But swap can be the difference between being able to run one app at a time, vs run three or four, as long as you only have one in the foreground at a time. Swap doesn't play well with GC, or pointerful programming languages generally. Swap is a latency hit every time you touch an evicted page, and the more pointers your structures have, the more frequently that occurs. In older times, when memory needed to be conserved, GC was less frequently used, and as a consequence of the accounting overheads of manual memory allocation, pointerful data structures weren't quite as common. Instead, you'd have pre-allocated arrays which supported up to N items of a particular type (and tough luck if you wanted more), or in more sophisticated applications, arena allocators. In such systems, the working set of memory is more likely to be contiguous; paging works better because every page load is more likely to bring in relevant data. Switching between two open applications was the moment where you'd typically hit swap; that, and every now and then when you touched an area of the app you hadn't in a while, or jumped to a distant area of the document you were working on. The HDD light would come on, and you'd have to wait a few seconds while the computer ground through (HDDs being audible, some much louder than others) before you'd switch. At some point in the mid 2000s, GC languages tipped the balance, especially JS. The browser became the main application, and wouldn't tolerate swap well. It became a mini OS unto itself, and had to balance the competing needs of different tabs. Then mobile exploded, and didn't use swap at all in order to compete with the iPhone, remarkable for its smooth animations. If there was to be latency, the application had to hide it; OS-level introduced latency on a page fault wasn't good enough, because if it happened on the UI thread, you dropped frames. And of course swap never worked well on servers with frequent requests. Swapping on a server is a great way to back up all your queues.
  - justapassenger 6y ago
    Yeah, and for exactly the same reasons. In old days swap was necessity to keep system up, but nowadays you need it only if your dataset is really big (other than some weirdness in kernel, that actually like to have some small amount of swap to do cleanup and maintenance) And if you have that problem, you have custom solutions that implement memory hierarchy for you, that give you much better result, as they can work across all the layers (cache/hdd/sdd/net/etc) in much more effective way than generic, software agnostic solutions focused just on one dimension.
dirtydroog 6y ago
They seem to be different problems though, as that LWN article suggests. It's probably easier to checkpoint/restore an entire image's state than an individual process' state. But even calls like gettimeofday() work differently when running on hypervisors than they do when running on bare metal
colinchartier 6y ago
The hypervisor migration features you mention are really powerful and stable. I founded a startup that does fast CI using them. This is what it looks like: https://layerci.com/jobs/github/layerdemo/layer-gogs-demo/18 https://layerci.com/jobs/github/layerdemo/layer-gogs-demo/18 (in this run it restored a previous VM snapshot to skip the setup, and took a snapshot after the webserver started to give a lazy-loaded staging server)
colinchartier 6y ago
There's also a project called CRIU that is used experimentally by Docker for container save/load: https://www.criu.org/Main_Page https://www.criu.org/Main_Page
- jraph 6y ago
  You were faster than me. Yes, CRIU does checkpointing in userspace on Linux. They contributed a lot of patches to the Linux kernel to support this feature. So there isn't any support for checkpointing in the mainline kernel. Rather, there are many kernel features that allows building such a feature in userspace. For instance, you can set the PID of the next program you spawn on Linux thanks to CRIU, because CRIU needs to be able to restore processes with the same PIDs as when they were checkpointed [1]. CRIU is used by OpenVZ, which, if I remember correctly, are moving or moved away from a kernel-based approach. [1] http://efiop-notes.blogspot.com/2014/06/how-to-set-pid-using-nslastpid.html http://efiop-notes.blogspot.com/2014/06/how-to-set-pid-using...
  - jabedude 6y ago
    This was immensely informative, thank you for that link. CRIU looks like an awesome project. I can't believe I haven't heard of them prior to today.
- emmelaich 6y ago
  CRIU and DMTCP are mentioned in the article.
deleted 6y ago
[deleted]

justapassenger 6y ago

IMO, big reason for why Linux doesn’t support it, is that big tech companies, who drive a lot of development and funding, found that it’s just not worth it. Tech moved away from mainframe model of keeping single processes up for as long as possible and isolating software from the failures. Instead they embrace failure at the software layer, which gives you (when executed correctly) both same high level of availability, but also layers of protection against other issues (as node going down in expected way is the same, no matter what’s the root cause) and makes overall maintainability and upgradability systems higher. Basically, why deal with complicated checkpoints at kernel level, that doesn’t understand your software, when you ca instead deal with that at the software layer, with full control over it.

spullara 6y ago
Swap performance has been similarly underinvested.
- snazz 6y ago
  How commonly do people need to use swap in the first place? On my current computer, I don't even have a swap partition because I've never gotten close to filling my 16GB, even with a big virtual machine and hundreds of browser tabs. In my experience on desktop systems, it just makes your computer painfully slow when you write a program that allocates far too much memory. Without swap, the OOM killer gets involved much earlier and you don't have to do a hard power cut to stop the disk from thrashing.
  - cesarb 6y ago
    I just opened the site of the biggest retailer in my country, and looked at one entry-level laptop: it has 2GB of RAM. Not everybody has a high-end computer.
    - sjwright 6y ago
      A Chromebook?
      - saagarjha 6y ago
        My $150 Chromebook has twice that; most computers these days tend to ship with more than than I think.
      - cesarb 6y ago
        No, a normal laptop from a local brand.
  - haneefmubarak 6y ago
    When done well, swap can be pretty useful. I have a Fall 2019 MacBook Pro maxed out on CPU (8 cores) and RAM (64 GB) - MacOS still regularly compresses (highest I've seen so far: 12 GB) and/or swaps (highest I've seen so far: 16 GB) relatively unused memory to make more space for other applications and hot caches (which are still useful even with an PCIe SSD). Keep in mind that everything feels fully snappy and responsive while this is all happening.
  - barrkel 6y ago
    If you don't have enough memory for a single app, then your life is completely miserable - in that case, swapping is almost indistinguishable from crashed. But swap can be the difference between being able to run one app at a time, vs run three or four, as long as you only have one in the foreground at a time. Swap doesn't play well with GC, or pointerful programming languages generally. Swap is a latency hit every time you touch an evicted page, and the more pointers your structures have, the more frequently that occurs. In older times, when memory needed to be conserved, GC was less frequently used, and as a consequence of the accounting overheads of manual memory allocation, pointerful data structures weren't quite as common. Instead, you'd have pre-allocated arrays which supported up to N items of a particular type (and tough luck if you wanted more), or in more sophisticated applications, arena allocators. In such systems, the working set of memory is more likely to be contiguous; paging works better because every page load is more likely to bring in relevant data. Switching between two open applications was the moment where you'd typically hit swap; that, and every now and then when you touched an area of the app you hadn't in a while, or jumped to a distant area of the document you were working on. The HDD light would come on, and you'd have to wait a few seconds while the computer ground through (HDDs being audible, some much louder than others) before you'd switch. At some point in the mid 2000s, GC languages tipped the balance, especially JS. The browser became the main application, and wouldn't tolerate swap well. It became a mini OS unto itself, and had to balance the competing needs of different tabs. Then mobile exploded, and didn't use swap at all in order to compete with the iPhone, remarkable for its smooth animations. If there was to be latency, the application had to hide it; OS-level introduced latency on a page fault wasn't good enough, because if it happened on the UI thread, you dropped frames. And of course swap never worked well on servers with frequent requests. Swapping on a server is a great way to back up all your queues.
- justapassenger 6y ago
  Yeah, and for exactly the same reasons. In old days swap was necessity to keep system up, but nowadays you need it only if your dataset is really big (other than some weirdness in kernel, that actually like to have some small amount of swap to do cleanup and maintenance) And if you have that problem, you have custom solutions that implement memory hierarchy for you, that give you much better result, as they can work across all the layers (cache/hdd/sdd/net/etc) in much more effective way than generic, software agnostic solutions focused just on one dimension.

dirtydroog 6y ago

They seem to be different problems though, as that LWN article suggests. It's probably easier to checkpoint/restore an entire image's state than an individual process' state. But even calls like gettimeofday() work differently when running on hypervisors than they do when running on bare metal

colinchartier 6y ago

The hypervisor migration features you mention are really powerful and stable. I founded a startup that does fast CI using them. This is what it looks like: https://layerci.com/jobs/github/layerdemo/layer-gogs-demo/18 https://layerci.com/jobs/github/layerdemo/layer-gogs-demo/18 (in this run it restored a previous VM snapshot to skip the setup, and took a snapshot after the webserver started to give a lazy-loaded staging server)

colinchartier 6y ago

There's also a project called CRIU that is used experimentally by Docker for container save/load: https://www.criu.org/Main_Page https://www.criu.org/Main_Page

jraph 6y ago
You were faster than me. Yes, CRIU does checkpointing in userspace on Linux. They contributed a lot of patches to the Linux kernel to support this feature. So there isn't any support for checkpointing in the mainline kernel. Rather, there are many kernel features that allows building such a feature in userspace. For instance, you can set the PID of the next program you spawn on Linux thanks to CRIU, because CRIU needs to be able to restore processes with the same PIDs as when they were checkpointed [1]. CRIU is used by OpenVZ, which, if I remember correctly, are moving or moved away from a kernel-based approach. [1] http://efiop-notes.blogspot.com/2014/06/how-to-set-pid-using-nslastpid.html http://efiop-notes.blogspot.com/2014/06/how-to-set-pid-using...
- jabedude 6y ago
  This was immensely informative, thank you for that link. CRIU looks like an awesome project. I can't believe I haven't heard of them prior to today.
emmelaich 6y ago
CRIU and DMTCP are mentioned in the article.

deleted 6y ago

[deleted]