6 ms·

Seriously cool. That also reminds me of DragonFlyBSD's process checkpointing feature that offers suspend to disk. In Linux world there were many attempts, but A

by hawski 7y ago

Seriously cool. That also reminds me of DragonFlyBSD's process checkpointing feature that offers suspend to disk. In Linux world there were many attempts, but AFAIK nothing simple and complete enough. To be fair I don't know if DF's implementation is that either. https://www.dragonflybsd.org/cgi/web-man?command=sys_checkpoint&section=2 https://www.dragonflybsd.org/cgi/web-man?command=sys_checkpo... https://www.dragonflybsd.org/cgi/web-man?command=checkpoint&section=ANY https://www.dragonflybsd.org/cgi/web-man?command=checkpoint&...

jabedude 7y ago
This article[0] on LWN seems to suggest that Linux has no kernel support for checkpoint/restore because it's hard to implement (no arguments there). But hypervisors support checkpoint/restore for virtual machines, e.g. ESXi VMotion and KVM live migration, so it seems like these technical problems are solvable. Indeed all the benefits of VM migration seem to also apply to process migration (load balancing, service uptime, etc). 0. https://lwn.net/Articles/293575/ https://lwn.net/Articles/293575/
- justapassenger 7y ago
  IMO, big reason for why Linux doesn’t support it, is that big tech companies, who drive a lot of development and funding, found that it’s just not worth it. Tech moved away from mainframe model of keeping single processes up for as long as possible and isolating software from the failures. Instead they embrace failure at the software layer, which gives you (when executed correctly) both same high level of availability, but also layers of protection against other issues (as node going down in expected way is the same, no matter what’s the root cause) and makes overall maintainability and upgradability systems higher. Basically, why deal with complicated checkpoints at kernel level, that doesn’t understand your software, when you ca instead deal with that at the software layer, with full control over it.
  - spullara 7y ago
    Swap performance has been similarly underinvested.
    - snazz 7y ago
      How commonly do people need to use swap in the first place? On my current computer, I don't even have a swap partition because I've never gotten close to filling my 16GB, even with a big virtual machine and hundreds of browser tabs. In my experience on desktop systems, it just makes your computer painfully slow when you write a program that allocates far too much memory. Without swap, the OOM killer gets involved much earlier and you don't have to do a hard power cut to stop the disk from thrashing.
      - cesarb 7y ago
        I just opened the site of the biggest retailer in my country, and looked at one entry-level laptop: it has 2GB of RAM. Not everybody has a high-end computer.
        sjwright 7y ago
        A Chromebook?
        saagarjha 7y ago
        My $150 Chromebook has twice that; most computers these days tend to ship with more than than I think.
        cesarb 7y ago
        No, a normal laptop from a local brand.
      - haneefmubarak 7y ago
        When done well, swap can be pretty useful. I have a Fall 2019 MacBook Pro maxed out on CPU (8 cores) and RAM (64 GB) - MacOS still regularly compresses (highest I've seen so far: 12 GB) and/or swaps (highest I've seen so far: 16 GB) relatively unused memory to make more space for other applications and hot caches (which are still useful even with an PCIe SSD). Keep in mind that everything feels fully snappy and responsive while this is all happening.
- dirtydroog 7y ago
  They seem to be different problems though, as that LWN article suggests. It's probably easier to checkpoint/restore an entire image's state than an individual process' state. But even calls like gettimeofday() work differently when running on hypervisors than they do when running on bare metal
- colinchartier 7y ago
  The hypervisor migration features you mention are really powerful and stable. I founded a startup that does fast CI using them. This is what it looks like: https://layerci.com/jobs/github/layerdemo/layer-gogs-demo/18 https://layerci.com/jobs/github/layerdemo/layer-gogs-demo/18 (in this run it restored a previous VM snapshot to skip the setup, and took a snapshot after the webserver started to give a lazy-loaded staging server)
- colinchartier 7y ago
  There's also a project called CRIU that is used experimentally by Docker for container save/load: https://www.criu.org/Main_Page https://www.criu.org/Main_Page
  - jraph 7y ago
    You were faster than me. Yes, CRIU does checkpointing in userspace on Linux. They contributed a lot of patches to the Linux kernel to support this feature. So there isn't any support for checkpointing in the mainline kernel. Rather, there are many kernel features that allows building such a feature in userspace. For instance, you can set the PID of the next program you spawn on Linux thanks to CRIU, because CRIU needs to be able to restore processes with the same PIDs as when they were checkpointed [1]. CRIU is used by OpenVZ, which, if I remember correctly, are moving or moved away from a kernel-based approach. [1] http://efiop-notes.blogspot.com/2014/06/how-to-set-pid-using-nslastpid.html http://efiop-notes.blogspot.com/2014/06/how-to-set-pid-using...
    - jabedude 7y ago
      This was immensely informative, thank you for that link. CRIU looks like an awesome project. I can't believe I haven't heard of them prior to today.
  - emmelaich 7y ago
    CRIU and DMTCP are mentioned in the article.
- deleted 7y ago
  [deleted]

by hawski 7y ago

jabedude 7y ago

This article[0] on LWN seems to suggest that Linux has no kernel support for checkpoint/restore because it's hard to implement (no arguments there). But hypervisors support checkpoint/restore for virtual machines, e.g. ESXi VMotion and KVM live migration, so it seems like these technical problems are solvable. Indeed all the benefits of VM migration seem to also apply to process migration (load balancing, service uptime, etc). 0. https://lwn.net/Articles/293575/ https://lwn.net/Articles/293575/

justapassenger 7y ago
IMO, big reason for why Linux doesn’t support it, is that big tech companies, who drive a lot of development and funding, found that it’s just not worth it. Tech moved away from mainframe model of keeping single processes up for as long as possible and isolating software from the failures. Instead they embrace failure at the software layer, which gives you (when executed correctly) both same high level of availability, but also layers of protection against other issues (as node going down in expected way is the same, no matter what’s the root cause) and makes overall maintainability and upgradability systems higher. Basically, why deal with complicated checkpoints at kernel level, that doesn’t understand your software, when you ca instead deal with that at the software layer, with full control over it.
- spullara 7y ago
  Swap performance has been similarly underinvested.
  - snazz 7y ago
    How commonly do people need to use swap in the first place? On my current computer, I don't even have a swap partition because I've never gotten close to filling my 16GB, even with a big virtual machine and hundreds of browser tabs. In my experience on desktop systems, it just makes your computer painfully slow when you write a program that allocates far too much memory. Without swap, the OOM killer gets involved much earlier and you don't have to do a hard power cut to stop the disk from thrashing.
    - cesarb 7y ago
      I just opened the site of the biggest retailer in my country, and looked at one entry-level laptop: it has 2GB of RAM. Not everybody has a high-end computer.
      - sjwright 7y ago
        A Chromebook?
        saagarjha 7y ago
        My $150 Chromebook has twice that; most computers these days tend to ship with more than than I think.
        cesarb 7y ago
        No, a normal laptop from a local brand.
    - haneefmubarak 7y ago
      When done well, swap can be pretty useful. I have a Fall 2019 MacBook Pro maxed out on CPU (8 cores) and RAM (64 GB) - MacOS still regularly compresses (highest I've seen so far: 12 GB) and/or swaps (highest I've seen so far: 16 GB) relatively unused memory to make more space for other applications and hot caches (which are still useful even with an PCIe SSD). Keep in mind that everything feels fully snappy and responsive while this is all happening.
dirtydroog 7y ago
They seem to be different problems though, as that LWN article suggests. It's probably easier to checkpoint/restore an entire image's state than an individual process' state. But even calls like gettimeofday() work differently when running on hypervisors than they do when running on bare metal
colinchartier 7y ago
The hypervisor migration features you mention are really powerful and stable. I founded a startup that does fast CI using them. This is what it looks like: https://layerci.com/jobs/github/layerdemo/layer-gogs-demo/18 https://layerci.com/jobs/github/layerdemo/layer-gogs-demo/18 (in this run it restored a previous VM snapshot to skip the setup, and took a snapshot after the webserver started to give a lazy-loaded staging server)
colinchartier 7y ago
There's also a project called CRIU that is used experimentally by Docker for container save/load: https://www.criu.org/Main_Page https://www.criu.org/Main_Page
- jraph 7y ago
  You were faster than me. Yes, CRIU does checkpointing in userspace on Linux. They contributed a lot of patches to the Linux kernel to support this feature. So there isn't any support for checkpointing in the mainline kernel. Rather, there are many kernel features that allows building such a feature in userspace. For instance, you can set the PID of the next program you spawn on Linux thanks to CRIU, because CRIU needs to be able to restore processes with the same PIDs as when they were checkpointed [1]. CRIU is used by OpenVZ, which, if I remember correctly, are moving or moved away from a kernel-based approach. [1] http://efiop-notes.blogspot.com/2014/06/how-to-set-pid-using-nslastpid.html http://efiop-notes.blogspot.com/2014/06/how-to-set-pid-using...
  - jabedude 7y ago
    This was immensely informative, thank you for that link. CRIU looks like an awesome project. I can't believe I haven't heard of them prior to today.
- emmelaich 7y ago
  CRIU and DMTCP are mentioned in the article.
deleted 7y ago
[deleted]