3 ms·

Different proposal: Let's say we have 2 Linux machines. Identical hardware, identical libs. I'd like to run a simple program on one machine, and then during m

Different proposal: Let's say we have 2 Linux machines. Identical hardware, identical libs. I'd like to run a simple program on one machine, and then during mid-calculation, would like to transfer the running program to the other machine. Is this doable?

new_user_final 2y ago
Unrelated, but somewhat similar in higher level, you can transfer state with durable execution, e.g temporal.io. Instead of RAM, program's state saved in DB and execution environment resume in the previous state when restarted
- WJW 2y ago
  How does such a method retain things like open network connections that have significant kernel state involved as well?
  - dilyevsky 2y ago
    it does not. all the state that you need to make "durable" needs to be explicitly committed in temporal via their sdk
dilyevsky 2y ago
Yes - using Criu[0] or docker checkpoint/restore api (which uses criu) [0] -https://criu.org/Main_Page https://criu.org/Main_Page
panki27 2y ago
Interesting thought, but highly dependant on the actual program. Let's assume it doesn't touch any files on disk (no opening sockets either). You would need to at least 1. Halt the process (SIGSTOP comes to mind) 2. Create a copy of the running program and /proc/$pid - which will also include memory and mmap details 3. Transfer everything to the other machine 4. Load memory, somehow spawn a spawn a new process with the info from /proc/$pid we saved, mmap the loaded memory into it 5. Continue the process on the new machine (SIGCONT) Let me admit that I do not have the slightest clue how to achieve step 4. I wonder if a systemd namespace could make things easier.
toast0 2y ago
A search for 'linux process live migration' picks up at least one repo that claims to have done it, and a bunch of other interesrting things. For a very simple program, with limited I/O, it's not too hard; especially if you don't mind a significant pause to move. Difficulty comes when you have FDs to migrate and if you need to reduce the pausing. If you need to keep FDs to the filesystem or the program will load/store to the filesystem periodically, you'd need to do a filesystem migration too... If you need to keep FDs for network sockets, you've got to transfer those somehow. If it's just stdin/out/err, you could probably do the migration in userspace with some difficulty if you need to keep pid constant (but maybe you don't need that either). Minimal pausing involves letting the program run on the initial machine while you copy memory, setting pages to read-only so you can catch writes, and only pausing the program once the copy is substantially finished. Then you pause execution on the initial machine. If there's a significant amount of modified pages to copy over when you pause, you can still start execution on the new machine, as long as the modified pages are marked unavailable, if you background copy them before they're used great... if not, you have to block until the modified data comes through. Probably you do this on two nearby machines with fast networking, and the program doesn't have a lot of writes all over memory, so the pause should be short.
- dilyevsky 2y ago
  If you're talking about Criu then it's not just a claim it actually does work well in production. I know Google was using it in prod on their internal systems and probably many others. It even can migrate TCP connections for you via socket repair api in Linux
- wang_li 2y ago
  >...keep FDs for network sockets, you've got to transfer those somehow. And if you have any shared memory segments, semaphores, or message queues, you have to drag along a bunch of other processes.
tryauuum 2y ago
if you put you program in a qemu/kvm VM then it just works I was completely blown away when I first experienced it. My code running in a VM never even noticed any downtime. All the network connections are preserved and so on.

new_user_final 2y ago

Unrelated, but somewhat similar in higher level, you can transfer state with durable execution, e.g temporal.io. Instead of RAM, program's state saved in DB and execution environment resume in the previous state when restarted

WJW 2y ago
How does such a method retain things like open network connections that have significant kernel state involved as well?
- dilyevsky 2y ago
  it does not. all the state that you need to make "durable" needs to be explicitly committed in temporal via their sdk

dilyevsky 2y ago

Yes - using Criu[0] or docker checkpoint/restore api (which uses criu) [0] -https://criu.org/Main_Page https://criu.org/Main_Page

panki27 2y ago

Interesting thought, but highly dependant on the actual program. Let's assume it doesn't touch any files on disk (no opening sockets either). You would need to at least 1. Halt the process (SIGSTOP comes to mind) 2. Create a copy of the running program and /proc/$pid - which will also include memory and mmap details 3. Transfer everything to the other machine 4. Load memory, somehow spawn a spawn a new process with the info from /proc/$pid we saved, mmap the loaded memory into it 5. Continue the process on the new machine (SIGCONT) Let me admit that I do not have the slightest clue how to achieve step 4. I wonder if a systemd namespace could make things easier.

toast0 2y ago

A search for 'linux process live migration' picks up at least one repo that claims to have done it, and a bunch of other interesrting things. For a very simple program, with limited I/O, it's not too hard; especially if you don't mind a significant pause to move. Difficulty comes when you have FDs to migrate and if you need to reduce the pausing. If you need to keep FDs to the filesystem or the program will load/store to the filesystem periodically, you'd need to do a filesystem migration too... If you need to keep FDs for network sockets, you've got to transfer those somehow. If it's just stdin/out/err, you could probably do the migration in userspace with some difficulty if you need to keep pid constant (but maybe you don't need that either). Minimal pausing involves letting the program run on the initial machine while you copy memory, setting pages to read-only so you can catch writes, and only pausing the program once the copy is substantially finished. Then you pause execution on the initial machine. If there's a significant amount of modified pages to copy over when you pause, you can still start execution on the new machine, as long as the modified pages are marked unavailable, if you background copy them before they're used great... if not, you have to block until the modified data comes through. Probably you do this on two nearby machines with fast networking, and the program doesn't have a lot of writes all over memory, so the pause should be short.

dilyevsky 2y ago
If you're talking about Criu then it's not just a claim it actually does work well in production. I know Google was using it in prod on their internal systems and probably many others. It even can migrate TCP connections for you via socket repair api in Linux
wang_li 2y ago
>...keep FDs for network sockets, you've got to transfer those somehow. And if you have any shared memory segments, semaphores, or message queues, you have to drag along a bunch of other processes.

tryauuum 2y ago

if you put you program in a qemu/kvm VM then it just works I was completely blown away when I first experienced it. My code running in a VM never even noticed any downtime. All the network connections are preserved and so on.