4 ms·

For my OS class's final project last quarter, I built a way to live-migrate a process (running on a custom OS we built from scratch) from one Raspberry Pi to an

by arjvik 2y ago

For my OS class's final project last quarter, I built a way to live-migrate a process (running on a custom OS we built from scratch) from one Raspberry Pi to another, essentially using checkpoint/restore! Getting the code cleaned up enough to post it has been on my to-do list for quite some time, and this has inspired me to do it soon!

arjvik 2y ago
Should mention, the coolest part is that I never sent over "all" the memory used by the process, because it was difficult to tell what is needed and what isn't. Instead, I was clever with virtual memory, and when a page of memory was needed that wasn't loaded by the recipient Pi, it would request and lazy-load just that page from the provider Pi, and with some careful bookkeeping mark that the page was owned by the recipient Pi.
- zozbot234 2y ago
  > Instead, I was clever with virtual memory, and when a page of memory was needed that wasn't loaded by the recipient Pi, it would request and lazy-load just that page from the provider Pi, and with some careful bookkeeping mark that the page was owned by the recipient Pi. I wonder if that "trick" can be extended to a full implementation of distributed shared memory, i.e. multiple nodes running separate tasks in a single address space and implementing cache coherence over the network. Probably needs quite a bit of extra compiler/runtime support so it wouldn't really apply to standard binaries, but it might still be useful nonetheless.
  - nradclif 2y ago
    Partitioned Global Address Space (PGAS) compilers/runtimes do something similar to that. Unified Parallel C (UPC,https://upc.lbl.gov/ https://upc.lbl.gov/) and Coarray Fortran/Coarray C++ (https://docs.nersc.gov/development/programming-models/coarrays/ https://docs.nersc.gov/development/programming-models/coarra...) are good examples commonly used in HPC. Fabric Attached Memory (OpenFAM, https://openfam.github.io/ https://openfam.github.io/) is another example.
    - pklausler 2y ago
      “commonly used in HPC” is a bit of a stretch if you’re talking about production applications.
  - arjvik 2y ago
    That's actually basically what I was doing! Was able to run programs compiled for a "normal" OS on a single unified distributed virtual address space!
- alksjdalkj 2y ago
  Sounds like a cool class project! If I understand your approach correctly, this is how live virtual machine migration typically works (e.g., https://none.cs.umass.edu/~shenoy/courses/spring18/readings/xen-live-nsdi05.pdf https://none.cs.umass.edu/~shenoy/courses/spring18/readings/...). It also sounds similar to this "remote-fork" concept: https://www.usenix.org/system/files/osdi23-wei-rdma.pdf https://www.usenix.org/system/files/osdi23-wei-rdma.pdf.

by arjvik 2y ago

arjvik 2y ago

Should mention, the coolest part is that I never sent over "all" the memory used by the process, because it was difficult to tell what is needed and what isn't. Instead, I was clever with virtual memory, and when a page of memory was needed that wasn't loaded by the recipient Pi, it would request and lazy-load just that page from the provider Pi, and with some careful bookkeeping mark that the page was owned by the recipient Pi.

zozbot234 2y ago
> Instead, I was clever with virtual memory, and when a page of memory was needed that wasn't loaded by the recipient Pi, it would request and lazy-load just that page from the provider Pi, and with some careful bookkeeping mark that the page was owned by the recipient Pi. I wonder if that "trick" can be extended to a full implementation of distributed shared memory, i.e. multiple nodes running separate tasks in a single address space and implementing cache coherence over the network. Probably needs quite a bit of extra compiler/runtime support so it wouldn't really apply to standard binaries, but it might still be useful nonetheless.
- nradclif 2y ago
  Partitioned Global Address Space (PGAS) compilers/runtimes do something similar to that. Unified Parallel C (UPC,https://upc.lbl.gov/ https://upc.lbl.gov/) and Coarray Fortran/Coarray C++ (https://docs.nersc.gov/development/programming-models/coarrays/ https://docs.nersc.gov/development/programming-models/coarra...) are good examples commonly used in HPC. Fabric Attached Memory (OpenFAM, https://openfam.github.io/ https://openfam.github.io/) is another example.
  - pklausler 2y ago
    “commonly used in HPC” is a bit of a stretch if you’re talking about production applications.
- arjvik 2y ago
  That's actually basically what I was doing! Was able to run programs compiled for a "normal" OS on a single unified distributed virtual address space!
alksjdalkj 2y ago
Sounds like a cool class project! If I understand your approach correctly, this is how live virtual machine migration typically works (e.g., https://none.cs.umass.edu/~shenoy/courses/spring18/readings/xen-live-nsdi05.pdf https://none.cs.umass.edu/~shenoy/courses/spring18/readings/...). It also sounds similar to this "remote-fork" concept: https://www.usenix.org/system/files/osdi23-wei-rdma.pdf https://www.usenix.org/system/files/osdi23-wei-rdma.pdf.