4 ms·

Should mention, the coolest part is that I never sent over "all" the memory used by the process, because it was difficult to tell what is needed and what isn't.

by arjvik 2y ago

Should mention, the coolest part is that I never sent over "all" the memory used by the process, because it was difficult to tell what is needed and what isn't. Instead, I was clever with virtual memory, and when a page of memory was needed that wasn't loaded by the recipient Pi, it would request and lazy-load just that page from the provider Pi, and with some careful bookkeeping mark that the page was owned by the recipient Pi.

zozbot234 2y ago
> Instead, I was clever with virtual memory, and when a page of memory was needed that wasn't loaded by the recipient Pi, it would request and lazy-load just that page from the provider Pi, and with some careful bookkeeping mark that the page was owned by the recipient Pi. I wonder if that "trick" can be extended to a full implementation of distributed shared memory, i.e. multiple nodes running separate tasks in a single address space and implementing cache coherence over the network. Probably needs quite a bit of extra compiler/runtime support so it wouldn't really apply to standard binaries, but it might still be useful nonetheless.
- nradclif 2y ago
  Partitioned Global Address Space (PGAS) compilers/runtimes do something similar to that. Unified Parallel C (UPC,https://upc.lbl.gov/ https://upc.lbl.gov/) and Coarray Fortran/Coarray C++ (https://docs.nersc.gov/development/programming-models/coarrays/ https://docs.nersc.gov/development/programming-models/coarra...) are good examples commonly used in HPC. Fabric Attached Memory (OpenFAM, https://openfam.github.io/ https://openfam.github.io/) is another example.
  - pklausler 2y ago
    “commonly used in HPC” is a bit of a stretch if you’re talking about production applications.
- arjvik 2y ago
  That's actually basically what I was doing! Was able to run programs compiled for a "normal" OS on a single unified distributed virtual address space!
alksjdalkj 2y ago
Sounds like a cool class project! If I understand your approach correctly, this is how live virtual machine migration typically works (e.g., https://none.cs.umass.edu/~shenoy/courses/spring18/readings/xen-live-nsdi05.pdf https://none.cs.umass.edu/~shenoy/courses/spring18/readings/...). It also sounds similar to this "remote-fork" concept: https://www.usenix.org/system/files/osdi23-wei-rdma.pdf https://www.usenix.org/system/files/osdi23-wei-rdma.pdf.

by arjvik 2y ago

zozbot234 2y ago

> Instead, I was clever with virtual memory, and when a page of memory was needed that wasn't loaded by the recipient Pi, it would request and lazy-load just that page from the provider Pi, and with some careful bookkeeping mark that the page was owned by the recipient Pi. I wonder if that "trick" can be extended to a full implementation of distributed shared memory, i.e. multiple nodes running separate tasks in a single address space and implementing cache coherence over the network. Probably needs quite a bit of extra compiler/runtime support so it wouldn't really apply to standard binaries, but it might still be useful nonetheless.

nradclif 2y ago
Partitioned Global Address Space (PGAS) compilers/runtimes do something similar to that. Unified Parallel C (UPC,https://upc.lbl.gov/ https://upc.lbl.gov/) and Coarray Fortran/Coarray C++ (https://docs.nersc.gov/development/programming-models/coarrays/ https://docs.nersc.gov/development/programming-models/coarra...) are good examples commonly used in HPC. Fabric Attached Memory (OpenFAM, https://openfam.github.io/ https://openfam.github.io/) is another example.
- pklausler 2y ago
  “commonly used in HPC” is a bit of a stretch if you’re talking about production applications.
arjvik 2y ago
That's actually basically what I was doing! Was able to run programs compiled for a "normal" OS on a single unified distributed virtual address space!

alksjdalkj 2y ago

Sounds like a cool class project! If I understand your approach correctly, this is how live virtual machine migration typically works (e.g., https://none.cs.umass.edu/~shenoy/courses/spring18/readings/xen-live-nsdi05.pdf https://none.cs.umass.edu/~shenoy/courses/spring18/readings/...). It also sounds similar to this "remote-fork" concept: https://www.usenix.org/system/files/osdi23-wei-rdma.pdf https://www.usenix.org/system/files/osdi23-wei-rdma.pdf.