4 ms·

The LASP author also has a good reading list for CRDTs. [0] Here's why I find LASP intriguing. It started with a simple question: can't the OS just migrate a "

by sigil 9y ago

The LASP author also has a good reading list for CRDTs. [0] Here's why I find LASP intriguing. It started with a simple question: can't the OS just migrate a "process" to another machine for me? For a couple decades now, we've been taking hardware and software originally designed to run on little islands called PCs and trying to build datacenter-scale distributed systems out of them. We've got kernels that can load balance workloads across many CPU cores, but we can't reuse those to distribute work across a large number of physical machines -- because of CAP, because of unreliable networks, and because there are performance issues. So we're reimplementing OS schedulers now [1] on machines instead of cores, but the basic building blocks we're using don't seem to have anticipated distributed systems very well. So there's tons of resource waste, we're still solving lots of hard distributed systems problems at the application layer, and we're maintaining a huge, somewhat-historically-accidental pile of abstractions between the hardware and the distributed application running on it. One wonders if and when that pile may collapse. One also wonders what a distributed-first OS might look like. LASP might be the first step towards a whole new computing architecture. The syntax is hideous to my eye, and I have serious questions about performance, but if I can write a long-running program that efficiently scales from 1 machine to 100,000 machines with no code changes, that might be a game changer. PS. I think Meiklejohn's decision to start by modernizing Distributed Erlang was a solid one. Distributed Erlang's fully connected graph topology and pre-CAP approach to state probably wasn't workable...we've learned a ton about distributed systems since the 80s. [0] https://webcache.googleusercontent.com/search?q=cache:QJlbCkMI68YJ:https://christophermeiklejohn.com/crdt/2014/07/22/readings-in-crdts.html+&cd=1&hl=en&ct=clnk&gl=us https://webcache.googleusercontent.com/search?q=cache:QJlbCk... [1] https://kubernetes.io/docs/admin/kube-scheduler/ https://kubernetes.io/docs/admin/kube-scheduler/

polskibus 9y ago
Migration of processes has been done in hpc for instance: http://fox.eti.pg.gda.pl/~pczarnul/DAMPVM.html http://fox.eti.pg.gda.pl/~pczarnul/DAMPVM.html
- sigil 9y ago
  If you're the author of DAMPVM or are familiar with the internals, can you comment on: 1. The costs of migrating a process. How long does it typically take to an order of magnitude -- ms, secs, mins? How much state needs to be transferred? 2. Limitations. Can all processes be migrated, or are there certain types of process state that prevent migration? 3. What process state needs to be migrated, and how does that process state get enumerated? 4. Are there cases where a process is migrated, but some of its state stays behind, and the new process uses a proxy to continue to read and write it? Been curious about past attempts at this lately, and I've found a few by digging around. For instance, Sprite OS was a BSD offshoot in the late 80s that had some form of cross-machine process migration: https://web.stanford.edu/~ouster/cgi-bin/papers/migration-spe.pdf https://web.stanford.edu/~ouster/cgi-bin/papers/migration-sp...
  - polskibus 9y ago
    I'm not the author, I attended class by DAMPVM author long time ago. There are some references to publications on the main page that should lead to measurements that you're interested in. look around years 2000-2005. As far as I remember it was very efficient and shined in dynamic load scheduling using process migration. Very cool at that time. It was aimed at the HPC world (PVM), there were many interesting things done in that space that we're replaced by the Beowulf attitude towards off-the-shelf components.
xj9 9y ago
i've been thinking along these lines for a long time. i was originally very inspired by the plan 9 paper, but that model has a lot of weaknesses too. i'll admit that i don't have (m)any solutions, i've been busy with other projects, but this is definitely an area that i would like to have time to explore properly.
nickpsecurity 9y ago
"One wonders if and when that pile may collapse. One also wonders what a distributed-first OS might look like. LASP might be the first step towards a whole new computing architecture." What about Tannenbaum et al's Amoeba OS and Globe toolkit? Or the grid computing platforms that have been around a long time? There's been steady work in this area with some of it used in production in both CompSci and industry.
- sigil 9y ago
  > What about Tannenbaum et al's Amoeba OS? Unlike the contemporary Sprite, Amoeba does not support process migration. https://en.wikipedia.org/wiki/Amoeba_(operating_system) https://en.wikipedia.org/wiki/Amoeba_(operating_system) > Globe toolkit Got a link to this one? > Or the grid computing platforms that have been around a long time? There's been steady work in this area with some of it used in production in both CompSci and industry. Curious if any of these grid computing platforms support process migration (aka "don't make me think about machine boundaries"), and whether any of the more recent ones have been informed by CAP / designed to withstand unreliable networks. LASP is interesting because it has been. As I understand it, process state is a CRDT, so computations should survive network partitions and state should reconverge afterwards. The biggest questions for me are around the performance characteristics of this approach.
  - nickpsecurity 9y ago
    "Unlike the contemporary Sprite, Amoeba does not support process migration" Oops. My bad. It did a lot as a distributed system except for that apparently. You also found the one that did. "Globe toolkit Got a link to this one?" It's an object-based system. The old site is dead I think with me having to constantly dig it up. Here's a nice summary: https://cds.cern.ch/record/400321/files/p117.pdf https://cds.cern.ch/record/400321/files/p117.pdf They also mapped the WWW protocols to it to some degree to show we could reuse those legacy systems benefiting from Globe's capabilities. Or start the transition that way. "Curious if any of these grid computing platforms support process migration " The first I saw was MOSIX: https://en.wikipedia.org/wiki/MOSIX https://en.wikipedia.org/wiki/MOSIX Modern, smaller experiment that's similar in Linux: http://citeseerx.ist.psu.edu/viewdoc/download?rep=rep1&type=pdf&doi=10.1.1.103.417 http://citeseerx.ist.psu.edu/viewdoc/download?rep=rep1&type=... I found a nice survey paper that includes references to a number of systems from 2006: https://www.cse.buffalo.edu/~vipin/book_Chapters/2006/2006_1.pdf https://www.cse.buffalo.edu/~vipin/book_Chapters/2006/2006_1... I think most of the research is going into "VM" rather than "process" migrations given most computing is going in that direction. If you research that, you will get many more results whose techniques might apply to process migration. Quick example: http://mvapich.cse.ohio-state.edu/static/media/publications/abstract/huang-cluster07.pdf http://mvapich.cse.ohio-state.edu/static/media/publications/... The high-performance part of CompSci has mostly shifted in that direction since most industry and FOSS have shifted toward containers/VM's. The point of my original comment, though, was to say that what you say needs to be done has been happening for a while. It's just a small few in academia who dig in that deep to develop those mechanisms. And resulting in something worth finding with my quick searches. :)
Quequau 9y ago
Doesn't IBM have the facility to migrate processes (by virtue of Virtual Machines) from core-to-core, processor module to processor module, and machine to machine. I'm fairly confident I've seen that in their hypervisor on Power.

by sigil 9y ago

polskibus 9y ago

Migration of processes has been done in hpc for instance: http://fox.eti.pg.gda.pl/~pczarnul/DAMPVM.html http://fox.eti.pg.gda.pl/~pczarnul/DAMPVM.html

sigil 9y ago
If you're the author of DAMPVM or are familiar with the internals, can you comment on: 1. The costs of migrating a process. How long does it typically take to an order of magnitude -- ms, secs, mins? How much state needs to be transferred? 2. Limitations. Can all processes be migrated, or are there certain types of process state that prevent migration? 3. What process state needs to be migrated, and how does that process state get enumerated? 4. Are there cases where a process is migrated, but some of its state stays behind, and the new process uses a proxy to continue to read and write it? Been curious about past attempts at this lately, and I've found a few by digging around. For instance, Sprite OS was a BSD offshoot in the late 80s that had some form of cross-machine process migration: https://web.stanford.edu/~ouster/cgi-bin/papers/migration-spe.pdf https://web.stanford.edu/~ouster/cgi-bin/papers/migration-sp...
- polskibus 9y ago
  I'm not the author, I attended class by DAMPVM author long time ago. There are some references to publications on the main page that should lead to measurements that you're interested in. look around years 2000-2005. As far as I remember it was very efficient and shined in dynamic load scheduling using process migration. Very cool at that time. It was aimed at the HPC world (PVM), there were many interesting things done in that space that we're replaced by the Beowulf attitude towards off-the-shelf components.

xj9 9y ago

i've been thinking along these lines for a long time. i was originally very inspired by the plan 9 paper, but that model has a lot of weaknesses too. i'll admit that i don't have (m)any solutions, i've been busy with other projects, but this is definitely an area that i would like to have time to explore properly.

nickpsecurity 9y ago

"One wonders if and when that pile may collapse. One also wonders what a distributed-first OS might look like. LASP might be the first step towards a whole new computing architecture." What about Tannenbaum et al's Amoeba OS and Globe toolkit? Or the grid computing platforms that have been around a long time? There's been steady work in this area with some of it used in production in both CompSci and industry.

sigil 9y ago
> What about Tannenbaum et al's Amoeba OS? Unlike the contemporary Sprite, Amoeba does not support process migration. https://en.wikipedia.org/wiki/Amoeba_(operating_system) https://en.wikipedia.org/wiki/Amoeba_(operating_system) > Globe toolkit Got a link to this one? > Or the grid computing platforms that have been around a long time? There's been steady work in this area with some of it used in production in both CompSci and industry. Curious if any of these grid computing platforms support process migration (aka "don't make me think about machine boundaries"), and whether any of the more recent ones have been informed by CAP / designed to withstand unreliable networks. LASP is interesting because it has been. As I understand it, process state is a CRDT, so computations should survive network partitions and state should reconverge afterwards. The biggest questions for me are around the performance characteristics of this approach.
- nickpsecurity 9y ago
  "Unlike the contemporary Sprite, Amoeba does not support process migration" Oops. My bad. It did a lot as a distributed system except for that apparently. You also found the one that did. "Globe toolkit Got a link to this one?" It's an object-based system. The old site is dead I think with me having to constantly dig it up. Here's a nice summary: https://cds.cern.ch/record/400321/files/p117.pdf https://cds.cern.ch/record/400321/files/p117.pdf They also mapped the WWW protocols to it to some degree to show we could reuse those legacy systems benefiting from Globe's capabilities. Or start the transition that way. "Curious if any of these grid computing platforms support process migration " The first I saw was MOSIX: https://en.wikipedia.org/wiki/MOSIX https://en.wikipedia.org/wiki/MOSIX Modern, smaller experiment that's similar in Linux: http://citeseerx.ist.psu.edu/viewdoc/download?rep=rep1&type=pdf&doi=10.1.1.103.417 http://citeseerx.ist.psu.edu/viewdoc/download?rep=rep1&type=... I found a nice survey paper that includes references to a number of systems from 2006: https://www.cse.buffalo.edu/~vipin/book_Chapters/2006/2006_1.pdf https://www.cse.buffalo.edu/~vipin/book_Chapters/2006/2006_1... I think most of the research is going into "VM" rather than "process" migrations given most computing is going in that direction. If you research that, you will get many more results whose techniques might apply to process migration. Quick example: http://mvapich.cse.ohio-state.edu/static/media/publications/abstract/huang-cluster07.pdf http://mvapich.cse.ohio-state.edu/static/media/publications/... The high-performance part of CompSci has mostly shifted in that direction since most industry and FOSS have shifted toward containers/VM's. The point of my original comment, though, was to say that what you say needs to be done has been happening for a while. It's just a small few in academia who dig in that deep to develop those mechanisms. And resulting in something worth finding with my quick searches. :)

Quequau 9y ago

Doesn't IBM have the facility to migrate processes (by virtue of Virtual Machines) from core-to-core, processor module to processor module, and machine to machine. I'm fairly confident I've seen that in their hypervisor on Power.