9 ms·

Teleforking a Process onto a Different Computer

218 points by kaladin-jasnah 4y ago

cl0ckt0wer 4y ago
This sounds like Kafka, but more low level.
- latenightcoding 4y ago
  Not even close
basementcat 4y ago
Both MOSIX and openMOSIX supported fork()ing to another node on the network. https://en.m.wikipedia.org/wiki/MOSIX https://en.m.wikipedia.org/wiki/MOSIX
- dublin 4y ago
  The old 'dsh' (distributed shell) command in DEC Ultrix gave you the ability to do something similar from the shell - kind of rsh on steroids. (FWIW, Ultrix remains my overall favorite Unix - it was very well thought-out, and just thought-about period. It's a shame they abandoned it for OSF/1 on Alpha, which never really worked (well, the DEC Alpha hardware worked, but OSF/1 and the compilers never did...) Way back in the early 90s, inspired by this and a few other things, a coworker of mine (one of only three true geniuses I've ever met) came up with an even more powerful way (called vArray) to spread processes across any arbitrary number of POSIX machines. One of the things vArray could do was make remote memory look local (if painfully slow). He was the company's Cray expert, and developed it for the major oil company we worked for to run jobs too large to fit into the Cray's memory. Since some seismic deconvolution vectorizes very well, with fairly infrequent memory access, it worked better than you might expect, even over the slow networks of the day. Obviously, it hammered the network, but it was quite a sight to see every Sun, DEC, IBM, and HP Unix machine on the company's network light up at once, all coordinating to process jobs that were probably bigger than anything run outside the Black world of the Spooks... Fun days.
- flyinghamster 4y ago
  I think multicore CPUs did a lot to kill the momentum of OpenMOSIX. I remember playing around with clusterKNOPPIX back in the day and being quite impressed by it, but it seems that multiple cores and threads per core became the norm and the cores themselves got a lot faster. On the other hand, combining multiple 64-core Threadrippers into a single-system-image cluster has a certain appeal to it.
- spaceprison 4y ago
  Open mosix was the coolest project.
- marcodiego 4y ago
  OpenMOSIX is something I miss so much. I watched people build entire functional clusters using off-the-shelf hardware in a very tight budget and in just a few hours (most of the time was just unpacking things) using network boot and (IIRC) ClusterKnoppix. It was just "fork and forget" and boom, your job was now running in multiple processors. I'd like to do that with my spare ARM SBC's.
Animats 4y ago
That used to be in some UNIX variants, such as UCLA Locus and the IBM derivatives of that. But it never got to be a Linux thing.
- Fnoord 4y ago
  Was VMS capable of achieving this as well?
jonathaneunice 4y ago
Congratulations! You have just reinvented the core idea of UCLA's LOCUS distributed computing project from 1979. https://en.wikipedia.org/wiki/LOCUS https://en.wikipedia.org/wiki/LOCUS Reinventing LOCUS also has a strong heritage. Bell Lab's Plan 9, for example, did so in part in the late 1980s. While never a breakout commercial success, tele-forking and its slightly more advanced cousins machine-to-machine process migration and cluster-wide process pools intrigued some of the best minds in distributed computing for 20+ years. Unfortunately "it's complicated" to implement well, especially when you try to tele-spawn and manage resources beyond compute cycles (network connections, files, file handles, ...) that are important to scale up the idea.
- zozbot234 4y ago
  > Unfortunately "it's complicated" to implement well, especially when you try to tele-spawn and manage resources beyond compute cycles (network connections, files, file handles, ...) Aren't all of these resources namespaced/containerized in modern Linux? This should make it feasible to checkpoint and restore them on the same machine (via, e.g. the CRIU patchset) and true location-independence is not that much harder. One of the hardest parts (not even implemented in plan9, AFAICT) is distributed shared memory (allowing for sharing a single virtual address space across cluster nodes), but even that AIUI has some research-level implementations.
  - convolvatron 4y ago
    you still have to migrate the kernel state of underlying resources, so containers don't really buy you very much. as you say, in checkpoint restart you start with the big stuff like tcp sockets, but after a while you get tired of tracking down little flecks of state. distributed shared memory though does deserve a new look now that baseline network speeds are 100x what the max was when it was first investigated. unfortunately temporal consistency is still going to be a major factor. some workloads will run great with some heuristics, and some won't. you'll almost certainly need to migrate threads along with pages to try to keep them running co-local without exhausting per-node resources.
daenz 4y ago
Lambda/Cloud Functions are starting to converge on this idea. It will eventually get streamlined and ergonomic enough that it appears you're executing an expensive or async function locally, except you aren't.
- SteadBytes 4y ago
  "From laptop to lambda"[0] from a couple of years ago is an interesting step in this direction. [0]https://dl.acm.org/doi/10.5555/3358807.3358848 https://dl.acm.org/doi/10.5555/3358807.3358848
a-dub 4y ago
this is a fun hack. it would be interesting to look at some real world workloads and compare whether this sort of init once ship initialized memory image everywhere style is faster than just initializing everywhere.
mghfreud 4y ago
isn’t this exactly what the vm migration in cloud is?
- mlyle 4y ago
  No. VM migration moves entire virtual computers. Forking makes a copy of a process with the current state; this moves that single duplicated process to a different machine.
  - mghfreud 4y ago
    Virtual computer is a bunch of processes.
    - speed_spread 4y ago
      And a kernel. And drivers. And devices. And busses. And interrupts.
    - mlyle 4y ago
      The word "entire" is in there.
  - sitkack 4y ago
    Is Star Trek's Transporter Actually a Murder Machine? https://www.youtube.com/watch?v=f-8zEkIaB0c https://www.youtube.com/watch?v=f-8zEkIaB0c
- sitkack 4y ago
  Yes. All live migration systems basically follow the same pattern https://cloud.google.com/compute/docs/instances/live-migration https://cloud.google.com/compute/docs/instances/live-migrati... And as others have mentioned upthread, Mosix was an extension to Linux that also implemented fork() in such a way that the child could be local or remote with file handles retained across the cluster. We had a Linux lan party when I was in college and managed to scrape together a Mosix cluster across a bunch of machines. Ignore the uncreative pedants.
felixgallo 4y ago
Always worth a reread: https://joearms.github.io/published/2013-11-21-My-favorite-erlang-program.html https://joearms.github.io/published/2013-11-21-My-favorite-e...
- eointierney 4y ago
  <About a year later I had to write a paper. One of the disadvantages of being a researcher is that in order to get money you have to write a paper about something or other, the paper is never about what currently interests you at the moment, but must be about what the project that financed your research expects to read about. Well I had my gossip network setup on planet lab and I could tell it to become anything, so I told it to become a content distribution networks and used a gossip algorithm to make copies of the same file on all machine on the network and wrote a paper about it and everybody was happy.> I miss Joe, not that I ever met him, but his attitude and good humour are inspiring.
  - ungamedplayer 4y ago
    He was just as good of a man in real life.
- ryukafalz 4y ago
  Wow, thanks for posting this! I've never used Erlang before, but the "become" functionality and the message-passing style is similar enough to another actor-model system I'm familiar with (Spritely Goblins) that I was inspired to write a version of this program for it: https://git.sr.ht/~jfred/goblins-test/tree/master/item/universal.rkt https://git.sr.ht/~jfred/goblins-test/tree/master/item/unive...
AaronFriel 4y ago
The Cloud Haskell project and language is likely the only one to get this right, thanks to strictly enforced purity. It's much simpler to understand, absence global mutable state, whether it's safe and possible to serialize a closure and run it somewhere else. (fork(2) being a closure by another name.) In almost all other languages there's just no way to know if a closure is holding on to a file descriptor. Critics may say the Haskell closures could contain `unsafePerformIO`, but as the saying goes: now you have two problems.
- gpderetta 4y ago
  Isn't fork more of a continuation? /pedantic
- gnufx 4y ago
  Is Cloud [sigh] Haskell still alive? For comparison, two old distributed lexical scope systems were Cardelli's Obliq and Kelsey's(?) Kali Scheme. From what I remember, not like remote forking, though.
  - AaronFriel 4y ago
    I don't think so, sadly.
gnufx 4y ago
For what it's worth, at least for HPC-ish distributed computing, this sort of thing turns out not to be terribly worthwhile. We have a standard for distribution of computation, shared memory, i/o, and process starting in MPI (and, for instance, DMTCP to migrate the distributed application if necessary, though I think DMTCP needs a release). I don't know what its current status is, but the HPC-ish Bproc system has/had an rfork [1]. Probably the most HPC-oriented SSI system, Kerrighed died, as did the Plan 9-ish xcpu, though that was a bit different. 1. https://www.penguinsolutions.com/computing/documentation/scyld-clusterware/7/admin-guide/ref-guide/libraries/bproc/bproc-rfork.html https://www.penguinsolutions.com/computing/documentation/scy...
- zozbot234 4y ago
  The biggest benefit is arguably that codes that are designed for "telefork" and perhaps remote threads can also be scaled down to a single shared-memory machine, and run way more efficiently than if they had been coded using the MPI approach. Whilst you don't really add much of any overhead when running in a cluster, assuming that the codes are designed properly.
  - gnufx 4y ago
    Just doing a fork may be sufficient for something embarrassingly parallel, but interesting things are tightly coupled. Obviously MPI scales down to a single node (a distributed system anyway, these days), typically as real forked processes, but possibly with all the ranks in a single processwith an appropriate implementation. Citation needed, as they say, for “run way more efficiently”, particularly as the conventional wisdom says shared memory in a single process (e.g. OpenMP). “Acknowledgements: ... NUMA and Amdahl’s Law, for holding OpenMP back and keeping MPI-only competitive in spite of the ridiculous cost of Send-Recv within a shared-memory domain.” — Jeff Hammond, ‘MPI+MPI’
- bluedino 4y ago
  I would kill for "vmotion" on our HPC cluster. Tired of draining a node then pestering the remaining users to log their jobs out.
fleddr 4y ago
In the late 90s I attended a tour at Holland Signaal, a dutch defense company producing radar and anti-missile systems. I remember vividly how they demonstrated an unbreakable process. They had a computer running a process and no matter what happened to that computer, the next one would flawlessly continue the process down to the cycle, with no change or corruption or skipping a beat. It may very well be that this is actually not very difficult, but it seemed difficult and impressive. Perhaps more shocking were ultra high resolution radar screens, some 3 generations ahead of anything I had seen in the consumer space, showing an incredible visualization of the air space, live. Showing exactly which plane is where, the model/type, age, fuel on board, hostile/friendly, all of it. They even had a "situation room" with a holodeck chair in the middle, full of controls. The entire room was covered in wall-size screens basically showing the air space of the entire country, being live analyzed. Sounds very 2022, not 1998.
- zamadatix 4y ago
  Of all of the amazing things I've learned/seen in IT over the years the one that had the greatest "jaw to floor" effect was when I first learned of VMware's Fault Tolerance feature (shortly after I learned about VMs and live VM migration, which was the previous "jaw to floor" winner). Fault Tolerance allows for a very similar cycle accurate continuation concept but trades off specialized synchronous design and near instant failover for a ~1 second hitch to gain the ability to do this with a non-specialized VM and software stack. For even more fun if you have more than 2 hosts when 1 is lost it'll rebuild a fault tolerant state automatically with another host.
  - fleddr 4y ago
    That's so cool, I'm only a casual VMWare user and had no idea. With things like this possible, and possible for a long time now, it honestly makes me cringe at how absolutely terrible (slow, inefficient, unreliable, complex, not robust) the typical modern software stack is.
    - xwolfi 4y ago
      It's not true ! The typical software stack is optimized with need as it arises You wouldnt want to write ultrafast software for your idea to deliver cats back from the vets with a network of teenagers om bicycle. Imagine, years to write your website in C++/ASM to end up failing the first week. What s embarrassing is how much software cost to build in the 80s.
tenken 4y ago
Doesn't Erlang support these ideas of distributed computing .... And if I recall correctly Clipper supported remote execution of objects, or sharing object code in a distributed fashion.
- linux2647 4y ago
  Yep. spawn is the basic primitive: https://www.erlang.org/doc/man/erlang.html#spawn-4 https://www.erlang.org/doc/man/erlang.html#spawn-4
kristjansson 4y ago
Off-topic, but google-fu is failing me: wasn't there a similar project posted here a while back, that instead shipped a binary + a virtual file system that mirrored all local resources to a remote host so that the entire process could run there? IIRC the author demonstrated it running an ffmepg transcode? edit: Got it, after a while: https://github.com/Overv/outrun https://github.com/Overv/outrun
- sitkack 4y ago
  This is cool, when it talks about packaging, I was reminded of statifier. It would take a dynamically linked ELF executable and package everything up into a single binary. I last used in the early 2000s, but it looks like it had an update as recently as 2016. https://sourceforge.net/projects/statifier/ https://sourceforge.net/projects/statifier/
paulgb 4y ago
I've been playing with adjacent ideas in WebAssembly. The idea is that you can write a function that sends and receives messages of arbitrary Rust types, compile it into WebAssembly, and then snapshot and restore it (including across space and time) as you like. It's very experimental, but I got as far as being able to freeze a JavaScript interpreter (compiled to WebAssembly) and restore it later. The code is here: https://github.com/drifting-in-space/wasmbox https://github.com/drifting-in-space/wasmbox
- sitkack 4y ago
  Me too! The entire heap of the Wasm env is up for inspection and mutation. If your data structures are stable, then they can be moved across envs, or use accessors from both sides. Fun stuff.
  - paulgb 4y ago
    Yeah! I think it’s an under-appreciated aspect of WebAssembly.
aag 4y ago
Termite Scheme did something like this for Scheme programs. It combines Erlang-style concurrency with process migration. http://www.iro.umontreal.ca/~feeley/papers/GermainFeeleyMonnierELSW05.pdf http://www.iro.umontreal.ca/~feeley/papers/GermainFeeleyMonn...
OJFord 4y ago
(2020) and discussed at the time here: https://news.ycombinator.com/item?id=22987747 https://news.ycombinator.com/item?id=22987747
megamindbrian2 4y ago
samsquire 4y ago
I also had this idea but I called it networkfork.
bediger4000 4y ago
I don't think anyone has mentioned Sprite https://en.m.wikipedia.org/wiki/Sprite_(operating_system) https://en.m.wikipedia.org/wiki/Sprite_(operating_system) Sprite called this "process migration".
cryptonector 4y ago
You could do what CRIU does, but leave the original process running, and arrange to have a composite {instance ID, PID} as the return value (different on both sides). (Trivia: in Unix VI, fork() returns both PIDs, and the C library stub arranges to return 0 to the child and the child's PID to the parent.)
phs318u 4y ago
I can't state with absolute certainty but I seem to recall OpenVMS clusters back in the 90's transparently supporting "portable" processes across cluster nodes.
innocentoldguy 4y ago
If I am understanding the article correctly, BEAM does this for Elixir and Erlang (and other BEAM languages) out of the box.
- tashbarg 4y ago
  No, it does not. Not _this_. Erlang has, however, excellent support for distributed computing using its own kind of processes.
midislack 4y ago
This is really cool, Distributed Unix could do this IIRC.
rcarmo 4y ago
This is fun. Reminded me of mincemeat (https://github.com/ziyuang/mincemeatpy https://github.com/ziyuang/mincemeatpy), which actually serializes the mapping functions and sends them to workers.
deleted 4y ago
[deleted]

This sounds like Kafka, but more low level.

latenightcoding 4y ago
Not even close

Both MOSIX and openMOSIX supported fork()ing to another node on the network. https://en.m.wikipedia.org/wiki/MOSIX https://en.m.wikipedia.org/wiki/MOSIX

dublin 4y ago
The old 'dsh' (distributed shell) command in DEC Ultrix gave you the ability to do something similar from the shell - kind of rsh on steroids. (FWIW, Ultrix remains my overall favorite Unix - it was very well thought-out, and just thought-about period. It's a shame they abandoned it for OSF/1 on Alpha, which never really worked (well, the DEC Alpha hardware worked, but OSF/1 and the compilers never did...) Way back in the early 90s, inspired by this and a few other things, a coworker of mine (one of only three true geniuses I've ever met) came up with an even more powerful way (called vArray) to spread processes across any arbitrary number of POSIX machines. One of the things vArray could do was make remote memory look local (if painfully slow). He was the company's Cray expert, and developed it for the major oil company we worked for to run jobs too large to fit into the Cray's memory. Since some seismic deconvolution vectorizes very well, with fairly infrequent memory access, it worked better than you might expect, even over the slow networks of the day. Obviously, it hammered the network, but it was quite a sight to see every Sun, DEC, IBM, and HP Unix machine on the company's network light up at once, all coordinating to process jobs that were probably bigger than anything run outside the Black world of the Spooks... Fun days.
flyinghamster 4y ago
I think multicore CPUs did a lot to kill the momentum of OpenMOSIX. I remember playing around with clusterKNOPPIX back in the day and being quite impressed by it, but it seems that multiple cores and threads per core became the norm and the cores themselves got a lot faster. On the other hand, combining multiple 64-core Threadrippers into a single-system-image cluster has a certain appeal to it.
spaceprison 4y ago
Open mosix was the coolest project.
marcodiego 4y ago
OpenMOSIX is something I miss so much. I watched people build entire functional clusters using off-the-shelf hardware in a very tight budget and in just a few hours (most of the time was just unpacking things) using network boot and (IIRC) ClusterKnoppix. It was just "fork and forget" and boom, your job was now running in multiple processors. I'd like to do that with my spare ARM SBC's.

Animats 4y ago

That used to be in some UNIX variants, such as UCLA Locus and the IBM derivatives of that. But it never got to be a Linux thing.

Fnoord 4y ago
Was VMS capable of achieving this as well?

jonathaneunice 4y ago

Congratulations! You have just reinvented the core idea of UCLA's LOCUS distributed computing project from 1979. https://en.wikipedia.org/wiki/LOCUS https://en.wikipedia.org/wiki/LOCUS Reinventing LOCUS also has a strong heritage. Bell Lab's Plan 9, for example, did so in part in the late 1980s. While never a breakout commercial success, tele-forking and its slightly more advanced cousins machine-to-machine process migration and cluster-wide process pools intrigued some of the best minds in distributed computing for 20+ years. Unfortunately "it's complicated" to implement well, especially when you try to tele-spawn and manage resources beyond compute cycles (network connections, files, file handles, ...) that are important to scale up the idea.

zozbot234 4y ago
> Unfortunately "it's complicated" to implement well, especially when you try to tele-spawn and manage resources beyond compute cycles (network connections, files, file handles, ...) Aren't all of these resources namespaced/containerized in modern Linux? This should make it feasible to checkpoint and restore them on the same machine (via, e.g. the CRIU patchset) and true location-independence is not that much harder. One of the hardest parts (not even implemented in plan9, AFAICT) is distributed shared memory (allowing for sharing a single virtual address space across cluster nodes), but even that AIUI has some research-level implementations.
- convolvatron 4y ago
  you still have to migrate the kernel state of underlying resources, so containers don't really buy you very much. as you say, in checkpoint restart you start with the big stuff like tcp sockets, but after a while you get tired of tracking down little flecks of state. distributed shared memory though does deserve a new look now that baseline network speeds are 100x what the max was when it was first investigated. unfortunately temporal consistency is still going to be a major factor. some workloads will run great with some heuristics, and some won't. you'll almost certainly need to migrate threads along with pages to try to keep them running co-local without exhausting per-node resources.

daenz 4y ago

Lambda/Cloud Functions are starting to converge on this idea. It will eventually get streamlined and ergonomic enough that it appears you're executing an expensive or async function locally, except you aren't.

SteadBytes 4y ago
"From laptop to lambda"[0] from a couple of years ago is an interesting step in this direction. [0]https://dl.acm.org/doi/10.5555/3358807.3358848 https://dl.acm.org/doi/10.5555/3358807.3358848

a-dub 4y ago

this is a fun hack. it would be interesting to look at some real world workloads and compare whether this sort of init once ship initialized memory image everywhere style is faster than just initializing everywhere.

mghfreud 4y ago

isn’t this exactly what the vm migration in cloud is?

mlyle 4y ago
No. VM migration moves entire virtual computers. Forking makes a copy of a process with the current state; this moves that single duplicated process to a different machine.
- mghfreud 4y ago
  Virtual computer is a bunch of processes.
  - speed_spread 4y ago
    And a kernel. And drivers. And devices. And busses. And interrupts.
  - mlyle 4y ago
    The word "entire" is in there.
- sitkack 4y ago
  Is Star Trek's Transporter Actually a Murder Machine? https://www.youtube.com/watch?v=f-8zEkIaB0c https://www.youtube.com/watch?v=f-8zEkIaB0c
sitkack 4y ago
Yes. All live migration systems basically follow the same pattern https://cloud.google.com/compute/docs/instances/live-migration https://cloud.google.com/compute/docs/instances/live-migrati... And as others have mentioned upthread, Mosix was an extension to Linux that also implemented fork() in such a way that the child could be local or remote with file handles retained across the cluster. We had a Linux lan party when I was in college and managed to scrape together a Mosix cluster across a bunch of machines. Ignore the uncreative pedants.

felixgallo 4y ago

Always worth a reread: https://joearms.github.io/published/2013-11-21-My-favorite-erlang-program.html https://joearms.github.io/published/2013-11-21-My-favorite-e...

eointierney 4y ago
<About a year later I had to write a paper. One of the disadvantages of being a researcher is that in order to get money you have to write a paper about something or other, the paper is never about what currently interests you at the moment, but must be about what the project that financed your research expects to read about. Well I had my gossip network setup on planet lab and I could tell it to become anything, so I told it to become a content distribution networks and used a gossip algorithm to make copies of the same file on all machine on the network and wrote a paper about it and everybody was happy.> I miss Joe, not that I ever met him, but his attitude and good humour are inspiring.
- ungamedplayer 4y ago
  He was just as good of a man in real life.
ryukafalz 4y ago
Wow, thanks for posting this! I've never used Erlang before, but the "become" functionality and the message-passing style is similar enough to another actor-model system I'm familiar with (Spritely Goblins) that I was inspired to write a version of this program for it: https://git.sr.ht/~jfred/goblins-test/tree/master/item/universal.rkt https://git.sr.ht/~jfred/goblins-test/tree/master/item/unive...

AaronFriel 4y ago

The Cloud Haskell project and language is likely the only one to get this right, thanks to strictly enforced purity. It's much simpler to understand, absence global mutable state, whether it's safe and possible to serialize a closure and run it somewhere else. (fork(2) being a closure by another name.) In almost all other languages there's just no way to know if a closure is holding on to a file descriptor. Critics may say the Haskell closures could contain `unsafePerformIO`, but as the saying goes: now you have two problems.

gpderetta 4y ago
Isn't fork more of a continuation? /pedantic
gnufx 4y ago
Is Cloud [sigh] Haskell still alive? For comparison, two old distributed lexical scope systems were Cardelli's Obliq and Kelsey's(?) Kali Scheme. From what I remember, not like remote forking, though.
- AaronFriel 4y ago
  I don't think so, sadly.

gnufx 4y ago

For what it's worth, at least for HPC-ish distributed computing, this sort of thing turns out not to be terribly worthwhile. We have a standard for distribution of computation, shared memory, i/o, and process starting in MPI (and, for instance, DMTCP to migrate the distributed application if necessary, though I think DMTCP needs a release). I don't know what its current status is, but the HPC-ish Bproc system has/had an rfork [1]. Probably the most HPC-oriented SSI system, Kerrighed died, as did the Plan 9-ish xcpu, though that was a bit different. 1. https://www.penguinsolutions.com/computing/documentation/scyld-clusterware/7/admin-guide/ref-guide/libraries/bproc/bproc-rfork.html https://www.penguinsolutions.com/computing/documentation/scy...

zozbot234 4y ago
The biggest benefit is arguably that codes that are designed for "telefork" and perhaps remote threads can also be scaled down to a single shared-memory machine, and run way more efficiently than if they had been coded using the MPI approach. Whilst you don't really add much of any overhead when running in a cluster, assuming that the codes are designed properly.
- gnufx 4y ago
  Just doing a fork may be sufficient for something embarrassingly parallel, but interesting things are tightly coupled. Obviously MPI scales down to a single node (a distributed system anyway, these days), typically as real forked processes, but possibly with all the ranks in a single processwith an appropriate implementation. Citation needed, as they say, for “run way more efficiently”, particularly as the conventional wisdom says shared memory in a single process (e.g. OpenMP). “Acknowledgements: ... NUMA and Amdahl’s Law, for holding OpenMP back and keeping MPI-only competitive in spite of the ridiculous cost of Send-Recv within a shared-memory domain.” — Jeff Hammond, ‘MPI+MPI’
bluedino 4y ago
I would kill for "vmotion" on our HPC cluster. Tired of draining a node then pestering the remaining users to log their jobs out.

fleddr 4y ago

In the late 90s I attended a tour at Holland Signaal, a dutch defense company producing radar and anti-missile systems. I remember vividly how they demonstrated an unbreakable process. They had a computer running a process and no matter what happened to that computer, the next one would flawlessly continue the process down to the cycle, with no change or corruption or skipping a beat. It may very well be that this is actually not very difficult, but it seemed difficult and impressive. Perhaps more shocking were ultra high resolution radar screens, some 3 generations ahead of anything I had seen in the consumer space, showing an incredible visualization of the air space, live. Showing exactly which plane is where, the model/type, age, fuel on board, hostile/friendly, all of it. They even had a "situation room" with a holodeck chair in the middle, full of controls. The entire room was covered in wall-size screens basically showing the air space of the entire country, being live analyzed. Sounds very 2022, not 1998.

zamadatix 4y ago
Of all of the amazing things I've learned/seen in IT over the years the one that had the greatest "jaw to floor" effect was when I first learned of VMware's Fault Tolerance feature (shortly after I learned about VMs and live VM migration, which was the previous "jaw to floor" winner). Fault Tolerance allows for a very similar cycle accurate continuation concept but trades off specialized synchronous design and near instant failover for a ~1 second hitch to gain the ability to do this with a non-specialized VM and software stack. For even more fun if you have more than 2 hosts when 1 is lost it'll rebuild a fault tolerant state automatically with another host.
- fleddr 4y ago
  That's so cool, I'm only a casual VMWare user and had no idea. With things like this possible, and possible for a long time now, it honestly makes me cringe at how absolutely terrible (slow, inefficient, unreliable, complex, not robust) the typical modern software stack is.
  - xwolfi 4y ago
    It's not true ! The typical software stack is optimized with need as it arises You wouldnt want to write ultrafast software for your idea to deliver cats back from the vets with a network of teenagers om bicycle. Imagine, years to write your website in C++/ASM to end up failing the first week. What s embarrassing is how much software cost to build in the 80s.

tenken 4y ago

Doesn't Erlang support these ideas of distributed computing .... And if I recall correctly Clipper supported remote execution of objects, or sharing object code in a distributed fashion.

linux2647 4y ago
Yep. spawn is the basic primitive: https://www.erlang.org/doc/man/erlang.html#spawn-4 https://www.erlang.org/doc/man/erlang.html#spawn-4

kristjansson 4y ago

Off-topic, but google-fu is failing me: wasn't there a similar project posted here a while back, that instead shipped a binary + a virtual file system that mirrored all local resources to a remote host so that the entire process could run there? IIRC the author demonstrated it running an ffmepg transcode? edit: Got it, after a while: https://github.com/Overv/outrun https://github.com/Overv/outrun

sitkack 4y ago
This is cool, when it talks about packaging, I was reminded of statifier. It would take a dynamically linked ELF executable and package everything up into a single binary. I last used in the early 2000s, but it looks like it had an update as recently as 2016. https://sourceforge.net/projects/statifier/ https://sourceforge.net/projects/statifier/

paulgb 4y ago

I've been playing with adjacent ideas in WebAssembly. The idea is that you can write a function that sends and receives messages of arbitrary Rust types, compile it into WebAssembly, and then snapshot and restore it (including across space and time) as you like. It's very experimental, but I got as far as being able to freeze a JavaScript interpreter (compiled to WebAssembly) and restore it later. The code is here: https://github.com/drifting-in-space/wasmbox https://github.com/drifting-in-space/wasmbox

sitkack 4y ago
Me too! The entire heap of the Wasm env is up for inspection and mutation. If your data structures are stable, then they can be moved across envs, or use accessors from both sides. Fun stuff.
- paulgb 4y ago
  Yeah! I think it’s an under-appreciated aspect of WebAssembly.

aag 4y ago

Termite Scheme did something like this for Scheme programs. It combines Erlang-style concurrency with process migration. http://www.iro.umontreal.ca/~feeley/papers/GermainFeeleyMonnierELSW05.pdf http://www.iro.umontreal.ca/~feeley/papers/GermainFeeleyMonn...

OJFord 4y ago

(2020) and discussed at the time here: https://news.ycombinator.com/item?id=22987747 https://news.ycombinator.com/item?id=22987747

megamindbrian2 4y ago

samsquire 4y ago

I also had this idea but I called it networkfork.

bediger4000 4y ago

I don't think anyone has mentioned Sprite https://en.m.wikipedia.org/wiki/Sprite_(operating_system) https://en.m.wikipedia.org/wiki/Sprite_(operating_system) Sprite called this "process migration".

cryptonector 4y ago

You could do what CRIU does, but leave the original process running, and arrange to have a composite {instance ID, PID} as the return value (different on both sides). (Trivia: in Unix VI, fork() returns both PIDs, and the C library stub arranges to return 0 to the child and the child's PID to the parent.)

phs318u 4y ago

I can't state with absolute certainty but I seem to recall OpenVMS clusters back in the 90's transparently supporting "portable" processes across cluster nodes.

innocentoldguy 4y ago

If I am understanding the article correctly, BEAM does this for Elixir and Erlang (and other BEAM languages) out of the box.

tashbarg 4y ago
No, it does not. Not _this_. Erlang has, however, excellent support for distributed computing using its own kind of processes.

midislack 4y ago

This is really cool, Distributed Unix could do this IIRC.

rcarmo 4y ago

This is fun. Reminded me of mincemeat (https://github.com/ziyuang/mincemeatpy https://github.com/ziyuang/mincemeatpy), which actually serializes the mapping functions and sends them to workers.

deleted 4y ago

[deleted]