4 ms·

Congratulations! You have just reinvented the core idea of UCLA's LOCUS distributed computing project from 1979. https://en.wikipedia.org/wiki/LOCUS https://en.

Congratulations! You have just reinvented the core idea of UCLA's LOCUS distributed computing project from 1979. https://en.wikipedia.org/wiki/LOCUS https://en.wikipedia.org/wiki/LOCUS Reinventing LOCUS also has a strong heritage. Bell Lab's Plan 9, for example, did so in part in the late 1980s. While never a breakout commercial success, tele-forking and its slightly more advanced cousins machine-to-machine process migration and cluster-wide process pools intrigued some of the best minds in distributed computing for 20+ years. Unfortunately "it's complicated" to implement well, especially when you try to tele-spawn and manage resources beyond compute cycles (network connections, files, file handles, ...) that are important to scale up the idea.

zozbot234 4y ago
> Unfortunately "it's complicated" to implement well, especially when you try to tele-spawn and manage resources beyond compute cycles (network connections, files, file handles, ...) Aren't all of these resources namespaced/containerized in modern Linux? This should make it feasible to checkpoint and restore them on the same machine (via, e.g. the CRIU patchset) and true location-independence is not that much harder. One of the hardest parts (not even implemented in plan9, AFAICT) is distributed shared memory (allowing for sharing a single virtual address space across cluster nodes), but even that AIUI has some research-level implementations.
- convolvatron 4y ago
  you still have to migrate the kernel state of underlying resources, so containers don't really buy you very much. as you say, in checkpoint restart you start with the big stuff like tcp sockets, but after a while you get tired of tracking down little flecks of state. distributed shared memory though does deserve a new look now that baseline network speeds are 100x what the max was when it was first investigated. unfortunately temporal consistency is still going to be a major factor. some workloads will run great with some heuristics, and some won't. you'll almost certainly need to migrate threads along with pages to try to keep them running co-local without exhausting per-node resources.
  - VWWHFSfQ 4y ago
    > you still have to migrate the kernel state of underlying resources DragonflyBSD has been experimenting with this concept for several years now. At the moment it's mostly only useful for snapshotting the kernel for debugging purposes. But there's no reason why it couldn't be extended to transport the state of the computer to another computer and resume execution! https://www.dragonflybsd.org/docs/handbook/vkernel/ https://www.dragonflybsd.org/docs/handbook/vkernel/
- emmelaich 4y ago
  https://criu.org/ https://criu.org/ indicates it supports migration already; at least at the container level.
gowld 4y ago
Also , Haskell Control.Distributed.Fork > This module provides a common interface for offloading an IO action to remote executors. > It uses StaticPointers language extension and distributed-closure library for serializing closures to run remotely. This blog post[1] is a good introduction for those. > In short, if you you need a Closure a: > One important constraint when using this library is that it assumes the remote environment is capable of executing the exact same binary. On most cases, this requires your host environment to be Linux. https://hackage.haskell.org/package/distributed-fork-0.0.1.3/docs/Control-Distributed-Fork.html https://hackage.haskell.org/package/distributed-fork-0.0.1.3... [1] https://blog.ocharles.org.uk/blog/guest-posts/2014-12-23-static-pointers.html https://blog.ocharles.org.uk/blog/guest-posts/2014-12-23-sta...
- CMCDragonkai 4y ago
  Unless your closure closed over the entire operating system.
  - nine_k 4y ago
    With a pure language like Haskell, and the now-standard "functional core, imperative shell" approach, it's rather easy to limit the extent of external dependencies of a meaningful computation.
Terry_Roll 4y ago
I would image the process of pausing virtual machines and then loading them on other machines is something similar.
- deleted 4y ago
  [deleted]
cbsmith 4y ago
...and many, many more distributed systems. It's basically what PVM was all about. These days most of the work for this is handled by the CRIU project (https://criu.org/ https://criu.org/), so the "hard" work has been done.
daniel-cussen 4y ago
Hey if they just came up with the question and struggled at the answer, that's half the battle. It's not all just solving open problems, it's also opening new problems. There was a sociology paper that found 100% of great scientists it reviewed said questions were more important than answers. (Then the paper went on to make some other point with a more impressive higher margin of error, p-hacking upward, 0% error was too low.)
hinkley 4y ago
Berkeley Sprite, ~1991: https://www2.eecs.berkeley.edu/Research/Projects/CS/sprite/sprite.html https://www2.eecs.berkeley.edu/Research/Projects/CS/sprite/s... Everything that wasn't immediately successful is tried again in 15 years. This is why the old farts on your project are grumpy about 'new' things that... aren't.
- hdjjhhvvhga 4y ago
  Yes, this is exactly what is happening again and again. I remember when mSQL (and later MySQL)+PHP was all the rage and everybody was writing CMSes with it (could be late 90s) and people routinely complained about efficiency, it occurred to me: since these are mostly reads, and writes are quite rare, why don't we regenerate the necessary elements only on write, and not whenever a user visits a page? I created a simple proof of concept with multiple themes and it worked perfectly. Fast forward 15 years and static site generators were all the rage. I believe they will resurge again around 2030.
  - rubyfan 4y ago
    I took that approach with a high traffic site about 10 years ago. The property was mostly write once content from official staff writers with social engagement, comments, follows and light user generated content, etc. It was built originally with Ruby on Rails and over time had lots of layers of caching. Eventually I moved to a system of partial caching which was stored on the file system and nginx would use SSI to reassemble the partials into complete pages. All dynamic content would be loaded secondarily via javascript with mustache templates and lots of in memory json caches. It would even rsync the file system across a handful of hosts which turned out to be a pain in the butt - then distributed file systems were probably less mature than they are now so I never ended up implementing but did much research. Funny, I think probably all of the above would be trivial now on AWS or Cloudflare.
  - hinkley 4y ago
    My job satisfaction dropped like a rock the day I realized we're still basically implementing CMSes over, and over, and over again. I think the 'all apps will evolve until they can send email' axiom is the wrong one. Everything turns into content management. And I don't think you'll have to wait until 2030. I suspect 2027 ±2 years.
RandomBK 4y ago
There's elements of this also represented in GNU Parallel, a really nifty utility that lets you create a fork-join pool out of one or many systems. IMO it is severely underused these days.
folli 4y ago
So if I don't want to reinvent the wheel, what would I use nowadays to approach this problem?
catern 4y ago
>Bell Lab's Plan 9, for example, did so in part in the late 1980s. Plan 9 did not support forking to remote machines and was not a single system image system. Its support for running a program on a remote machine was a completely userspace program. If you say Plan 9 was an example of teleforking, then so is ssh. That might be a reasonable categorization, but both Plan 9 and ssh are very different from LOCUS and other SSI systems.

zozbot234 4y ago

> Unfortunately "it's complicated" to implement well, especially when you try to tele-spawn and manage resources beyond compute cycles (network connections, files, file handles, ...) Aren't all of these resources namespaced/containerized in modern Linux? This should make it feasible to checkpoint and restore them on the same machine (via, e.g. the CRIU patchset) and true location-independence is not that much harder. One of the hardest parts (not even implemented in plan9, AFAICT) is distributed shared memory (allowing for sharing a single virtual address space across cluster nodes), but even that AIUI has some research-level implementations.

convolvatron 4y ago
you still have to migrate the kernel state of underlying resources, so containers don't really buy you very much. as you say, in checkpoint restart you start with the big stuff like tcp sockets, but after a while you get tired of tracking down little flecks of state. distributed shared memory though does deserve a new look now that baseline network speeds are 100x what the max was when it was first investigated. unfortunately temporal consistency is still going to be a major factor. some workloads will run great with some heuristics, and some won't. you'll almost certainly need to migrate threads along with pages to try to keep them running co-local without exhausting per-node resources.
- VWWHFSfQ 4y ago
  > you still have to migrate the kernel state of underlying resources DragonflyBSD has been experimenting with this concept for several years now. At the moment it's mostly only useful for snapshotting the kernel for debugging purposes. But there's no reason why it couldn't be extended to transport the state of the computer to another computer and resume execution! https://www.dragonflybsd.org/docs/handbook/vkernel/ https://www.dragonflybsd.org/docs/handbook/vkernel/
emmelaich 4y ago
https://criu.org/ https://criu.org/ indicates it supports migration already; at least at the container level.

gowld 4y ago

Also , Haskell Control.Distributed.Fork > This module provides a common interface for offloading an IO action to remote executors. > It uses StaticPointers language extension and distributed-closure library for serializing closures to run remotely. This blog post[1] is a good introduction for those. > In short, if you you need a Closure a: > One important constraint when using this library is that it assumes the remote environment is capable of executing the exact same binary. On most cases, this requires your host environment to be Linux. https://hackage.haskell.org/package/distributed-fork-0.0.1.3/docs/Control-Distributed-Fork.html https://hackage.haskell.org/package/distributed-fork-0.0.1.3... [1] https://blog.ocharles.org.uk/blog/guest-posts/2014-12-23-static-pointers.html https://blog.ocharles.org.uk/blog/guest-posts/2014-12-23-sta...

CMCDragonkai 4y ago
Unless your closure closed over the entire operating system.
- nine_k 4y ago
  With a pure language like Haskell, and the now-standard "functional core, imperative shell" approach, it's rather easy to limit the extent of external dependencies of a meaningful computation.

Terry_Roll 4y ago

I would image the process of pausing virtual machines and then loading them on other machines is something similar.

deleted 4y ago
[deleted]

cbsmith 4y ago

...and many, many more distributed systems. It's basically what PVM was all about. These days most of the work for this is handled by the CRIU project (https://criu.org/ https://criu.org/), so the "hard" work has been done.

daniel-cussen 4y ago

Hey if they just came up with the question and struggled at the answer, that's half the battle. It's not all just solving open problems, it's also opening new problems. There was a sociology paper that found 100% of great scientists it reviewed said questions were more important than answers. (Then the paper went on to make some other point with a more impressive higher margin of error, p-hacking upward, 0% error was too low.)

hinkley 4y ago

Berkeley Sprite, ~1991: https://www2.eecs.berkeley.edu/Research/Projects/CS/sprite/sprite.html https://www2.eecs.berkeley.edu/Research/Projects/CS/sprite/s... Everything that wasn't immediately successful is tried again in 15 years. This is why the old farts on your project are grumpy about 'new' things that... aren't.

hdjjhhvvhga 4y ago
Yes, this is exactly what is happening again and again. I remember when mSQL (and later MySQL)+PHP was all the rage and everybody was writing CMSes with it (could be late 90s) and people routinely complained about efficiency, it occurred to me: since these are mostly reads, and writes are quite rare, why don't we regenerate the necessary elements only on write, and not whenever a user visits a page? I created a simple proof of concept with multiple themes and it worked perfectly. Fast forward 15 years and static site generators were all the rage. I believe they will resurge again around 2030.
- rubyfan 4y ago
  I took that approach with a high traffic site about 10 years ago. The property was mostly write once content from official staff writers with social engagement, comments, follows and light user generated content, etc. It was built originally with Ruby on Rails and over time had lots of layers of caching. Eventually I moved to a system of partial caching which was stored on the file system and nginx would use SSI to reassemble the partials into complete pages. All dynamic content would be loaded secondarily via javascript with mustache templates and lots of in memory json caches. It would even rsync the file system across a handful of hosts which turned out to be a pain in the butt - then distributed file systems were probably less mature than they are now so I never ended up implementing but did much research. Funny, I think probably all of the above would be trivial now on AWS or Cloudflare.
- hinkley 4y ago
  My job satisfaction dropped like a rock the day I realized we're still basically implementing CMSes over, and over, and over again. I think the 'all apps will evolve until they can send email' axiom is the wrong one. Everything turns into content management. And I don't think you'll have to wait until 2030. I suspect 2027 ±2 years.

RandomBK 4y ago

There's elements of this also represented in GNU Parallel, a really nifty utility that lets you create a fork-join pool out of one or many systems. IMO it is severely underused these days.

folli 4y ago

So if I don't want to reinvent the wheel, what would I use nowadays to approach this problem?

catern 4y ago

>Bell Lab's Plan 9, for example, did so in part in the late 1980s. Plan 9 did not support forking to remote machines and was not a single system image system. Its support for running a program on a remote machine was a completely userspace program. If you say Plan 9 was an example of teleforking, then so is ssh. That might be a reasonable categorization, but both Plan 9 and ssh are very different from LOCUS and other SSI systems.