13 ms·

Teleforking a Process onto a Different Computer

572 points by trishume 6y ago

anthk 6y ago
What I'd love it's bingding easily remote directories as local. Not NFS, but a braindead 9p. If I don't have a tool, I'd love to have a bind-mount of a directory from a stranger, and run a binarye from within (or piping it) without he being able to trace the I/O. If the remote FS is a diff arch, I'd should be able to run the same binary remotely as a fallback option, seamless.
londons_explore 6y ago
Bonus points if you can effectively implement the "copy on write" ability of the linux kernel to only send over pages to the remote machine that are changed either in the local or remote fork, or read in the remote fork. A rsync-like diff algorithm might also substantially reduce copied pages if the same or a similar process is teleforked multiple times. Many processes have a lot of memory which is never read or written, and there's no reason that should be moved, or at least no reason it should be moved quickly. Using that, you ought to be able to resume the remote fork in milliseconds rather than seconds. userfaultfd() or mapping everything to files on a FUSE filesystem both look like promising implementation options.
- trishume 6y ago
  I do in fact mention this idea in the article. In fact userfaultfd was added to the kernel so that CRIU and KVM live migration could implement exactly this. Another cool project that does something like this is https://github.com/gamozolabs/chocolate_milk https://github.com/gamozolabs/chocolate_milk which is a fuzzing hypervisor kernel which can back a VM snapshot memory mapping over the network to only pull down the pages that the VM actually reads during the fuzz case.
- mlyle 6y ago
  If you just pull things on demand, you're going to get a lot of round-trip-time penalties to page things in. I think you should still be pushing the memory as fast as you can, but maybe you start the child while this is still in progress, and prioritize sending stuff the child asks for (reorder to send that stuff "next"), if you've not already sent it.
  - lunixbochs 6y ago
    That’s a great idea. One of my thoughts was to “pre-heat” the process by executing a bit locally with side effects disabled to see what would get immediately accessed and send that first. If your systems strictly match somehow (machine image with auto update disabled? or regularly hash and timestamp files on both systems) you can also cheat by mapping some of the files locally on the other side.
  - trishume 6y ago
    Yah that is indeed a super important optimization for avoiding round trips. CRIU does this and calls it "pre-paging", their wiki also mentions that they adapt their page streaming to try to pre-stream pages around pages that have been faulted: https://en.wikipedia.org/wiki/Live_migration#Post-copy_memory_migration https://en.wikipedia.org/wiki/Live_migration#Post-copy_memor... edit: lol I didn't realized that isn't CRIU's wiki since they just linked to a Wikipedia page and both use WikiMedia software. This is the actual CRIU wiki page, and it's way harder to tell if they do this, although I suspect they do and it's in the "copy images" step of the diagram https://criu.org/Userfaultfd https://criu.org/Userfaultfd
- deleted 6y ago
  [deleted]
- touisteur 6y ago
  If you're at the hypervisor level you can also use Intel PML, seems it's made for this. https://arxiv.org/abs/2001.09991 https://arxiv.org/abs/2001.09991 I'm guessing @gamozolabs (twitter) is going to use it some day soon to be fuzz billions of cases per second, with snapshot fuzzing...
- juancampa 6y ago
  If you ever needed to bring the process back, you could use soft-dirty-bit[1] to determine which pages were modified since forking and only transfer those. CRIU uses it for incremental snapshots (in fact, they wrote the kernel patch afaik) [1] https://www.kernel.org/doc/Documentation/vm/soft-dirty.txt https://www.kernel.org/doc/Documentation/vm/soft-dirty.txt
anticensor 6y ago
hoard() would be a better name.
- jabedude 6y ago
  I don't understand, could you explain? `tfork(2)` seems more in line with Linux naming conventions.
  - anticensor 6y ago
    Telefork as a verb does not map to a real word concept. Hoard fits better, as in hoarding another computer for your process.
    - carapace 6y ago
      Ha! I thought you had misspelled horde! ;-)
      - buckminster 6y ago
        I think you mishurd.
    - jabedude 6y ago
      That is funny. hoard() and steal() could work.
      - anticensor 6y ago
        Agree, steal() instead of telepad() would be a good idea.
zozbot234 6y ago
The page states that CRIU requires kernel patches, but other sources say that the kernel code for CRIU is already in the mainline kernel. What's up with that?
- trishume 6y ago
  I didn't say it requires kernel patches, just that it requires a build flag and new enough versions. It does happen that most distros do enable that build flag though. There is a patch set for write protection support in userfaultfd that was just merged recently and I'm not sure if it's made it into any actual releases yet.
- cyphar 6y ago
  CRIU does work on mainline kernels and has for several years. However, it is still being actively developed (userfaultfd is the example the article uses -- a mainline feature which is still seeing lots of development). It doesn't usually require additional patches to work, though they do have a fairly big job (every new kernel feature needs to be added to CRIU so it can both checkpoint and restore it -- and not all features are trivial to checkpoint). It's entirely possible that programs using very new kernel features may struggle to get CRIU to work.
- en4bz 6y ago
  CRIU can use two mechanisms to detect page changes. One is the soft-dirty kernel feature which is mainlined and can be accessed via /proc/PID/pagemap [1]. The other is userfaultfd which is only partially merged in the newest kernel. userfaultfd lacks write detection which the article mentioned. My understanding is that using pagemap requires the entire process to be frozen while it is scanned for memory changes and the memory is copied while uffd allows for a more streaming/on-demand approach that doesn't require stopping the entire process. [1] https://www.kernel.org/doc/Documentation/vm/soft-dirty.txt https://www.kernel.org/doc/Documentation/vm/soft-dirty.txt
saagarjha 6y ago
It’s touched on at the very end, but this kind of work is somewhat similar to what the kernel needs to do on a fork or context switch, so you can really figure out what state you need to keep track of from there. Once you have that, scheduling one of these network processes isn’t really all that different than scheduling a normal process, except the of course syscalls on the remote machine will possibly go to a kernel that doesn’t know what to do with them.
new_realist 6y ago
See https://criu.org/Live_migration https://criu.org/Live_migration
ISL 6y ago
What's old is new again -- I'm pretty sure QNX could do this in the 1990s. QNX had a really cool way of doing inter-process communication over the LAN that worked as if it were local. Used it in my first lab job in 2001. You might not find it on the web, though. The API references were all (thick!) dead trees. Edit: Looks like QNX4 couldn't fork over the LAN. It had a separate "spawn()" call that could operate across nodes. https://www.qnx.com/developers/docs/qnx_4.25_docs/qnx4/sysarch/proc.html https://www.qnx.com/developers/docs/qnx_4.25_docs/qnx4/sysar...
- checker659 6y ago
  > I'm pretty sure QNX could do this in the 1990s. Plan 9, SmallTalk
  - imglorp 6y ago
    Erlang of course. All spawns and messages can be local or remote.
    - masklinn 6y ago
      Indeed, weird to only find it that low as a sub-comment: spawn(Node, Module, Function, Args) -> pid() Returns the process identifier (pid) of a new process started by the application of Module:Function to Args on Node. If Node does not exist, a useless pid is returned. Otherwise works like spawn/3.
    - russellbeattie 6y ago
      Any time I see something about remote processes, I immediately think, "Erlang could probably do this." I think Erlang would have been the programming language of the 21st century... If only the syntax wasn't like line noise and a printer error code had a baby, and raised it to think like Lisp.
      - sosodev 6y ago
        Elixir is pretty cool. You get all the power of Erlang wrapped up in a much more developer friendly language.
synack 6y ago
This reminds me of OpenMOSIX, which implemented a good chunk of POSIX in a distributed fashion. MPI also comes to mind, but it's more focused on the IPC mechanisms. I always liked Plan 9's approach, where every CPU is just a file and you execute code by writing to that file, even if it's on a remote filesystem.
- anfractuosity 6y ago
  I recall OpenMosix too, I remember playing with it on a couple of old machines a while ago. I thought it seemed quite a cool idea. I think I was trying to do mp3 encoding from CDs with it in a distributed fashion.
- synack 6y ago
  Ah, the Plan 9 bits I was remembering are actually in Inferno... http://www.vitanuova.com/inferno/man/4/grid-cpu.html http://www.vitanuova.com/inferno/man/4/grid-cpu.html
- notacoward 6y ago
  Same reaction here. MOSIX was basically this idea developed for real. Turns out there are a bunch of secondary problems to do with namespaces and security, scheduling and resource use, topology, performance, and so on. In the end it never turned out to be all that practical, and I know of several efforts (e.g. LOCUS) that went quite far to reach the same conclusion. There's a reason that non-transparent "shared nothing" distributed computing came to dominate the computing landscape.
- adrianpike 6y ago
  Whoa, I was just talking about OpenMOSIX the other day - at my college job, when we retired workstations, they would just sit in the back closet for years until facilities would get around to getting them up for auction. We set up a TFTP boot setup and had a few dozen nodes at any given time. It wasn't high performance by any fashion, but it worked pretty transparently and was always fun to throw heavy synthetic workloads at it and watch the cluster rebalance and hear the fans spin up on the clunky old pentium's.
- dghughes 6y ago
dekhn 6y ago
Condor, a distributed computing environment, has done IO remoting (where all calls to IO on the target machine get sent back to the source) for several decades. The origin of Linux Containers was process migration. I believe people have found other ways to do this, personally I think the ECS model (like k8s, but the cloud provider hosts the k8s environment) where the user packages up all the dependencies and clearly specifies the IO mechanisms through late biniding, makes a lot more sense for distributed computing.
- vidarh 6y ago
  I clicked through to mention Condor too... I first came across it in the 90's, and it seems like one of those obvious hacks that keeps being reinvented.
  - dekhn 6y ago
    I was actually channeling the creator of Condor, Miron Livny, who has a history of going to talks about distributed computing and pointing out that "Condor already does that" for nearly everything that people try to tout as new and cool. He's not wrong, but few people use Condor.
    - iaresee 6y ago
      Few people outside academia, maybe? But inside it still seems to dominate in areas like physics computation. CERN uses a world wide Condor grid. LIGO too. It’s excellent for sharing cycles for those slow-burn, highly parallel, massive data scale problems. I spent more than a decade bringing Condor to semi-conductor, financial and biomedical institutions. It was always a fight to show them there was a better way to utilize their massive server farms that didn’t require paying the LSF Tax. Without a shiny sales or marketing department, Condor was hard to pitch to IT departments. Still, to this day, I see people doing things with “modern” platforms like Kubernetes and such and I chuckle. Had that in Condor 15 years ago in many cases. :)
      - 0xdeadbeefbabe 6y ago
touisteur 6y ago
I wonder whether the effort by the syzkaller people (@dyukov) could help with the actual description of all the syscalls (that the author says people gave up on for now, because too complex), since they need them to be able to fuzz efficiently...
carapace 6y ago
"Somebody else has had this problem." Don't get me wrong, this is great hacking and great fun. And this is a good point: > I think this stuff is really cool because it’s an instance of one of my favourite techniques, which is diving in to find a lesser-known layer of abstraction that makes something that seems nigh-impossible actually not that much work. Teleporting a computation may seem impossible, or like it would require techniques like serializing all your state, copying a binary executable to the remote machine, and running it there with special command line flags to reload the state.
fitzn 6y ago
Really cool idea! Thanks for providing so much detail in the post. I enjoyed it. A somewhat related project is the PIOS operating system written 10 years ago but still used today to teach the operating systems class there. The OS has different goals than your project but it does support forking processes to different machines and then deterministically merging their results back into the parent process. Your post remind me of it. There's a handful of papers that talks about the different things they did with the OS, as well as their best paper award at OSDI 2010. https://dedis.cs.yale.edu/2010/det/ https://dedis.cs.yale.edu/2010/det/
hawski 6y ago
Seriously cool. That also reminds me of DragonFlyBSD's process checkpointing feature that offers suspend to disk. In Linux world there were many attempts, but AFAIK nothing simple and complete enough. To be fair I don't know if DF's implementation is that either. https://www.dragonflybsd.org/cgi/web-man?command=sys_checkpoint&section=2 https://www.dragonflybsd.org/cgi/web-man?command=sys_checkpo... https://www.dragonflybsd.org/cgi/web-man?command=checkpoint&section=ANY https://www.dragonflybsd.org/cgi/web-man?command=checkpoint&...
- jabedude 6y ago
  This article[0] on LWN seems to suggest that Linux has no kernel support for checkpoint/restore because it's hard to implement (no arguments there). But hypervisors support checkpoint/restore for virtual machines, e.g. ESXi VMotion and KVM live migration, so it seems like these technical problems are solvable. Indeed all the benefits of VM migration seem to also apply to process migration (load balancing, service uptime, etc). 0. https://lwn.net/Articles/293575/ https://lwn.net/Articles/293575/
  - justapassenger 6y ago
    IMO, big reason for why Linux doesn’t support it, is that big tech companies, who drive a lot of development and funding, found that it’s just not worth it. Tech moved away from mainframe model of keeping single processes up for as long as possible and isolating software from the failures. Instead they embrace failure at the software layer, which gives you (when executed correctly) both same high level of availability, but also layers of protection against other issues (as node going down in expected way is the same, no matter what’s the root cause) and makes overall maintainability and upgradability systems higher. Basically, why deal with complicated checkpoints at kernel level, that doesn’t understand your software, when you ca instead deal with that at the software layer, with full control over it.
    - spullara 6y ago
      Swap performance has been similarly underinvested.
abotsis 6y ago
Also of interest might be Sprite- a Berkeley research os developed “back in the day” by Ken Shirriff And others. It boasted a lot of innovations like a logging filesystem (not just metadata) and a distributed process model and filesystem allowing for live migrations between nodes. https://www2.eecs.berkeley.edu/Research/Projects/CS/sprite/sprite.html https://www2.eecs.berkeley.edu/Research/Projects/CS/sprite/s...
- kens 6y ago
  To be clear, John Ousterhout was the creator of the Sprite operating system. I was one of the grad students working on it.
lachlan-sneff 6y ago
Wow, this is really interesting. I bet that there's a way of doing this robustly by streaming wasm modules instead of full executables to every server in the cluster.
deleted 6y ago
[deleted]
jka 6y ago
This reminds me a little bit of the idea of 'Single System Image'[1] computing. The idea, in abstract, is that you login to an environment where you can list running processes, perform filesystem I/O, list and create network connections, etc -- and any and all of these are in fact running across a cluster of distributed machines. (in a trivial case that cluster might be a single machine, in which case it's essentially no different to logging in to a standalone server) The wikipedia page referenced has a good description and a list of implementations; sadly the set of {has-recent-release && is-open-source && supports-process-migration} seems empty. [1] - https://en.wikipedia.org/wiki/Single_system_image https://en.wikipedia.org/wiki/Single_system_image
- macintux 6y ago
  That was the original concept that led Ian Murdock and John Hartman to found Progeny. The idea was that overnight, while no one was working at their desks, companies could reboot their Windows boxes into a SSI network of Linux nodes to run parallel compute tasks. Roughly, anyway, I got the sales pitch 20 years ago so my memories are fuzzy. I wasn’t remotely sold on it but was so anxious to work for a Linux R&D company in Indianapolis of all places that I accepted the job anyway. Sadly we didn’t get far on the concept before the dot-com crash. Absent more venture capital we pivoted to focus on something we could sell, Progeny Linux, and tried to turn that into a managed platform for companies who wanted to run Linux on their appliances.
pcr910303 6y ago
This makes me think about Urbit[0] OS — Urbit OS represents the entire OS state in a simple tree, this would be very simple to implement. [0]: https://urbit.org/ https://urbit.org/
deleted 6y ago
[deleted]
Animats 6y ago
That goes back to the 1980s, with UCLA Locus. This was a distributed UNIX-like system. You could launch a process on another machine and keep I/O and pipes connected. Even on a machine with a different CPU architecture. They even shared file position between tasks across the network. Locus was eventually part of an IBM product. A big part of the problem is "fork", which is a primitive designed to work on a PDP-11 with very limited memory. The way "fork" originally worked was to swap out the process, and instead of discarding the in-memory copy, duplicate the process table entry for it, making the swapped-out version and the in-memory version separate processes. This copied code, data, and the process header with the file info. This is a strange way to launch a new process, but it was really easy to implement in early Unix. Most other systems had some variant on "run" - launch and run the indicated image. That distributes much better.
- wrs 6y ago
  No need to use the past tense — CreateProcess is the primitive in Windows NT. (It's been 20 years...can we just call that Windows now?)
  - zozbot234 6y ago
    posix_spawn is a thing as well.
    - nwmcsween 6y ago
      posix_spawn is horrendous
- barrkel 6y ago
  There's also an ergonomics to process creation APIs - rather than needing separate APIs for manipulating your child process vs manipulating your own process, fork() lets you use one to implement the other: fork(), configure the resulting process, then exec(). CreateProcess* on Windows is a relative monstrosity of complexity compared to fork/exec.
- slobiwan 6y ago
0xbadcafebee 6y ago
It's nice to see people re-discover old school tech. In cluster computing this was generally called "application checkpointing"[1] and it's still in use in many different systems today. If you want to build this into your app for parallel computing you'd typically use PVM[2]/MPI[3]. SSI[4] clusters tried to simplify all this by making any process "telefork" and run on any node (based on a load balancing algorithm), but the most persistent and difficult challenge was getting shared memory and threading to work reliably. It looks like CRIU support is bundled in kernels since 3.11[5], and works for me in Ubuntu 18.04, so you can basically do this now without custom apps. [1] https://en.wikipedia.org/wiki/Application_checkpointing https://en.wikipedia.org/wiki/Application_checkpointing [2] https://en.wikipedia.org/wiki/Parallel_Virtual_Machine https://en.wikipedia.org/wiki/Parallel_Virtual_Machine [3] https://en.wikipedia.org/wiki/Message_Passing_Interface https://en.wikipedia.org/wiki/Message_Passing_Interface [4] https://en.wikipedia.org/wiki/Single_system_image https://en.wikipedia.org/wiki/Single_system_image [5] https://en.wikipedia.org/wiki/CRIU#Use https://en.wikipedia.org/wiki/CRIU#Use
dreamcompiler 6y ago
Telescript [0] is based on this idea, although at a higher level. I wish we could just build Actor-based operating systems and then we wouldn't need to keep reinventing flexible distributed computation, but alas...[1] [0] https://en.wikipedia.org/wiki/Telescript_(programming_language) https://en.wikipedia.org/wiki/Telescript_(programming_langua... [1] Yes I know Erlang exists. I wish more people would use it.
- peterwwillis 6y ago
  We'll keep poorly reinventing distributed computing features until we have a real distributed operating system. We're actually not that far off from it, but good luck convincing a mainline kernel dev to accept your patches.
cecilpl2 6y ago
This is similar to what Incredibuild does. It distributes compile and compute jobs across a network, effectively sandboxing the remote process and forwarding all filesystem calls back to the initiating agent.
cjbprime 6y ago
Amazing work.
- ignoramous 6y ago
  Yep. Reminds me of u/xal praising u/trishume: https://news.ycombinator.com/item?id=20490964 https://news.ycombinator.com/item?id=20490964
userbinator 6y ago
This can let you stream in new pages of memory only as they are accessed by the program, allowing you to teleport processes with lower latency since they can start running basically right away. That's what "live migration" does; it can be done with an entire VM: https://en.wikipedia.org/wiki/Live_migration https://en.wikipedia.org/wiki/Live_migration
- dmitrygr 6y ago
  Hey what's the best way to contact you? Have a question. If possible my email is in my profile.
  - sitkack 6y ago
    https://www.usenix.org/node/170864 https://www.usenix.org/node/170864
- trishume 6y ago
  It's what a certain kind of live migration strategy does. There's a discussion thread about this further down where I link to that same page.
- sitkack 6y ago
  https://cloud.google.com/compute/docs/instances/live-migration https://cloud.google.com/compute/docs/instances/live-migrati... gives a great rundown on the mechanics as they pertain to GCP. Is probably somewhat valid for other systems.
- kevinchen 6y ago
  I guess this post is a little bit different, because VMs are designed to be portable across different hosts. Even hypervisor software without live migration still lets you freeze the VM’s state to a file which can be copied to a new host. However, an already running process is not designed to be portable in the same way.
vladbb 6y ago
I implemented something similar ten years ago for a class project: https://youtu.be/0am-5noTrWk https://youtu.be/0am-5noTrWk
peterkelly 6y ago
There's been a bunch of interesting work done on this over the years. Here's a literature survey on the topic: https://dl.acm.org/doi/abs/10.1145/367701.367728 https://dl.acm.org/doi/abs/10.1145/367701.367728
crashdelta 6y ago
THIS IS REVOLUTIONARY!
- systemBuilder 6y ago
  Hardly. See my rebuttal above.
crashdelta 6y ago
This is one of the best side projects I've ever seen, hands down.
- trishume 6y ago
  <3
YesThatTom2 6y ago
Condor did this in the early 90s.
concernedctzn 6y ago
side note: take a look at this guy's other blog posts, they're all very good
TazeTSchnitzel 6y ago
In essence this is manually implementing forking — spawning a new process and copying the bytes over without getting the kernel to help you, except over a network too. It reminds me a bit of when I wanted to parallelise the PHP test suite but didn't want to (couldn't?) use fork(), yet I didn't want to substantially rewrite the code to be amenable to cleanly re-initialising the state in the right way. But conveniently, this program used mostly global variables, and you can access global variables in PHP as one magic big associative array called $GLOBALS. So I moved most of the program's code into two functions (mostly just adding the enclosing function declaration syntax and indentation, plus `global` imports), made the program re-invoke itself NPROCS times mid-way, sending its children `serialize($GLOBALS)` over a loopback TCP connection, then had the spawned children detect an environment variable to receieve the serialized array over TCP, unserialize() it and copy it into `$GLOBALS`, and call the second function… lo and behold, it worked perfectly. :D (Of course I needed to make some other changes to make it useful, but they were also similar small incisions that tried to avoid refactoring the code as much as possible.) PHP's test suite uses this horrible hack to this day. It's… easier than rewriting the legacy code…
- trishume 6y ago
  Indeed! I was talking to someone about their attempt to fork a Graal Java process and recreate all the compiler and GC threads, and I said if I had that task I'd be tempted to just use my new knowledge to implement a fork that also recreated those threads rather than trying to understand how to shut them down and restore them properly.
- i15e 6y ago
  I did something similar using the 'at' daemon's 'now' time specification to "fork" off background tasks from a web request using the same .php file. It actually worked well for what I needed at the time!
bamboozled 6y ago
I could imagine having a build system which produces a process as an artifact and then just forks it in the cloud without distributing those pesky archives!
- keeganpoppen 6y ago
  that... is a really fucking cool idea. i mean, it's of limited utility in the sense that builds really should generally work most of the time & be reproducible, so even though there is a bit of wasted effort, bandwidth, etc. in running the same build in multiple places at once, it's not like the worst thing in the world... but it just feels like there have to be cool / "useful" scenarios where you want to distribute a thing, but not the environment, etc. required to build the thing. oh wait, that already exists: it's called "a binary"... lol. iono. i mean i guess if you have a process byte-for-byte frozen in time, you're a bit more certain (provably so? maybe?) maybe that when it is resurrected on some remote host that remote process and local process (as of t_freeze) are in ~"exactly" the same state? or if you continuously snapshotted a process while it was running, and then when it crashes you (hopefully) have a snapshot from right before when it crashed that can be replayed, modified, etc. at will? but that's a bit of a stretch, and there are simpler ways of accomplishing that for the most part... the only other thing i could think of is that this could allow you to run completely diskless servers, because you can just beam all the processes in, without ever having to "install" anything, beyond copying all the shared libs, etc. which i guess could get the "1 have one of these and i want 100 of them, in the cloud" latency down by an order of magnitude or two maybe.
justicezyx 6y ago
There is some company using CRIU to implement general purpose process live migration (not specifically VMs).
systemBuilder 6y ago
I think the problem with a lot of these ideas is that the value of fork() is only marginally higher than the value of starting a fresh process with arguments on a remote machine. The complexity of moving a full process to another machine is 10 times higher than just starting a new process on the remote machine with all the binaries present already. Quite frankly, vfork only exists and gets used because it's so damned cheap to copy the pagetable entries and use copy-on-write, to save RAM. Take away the cheapness by copying the whole address space over a network, adding slowness, and nobody will be interested any more. And both techniques are inferior to having a standing service on the remote machine that can accept an RPC and begin doing useful work in under 10 microseconds. RPC is how we launch mapshards at Google with a worker process that is a long-running server and it just receives a job spec over the network and can execute right away against the job spec.
p4bl0 6y ago
Very cool :). Apart from Plan9 that many people already talked about here, it also made me think of Emacs `unexec` [0]. [0] http://git.savannah.gnu.org/cgit/emacs.git/tree/src/unexelf.c http://git.savannah.gnu.org/cgit/emacs.git/tree/src/unexelf....
rapjr9 6y ago
There was a lot of work on mobile agents 20 years ago, Java programs that could jump from machine to machine over the network and continue executing wherever they landed. The field stagnated because there were some really difficult security problems (how can you trust the code to execute on your machine? How can the code trust whatever machine it lands on and use it's services?). I think later work resolved the security issues but the field has not re-surged. Might be a good place to start to see what the issues and risks of mobile task execution are.
totorovirus 6y ago
teleforkbomb..
tpetry 6y ago
It‘s a really nice idea. But when reading it i came to the conclusion that web workers are a really genious idea which could be used for c-like software equally good: Everything which should run on a different server is an extra executable, so this executable would be shipped to the destination, started and then the two processes can talk by message passing. This concept is so generic there could be dozens of „schedulers“ to start a process on a remote location, like ssh connect, start a cloud vm, ...
sharno 6y ago
I think Unison [0] is going to make this trivial [0] https://www.unisonweb.org/ https://www.unisonweb.org/
rhabarba 6y ago
I love how people reimplement Plan 9 (poorly).

572 points by trishume 6y ago

What I'd love it's bingding easily remote directories as local. Not NFS, but a braindead 9p. If I don't have a tool, I'd love to have a bind-mount of a directory from a stranger, and run a binarye from within (or piping it) without he being able to trace the I/O. If the remote FS is a diff arch, I'd should be able to run the same binary remotely as a fallback option, seamless.

londons_explore 6y ago

Bonus points if you can effectively implement the "copy on write" ability of the linux kernel to only send over pages to the remote machine that are changed either in the local or remote fork, or read in the remote fork. A rsync-like diff algorithm might also substantially reduce copied pages if the same or a similar process is teleforked multiple times. Many processes have a lot of memory which is never read or written, and there's no reason that should be moved, or at least no reason it should be moved quickly. Using that, you ought to be able to resume the remote fork in milliseconds rather than seconds. userfaultfd() or mapping everything to files on a FUSE filesystem both look like promising implementation options.

trishume 6y ago
I do in fact mention this idea in the article. In fact userfaultfd was added to the kernel so that CRIU and KVM live migration could implement exactly this. Another cool project that does something like this is https://github.com/gamozolabs/chocolate_milk https://github.com/gamozolabs/chocolate_milk which is a fuzzing hypervisor kernel which can back a VM snapshot memory mapping over the network to only pull down the pages that the VM actually reads during the fuzz case.
mlyle 6y ago
If you just pull things on demand, you're going to get a lot of round-trip-time penalties to page things in. I think you should still be pushing the memory as fast as you can, but maybe you start the child while this is still in progress, and prioritize sending stuff the child asks for (reorder to send that stuff "next"), if you've not already sent it.
- lunixbochs 6y ago
  That’s a great idea. One of my thoughts was to “pre-heat” the process by executing a bit locally with side effects disabled to see what would get immediately accessed and send that first. If your systems strictly match somehow (machine image with auto update disabled? or regularly hash and timestamp files on both systems) you can also cheat by mapping some of the files locally on the other side.
- trishume 6y ago
  Yah that is indeed a super important optimization for avoiding round trips. CRIU does this and calls it "pre-paging", their wiki also mentions that they adapt their page streaming to try to pre-stream pages around pages that have been faulted: https://en.wikipedia.org/wiki/Live_migration#Post-copy_memory_migration https://en.wikipedia.org/wiki/Live_migration#Post-copy_memor... edit: lol I didn't realized that isn't CRIU's wiki since they just linked to a Wikipedia page and both use WikiMedia software. This is the actual CRIU wiki page, and it's way harder to tell if they do this, although I suspect they do and it's in the "copy images" step of the diagram https://criu.org/Userfaultfd https://criu.org/Userfaultfd
deleted 6y ago
[deleted]
touisteur 6y ago
If you're at the hypervisor level you can also use Intel PML, seems it's made for this. https://arxiv.org/abs/2001.09991 https://arxiv.org/abs/2001.09991 I'm guessing @gamozolabs (twitter) is going to use it some day soon to be fuzz billions of cases per second, with snapshot fuzzing...
juancampa 6y ago
If you ever needed to bring the process back, you could use soft-dirty-bit[1] to determine which pages were modified since forking and only transfer those. CRIU uses it for incremental snapshots (in fact, they wrote the kernel patch afaik) [1] https://www.kernel.org/doc/Documentation/vm/soft-dirty.txt https://www.kernel.org/doc/Documentation/vm/soft-dirty.txt

anticensor 6y ago

hoard() would be a better name.

jabedude 6y ago
I don't understand, could you explain? `tfork(2)` seems more in line with Linux naming conventions.
- anticensor 6y ago
  Telefork as a verb does not map to a real word concept. Hoard fits better, as in hoarding another computer for your process.
  - carapace 6y ago
    Ha! I thought you had misspelled horde! ;-)
    - buckminster 6y ago
      I think you mishurd.
  - jabedude 6y ago
    That is funny. hoard() and steal() could work.
    - anticensor 6y ago
      Agree, steal() instead of telepad() would be a good idea.

zozbot234 6y ago

The page states that CRIU requires kernel patches, but other sources say that the kernel code for CRIU is already in the mainline kernel. What's up with that?

trishume 6y ago
I didn't say it requires kernel patches, just that it requires a build flag and new enough versions. It does happen that most distros do enable that build flag though. There is a patch set for write protection support in userfaultfd that was just merged recently and I'm not sure if it's made it into any actual releases yet.
cyphar 6y ago
CRIU does work on mainline kernels and has for several years. However, it is still being actively developed (userfaultfd is the example the article uses -- a mainline feature which is still seeing lots of development). It doesn't usually require additional patches to work, though they do have a fairly big job (every new kernel feature needs to be added to CRIU so it can both checkpoint and restore it -- and not all features are trivial to checkpoint). It's entirely possible that programs using very new kernel features may struggle to get CRIU to work.
en4bz 6y ago
CRIU can use two mechanisms to detect page changes. One is the soft-dirty kernel feature which is mainlined and can be accessed via /proc/PID/pagemap [1]. The other is userfaultfd which is only partially merged in the newest kernel. userfaultfd lacks write detection which the article mentioned. My understanding is that using pagemap requires the entire process to be frozen while it is scanned for memory changes and the memory is copied while uffd allows for a more streaming/on-demand approach that doesn't require stopping the entire process. [1] https://www.kernel.org/doc/Documentation/vm/soft-dirty.txt https://www.kernel.org/doc/Documentation/vm/soft-dirty.txt

saagarjha 6y ago

It’s touched on at the very end, but this kind of work is somewhat similar to what the kernel needs to do on a fork or context switch, so you can really figure out what state you need to keep track of from there. Once you have that, scheduling one of these network processes isn’t really all that different than scheduling a normal process, except the of course syscalls on the remote machine will possibly go to a kernel that doesn’t know what to do with them.

new_realist 6y ago

See https://criu.org/Live_migration https://criu.org/Live_migration

ISL 6y ago

What's old is new again -- I'm pretty sure QNX could do this in the 1990s. QNX had a really cool way of doing inter-process communication over the LAN that worked as if it were local. Used it in my first lab job in 2001. You might not find it on the web, though. The API references were all (thick!) dead trees. Edit: Looks like QNX4 couldn't fork over the LAN. It had a separate "spawn()" call that could operate across nodes. https://www.qnx.com/developers/docs/qnx_4.25_docs/qnx4/sysarch/proc.html https://www.qnx.com/developers/docs/qnx_4.25_docs/qnx4/sysar...

checker659 6y ago
> I'm pretty sure QNX could do this in the 1990s. Plan 9, SmallTalk
- imglorp 6y ago
  Erlang of course. All spawns and messages can be local or remote.
  - masklinn 6y ago
    Indeed, weird to only find it that low as a sub-comment: spawn(Node, Module, Function, Args) -> pid() Returns the process identifier (pid) of a new process started by the application of Module:Function to Args on Node. If Node does not exist, a useless pid is returned. Otherwise works like spawn/3.
  - russellbeattie 6y ago
    Any time I see something about remote processes, I immediately think, "Erlang could probably do this." I think Erlang would have been the programming language of the 21st century... If only the syntax wasn't like line noise and a printer error code had a baby, and raised it to think like Lisp.
    - sosodev 6y ago
      Elixir is pretty cool. You get all the power of Erlang wrapped up in a much more developer friendly language.

synack 6y ago

This reminds me of OpenMOSIX, which implemented a good chunk of POSIX in a distributed fashion. MPI also comes to mind, but it's more focused on the IPC mechanisms. I always liked Plan 9's approach, where every CPU is just a file and you execute code by writing to that file, even if it's on a remote filesystem.

anfractuosity 6y ago
I recall OpenMosix too, I remember playing with it on a couple of old machines a while ago. I thought it seemed quite a cool idea. I think I was trying to do mp3 encoding from CDs with it in a distributed fashion.
synack 6y ago
Ah, the Plan 9 bits I was remembering are actually in Inferno... http://www.vitanuova.com/inferno/man/4/grid-cpu.html http://www.vitanuova.com/inferno/man/4/grid-cpu.html
notacoward 6y ago
Same reaction here. MOSIX was basically this idea developed for real. Turns out there are a bunch of secondary problems to do with namespaces and security, scheduling and resource use, topology, performance, and so on. In the end it never turned out to be all that practical, and I know of several efforts (e.g. LOCUS) that went quite far to reach the same conclusion. There's a reason that non-transparent "shared nothing" distributed computing came to dominate the computing landscape.
adrianpike 6y ago
Whoa, I was just talking about OpenMOSIX the other day - at my college job, when we retired workstations, they would just sit in the back closet for years until facilities would get around to getting them up for auction. We set up a TFTP boot setup and had a few dozen nodes at any given time. It wasn't high performance by any fashion, but it worked pretty transparently and was always fun to throw heavy synthetic workloads at it and watch the cluster rebalance and hear the fans spin up on the clunky old pentium's.
dghughes 6y ago

dekhn 6y ago

Condor, a distributed computing environment, has done IO remoting (where all calls to IO on the target machine get sent back to the source) for several decades. The origin of Linux Containers was process migration. I believe people have found other ways to do this, personally I think the ECS model (like k8s, but the cloud provider hosts the k8s environment) where the user packages up all the dependencies and clearly specifies the IO mechanisms through late biniding, makes a lot more sense for distributed computing.

vidarh 6y ago
I clicked through to mention Condor too... I first came across it in the 90's, and it seems like one of those obvious hacks that keeps being reinvented.
- dekhn 6y ago
  I was actually channeling the creator of Condor, Miron Livny, who has a history of going to talks about distributed computing and pointing out that "Condor already does that" for nearly everything that people try to tout as new and cool. He's not wrong, but few people use Condor.
  - iaresee 6y ago
    Few people outside academia, maybe? But inside it still seems to dominate in areas like physics computation. CERN uses a world wide Condor grid. LIGO too. It’s excellent for sharing cycles for those slow-burn, highly parallel, massive data scale problems. I spent more than a decade bringing Condor to semi-conductor, financial and biomedical institutions. It was always a fight to show them there was a better way to utilize their massive server farms that didn’t require paying the LSF Tax. Without a shiny sales or marketing department, Condor was hard to pitch to IT departments. Still, to this day, I see people doing things with “modern” platforms like Kubernetes and such and I chuckle. Had that in Condor 15 years ago in many cases. :)
    - 0xdeadbeefbabe 6y ago

touisteur 6y ago

I wonder whether the effort by the syzkaller people (@dyukov) could help with the actual description of all the syscalls (that the author says people gave up on for now, because too complex), since they need them to be able to fuzz efficiently...

carapace 6y ago

"Somebody else has had this problem." Don't get me wrong, this is great hacking and great fun. And this is a good point: > I think this stuff is really cool because it’s an instance of one of my favourite techniques, which is diving in to find a lesser-known layer of abstraction that makes something that seems nigh-impossible actually not that much work. Teleporting a computation may seem impossible, or like it would require techniques like serializing all your state, copying a binary executable to the remote machine, and running it there with special command line flags to reload the state.

fitzn 6y ago

Really cool idea! Thanks for providing so much detail in the post. I enjoyed it. A somewhat related project is the PIOS operating system written 10 years ago but still used today to teach the operating systems class there. The OS has different goals than your project but it does support forking processes to different machines and then deterministically merging their results back into the parent process. Your post remind me of it. There's a handful of papers that talks about the different things they did with the OS, as well as their best paper award at OSDI 2010. https://dedis.cs.yale.edu/2010/det/ https://dedis.cs.yale.edu/2010/det/

hawski 6y ago

Seriously cool. That also reminds me of DragonFlyBSD's process checkpointing feature that offers suspend to disk. In Linux world there were many attempts, but AFAIK nothing simple and complete enough. To be fair I don't know if DF's implementation is that either. https://www.dragonflybsd.org/cgi/web-man?command=sys_checkpoint&section=2 https://www.dragonflybsd.org/cgi/web-man?command=sys_checkpo... https://www.dragonflybsd.org/cgi/web-man?command=checkpoint&section=ANY https://www.dragonflybsd.org/cgi/web-man?command=checkpoint&...

jabedude 6y ago
This article[0] on LWN seems to suggest that Linux has no kernel support for checkpoint/restore because it's hard to implement (no arguments there). But hypervisors support checkpoint/restore for virtual machines, e.g. ESXi VMotion and KVM live migration, so it seems like these technical problems are solvable. Indeed all the benefits of VM migration seem to also apply to process migration (load balancing, service uptime, etc). 0. https://lwn.net/Articles/293575/ https://lwn.net/Articles/293575/
- justapassenger 6y ago
  IMO, big reason for why Linux doesn’t support it, is that big tech companies, who drive a lot of development and funding, found that it’s just not worth it. Tech moved away from mainframe model of keeping single processes up for as long as possible and isolating software from the failures. Instead they embrace failure at the software layer, which gives you (when executed correctly) both same high level of availability, but also layers of protection against other issues (as node going down in expected way is the same, no matter what’s the root cause) and makes overall maintainability and upgradability systems higher. Basically, why deal with complicated checkpoints at kernel level, that doesn’t understand your software, when you ca instead deal with that at the software layer, with full control over it.
  - spullara 6y ago
    Swap performance has been similarly underinvested.

abotsis 6y ago

Also of interest might be Sprite- a Berkeley research os developed “back in the day” by Ken Shirriff And others. It boasted a lot of innovations like a logging filesystem (not just metadata) and a distributed process model and filesystem allowing for live migrations between nodes. https://www2.eecs.berkeley.edu/Research/Projects/CS/sprite/sprite.html https://www2.eecs.berkeley.edu/Research/Projects/CS/sprite/s...

kens 6y ago
To be clear, John Ousterhout was the creator of the Sprite operating system. I was one of the grad students working on it.

lachlan-sneff 6y ago

Wow, this is really interesting. I bet that there's a way of doing this robustly by streaming wasm modules instead of full executables to every server in the cluster.

deleted 6y ago

[deleted]

jka 6y ago

This reminds me a little bit of the idea of 'Single System Image'[1] computing. The idea, in abstract, is that you login to an environment where you can list running processes, perform filesystem I/O, list and create network connections, etc -- and any and all of these are in fact running across a cluster of distributed machines. (in a trivial case that cluster might be a single machine, in which case it's essentially no different to logging in to a standalone server) The wikipedia page referenced has a good description and a list of implementations; sadly the set of {has-recent-release && is-open-source && supports-process-migration} seems empty. [1] - https://en.wikipedia.org/wiki/Single_system_image https://en.wikipedia.org/wiki/Single_system_image

macintux 6y ago
That was the original concept that led Ian Murdock and John Hartman to found Progeny. The idea was that overnight, while no one was working at their desks, companies could reboot their Windows boxes into a SSI network of Linux nodes to run parallel compute tasks. Roughly, anyway, I got the sales pitch 20 years ago so my memories are fuzzy. I wasn’t remotely sold on it but was so anxious to work for a Linux R&D company in Indianapolis of all places that I accepted the job anyway. Sadly we didn’t get far on the concept before the dot-com crash. Absent more venture capital we pivoted to focus on something we could sell, Progeny Linux, and tried to turn that into a managed platform for companies who wanted to run Linux on their appliances.

pcr910303 6y ago

This makes me think about Urbit[0] OS — Urbit OS represents the entire OS state in a simple tree, this would be very simple to implement. [0]: https://urbit.org/ https://urbit.org/

deleted 6y ago

[deleted]

Animats 6y ago

That goes back to the 1980s, with UCLA Locus. This was a distributed UNIX-like system. You could launch a process on another machine and keep I/O and pipes connected. Even on a machine with a different CPU architecture. They even shared file position between tasks across the network. Locus was eventually part of an IBM product. A big part of the problem is "fork", which is a primitive designed to work on a PDP-11 with very limited memory. The way "fork" originally worked was to swap out the process, and instead of discarding the in-memory copy, duplicate the process table entry for it, making the swapped-out version and the in-memory version separate processes. This copied code, data, and the process header with the file info. This is a strange way to launch a new process, but it was really easy to implement in early Unix. Most other systems had some variant on "run" - launch and run the indicated image. That distributes much better.

wrs 6y ago
No need to use the past tense — CreateProcess is the primitive in Windows NT. (It's been 20 years...can we just call that Windows now?)
- zozbot234 6y ago
  posix_spawn is a thing as well.
  - nwmcsween 6y ago
    posix_spawn is horrendous
barrkel 6y ago
There's also an ergonomics to process creation APIs - rather than needing separate APIs for manipulating your child process vs manipulating your own process, fork() lets you use one to implement the other: fork(), configure the resulting process, then exec(). CreateProcess* on Windows is a relative monstrosity of complexity compared to fork/exec.
slobiwan 6y ago

0xbadcafebee 6y ago

It's nice to see people re-discover old school tech. In cluster computing this was generally called "application checkpointing"[1] and it's still in use in many different systems today. If you want to build this into your app for parallel computing you'd typically use PVM[2]/MPI[3]. SSI[4] clusters tried to simplify all this by making any process "telefork" and run on any node (based on a load balancing algorithm), but the most persistent and difficult challenge was getting shared memory and threading to work reliably. It looks like CRIU support is bundled in kernels since 3.11[5], and works for me in Ubuntu 18.04, so you can basically do this now without custom apps. [1] https://en.wikipedia.org/wiki/Application_checkpointing https://en.wikipedia.org/wiki/Application_checkpointing [2] https://en.wikipedia.org/wiki/Parallel_Virtual_Machine https://en.wikipedia.org/wiki/Parallel_Virtual_Machine [3] https://en.wikipedia.org/wiki/Message_Passing_Interface https://en.wikipedia.org/wiki/Message_Passing_Interface [4] https://en.wikipedia.org/wiki/Single_system_image https://en.wikipedia.org/wiki/Single_system_image [5] https://en.wikipedia.org/wiki/CRIU#Use https://en.wikipedia.org/wiki/CRIU#Use

dreamcompiler 6y ago

Telescript [0] is based on this idea, although at a higher level. I wish we could just build Actor-based operating systems and then we wouldn't need to keep reinventing flexible distributed computation, but alas...[1] [0] https://en.wikipedia.org/wiki/Telescript_(programming_language) https://en.wikipedia.org/wiki/Telescript_(programming_langua... [1] Yes I know Erlang exists. I wish more people would use it.

peterwwillis 6y ago
We'll keep poorly reinventing distributed computing features until we have a real distributed operating system. We're actually not that far off from it, but good luck convincing a mainline kernel dev to accept your patches.

cecilpl2 6y ago

This is similar to what Incredibuild does. It distributes compile and compute jobs across a network, effectively sandboxing the remote process and forwarding all filesystem calls back to the initiating agent.

cjbprime 6y ago

Amazing work.

ignoramous 6y ago
Yep. Reminds me of u/xal praising u/trishume: https://news.ycombinator.com/item?id=20490964 https://news.ycombinator.com/item?id=20490964

userbinator 6y ago

This can let you stream in new pages of memory only as they are accessed by the program, allowing you to teleport processes with lower latency since they can start running basically right away. That's what "live migration" does; it can be done with an entire VM: https://en.wikipedia.org/wiki/Live_migration https://en.wikipedia.org/wiki/Live_migration

dmitrygr 6y ago
Hey what's the best way to contact you? Have a question. If possible my email is in my profile.
- sitkack 6y ago
  https://www.usenix.org/node/170864 https://www.usenix.org/node/170864
trishume 6y ago
It's what a certain kind of live migration strategy does. There's a discussion thread about this further down where I link to that same page.
sitkack 6y ago
https://cloud.google.com/compute/docs/instances/live-migration https://cloud.google.com/compute/docs/instances/live-migrati... gives a great rundown on the mechanics as they pertain to GCP. Is probably somewhat valid for other systems.
kevinchen 6y ago
I guess this post is a little bit different, because VMs are designed to be portable across different hosts. Even hypervisor software without live migration still lets you freeze the VM’s state to a file which can be copied to a new host. However, an already running process is not designed to be portable in the same way.

vladbb 6y ago

I implemented something similar ten years ago for a class project: https://youtu.be/0am-5noTrWk https://youtu.be/0am-5noTrWk

peterkelly 6y ago

There's been a bunch of interesting work done on this over the years. Here's a literature survey on the topic: https://dl.acm.org/doi/abs/10.1145/367701.367728 https://dl.acm.org/doi/abs/10.1145/367701.367728

crashdelta 6y ago

THIS IS REVOLUTIONARY!

systemBuilder 6y ago
Hardly. See my rebuttal above.

crashdelta 6y ago

This is one of the best side projects I've ever seen, hands down.

trishume 6y ago
<3

YesThatTom2 6y ago

Condor did this in the early 90s.

concernedctzn 6y ago

side note: take a look at this guy's other blog posts, they're all very good

TazeTSchnitzel 6y ago

In essence this is manually implementing forking — spawning a new process and copying the bytes over without getting the kernel to help you, except over a network too. It reminds me a bit of when I wanted to parallelise the PHP test suite but didn't want to (couldn't?) use fork(), yet I didn't want to substantially rewrite the code to be amenable to cleanly re-initialising the state in the right way. But conveniently, this program used mostly global variables, and you can access global variables in PHP as one magic big associative array called $GLOBALS. So I moved most of the program's code into two functions (mostly just adding the enclosing function declaration syntax and indentation, plus `global` imports), made the program re-invoke itself NPROCS times mid-way, sending its children `serialize($GLOBALS)` over a loopback TCP connection, then had the spawned children detect an environment variable to receieve the serialized array over TCP, unserialize() it and copy it into `$GLOBALS`, and call the second function… lo and behold, it worked perfectly. :D (Of course I needed to make some other changes to make it useful, but they were also similar small incisions that tried to avoid refactoring the code as much as possible.) PHP's test suite uses this horrible hack to this day. It's… easier than rewriting the legacy code…

trishume 6y ago
Indeed! I was talking to someone about their attempt to fork a Graal Java process and recreate all the compiler and GC threads, and I said if I had that task I'd be tempted to just use my new knowledge to implement a fork that also recreated those threads rather than trying to understand how to shut them down and restore them properly.
i15e 6y ago
I did something similar using the 'at' daemon's 'now' time specification to "fork" off background tasks from a web request using the same .php file. It actually worked well for what I needed at the time!

bamboozled 6y ago

I could imagine having a build system which produces a process as an artifact and then just forks it in the cloud without distributing those pesky archives!

keeganpoppen 6y ago
that... is a really fucking cool idea. i mean, it's of limited utility in the sense that builds really should generally work most of the time & be reproducible, so even though there is a bit of wasted effort, bandwidth, etc. in running the same build in multiple places at once, it's not like the worst thing in the world... but it just feels like there have to be cool / "useful" scenarios where you want to distribute a thing, but not the environment, etc. required to build the thing. oh wait, that already exists: it's called "a binary"... lol. iono. i mean i guess if you have a process byte-for-byte frozen in time, you're a bit more certain (provably so? maybe?) maybe that when it is resurrected on some remote host that remote process and local process (as of t_freeze) are in ~"exactly" the same state? or if you continuously snapshotted a process while it was running, and then when it crashes you (hopefully) have a snapshot from right before when it crashed that can be replayed, modified, etc. at will? but that's a bit of a stretch, and there are simpler ways of accomplishing that for the most part... the only other thing i could think of is that this could allow you to run completely diskless servers, because you can just beam all the processes in, without ever having to "install" anything, beyond copying all the shared libs, etc. which i guess could get the "1 have one of these and i want 100 of them, in the cloud" latency down by an order of magnitude or two maybe.

justicezyx 6y ago

There is some company using CRIU to implement general purpose process live migration (not specifically VMs).

systemBuilder 6y ago

I think the problem with a lot of these ideas is that the value of fork() is only marginally higher than the value of starting a fresh process with arguments on a remote machine. The complexity of moving a full process to another machine is 10 times higher than just starting a new process on the remote machine with all the binaries present already. Quite frankly, vfork only exists and gets used because it's so damned cheap to copy the pagetable entries and use copy-on-write, to save RAM. Take away the cheapness by copying the whole address space over a network, adding slowness, and nobody will be interested any more. And both techniques are inferior to having a standing service on the remote machine that can accept an RPC and begin doing useful work in under 10 microseconds. RPC is how we launch mapshards at Google with a worker process that is a long-running server and it just receives a job spec over the network and can execute right away against the job spec.

p4bl0 6y ago

Very cool :). Apart from Plan9 that many people already talked about here, it also made me think of Emacs `unexec` [0]. [0] http://git.savannah.gnu.org/cgit/emacs.git/tree/src/unexelf.c http://git.savannah.gnu.org/cgit/emacs.git/tree/src/unexelf....

rapjr9 6y ago

There was a lot of work on mobile agents 20 years ago, Java programs that could jump from machine to machine over the network and continue executing wherever they landed. The field stagnated because there were some really difficult security problems (how can you trust the code to execute on your machine? How can the code trust whatever machine it lands on and use it's services?). I think later work resolved the security issues but the field has not re-surged. Might be a good place to start to see what the issues and risks of mobile task execution are.

totorovirus 6y ago

teleforkbomb..

tpetry 6y ago

It‘s a really nice idea. But when reading it i came to the conclusion that web workers are a really genious idea which could be used for c-like software equally good: Everything which should run on a different server is an extra executable, so this executable would be shipped to the destination, started and then the two processes can talk by message passing. This concept is so generic there could be dozens of „schedulers“ to start a process on a remote location, like ssh connect, start a cloud vm, ...

sharno 6y ago

I think Unison [0] is going to make this trivial [0] https://www.unisonweb.org/ https://www.unisonweb.org/

rhabarba 6y ago

I love how people reimplement Plan 9 (poorly).