5 ms·

It is interesting to use fork() to snapshot application state. It seems so high-level and powerful (Compared to other C/POSIX level stuff - so I'm not comparing

by szemet 8y ago

It is interesting to use fork() to snapshot application state. It seems so high-level and powerful (Compared to other C/POSIX level stuff - so I'm not comparing it to Scheme/Lisp/"your favorite high/low level environment" here!). We may collect some other examples here - just for fun: - I know that Redis does this (RDB snaphots) too, while the child process saves the actual snapshot, the parent can operate continously relying on the system provided copy on write mechanism - I have also found this: https://github.com/thomasballinger/rlundo https://github.com/thomasballinger/rlundo - It overwrites readline library to get undo function in any REPL using it Other examples?

chrisseaton 8y ago
There is a time-travelling debugger that forks the process at each statement, allowing you to go back to a previous statement and continue from that point.
- dooglius 8y ago
  It seems like forking so often would use up a system's resources very quickly. I'd imagine that each fork needs at least a handful of individual pages for OS bookkeeping overhead, so you'd get into the gigabyte range fast even with perfect copy-on-write.
  - chrisseaton 8y ago
    Yes you can't run like that forever. But you could break, step forward and backward and bit, and then resume. I often step forward while debugging, realise that whatever is going wrong has already gone wrong, and would like to step back a few steps.
  - caf 8y ago
    It's more than a handful of pages - each forked process requires its own set of page tables.
- adtac 8y ago
  Very interesting. Do you have a link where I can read more about this?
  - CyberShadow 8y ago
    Mozilla's rr (mentioned in the article) is a time-traveling debugger, but as far as I know it doesn't use fork() for snapshots / time-travel.
    - roca 8y ago
      rr does use fork() internally to create checkpoints, and checkpoints are created implicitly as the program executes to support time-travel. The COW behaviour of fork() is extremely helpful for performance. There's more to checkpoints than just fork(), though; for example, when two processes are sharing memory, checkpointing the process group fork()s both processes and then must explicitly duplicate the shared memory region, otherwise the checkpoint would share memory with the original, which would be disastrous. Unfortunately this means we don't get COW behaviour for shared memory regions; unfortunately the Linux/POSIX APIs are a bit deficient here.
  - chrisseaton 8y ago
    https://github.com/awwaiid/pry-timetravel https://github.com/awwaiid/pry-timetravel But I think the idea has been around forever.
  - Pete_D 8y ago
    gdb has checkpoint/restart that uses fork(), though you have to specify the checkpoints manually. https://sourceware.org/gdb/onlinedocs/gdb/Checkpoint_002fRestart.html https://sourceware.org/gdb/onlinedocs/gdb/Checkpoint_002fRes... It has another mode where you can record and rewind execution traces, which sounds more like the description, but I'm almost certain that doesn't use fork. https://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-Replay.html#Process-Record-and-Replay https://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and...
- arcticbull 8y ago
  It's cool but you can't un-write to disk, or un-send network packets making its applicability limited.
  - tylerhou 8y ago
    > you can’t un-write to disk Why not? (Experimental) versioned file systems exist.
    - arcticbull 8y ago
      I think the broader point I'm making is that these time traveling debuggers can't work in the general case due to externalities.
c0l0 8y ago
Only of historical relevance, but sure gave me a good chuckle of the "omg I can't believe that's how they made that work!"-sort when I first read it: Ultima Online fork()ed its server process(es) for marshalling in-process data into files on NFS, and used that mechanism to persist world state: https://news.ycombinator.com/item?id=9820645 https://news.ycombinator.com/item?id=9820645
Asooka 8y ago
Not exactly a fork, but Emacs is (famous/notorious) for saving its initial environment by basically coredumping and loading the coredump on startup if the startup files haven't changed: https://lwn.net/Articles/673724/ https://lwn.net/Articles/673724/ http://emacshorrors.com/posts/unexecute.html(quite http://emacshorrors.com/posts/unexecute.html(quite unflattering as the link suggests, but factually accurate nonetheless)
Filligree 8y ago
The Factorio dedicated server has an 'experimental' fork-to-save option, which is massively helpful when late-game saves take 10-15 seconds.
- reificator 8y ago
  Alright, I'm geared up, I'm going to go take out this hive- > Saving > Crash the already difficult to drive vehicle into a nearby rock > You died
CGamesPlay 8y ago
Web servers did this all the time to skip parsing all of your cgi scripts on each request. https://stackoverflow.com/questions/25834333/what-exactly-is-a-pre-fork-web-server-model https://stackoverflow.com/questions/25834333/what-exactly-is...
pmarreck 8y ago
This is, incidentally, one of the built-in advantages of immutable data structures (in the languages that feature them). For example I recently had to write a test that worked with a fairly complex connection object, the execution of the test changed that object but later on I wanted to reset it to its earlier state- this was as simple as doing something like this in Elixir original_conn = conn conn = do_something_expensive(conn, params) assert <whatever> conn = original_conn # this "grabs" the earlier state, no matter how complex it is conn = do_something_else(conn, params) assert <something having to do with both calls> Basically, if your language is immutable, then any reference to any state (no matter how complex the data structure) is the state it was at that point in time You could say that changes to the state are forked from the earlier state.
amelius 8y ago
But I suppose immutable data structures can be more efficiently stored using a heap with byte-size-granularity, rather than by forking, which uses pages of perhaps thousands of bytes.
- kccqzy 8y ago
  Why would you think so? A page is 4096 bytes and it's hard coded into the kernel (and the hardware). With byte sized granularity, you are just introducing extreme large overhead for page table entries and make the TLB a lot less effective.
  - amelius 8y ago
    I wasn't suggesting to make the page size smaller, but instead to not fork the compiler and use a single heap for all versions of the data.
- szemet 8y ago
  There is another fairly similar design decision split: LVM - low (block) level, hardware supported, generic, unavare of filesystem layer above (so it may not be optimal for some usecases) but at least this gives some separation of concerns, and still can provide some impressive features: resize, encrypt, snapshots CoW filesystems (btrfs, zfs, ...) - specialized, carefully designed data structures/layout, can have more advanced features and more optimized to given ussecases (they have more information what is actually happening), but they are not a generic separatable layer - all the fancy advanced features are tied to the high level fs implementation
wrs 8y ago
- The Resque background job queue (popular in the Ruby world) forks itself to run each job, to isolate the scheduler from any badness in the jobs. - I’ve used fork() in Ruby for similar reasons when a library had instability or memory leaks and I wanted to isolate it rather than try to find the bug (I’m looking at you, ImageMagick). A while back, MRI Ruby got an improved garbage collector that keeps the mark bits separate from the objects just so copy-on-write would (mostly) work in cases like this.
chubot 8y ago
Chrome and Android have both forked() warmed-up processes to speed application startup. Although looking at this 2011 post it's more complicated than I thought in Chrome: http://neugierig.org/software/chromium/notes/2011/08/zygote.html http://neugierig.org/software/chromium/notes/2011/08/zygote.... This one is also kind of old: http://coltf.blogspot.com/p/android-os-processes-and-zygote.html http://coltf.blogspot.com/p/android-os-processes-and-zygote.... If anyone has any updates on these I'd be interested. As far as I remember App Engine also did a similar thing to avoid reinitializing a Python VM for every request.
svat 8y ago
The TeX program (by Donald Knuth) does this. More precisely, the first version of the TeX program (now called TeX78 or TeX80) which was written in the SAIL programming language at Stanford was written in this way, because it was apparently the natural thing to do and lots of programs at the time were written like that. The present version (rewrite) of the TeX program (briefly called TeX82 but now just called TeX), was written in (a language/system based on) Pascal and meant to be portable for different OSes. So in addition to using the system's fork or dump/undump (if available) to snapshot the application state, it contains its own implementation of dumping all the relevant application state into a "format file", and loading from it. In fact, the dumping and undumping were traditionally done by different binaries: INITEX -> VIRTEX -> TEX See https://news.ycombinator.com/item?id=13076098 https://news.ycombinator.com/item?id=13076098 where drfuchs (David Fuchs, who was Knuth's "right-hand man" during his rewrite of TeX) describes it in more detail. (Originally posted on https://news.ycombinator.com/item?id=13073566 https://news.ycombinator.com/item?id=13073566 "The Emacs dumper dispute", and also quoted in https://news.ycombinator.com/item?id=14140421 https://news.ycombinator.com/item?id=14140421 "Improving startup time in Atom with V8 snapshots".) See also https://tex.stackexchange.com/questions/417624/installation-procedures-of-early-tex-installations https://tex.stackexchange.com/questions/417624/installation-... for some elaboration (though it's confusing and now I realize it has at least one error).
deleted 8y ago
[deleted]

by szemet 8y ago

chrisseaton 8y ago

There is a time-travelling debugger that forks the process at each statement, allowing you to go back to a previous statement and continue from that point.

dooglius 8y ago
It seems like forking so often would use up a system's resources very quickly. I'd imagine that each fork needs at least a handful of individual pages for OS bookkeeping overhead, so you'd get into the gigabyte range fast even with perfect copy-on-write.
- chrisseaton 8y ago
  Yes you can't run like that forever. But you could break, step forward and backward and bit, and then resume. I often step forward while debugging, realise that whatever is going wrong has already gone wrong, and would like to step back a few steps.
- caf 8y ago
  It's more than a handful of pages - each forked process requires its own set of page tables.
adtac 8y ago
Very interesting. Do you have a link where I can read more about this?
- CyberShadow 8y ago
  Mozilla's rr (mentioned in the article) is a time-traveling debugger, but as far as I know it doesn't use fork() for snapshots / time-travel.
  - roca 8y ago
    rr does use fork() internally to create checkpoints, and checkpoints are created implicitly as the program executes to support time-travel. The COW behaviour of fork() is extremely helpful for performance. There's more to checkpoints than just fork(), though; for example, when two processes are sharing memory, checkpointing the process group fork()s both processes and then must explicitly duplicate the shared memory region, otherwise the checkpoint would share memory with the original, which would be disastrous. Unfortunately this means we don't get COW behaviour for shared memory regions; unfortunately the Linux/POSIX APIs are a bit deficient here.
- chrisseaton 8y ago
  https://github.com/awwaiid/pry-timetravel https://github.com/awwaiid/pry-timetravel But I think the idea has been around forever.
- Pete_D 8y ago
  gdb has checkpoint/restart that uses fork(), though you have to specify the checkpoints manually. https://sourceware.org/gdb/onlinedocs/gdb/Checkpoint_002fRestart.html https://sourceware.org/gdb/onlinedocs/gdb/Checkpoint_002fRes... It has another mode where you can record and rewind execution traces, which sounds more like the description, but I'm almost certain that doesn't use fork. https://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and-Replay.html#Process-Record-and-Replay https://sourceware.org/gdb/onlinedocs/gdb/Process-Record-and...
arcticbull 8y ago
It's cool but you can't un-write to disk, or un-send network packets making its applicability limited.
- tylerhou 8y ago
  > you can’t un-write to disk Why not? (Experimental) versioned file systems exist.
  - arcticbull 8y ago
    I think the broader point I'm making is that these time traveling debuggers can't work in the general case due to externalities.

c0l0 8y ago

Only of historical relevance, but sure gave me a good chuckle of the "omg I can't believe that's how they made that work!"-sort when I first read it: Ultima Online fork()ed its server process(es) for marshalling in-process data into files on NFS, and used that mechanism to persist world state: https://news.ycombinator.com/item?id=9820645 https://news.ycombinator.com/item?id=9820645

Asooka 8y ago

Not exactly a fork, but Emacs is (famous/notorious) for saving its initial environment by basically coredumping and loading the coredump on startup if the startup files haven't changed: https://lwn.net/Articles/673724/ https://lwn.net/Articles/673724/ http://emacshorrors.com/posts/unexecute.html(quite http://emacshorrors.com/posts/unexecute.html(quite unflattering as the link suggests, but factually accurate nonetheless)

Filligree 8y ago

The Factorio dedicated server has an 'experimental' fork-to-save option, which is massively helpful when late-game saves take 10-15 seconds.

reificator 8y ago
Alright, I'm geared up, I'm going to go take out this hive- > Saving > Crash the already difficult to drive vehicle into a nearby rock > You died

CGamesPlay 8y ago

Web servers did this all the time to skip parsing all of your cgi scripts on each request. https://stackoverflow.com/questions/25834333/what-exactly-is-a-pre-fork-web-server-model https://stackoverflow.com/questions/25834333/what-exactly-is...

pmarreck 8y ago

This is, incidentally, one of the built-in advantages of immutable data structures (in the languages that feature them). For example I recently had to write a test that worked with a fairly complex connection object, the execution of the test changed that object but later on I wanted to reset it to its earlier state- this was as simple as doing something like this in Elixir original_conn = conn conn = do_something_expensive(conn, params) assert <whatever> conn = original_conn # this "grabs" the earlier state, no matter how complex it is conn = do_something_else(conn, params) assert <something having to do with both calls> Basically, if your language is immutable, then any reference to any state (no matter how complex the data structure) is the state it was at that point in time You could say that changes to the state are forked from the earlier state.

amelius 8y ago

But I suppose immutable data structures can be more efficiently stored using a heap with byte-size-granularity, rather than by forking, which uses pages of perhaps thousands of bytes.

kccqzy 8y ago
Why would you think so? A page is 4096 bytes and it's hard coded into the kernel (and the hardware). With byte sized granularity, you are just introducing extreme large overhead for page table entries and make the TLB a lot less effective.
- amelius 8y ago
  I wasn't suggesting to make the page size smaller, but instead to not fork the compiler and use a single heap for all versions of the data.
szemet 8y ago
There is another fairly similar design decision split: LVM - low (block) level, hardware supported, generic, unavare of filesystem layer above (so it may not be optimal for some usecases) but at least this gives some separation of concerns, and still can provide some impressive features: resize, encrypt, snapshots CoW filesystems (btrfs, zfs, ...) - specialized, carefully designed data structures/layout, can have more advanced features and more optimized to given ussecases (they have more information what is actually happening), but they are not a generic separatable layer - all the fancy advanced features are tied to the high level fs implementation

wrs 8y ago

- The Resque background job queue (popular in the Ruby world) forks itself to run each job, to isolate the scheduler from any badness in the jobs. - I’ve used fork() in Ruby for similar reasons when a library had instability or memory leaks and I wanted to isolate it rather than try to find the bug (I’m looking at you, ImageMagick). A while back, MRI Ruby got an improved garbage collector that keeps the mark bits separate from the objects just so copy-on-write would (mostly) work in cases like this.

chubot 8y ago

Chrome and Android have both forked() warmed-up processes to speed application startup. Although looking at this 2011 post it's more complicated than I thought in Chrome: http://neugierig.org/software/chromium/notes/2011/08/zygote.html http://neugierig.org/software/chromium/notes/2011/08/zygote.... This one is also kind of old: http://coltf.blogspot.com/p/android-os-processes-and-zygote.html http://coltf.blogspot.com/p/android-os-processes-and-zygote.... If anyone has any updates on these I'd be interested. As far as I remember App Engine also did a similar thing to avoid reinitializing a Python VM for every request.

svat 8y ago

The TeX program (by Donald Knuth) does this. More precisely, the first version of the TeX program (now called TeX78 or TeX80) which was written in the SAIL programming language at Stanford was written in this way, because it was apparently the natural thing to do and lots of programs at the time were written like that. The present version (rewrite) of the TeX program (briefly called TeX82 but now just called TeX), was written in (a language/system based on) Pascal and meant to be portable for different OSes. So in addition to using the system's fork or dump/undump (if available) to snapshot the application state, it contains its own implementation of dumping all the relevant application state into a "format file", and loading from it. In fact, the dumping and undumping were traditionally done by different binaries: INITEX -> VIRTEX -> TEX See https://news.ycombinator.com/item?id=13076098 https://news.ycombinator.com/item?id=13076098 where drfuchs (David Fuchs, who was Knuth's "right-hand man" during his rewrite of TeX) describes it in more detail. (Originally posted on https://news.ycombinator.com/item?id=13073566 https://news.ycombinator.com/item?id=13073566 "The Emacs dumper dispute", and also quoted in https://news.ycombinator.com/item?id=14140421 https://news.ycombinator.com/item?id=14140421 "Improving startup time in Atom with V8 snapshots".) See also https://tex.stackexchange.com/questions/417624/installation-procedures-of-early-tex-installations https://tex.stackexchange.com/questions/417624/installation-... for some elaboration (though it's confusing and now I realize it has at least one error).

deleted 8y ago

[deleted]