6 ms·

fork() is not a small tool. It is a MASSIVE, unwieldy tool that is difficult to use correctly. It happens to have a small interface.

adastra22 4y ago
...is it? In it is original implementation, fork() was pretty trivial. All it did was create a new process entry in the kernel table, with all the pages and capabilities and such copied from the original process. Then mark all pages as copy-on-write, and return to the caller. Maybe not trivial, but much less complicated than loading an executable file from disk. My understanding of Linux internals is maybe 20 years out of date, so I am legitimately curious what makes fork() so complicated these days.
- cryptonector 4y ago
  fork() is not trivial now. Processes are huge now -- they have huge heaps among other things. Copying all that is expensive. In the 80s we tried COW, but that turns out to be very slow as well. What operating systems do now is immediately copy the resident set, then do COW for the rest of writable memory, but in large, multi-threaded processes, this is still too slow. Use vfork() or posix_spawn().
  - adastra22 4y ago
    Hrm. Googling "fork linux copy-on-write" seems to find a lot of stack overflow answers from 2014-2015 claiming Linux marks pages as copy-on-write when fork() is called. I didn't see anything more recent in the first page of results. I could see it being worthwhile to immediately copy a few pages, like the top of the stack, but copying the whole resident set seems excessive. Especially since some of that data might not even be written to.
    - CHY872 4y ago
      So the problem is what happens to the old and new processes after the fork. To CoW, you need to mark all the pages read only in _both_ old and new which means that every memory write in the caller will now pagefault since the OS now has to lazily copy on both sides. So with true copy on write the fixed costs may be low but the marginal cost per memory write may be high in both parent and child. In this case you can see why the resident set is copied, yes? It’s the smallest amount of memory that guarantees predictable performance subsequent to the call returning.
      - cryptonector 4y ago
        If the parent is threaded and the host has more than one CPU, then fork() == TLB shootdowns, which are slow. As well there's the cost of all those page faults that the two processes are likely to take to do the copying. And lastly there's all sorts of complexity involving multiple parent threads calling fork(), or the child calling fork() again (or vfork()) before calling exec. It's just much easier to copy the resident set and mark the address space as being CoW, because now you only have to worry about page faults for pages that are not in core anyways and so were going to fault anyways, and that means you don't have to worry about TLB shootdowns either (if a page is not in core, it's not referenced by any TLB either). You still have the multi-fork issues, but now you can use an atomic reference count on the address space.
      - btilly 4y ago
        The classic pattern was that you'd spin up something like Apache, load it full of read-only data, then start forking its children. Hoping that you'd share memory between children. With what you describe, you'd share nothing because at the point you've loaded it up, all the data you just loaded is resident. :-(
        cryptonector 4y ago
        If loading == mmap(2)ing read-only, then it's not a problem. Read-only pages don't get CoW'ed.
        4y ago
  - spc476 4y ago
    Hmmm, from <https://www.man7.org/linux/man-pages/man3/posix_spawn.3.html https://www.man7.org/linux/man-pages/man3/posix_spawn.3.html>: The posix_spawn() and posix_spawnp() functions provide the functionality of a combined fork(2) and exec(3), with some optional housekeeping steps in the child process before the exec(3). These functions are not meant to replace the fork(2) and execve(2) system calls. In fact, they provide only a subset of the functionality that can be achieved by using the system calls. Also, there's no way to set resource limits in the child process, nor switch user or group ID, using posix_spawn().
    - GoblinSlayer 4y ago
      For that you may need posix_spawn and exec, but still can evade fork completely.
  - duskwuff 4y ago
    Using fork() also means you end up with shared ownership of resources like file descriptors, which can have some pretty weird consequences.
    - cryptonector 4y ago
      This is true with all process creation APIs. Windows defaults to CLOEXEC semantics and you have to opt-in to child process inheriting open file handles, and that has caused problems. Unix defaults to not-CLOEXEC sematincs, and that too has caused problems.
      - temac 4y ago
        The Windows default can cause problems because of simple logic bugs. The Unix default can cause unsolvable problems because of races between threads. You should use CLOEXEC everywhere. Except you can't because you are using libraries.
        cryptonector 4y ago
        True. The main problem with the Unix default is that there wasn't a way to set O_CLOEXEC on all new FDs race-free until recently. That's a real problem. FD leaks to children can be bad, but most of the time they are not the end of the world, and often one can steal a closefrom() implementation from a BSD or Illumos as a workaround when you know exactly what you want to allow the child to inherit.
      - asveikau 4y ago
        closefrom() comes in handy for this. It's missing on some platforms (notably glibc and mac iirc) but actually not too hard to implement a work-alike.
        cryptonector 4y ago
        I've copied a closefrom() many a time. A common hack I've had to add is an argument FD that is not to be closed because. e.g., an flock is held on it.
- moomin 4y ago
  It was simple when processes were simple, but requirements got serious.
- turminal 4y ago
  > Then mark all pages as copy-on-write, and return to the caller. Unix actually copied the memory over initially.
deleted 4y ago
[deleted]

adastra22 4y ago

...is it? In it is original implementation, fork() was pretty trivial. All it did was create a new process entry in the kernel table, with all the pages and capabilities and such copied from the original process. Then mark all pages as copy-on-write, and return to the caller. Maybe not trivial, but much less complicated than loading an executable file from disk. My understanding of Linux internals is maybe 20 years out of date, so I am legitimately curious what makes fork() so complicated these days.

cryptonector 4y ago
fork() is not trivial now. Processes are huge now -- they have huge heaps among other things. Copying all that is expensive. In the 80s we tried COW, but that turns out to be very slow as well. What operating systems do now is immediately copy the resident set, then do COW for the rest of writable memory, but in large, multi-threaded processes, this is still too slow. Use vfork() or posix_spawn().
- adastra22 4y ago
  Hrm. Googling "fork linux copy-on-write" seems to find a lot of stack overflow answers from 2014-2015 claiming Linux marks pages as copy-on-write when fork() is called. I didn't see anything more recent in the first page of results. I could see it being worthwhile to immediately copy a few pages, like the top of the stack, but copying the whole resident set seems excessive. Especially since some of that data might not even be written to.
  - CHY872 4y ago
    So the problem is what happens to the old and new processes after the fork. To CoW, you need to mark all the pages read only in _both_ old and new which means that every memory write in the caller will now pagefault since the OS now has to lazily copy on both sides. So with true copy on write the fixed costs may be low but the marginal cost per memory write may be high in both parent and child. In this case you can see why the resident set is copied, yes? It’s the smallest amount of memory that guarantees predictable performance subsequent to the call returning.
    - cryptonector 4y ago
      If the parent is threaded and the host has more than one CPU, then fork() == TLB shootdowns, which are slow. As well there's the cost of all those page faults that the two processes are likely to take to do the copying. And lastly there's all sorts of complexity involving multiple parent threads calling fork(), or the child calling fork() again (or vfork()) before calling exec. It's just much easier to copy the resident set and mark the address space as being CoW, because now you only have to worry about page faults for pages that are not in core anyways and so were going to fault anyways, and that means you don't have to worry about TLB shootdowns either (if a page is not in core, it's not referenced by any TLB either). You still have the multi-fork issues, but now you can use an atomic reference count on the address space.
    - btilly 4y ago
      The classic pattern was that you'd spin up something like Apache, load it full of read-only data, then start forking its children. Hoping that you'd share memory between children. With what you describe, you'd share nothing because at the point you've loaded it up, all the data you just loaded is resident. :-(
      - cryptonector 4y ago
        If loading == mmap(2)ing read-only, then it's not a problem. Read-only pages don't get CoW'ed.
        4y ago
- spc476 4y ago
  Hmmm, from <https://www.man7.org/linux/man-pages/man3/posix_spawn.3.html https://www.man7.org/linux/man-pages/man3/posix_spawn.3.html>: The posix_spawn() and posix_spawnp() functions provide the functionality of a combined fork(2) and exec(3), with some optional housekeeping steps in the child process before the exec(3). These functions are not meant to replace the fork(2) and execve(2) system calls. In fact, they provide only a subset of the functionality that can be achieved by using the system calls. Also, there's no way to set resource limits in the child process, nor switch user or group ID, using posix_spawn().
  - GoblinSlayer 4y ago
    For that you may need posix_spawn and exec, but still can evade fork completely.
- duskwuff 4y ago
  Using fork() also means you end up with shared ownership of resources like file descriptors, which can have some pretty weird consequences.
  - cryptonector 4y ago
    This is true with all process creation APIs. Windows defaults to CLOEXEC semantics and you have to opt-in to child process inheriting open file handles, and that has caused problems. Unix defaults to not-CLOEXEC sematincs, and that too has caused problems.
    - temac 4y ago
      The Windows default can cause problems because of simple logic bugs. The Unix default can cause unsolvable problems because of races between threads. You should use CLOEXEC everywhere. Except you can't because you are using libraries.
      - cryptonector 4y ago
        True. The main problem with the Unix default is that there wasn't a way to set O_CLOEXEC on all new FDs race-free until recently. That's a real problem. FD leaks to children can be bad, but most of the time they are not the end of the world, and often one can steal a closefrom() implementation from a BSD or Illumos as a workaround when you know exactly what you want to allow the child to inherit.
    - asveikau 4y ago
      closefrom() comes in handy for this. It's missing on some platforms (notably glibc and mac iirc) but actually not too hard to implement a work-alike.
      - cryptonector 4y ago
        I've copied a closefrom() many a time. A common hack I've had to add is an argument FD that is not to be closed because. e.g., an flock is held on it.
moomin 4y ago
It was simple when processes were simple, but requirements got serious.
turminal 4y ago
> Then mark all pages as copy-on-write, and return to the caller. Unix actually copied the memory over initially.

deleted 4y ago

[deleted]