12 ms·

Fork() without exec() is dangerous in large programs (2016)

128 points by goranmoomin 4y ago

Skunkleton 4y ago
Odd to not see any mention of vfork. vfork solves the problems with fork/exec for large programs.
- remexre 4y ago
  Isn't vfork much worse in terms of the problem the author is talking about, since the child can now acquire locks in the _parent's_ address space?
  - stefan_ 4y ago
    I thought the point of vfork is that they do not share an address space. But there are other things still shared and they should really just have a CreateProcess.
    - remexre 4y ago
      no, fork creates a new address space, vfork doesn't the posix_spawn mentioned in the article is effectively the equivalent of CreateProcess
      - medoc 4y ago
        Last time I looked, posix_spawn() just called fork/exec
        int_19h 4y ago
        That's an implementation detail at this point. The idea is to have a single syscall that takes all the information needed to spawn the process, and does so atomically, without the need to spread it across several calls. On Win32, that's CreateProcess(). On POSIX, the equivalent is posix_spawn().
        cryptonector 4y ago
        In glibc it uses vfork() in some cases. In Solaris/Illumos it uses vfork() or vforkx(). In principle posix_spawn() can be a system call.
        comex 4y ago
        On macOS it is a system call.
    - monocasa 4y ago
      They still share an address space until exec replaces it for one of them. Particularly awful is that they share the same mutable stack which is a pathway that only leads to the inner circle of hell.
      - stefan_ 4y ago
        Assuming you call exec, of course. To not call exec after vfork is not an option; one of the many ways the fork family of functions are fundamentally broken.
        monocasa 4y ago
        Well, without undefined behavior you can also call _exit(), continue within the same function, and receive conforming signals. Unfortunately this isn't always spelled out and there's code out there that definitely does other work invoking undefined behavior.
- tsimionescu 4y ago
  vfork() does the opposite of solving these problems. While there are a few functions that you can call after fork(), there is absolutely no function you can call after vfork() before exec(). You can't even write most local variables. vfork() solves the problem of not wasting so much time on fork() when you're just going to call exec() afterwards (fork() does A LOT of work - potentially, anyway).
  - cryptonector 4y ago
    > vfork() does the opposite of solving these problems. While there are a few functions that you can call after fork(), there is absolutely no function you can call after vfork() before exec(). You can't even write most local variables. Mostly wrong. You can call functions on the child side of vfork(), but you don't want to exec() in them -- you want to exec() in the same function that called vfork(). And you can write to local variables, but you have to be careful about it. There's a ton of vfork()-using code that does these things. Now, it's true that a compiler optimizer that knows nothing about vfork() but knows about _exit()'s semantics, could delete code it thinks is unreachable. So there is some issue, but you can just disable the optimizer if you run into this.
    - tsimionescu 4y ago
      That's all undefined behavior, under POSIX at least [0]: > The vfork() function has the same effect as fork(2), except that the behavior is undefined if the process created by vfork() either modifies any data other than a variable of type pid_t used to store the return value from vfork(), or returns from the function in which vfork() was called, or calls any other function before successfully calling _exit(2) or one of the exec(3) family of functions. So sure - you can do these things, but they have very little defined semantics after vfork(). It is true that Linux describes the semantics more clearly, so perhaps on Linux it is safer to use. [0] https://man7.org/linux/man-pages/man2/vfork.2.html https://man7.org/linux/man-pages/man2/vfork.2.html
      - cryptonector 4y ago
        If you can call _exit() and exec, then you can call other functions too. I do believe that the Open Group has changed the description of vfork() to discourage its use because of an old and incorrect paper from the 80s. Actual implementations of vfork() are not as dangerous as the Open Group text purports them to be. Moreover, most posix_spawn() implementations use vfork(), and they call more functions than _exit() and exec on the child side. Let's be reasonable about these things.
elankart 4y ago
Let’s also not forget to call our how APIs have to be “fork” aware. I’m surprised fork is still widely in used given all of these downsides
- tyingq 4y ago
  I imagine it comes up fairly often for languages that don't do well with threads. No shortage of those.
chubot 4y ago
FWIW this is the same reason you can't implement implement a portable Unix shell in portable Go. (And similar issues with an init daemon) Go only exports os.ForkExec() -- there is no os.Fork() or os.Exec(), because the things you can do between the calls could break Go's threaded runtime. (Goroutines are implemented with OS threads.) Some elaboration on that: https://lobste.rs/s/hj3np3/mvdan_sh_posix_shell_go#c_qszuer https://lobste.rs/s/hj3np3/mvdan_sh_posix_shell_go#c_qszuer That is, the space between fork and exec is where pipelines are implemented, but also entire subinterpreters/subshells. The shell actually uses copy-on-write usefully. (And yes I'm aware that there's a good argument that the shell is almost the ONLY program that needs fork() !) ---- A lot of people have asked me why not implement Oil in Go and various other languages, so I wrote this page: https://github.com/oilshell/oil/wiki/FAQ:-Why-Not-Write-Oil-in-X%3F https://github.com/oilshell/oil/wiki/FAQ:-Why-Not-Write-Oil-... So the funny thing is that Python is a lower level language than Go for this particular problem. It doesn't do anything weird with regard to syscalls. I'm still looking for help on this (and donations to pay people other than me): Oil Is Being Implemented "Middle Out" https://www.oilshell.org/blog/2022/03/middle-out.html https://www.oilshell.org/blog/2022/03/middle-out.html
- gizmo686 4y ago
  > The shell actually uses copy-on-write usefully. (And yes I'm aware that there's a good argument that the shell is almost the ONLY program that needs fork() !) It's been a while since I looked at it, but I believe Android uses fork for it's copy-on-write sementics to optimize app startup. On boot it initializes a single instance of the app runtime environment. Then when you launch apps that initial process is forked. As a result you do not need to reinitialize the runtime for every app launch.
  - chubot 4y ago
    Yes I think the argument is that Android (and Chrome) could use something like vfork or posix_spawn(). I'm not sure which, if any; I'd like to see an analysis of that... The issue is what kernel state is preserved/shared across the process creation call. Every process sort of has a "mirror" in kernel memory. The user memory is CoW, and I suppose you also have to choose whether to copy or reference every kernel data structure as well --- open files in FD tables which point to disk/pipes/sockets, locks which seem to be nonsensical, etc. But probably you can get the "warmup" property without the full semantics of fork(). That is the CoW of user memory is a somewhat separate choice from the kernel data structures. ---- As far as the shell .... In the recent linked thread, Ninja uses posix_spawn because it has a simple use of subprocesses: https://news.ycombinator.com/item?id=30503382 https://news.ycombinator.com/item?id=30503382 So there are definitely cases where a shell uses fork/exec like Ninja, so you could imagine optimizing it. But the subshell/subinterpreter case is probably the most general -- the language semantics depend on it. And it's actually useful, e.g. this "alternative shell challenge": https://www.oilshell.org/blog/2020/02/good-parts-sketch.html#the-alternative-shell-challenge https://www.oilshell.org/blog/2020/02/good-parts-sketch.html... and I use that pattern literally yesterday and this morning, etc. ---- edit: looks like fish already addresses this, i.e. where you can use posix_spawn in a shell, and where you can't! https://news.ycombinator.com/item?id=31743230 https://news.ycombinator.com/item?id=31743230
billpg 4y ago
I asked this a year or so ago. Interesting to read this article in light of that discussion. https://news.ycombinator.com/item?id=863871 https://news.ycombinator.com/item?id=863871 (13 years? Yikes!)
- cryptonector 4y ago
  fork() and the exec system calls exist because they were easy to implement in the 70s on PDPs, and fork() was cheap enough then, but much more importantly, it got the shell developers out of having to write and evolve a more complex API in the kernel. With fork/exec a shell developer could try lots of variations for executing a pipe command w/o having to develop any more kernel code. For example, until BSD came along, not much had to change in kernel land for any shell. Job control meant that the shell would need to put all the processes for a job in the same pgrp, and also there was a need to add `setsid()`.
  - chubot 4y ago
    Hm a lot of that doesn't match my understanding, which is more like: 1. Unix has fork() because it was influenced by Multics. I don't have the citation now, but I think some parts of Unix were from ITSS, perhaps the hierarchical file system, and some were from Multics. It was a drastic simplification of those systems, but with the same ideas. --- 2. The shell developer and the kernel developer were really the same person -- Ken Thompson. I link to his original paper in this post [1]. The original Thompson shell had pipelines and redirects, which are most of what happens between fork() and exec() in a shell. Also of note is a video by Stephen Bourne who says that Ken Thompson being away at Berkeley was a good time to turn his shell into a programming language (Bourne shell) [2]. Similarly, I read that Bill Joy added chroot() to Unix simply because he needed it for something he was doing one time (building another Unix system, I would imagine). --- 3. Bill Joy also added job control to the kernel and the terminal for his csh shell. It's a very tightly coupled and ugly design. So the key point is that we shouldn't assume any notion of "shell developers" or "APIs". The kernel developer and the shell developer were really the same person -- Thompson and Joy. Unix is more of a holistic system than a modular one; it's porous by design! --- Since job control, it appears that have been almost no system calls added for a shell. Although I have a feeling a few fcntl() operations were added for a shell, i.e. the difference between dup2() and fcntl(F_DUPFD). But I don't remember the argument offhand. So I think the history is basically what usually happens -- once the same person stops working on 2 sides of an interface, the interface calcifies. It's a little like the relationship between ISAs and C. So we will probably have fork() forever, but that doesn't mean there can't be evolution in a better direction. [1] Unix Shell: History and Trivia https://www.oilshell.org/blog/2021/08/history-trivia.html https://www.oilshell.org/blog/2021/08/history-trivia.html [2] https://www.oilshell.org/blog/2022/03/middle-out.html#more-writing-on-middle-out-and-translation https://www.oilshell.org/blog/2022/03/middle-out.html#more-w...
legalcorrection 4y ago
Won't stop the *nix know-nothings from criticizing Windows for not natively supporting fork().
- krylon 4y ago
  Funny side note, Perl fakes fork() on Windows using (I believe) threads. I am not sure if that is better or worse than Windows having fork() natively, though. Some people would probably argue that if you use Perl, you have much bigger problems to worry about, but that's another debate.
jordemort 4y ago
If you mix fork with threads, you're going to have a [undefined behavior] time. It seems like if you link with the sqlite that comes with macOS, you're using threads whether you like it or not. I think ending up at "you shouldn't use fork() at all" is a bit of an extreme conclusion, though. BTW, article title needs a (2016). It appears that the relevant Python bug has long since been closed, by avoiding linking with the system sqlite on macOS.
- masklinn 4y ago
  > I think ending up at "you shouldn't use fork() at all" is a bit of an extreme conclusion, though. Is it? There are more descriptive (as opposed to procedural) APIs which behave in a safer and more well-defined manner to do it these days. Unless you're implementing a shell, fork has never been a great tool. As one commenter noted 3 months back: > The dense fog lifts, tree branches part, a ray of light beams down on a pedestal revealing the hidden intentions of the ancients. A plaque states "The operational semantics of the most basic primitives of your operating system are designed to simplify the implementation of shells." You hesitantly lift your eyes to the item presented upon the pedestal, take a pause in respect, then turn away slumped and disappointed but not entirely surprised. As you walk you shake your head trying to evict the after image of a beam of light illuminating a turd. https://news.ycombinator.com/item?id=30504470 https://news.ycombinator.com/item?id=30504470
  - aidenn0 4y ago
    IMO, fork is strictly better than threads as a tool for having operations perform off of the main thread; they get all the state they need at the beginning and they can use IPC to return the result. TFA does allow for off-process operations, but all of the inputs to the operation would need to be passed explicitly. In this sense, I suppose TFA isn't arguing against multiprocessing per-se, but against the specific type that implicitly includes all of the current process state (which has both up- and down-sides).
    - quinoablast 4y ago
      That is basically how NGINX works if you run it in daemon mode. When you start or reload the server, the main process initializes common state then forks to become a worker process. Although I would recommend avoiding any IPC past that if possible
deleted 4y ago
[deleted]
medoc 4y ago
fork() also presents performance issues for programs with a large virtual space. Here vfork() helps, but it has even more pitfalls than fork(). I had written a small doc about converting the recollindex Recoll indexer from fork() to vfork() a while ago: https://www.lesbonscomptes.com/recoll/pages/idxthreads/forkingRecoll.html https://www.lesbonscomptes.com/recoll/pages/idxthreads/forki...
perryizgr8 4y ago
It's only dangerous if you use libraries without fully understanding what they're doing. And most well designed libraries will avoid creating threads, and will do so only when you make it explicit that you want it to happen. I also find that libraries that absolutely need to make their own threads are better off being their own process. Then you can use proper communication methods to pass data.
- aidenn0 4y ago
  Per TFA, a large fraction of OS X system libraries use threads, so if you're developing for the Macintosh, fork() is already out.
zokier 4y ago
See also this recent 340 (!) comment thread about the issues of fork https://news.ycombinator.com/item?id=30502392 https://news.ycombinator.com/item?id=30502392
- phendrenad2 4y ago
  Top comment there is gold (and makes a good point).
ttoinou 4y ago
> Only use fork in toy programs. The challenge is that successful toy programs grow into large ones, and large programs eventually use threads. It might be best just to not bother. How do you create a new process and pipe it data in a fast fashion without using fork, exec or posix_spawn ?
- ehvatum 4y ago
  Use a shared memory region. For example: https://github.com/erikhvatum/py_interprocess_shared_memory_blob https://github.com/erikhvatum/py_interprocess_shared_memory_...
- masklinn 4y ago
  > without using fork, exec or posix_spawn ? Did you manage to forget what you'd read two paragraphs earlier when you reached this bit? Because the essay's first recommendation is literally: > Only use fork to immediately call exec (or just use posix_spawn). It seems difficult to infer "don't use exec or posix_spawn" from this.
  - ttoinou 4y ago
    Ofc I read that. I think a "only" was missing in the sentence, hence the confusion
- cryptonector 4y ago
  posix_spawn or vfork+exec
garethrowlands 4y ago
"The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. In this paper, we argue that fork was a clever hack for machines and programs of the 1970s that has long outlived its usefulness and is now a liability. We catalog the ways in which fork is a terrible abstraction for the modern programmer to use, describe how it compromises OS implementations, and propose alternatives." from A Fork in the Road, <https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf https://www.microsoft.com/en-us/research/uploads/prod/2019/0...>
- wongarsu 4y ago
  fork() and exec() is a reflection of the unix philosophy: small tools for separate tasks, and as a result you have great composability. You can set up right etc for your new process by calling any number of system calls between fork() and exec(). Windows is a great example of the alternative (following "Windows philosophy"): There are about three different API calls for creating a new process, each with a heap of complicated optional arguments. The API becomes more complex, less composable, less extensible and less powerful. But also easier to reason about and easier for the kernel to provide, and arguably with fewer footguns.
  - klodolph 4y ago
    fork() is not a small tool. It is a MASSIVE, unwieldy tool that is difficult to use correctly. It happens to have a small interface.
    - adastra22 4y ago
      ...is it? In it is original implementation, fork() was pretty trivial. All it did was create a new process entry in the kernel table, with all the pages and capabilities and such copied from the original process. Then mark all pages as copy-on-write, and return to the caller. Maybe not trivial, but much less complicated than loading an executable file from disk. My understanding of Linux internals is maybe 20 years out of date, so I am legitimately curious what makes fork() so complicated these days.
tuxoko 4y ago
The suggestions here aren't really great. What you should do is already written in the fork(2) manpage https://man7.org/linux/man-pages/man2/fork.2.html https://man7.org/linux/man-pages/man2/fork.2.html "After a fork() in a multithreaded program, the child can safely call only async-signal-safe functions (see signal-safety(7)) until such time as it calls execve(2)." So just use only async-singal-safe function https://man7.org/linux/man-pages/man7/signal-safety.7.html https://man7.org/linux/man-pages/man7/signal-safety.7.html I don't know why so many people still hit this issue when it already told you what you can do and not do in the document. I've done this sort of things without any issue.
- the_mitsuhiko 4y ago
  Because in practice it’s almost impossible to know if you are single threaded still. Plenty of system libraries have background threads.
  - emmelaich 4y ago
    Not to disbelieve you but which ones?
- astrange 4y ago
  There are many languages where it's difficult to impossible to be async-signal-safe because you can't avoid memory allocations.
Strilanc 4y ago
Another danger using fork is it duplicates the internal state of pseudo random number generators. It's a great way to accidentally take the same random samples in every process, utterly trashing any statistics you were intending to do. Bonus: the python multiprocessing module silently uses fork by default. Person A writes a "make multiprocessing convenient" library, Person B writes a sampling library, you put them together and... whoops!.
- cryptonector 4y ago
  Libraries like that should use pthread_atfork() to automatically reset/reseed/whatever state as needed at fork() time.
  - Strilanc 4y ago
    I don't think that's really a viable strategy in practice in an ecosystem as complex as python's. There's too many libraries and too many little corner cases and interactions around what the behavior should be. For example, suppose I am using library A and I initialized the random number generator with a fixed seed. Clearly when I fork it's not appropriate for A to reseed, because I wanted fixed behavior. Something is very wrong so probably there should be an exception. But now suppose I was using library B which was using A and B handles getting system entropy to seed A. Now it is clear that when I fork I probably want B to reseed A, but alas A has already raised an exception because it was given a (from its perspective) fixed seed. So now A needs to be redesigned to be given a seed and like some sort of intent on what should happen when forking, and oh my god wow this is creating a lot of work for everyone everywhere this is not actually going to be done consistently and cannot be trusted.
    - cryptonector 4y ago
      If you're writing a simulation or a test, then you'll want the PRNG to stay unchanged, and you'll want to be in control of any reseeding. For all other RNG uses, you really do want it to reseed. A cryptographic PRNG vs. a simulation PRNG are very different things, and should be different libraries.
throwaway892238 4y ago
> When I ran into this problem, I was just trying to run all of Bluecore's unit tests on my Mac laptop. We use nose's multiprocess mode, which uses Python's multiprocessing module to utilize multiple CPUs. Unfortunately, the tests hung, even though they passed on our Linux test server. There will never be a time at which you can reliably expect any program developed on one system to "just work" on a different system. This person wasted a lot of time tracking down what was essentially a portability bug. Did they need this to be portable? Was this time well spent generating business value? Pick one system for development through production, stick to it. There will be portability bugs hiding in your code, but you will never have to fix them. You will be upset for a minute that you can't use a different system, but you will get over it.
- csours 4y ago
  I can write code that works NOW, but there is no guarantee that code will work in the future.
krylon 4y ago
The devil, as they say, is in the details. But to be fair, the only times I can recall using fork() without exec() were forking network servers, and that was mostly me learning about doing network stuff, and a forking server was the easiest to implement manually. Oh yeah, and that one time I accidentally wrote a fork bomb trying to stress test a DNS server. At least I learned something from my mistake. ;-) EDIT: To me, using fork() without exec() is kind of like operator overloading - there are cases where it absolutely is the right tool, but these aren't very numerous, so one should exercise caution. A lot.
dang 4y ago
Discussed at the time: Fork() without exec() is dangerous in large programs - https://news.ycombinator.com/item?id=12302539 https://news.ycombinator.com/item?id=12302539 - Aug 2016 (101 comments)
ridiculous_fish 4y ago
fish shell uses posix_spawn sometimes because of its performance benefits. We can't use it in the following cases: 1. No analog to tcsetpgrp, so it's no good if job control is enabled 2. No analog to fchdir, meaning you have to synchronize with fchdir elsewhere in the progarm 3. Error codes do not convey enough information for good error messages (e.g. if a file doesn't exist, posix_spawn doesn't tell you which file) 4. Inconsistent behavior around dup2 fd redirections and CLO_EXEC. 5. Inconsistent behavior for shebangless scripts These are basically deal-breakers so fish also supports a fork/exec path. However the performance benefits of posix_spawn are too real to ignore so fish uses posix_spawn when it can, and fork/exec when it must.
- cryptonector 4y ago
  Shells should really use `vfork()` to exec, and `fork()` for subshells (maybe).
  - mort96 4y ago
    Except that the set of things you are allowed to do after vfork() and before exec() is extremely small. I agree that things should use vfork or posix_spawn when possible, but it wouldn't surprise me at all if there's a similar list of cases when a vfork() path wouldn't be possible.
    - cryptonector 4y ago
      That set was made artificially small by the Open Group, but is not actually that small.
- chubot 4y ago
  OK very interesting ... is the performance benefit on certain platforms, or everywhere? A previous thread says Ninja is faster on OS X and Solaris because of it. Though that does seem like a large number of corner cases, probably learned through painful experience :-/
  - 4y ago
layer8 4y ago
New programming language implementations should maybe make fork() and multithreading be mutually exclusive at link time by default, and only allow them together in an unsafe-I-know-what-I’m-doing mode (if at all).
infogulch 4y ago
The dense fog lifts, tree branches part, a ray of light shines down on the ruins of a moss-covered pedestal revealing the hidden intentions of the ancients. A plaque states "The operational semantics of the most basic primitives of your operating system are optimized to simplify the implementation of command line shells." You look upon the pedestal, pause in respect, then turn away disappointed but unsurprised. As you walk you shake your head trying to evict the after image of a beam of light illuminating a turd.
- infogulch 4y ago
  I wrote this silly little bit on a previous fork() thread and touched it up a little. https://news.ycombinator.com/item?id=30504470 https://news.ycombinator.com/item?id=30504470
  - cryptonector 4y ago
    It was a glorious comment then, and it is now. BTW, it's unclear whether the turd is... the specific truth revealed, or the revelation itself (since it could be incorrect). It's still a glorious comment.
    - infogulch 4y ago
      Thank you! The goal was to evoke the same emotional response I felt when this idea struck me (I think when I was reading earlier comments in that thread in fact), so I guess there's as much in the sequence as the specific objects. The semantic design of fork was always incomprehensible to me (not what but why), so the setting and pedestal and plaque represent how strongly it struck me and the depth of its historical explanatory power. The turd is how I feel about it. I don't resent the designer's history or motivations or incentives and I'm happy to know the truth. I'm just sad/depressed that this is the reason why things are the way they are, that the original design intent is so misaligned with the needs of modern systems, how it limits our capabilities today even down to hardware, and how unlikely it is for this to change. I don't suppose a turd would last very long on a pedestal, or that the ancients would put a turd there on purpose. Maybe they put something beautiful there back then, but times have changed and now we're stuck with it and it's kind of shit.
wruza 4y ago
Curious why there isn’t an interface in which all required handles and resources could be passed to a child process explicitly. E.g.: execvpehm( ..., int *handles, size_t, void **pages, size_t, /* etc */ ); Would remove so many headaches with concurrency and accidental inheritance.
- emmelaich 4y ago
  I suppose Linux's clone3 call goes the closest to that.
londons_explore 4y ago
One other option is to fork all the threads too. Since you probably don't know what all the other threads in your process are up to, your only option is to attach a debugger to all of them, halt them all, and copy all their state into brand new threads in the child process. Do it all correctly and you end up with a multi-threaded-fork. You still need to fix up signal handlers, interrupted syscalls, various notification API's that no longer work, memory mapped temp files used for IPC, pipes and sockets, and a bunch of other things. But a fork of a complex process is possible. It just isn't easy.
olliej 4y ago
And it’s explicitly unsupported on macOS at least.
kazinator 4y ago
The problems with fork in the face of threads are caused by threads, not by fork. Fork was there first, and it is part of a system that is designed and integrated well. Threads were bolted onto Unix in a hamfisted way, breaking more than just fork. For instance, threads broke relative paths, requiring "at" functions like openat to be invented, an ugly stop-gap measure. Threads were badly integrated with signal handling too, another example. Blaming those existing mechanisms is purely an emotional argument, from the perspective of being infatuated with threads. The design of threads (coming from various efforts that became POSIX threads) came from such an infatuation: the desire to get any kinds of threads working at any cost, while ignoring the global state that exists in a Unix process, and the need to make a lot of it thread-local, or at least optionally so. A thread-local working directory or signal mask would have caused difficulties in hack thread implementations that used user space scheduling or M:N (M user space threads to N kernel tasks). The situation we have today largely comes from the initial reluctance to accept the fact that each thread has to be an entity known to the kernel; the belief that user space threads are viable into the long-term future.

128 points by goranmoomin 4y ago

Skunkleton 4y ago

Odd to not see any mention of vfork. vfork solves the problems with fork/exec for large programs.

remexre 4y ago
Isn't vfork much worse in terms of the problem the author is talking about, since the child can now acquire locks in the _parent's_ address space?
- stefan_ 4y ago
  I thought the point of vfork is that they do not share an address space. But there are other things still shared and they should really just have a CreateProcess.
  - remexre 4y ago
    no, fork creates a new address space, vfork doesn't the posix_spawn mentioned in the article is effectively the equivalent of CreateProcess
    - medoc 4y ago
      Last time I looked, posix_spawn() just called fork/exec
      - int_19h 4y ago
        That's an implementation detail at this point. The idea is to have a single syscall that takes all the information needed to spawn the process, and does so atomically, without the need to spread it across several calls. On Win32, that's CreateProcess(). On POSIX, the equivalent is posix_spawn().
      - cryptonector 4y ago
        In glibc it uses vfork() in some cases. In Solaris/Illumos it uses vfork() or vforkx(). In principle posix_spawn() can be a system call.
        comex 4y ago
        On macOS it is a system call.
  - monocasa 4y ago
    They still share an address space until exec replaces it for one of them. Particularly awful is that they share the same mutable stack which is a pathway that only leads to the inner circle of hell.
    - stefan_ 4y ago
      Assuming you call exec, of course. To not call exec after vfork is not an option; one of the many ways the fork family of functions are fundamentally broken.
      - monocasa 4y ago
        Well, without undefined behavior you can also call _exit(), continue within the same function, and receive conforming signals. Unfortunately this isn't always spelled out and there's code out there that definitely does other work invoking undefined behavior.
tsimionescu 4y ago
vfork() does the opposite of solving these problems. While there are a few functions that you can call after fork(), there is absolutely no function you can call after vfork() before exec(). You can't even write most local variables. vfork() solves the problem of not wasting so much time on fork() when you're just going to call exec() afterwards (fork() does A LOT of work - potentially, anyway).
- cryptonector 4y ago
  > vfork() does the opposite of solving these problems. While there are a few functions that you can call after fork(), there is absolutely no function you can call after vfork() before exec(). You can't even write most local variables. Mostly wrong. You can call functions on the child side of vfork(), but you don't want to exec() in them -- you want to exec() in the same function that called vfork(). And you can write to local variables, but you have to be careful about it. There's a ton of vfork()-using code that does these things. Now, it's true that a compiler optimizer that knows nothing about vfork() but knows about _exit()'s semantics, could delete code it thinks is unreachable. So there is some issue, but you can just disable the optimizer if you run into this.
  - tsimionescu 4y ago
    That's all undefined behavior, under POSIX at least [0]: > The vfork() function has the same effect as fork(2), except that the behavior is undefined if the process created by vfork() either modifies any data other than a variable of type pid_t used to store the return value from vfork(), or returns from the function in which vfork() was called, or calls any other function before successfully calling _exit(2) or one of the exec(3) family of functions. So sure - you can do these things, but they have very little defined semantics after vfork(). It is true that Linux describes the semantics more clearly, so perhaps on Linux it is safer to use. [0] https://man7.org/linux/man-pages/man2/vfork.2.html https://man7.org/linux/man-pages/man2/vfork.2.html
    - cryptonector 4y ago
      If you can call _exit() and exec, then you can call other functions too. I do believe that the Open Group has changed the description of vfork() to discourage its use because of an old and incorrect paper from the 80s. Actual implementations of vfork() are not as dangerous as the Open Group text purports them to be. Moreover, most posix_spawn() implementations use vfork(), and they call more functions than _exit() and exec on the child side. Let's be reasonable about these things.

elankart 4y ago

Let’s also not forget to call our how APIs have to be “fork” aware. I’m surprised fork is still widely in used given all of these downsides

tyingq 4y ago
I imagine it comes up fairly often for languages that don't do well with threads. No shortage of those.

chubot 4y ago

FWIW this is the same reason you can't implement implement a portable Unix shell in portable Go. (And similar issues with an init daemon) Go only exports os.ForkExec() -- there is no os.Fork() or os.Exec(), because the things you can do between the calls could break Go's threaded runtime. (Goroutines are implemented with OS threads.) Some elaboration on that: https://lobste.rs/s/hj3np3/mvdan_sh_posix_shell_go#c_qszuer https://lobste.rs/s/hj3np3/mvdan_sh_posix_shell_go#c_qszuer That is, the space between fork and exec is where pipelines are implemented, but also entire subinterpreters/subshells. The shell actually uses copy-on-write usefully. (And yes I'm aware that there's a good argument that the shell is almost the ONLY program that needs fork() !) ---- A lot of people have asked me why not implement Oil in Go and various other languages, so I wrote this page: https://github.com/oilshell/oil/wiki/FAQ:-Why-Not-Write-Oil-in-X%3F https://github.com/oilshell/oil/wiki/FAQ:-Why-Not-Write-Oil-... So the funny thing is that Python is a lower level language than Go for this particular problem. It doesn't do anything weird with regard to syscalls. I'm still looking for help on this (and donations to pay people other than me): Oil Is Being Implemented "Middle Out" https://www.oilshell.org/blog/2022/03/middle-out.html https://www.oilshell.org/blog/2022/03/middle-out.html

gizmo686 4y ago
> The shell actually uses copy-on-write usefully. (And yes I'm aware that there's a good argument that the shell is almost the ONLY program that needs fork() !) It's been a while since I looked at it, but I believe Android uses fork for it's copy-on-write sementics to optimize app startup. On boot it initializes a single instance of the app runtime environment. Then when you launch apps that initial process is forked. As a result you do not need to reinitialize the runtime for every app launch.
- chubot 4y ago
  Yes I think the argument is that Android (and Chrome) could use something like vfork or posix_spawn(). I'm not sure which, if any; I'd like to see an analysis of that... The issue is what kernel state is preserved/shared across the process creation call. Every process sort of has a "mirror" in kernel memory. The user memory is CoW, and I suppose you also have to choose whether to copy or reference every kernel data structure as well --- open files in FD tables which point to disk/pipes/sockets, locks which seem to be nonsensical, etc. But probably you can get the "warmup" property without the full semantics of fork(). That is the CoW of user memory is a somewhat separate choice from the kernel data structures. ---- As far as the shell .... In the recent linked thread, Ninja uses posix_spawn because it has a simple use of subprocesses: https://news.ycombinator.com/item?id=30503382 https://news.ycombinator.com/item?id=30503382 So there are definitely cases where a shell uses fork/exec like Ninja, so you could imagine optimizing it. But the subshell/subinterpreter case is probably the most general -- the language semantics depend on it. And it's actually useful, e.g. this "alternative shell challenge": https://www.oilshell.org/blog/2020/02/good-parts-sketch.html#the-alternative-shell-challenge https://www.oilshell.org/blog/2020/02/good-parts-sketch.html... and I use that pattern literally yesterday and this morning, etc. ---- edit: looks like fish already addresses this, i.e. where you can use posix_spawn in a shell, and where you can't! https://news.ycombinator.com/item?id=31743230 https://news.ycombinator.com/item?id=31743230

billpg 4y ago

I asked this a year or so ago. Interesting to read this article in light of that discussion. https://news.ycombinator.com/item?id=863871 https://news.ycombinator.com/item?id=863871 (13 years? Yikes!)

cryptonector 4y ago
fork() and the exec system calls exist because they were easy to implement in the 70s on PDPs, and fork() was cheap enough then, but much more importantly, it got the shell developers out of having to write and evolve a more complex API in the kernel. With fork/exec a shell developer could try lots of variations for executing a pipe command w/o having to develop any more kernel code. For example, until BSD came along, not much had to change in kernel land for any shell. Job control meant that the shell would need to put all the processes for a job in the same pgrp, and also there was a need to add `setsid()`.
- chubot 4y ago
  Hm a lot of that doesn't match my understanding, which is more like: 1. Unix has fork() because it was influenced by Multics. I don't have the citation now, but I think some parts of Unix were from ITSS, perhaps the hierarchical file system, and some were from Multics. It was a drastic simplification of those systems, but with the same ideas. --- 2. The shell developer and the kernel developer were really the same person -- Ken Thompson. I link to his original paper in this post [1]. The original Thompson shell had pipelines and redirects, which are most of what happens between fork() and exec() in a shell. Also of note is a video by Stephen Bourne who says that Ken Thompson being away at Berkeley was a good time to turn his shell into a programming language (Bourne shell) [2]. Similarly, I read that Bill Joy added chroot() to Unix simply because he needed it for something he was doing one time (building another Unix system, I would imagine). --- 3. Bill Joy also added job control to the kernel and the terminal for his csh shell. It's a very tightly coupled and ugly design. So the key point is that we shouldn't assume any notion of "shell developers" or "APIs". The kernel developer and the shell developer were really the same person -- Thompson and Joy. Unix is more of a holistic system than a modular one; it's porous by design! --- Since job control, it appears that have been almost no system calls added for a shell. Although I have a feeling a few fcntl() operations were added for a shell, i.e. the difference between dup2() and fcntl(F_DUPFD). But I don't remember the argument offhand. So I think the history is basically what usually happens -- once the same person stops working on 2 sides of an interface, the interface calcifies. It's a little like the relationship between ISAs and C. So we will probably have fork() forever, but that doesn't mean there can't be evolution in a better direction. [1] Unix Shell: History and Trivia https://www.oilshell.org/blog/2021/08/history-trivia.html https://www.oilshell.org/blog/2021/08/history-trivia.html [2] https://www.oilshell.org/blog/2022/03/middle-out.html#more-writing-on-middle-out-and-translation https://www.oilshell.org/blog/2022/03/middle-out.html#more-w...

legalcorrection 4y ago

Won't stop the *nix know-nothings from criticizing Windows for not natively supporting fork().

krylon 4y ago
Funny side note, Perl fakes fork() on Windows using (I believe) threads. I am not sure if that is better or worse than Windows having fork() natively, though. Some people would probably argue that if you use Perl, you have much bigger problems to worry about, but that's another debate.

jordemort 4y ago

If you mix fork with threads, you're going to have a [undefined behavior] time. It seems like if you link with the sqlite that comes with macOS, you're using threads whether you like it or not. I think ending up at "you shouldn't use fork() at all" is a bit of an extreme conclusion, though. BTW, article title needs a (2016). It appears that the relevant Python bug has long since been closed, by avoiding linking with the system sqlite on macOS.

masklinn 4y ago
> I think ending up at "you shouldn't use fork() at all" is a bit of an extreme conclusion, though. Is it? There are more descriptive (as opposed to procedural) APIs which behave in a safer and more well-defined manner to do it these days. Unless you're implementing a shell, fork has never been a great tool. As one commenter noted 3 months back: > The dense fog lifts, tree branches part, a ray of light beams down on a pedestal revealing the hidden intentions of the ancients. A plaque states "The operational semantics of the most basic primitives of your operating system are designed to simplify the implementation of shells." You hesitantly lift your eyes to the item presented upon the pedestal, take a pause in respect, then turn away slumped and disappointed but not entirely surprised. As you walk you shake your head trying to evict the after image of a beam of light illuminating a turd. https://news.ycombinator.com/item?id=30504470 https://news.ycombinator.com/item?id=30504470
- aidenn0 4y ago
  IMO, fork is strictly better than threads as a tool for having operations perform off of the main thread; they get all the state they need at the beginning and they can use IPC to return the result. TFA does allow for off-process operations, but all of the inputs to the operation would need to be passed explicitly. In this sense, I suppose TFA isn't arguing against multiprocessing per-se, but against the specific type that implicitly includes all of the current process state (which has both up- and down-sides).
  - quinoablast 4y ago
    That is basically how NGINX works if you run it in daemon mode. When you start or reload the server, the main process initializes common state then forks to become a worker process. Although I would recommend avoiding any IPC past that if possible

deleted 4y ago

[deleted]

medoc 4y ago

fork() also presents performance issues for programs with a large virtual space. Here vfork() helps, but it has even more pitfalls than fork(). I had written a small doc about converting the recollindex Recoll indexer from fork() to vfork() a while ago: https://www.lesbonscomptes.com/recoll/pages/idxthreads/forkingRecoll.html https://www.lesbonscomptes.com/recoll/pages/idxthreads/forki...

perryizgr8 4y ago

It's only dangerous if you use libraries without fully understanding what they're doing. And most well designed libraries will avoid creating threads, and will do so only when you make it explicit that you want it to happen. I also find that libraries that absolutely need to make their own threads are better off being their own process. Then you can use proper communication methods to pass data.

aidenn0 4y ago
Per TFA, a large fraction of OS X system libraries use threads, so if you're developing for the Macintosh, fork() is already out.

zokier 4y ago

See also this recent 340 (!) comment thread about the issues of fork https://news.ycombinator.com/item?id=30502392 https://news.ycombinator.com/item?id=30502392

phendrenad2 4y ago
Top comment there is gold (and makes a good point).

ttoinou 4y ago

> Only use fork in toy programs. The challenge is that successful toy programs grow into large ones, and large programs eventually use threads. It might be best just to not bother. How do you create a new process and pipe it data in a fast fashion without using fork, exec or posix_spawn ?

ehvatum 4y ago
Use a shared memory region. For example: https://github.com/erikhvatum/py_interprocess_shared_memory_blob https://github.com/erikhvatum/py_interprocess_shared_memory_...
masklinn 4y ago
> without using fork, exec or posix_spawn ? Did you manage to forget what you'd read two paragraphs earlier when you reached this bit? Because the essay's first recommendation is literally: > Only use fork to immediately call exec (or just use posix_spawn). It seems difficult to infer "don't use exec or posix_spawn" from this.
- ttoinou 4y ago
  Ofc I read that. I think a "only" was missing in the sentence, hence the confusion
cryptonector 4y ago
posix_spawn or vfork+exec

garethrowlands 4y ago

"The received wisdom suggests that Unix’s unusual combination of fork() and exec() for process creation was an inspired design. In this paper, we argue that fork was a clever hack for machines and programs of the 1970s that has long outlived its usefulness and is now a liability. We catalog the ways in which fork is a terrible abstraction for the modern programmer to use, describe how it compromises OS implementations, and propose alternatives." from A Fork in the Road, <https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf https://www.microsoft.com/en-us/research/uploads/prod/2019/0...>

wongarsu 4y ago
fork() and exec() is a reflection of the unix philosophy: small tools for separate tasks, and as a result you have great composability. You can set up right etc for your new process by calling any number of system calls between fork() and exec(). Windows is a great example of the alternative (following "Windows philosophy"): There are about three different API calls for creating a new process, each with a heap of complicated optional arguments. The API becomes more complex, less composable, less extensible and less powerful. But also easier to reason about and easier for the kernel to provide, and arguably with fewer footguns.
- klodolph 4y ago
  fork() is not a small tool. It is a MASSIVE, unwieldy tool that is difficult to use correctly. It happens to have a small interface.
  - adastra22 4y ago
    ...is it? In it is original implementation, fork() was pretty trivial. All it did was create a new process entry in the kernel table, with all the pages and capabilities and such copied from the original process. Then mark all pages as copy-on-write, and return to the caller. Maybe not trivial, but much less complicated than loading an executable file from disk. My understanding of Linux internals is maybe 20 years out of date, so I am legitimately curious what makes fork() so complicated these days.

tuxoko 4y ago

The suggestions here aren't really great. What you should do is already written in the fork(2) manpage https://man7.org/linux/man-pages/man2/fork.2.html https://man7.org/linux/man-pages/man2/fork.2.html "After a fork() in a multithreaded program, the child can safely call only async-signal-safe functions (see signal-safety(7)) until such time as it calls execve(2)." So just use only async-singal-safe function https://man7.org/linux/man-pages/man7/signal-safety.7.html https://man7.org/linux/man-pages/man7/signal-safety.7.html I don't know why so many people still hit this issue when it already told you what you can do and not do in the document. I've done this sort of things without any issue.

the_mitsuhiko 4y ago
Because in practice it’s almost impossible to know if you are single threaded still. Plenty of system libraries have background threads.
- emmelaich 4y ago
  Not to disbelieve you but which ones?
astrange 4y ago
There are many languages where it's difficult to impossible to be async-signal-safe because you can't avoid memory allocations.

Strilanc 4y ago

Another danger using fork is it duplicates the internal state of pseudo random number generators. It's a great way to accidentally take the same random samples in every process, utterly trashing any statistics you were intending to do. Bonus: the python multiprocessing module silently uses fork by default. Person A writes a "make multiprocessing convenient" library, Person B writes a sampling library, you put them together and... whoops!.

cryptonector 4y ago
Libraries like that should use pthread_atfork() to automatically reset/reseed/whatever state as needed at fork() time.
- Strilanc 4y ago
  I don't think that's really a viable strategy in practice in an ecosystem as complex as python's. There's too many libraries and too many little corner cases and interactions around what the behavior should be. For example, suppose I am using library A and I initialized the random number generator with a fixed seed. Clearly when I fork it's not appropriate for A to reseed, because I wanted fixed behavior. Something is very wrong so probably there should be an exception. But now suppose I was using library B which was using A and B handles getting system entropy to seed A. Now it is clear that when I fork I probably want B to reseed A, but alas A has already raised an exception because it was given a (from its perspective) fixed seed. So now A needs to be redesigned to be given a seed and like some sort of intent on what should happen when forking, and oh my god wow this is creating a lot of work for everyone everywhere this is not actually going to be done consistently and cannot be trusted.
  - cryptonector 4y ago
    If you're writing a simulation or a test, then you'll want the PRNG to stay unchanged, and you'll want to be in control of any reseeding. For all other RNG uses, you really do want it to reseed. A cryptographic PRNG vs. a simulation PRNG are very different things, and should be different libraries.

throwaway892238 4y ago

> When I ran into this problem, I was just trying to run all of Bluecore's unit tests on my Mac laptop. We use nose's multiprocess mode, which uses Python's multiprocessing module to utilize multiple CPUs. Unfortunately, the tests hung, even though they passed on our Linux test server. There will never be a time at which you can reliably expect any program developed on one system to "just work" on a different system. This person wasted a lot of time tracking down what was essentially a portability bug. Did they need this to be portable? Was this time well spent generating business value? Pick one system for development through production, stick to it. There will be portability bugs hiding in your code, but you will never have to fix them. You will be upset for a minute that you can't use a different system, but you will get over it.

csours 4y ago
I can write code that works NOW, but there is no guarantee that code will work in the future.

krylon 4y ago

The devil, as they say, is in the details. But to be fair, the only times I can recall using fork() without exec() were forking network servers, and that was mostly me learning about doing network stuff, and a forking server was the easiest to implement manually. Oh yeah, and that one time I accidentally wrote a fork bomb trying to stress test a DNS server. At least I learned something from my mistake. ;-) EDIT: To me, using fork() without exec() is kind of like operator overloading - there are cases where it absolutely is the right tool, but these aren't very numerous, so one should exercise caution. A lot.

dang 4y ago

Discussed at the time: Fork() without exec() is dangerous in large programs - https://news.ycombinator.com/item?id=12302539 https://news.ycombinator.com/item?id=12302539 - Aug 2016 (101 comments)

ridiculous_fish 4y ago

fish shell uses posix_spawn sometimes because of its performance benefits. We can't use it in the following cases: 1. No analog to tcsetpgrp, so it's no good if job control is enabled 2. No analog to fchdir, meaning you have to synchronize with fchdir elsewhere in the progarm 3. Error codes do not convey enough information for good error messages (e.g. if a file doesn't exist, posix_spawn doesn't tell you which file) 4. Inconsistent behavior around dup2 fd redirections and CLO_EXEC. 5. Inconsistent behavior for shebangless scripts These are basically deal-breakers so fish also supports a fork/exec path. However the performance benefits of posix_spawn are too real to ignore so fish uses posix_spawn when it can, and fork/exec when it must.

cryptonector 4y ago
Shells should really use `vfork()` to exec, and `fork()` for subshells (maybe).
- mort96 4y ago
  Except that the set of things you are allowed to do after vfork() and before exec() is extremely small. I agree that things should use vfork or posix_spawn when possible, but it wouldn't surprise me at all if there's a similar list of cases when a vfork() path wouldn't be possible.
  - cryptonector 4y ago
    That set was made artificially small by the Open Group, but is not actually that small.
chubot 4y ago
OK very interesting ... is the performance benefit on certain platforms, or everywhere? A previous thread says Ninja is faster on OS X and Solaris because of it. Though that does seem like a large number of corner cases, probably learned through painful experience :-/
- 4y ago

layer8 4y ago

New programming language implementations should maybe make fork() and multithreading be mutually exclusive at link time by default, and only allow them together in an unsafe-I-know-what-I’m-doing mode (if at all).

infogulch 4y ago

The dense fog lifts, tree branches part, a ray of light shines down on the ruins of a moss-covered pedestal revealing the hidden intentions of the ancients. A plaque states "The operational semantics of the most basic primitives of your operating system are optimized to simplify the implementation of command line shells." You look upon the pedestal, pause in respect, then turn away disappointed but unsurprised. As you walk you shake your head trying to evict the after image of a beam of light illuminating a turd.

infogulch 4y ago
I wrote this silly little bit on a previous fork() thread and touched it up a little. https://news.ycombinator.com/item?id=30504470 https://news.ycombinator.com/item?id=30504470
- cryptonector 4y ago
  It was a glorious comment then, and it is now. BTW, it's unclear whether the turd is... the specific truth revealed, or the revelation itself (since it could be incorrect). It's still a glorious comment.
  - infogulch 4y ago
    Thank you! The goal was to evoke the same emotional response I felt when this idea struck me (I think when I was reading earlier comments in that thread in fact), so I guess there's as much in the sequence as the specific objects. The semantic design of fork was always incomprehensible to me (not what but why), so the setting and pedestal and plaque represent how strongly it struck me and the depth of its historical explanatory power. The turd is how I feel about it. I don't resent the designer's history or motivations or incentives and I'm happy to know the truth. I'm just sad/depressed that this is the reason why things are the way they are, that the original design intent is so misaligned with the needs of modern systems, how it limits our capabilities today even down to hardware, and how unlikely it is for this to change. I don't suppose a turd would last very long on a pedestal, or that the ancients would put a turd there on purpose. Maybe they put something beautiful there back then, but times have changed and now we're stuck with it and it's kind of shit.

wruza 4y ago

Curious why there isn’t an interface in which all required handles and resources could be passed to a child process explicitly. E.g.: execvpehm( ..., int *handles, size_t, void **pages, size_t, /* etc */ ); Would remove so many headaches with concurrency and accidental inheritance.

emmelaich 4y ago
I suppose Linux's clone3 call goes the closest to that.

londons_explore 4y ago

One other option is to fork all the threads too. Since you probably don't know what all the other threads in your process are up to, your only option is to attach a debugger to all of them, halt them all, and copy all their state into brand new threads in the child process. Do it all correctly and you end up with a multi-threaded-fork. You still need to fix up signal handlers, interrupted syscalls, various notification API's that no longer work, memory mapped temp files used for IPC, pipes and sockets, and a bunch of other things. But a fork of a complex process is possible. It just isn't easy.

olliej 4y ago

And it’s explicitly unsupported on macOS at least.

kazinator 4y ago

The problems with fork in the face of threads are caused by threads, not by fork. Fork was there first, and it is part of a system that is designed and integrated well. Threads were bolted onto Unix in a hamfisted way, breaking more than just fork. For instance, threads broke relative paths, requiring "at" functions like openat to be invented, an ugly stop-gap measure. Threads were badly integrated with signal handling too, another example. Blaming those existing mechanisms is purely an emotional argument, from the perspective of being infatuated with threads. The design of threads (coming from various efforts that became POSIX threads) came from such an infatuation: the desire to get any kinds of threads working at any cost, while ignoring the global state that exists in a Unix process, and the need to make a lot of it thread-local, or at least optionally so. A thread-local working directory or signal mask would have caused difficulties in hack thread implementations that used user space scheduling or M:N (M user space threads to N kernel tasks). The situation we have today largely comes from the initial reluctance to accept the fact that each thread has to be an entity known to the kernel; the belief that user space threads are viable into the long-term future.