52 ms·

Fork() is evil; vfork() is goodness; afork() would be better; clone() is stupid

379 points by __s 5y ago

tych0 5y ago
The problem with this argument is that the set of programs that just fork() and then exec() is fairly small. Sure, shells are small and do this, but then the article argues that shells are a good use of fork(). In larger programs, you're forking because you need to diverge the work that's going to be done and probably where it's going to be done (maybe you want to create a new pid ns, you need a separate mm because you're going to allocate a bunch, whatever). Maybe the argument is that programs should never do this? I don't buy that. Then there's a lot of string-slinging through exec().
- pm215 5y ago
  That's backwards from my experience, which is that most users of fork() only do "fork; child does small amount of setup, eg closing file descriptors; exec". Shells are one of the few programs that do serious work in the child, because the POSIX shell semantics surface "create a subshell and do..." to the shell user, and then the natural way to implement that when you're evaluating an expression tree is "fork, and let the child process continue evaluating as a long-lived process continuing to execute as the same shell binary". (Depending on what's in that sub-tree of the expression, it might eventually exec, but it equally might not.) Many years back I worked on an rtos that had no fork(), only a 'spawn new process' primitive (it didn't use an MMU and all processes shared an address space, so fork would have been hard). Most unixy programs were easy to port, because you could just replace the fork-tweak-exec sequence with an appropriate spawn call. The shells (bash, ash I think were the two I looked at) were practically impossible to port -- at any rate, we never found it worth the effort, though I think with a lot of effort and willingness to carry invasive local patches it could have been done.
- olliej 5y ago
  The vast majority of programs that fork are doing fork() followed almost immediately by exec(), to the extent that on macOS for example a process is only really considered safe for exec() after fork() happens. Pretty much nothing else is considered safe.
  - NovemberWhiskey 5y ago
    Yeah; that would be my assumption too. I worked one time on a significant project that benefit from fork() without exec() and it was a monstrous pain - only if you own every single line of code in your project, have centralized resource management, and have no significant library dependencies should you ever consider doing this.
    - olliej 5y ago
      Yeah, you can't depend on pthreads or pthread mutexes (they're not defined as being fork safe). The entirety of Foundation (so presumably anything in Swift) is not fork safe either. To be clear: "not fork safe" in this case means "severely constrained environment": e.g. you can do things liker limits, set up pipes, etc but good luck with much more. I guess morally similar to the restrictions you have in a signal handler, albeit with different restrictions.
- kllrnohj 5y ago
  > In larger programs, you're forking because you need to diverge the work that's going to be done and probably where it's going to be done That's usually going to be done with clone() instead, no? You'll likely want to fiddle with the various flags for those usages and are unlikely to be happy with what fork() otherwise does.
- cryptonector 5y ago
  Oh no, there's tons of ProcessBuilder type APIs in Java, Python, and... every major language you can think of. The problems with fork() become very apparent in any Java apps that try to run external programs, especially in apps that have many threads and massive heaps and are very busy.
sys_64738 5y ago
The intent of fork() is to start a new process in its own address space. That *fork() variations that run in the SAME address space are confusing. A use case today for fork() might also be sandboxing apps. Certainly I expect browsers use this approach to spawn unique pages. But generally fork() is very specific from my recollection.
- cryptonector 5y ago
  > The intent of fork() is to start a new process in its own address space. True! > That *fork() variations that run in the SAME address space are confusing. Why is it confusing? They are distinct and different system calls, with different semantics. They are also sufficiently similar that they are also similarly named. But there's nothing confusing about their semantics. vfork() is not harder to use than fork() -- it's just subtly different. > A use case today for fork() might also be sandboxing apps. Certainly I expect browsers use this approach to spawn unique pages. I wouldn't expect that. Sandboxing is a large and complex topic.*
jph 5y ago
Is it a fair point to implement first with fork() because of memory protection, then optimize by using benchmarks and potentially vfork() for speed? Benchmark areas can look at synchronous locks, copy-on-write memory, stack sharing, etc. What are the good practices of security tradeoffs of fork() vs. vfork() especially in terms of ease of writing correct code? I'd thought that fork() + exec() tends to favor thinking about clearer separation/isolation. For example I've written small daemons using fork() + exec() because it seems safe and easy to do at the start.
- GrumpySloth 5y ago
  In short, fork() mixes poorly with multi-threaded code (and has some security footguns like needing to explicitly unshare elements of environment which may be sensitive, such as file descriptors (suddenly you need to know all the file descriptors used in the whole program from a single place in code)). Here is a well-written comment about fork() from David Chisnall: <https://lobste.rs/s/cowy6y/fork_road_2019#c_zec42d https://lobste.rs/s/cowy6y/fork_road_2019#c_zec42d> Additionally, the fork()+exec() idiom practically forces OS designers into a corner where they simply have to implement Copy-on-Write for virtual memory pages, or otherwise the whole userspace using this idiom is going to be terribly slow. Without the fork()+exec() idiom you don't need CoW to be efficient.
  - kevincox 5y ago
    TL;DR if another thread is holding a lock when you fork that lock will be stuck locked in the child, but that thread that was using that lock no longer exists. So if your multi-threaded program uses malloc you may fork while a global allocation lock is being held and you won't be able to use malloc or free in the child (thread-local caches aside). There are other problems but this is the basic idea. To be fork-safe you need to allow any thread to just disappear (or halt forever) at any point in your program.
    - kazinator 5y ago
      malloc has to guard its locks against fork, probably using pthread_atfork, or some lower level internal API related to that. The problem with pthread_atfork is third party libs. YOU will use it in YOUR code. The C library will correctly use it in its code. But you have no assurance that any other libraries are doing the right things with their locks.
londons_explore 5y ago
I was always disappointed by the performance of fork()/clone(). CompSci class told me it was a very cheap operation, because all the actual memory is copy-on-write, so its a great way to do all kinds of things. But the reality is that duplicating huge page tables, and hundreds of file handles is very slow. Like 10's of milliseconds slow for a big process. And then the process runs slowly for a long time after that because every memory access ends up causing lots of faults and page copying. I think my CompSci class lied to me... it might seem cheap and a neat thing to do, but the reality is there are very few usecases where it makes sense.
- smasher164 5y ago
  Also, mandating copy-on-write as an implementation strategy is a huge burden to place on the host. Now you’ve made the amount of memory a process is is using unquantifiable.
  - immibis 5y ago
    You also mandate a system complex enough to have an MMU.
  - vgel 5y ago
    It's not necessarily unquantifiable -- the kernel can count the not-yet-copied pages pessimistically as allocated memory, triggering OOM allocation failures if the amount of potential memory usage is greater than RAM. IIUC, this is how Linux vm.overcommit_memory[1] mode 2 works, if overcommit_ratio = 100. However, if an application is written to assume that it can fork a ton and rely on COW to not trigger OOM, it obviously won't work under mode 2. [1] https://www.kernel.org/doc/Documentation/vm/overcommit-accounting https://www.kernel.org/doc/Documentation/vm/overcommit-accou... > 2 - Don't overcommit. The total address space commit for the system is not permitted to exceed swap + a configurable amount (default is 50%) of physical RAM. > Depending on the amount you use, in most situations this means a process will not be killed while accessing pages but will receive errors on memory allocation as appropriate. > Useful for applications that want to guarantee their memory allocations will be available in the future without having to initialize every page.
lisper 5y ago
There is something about fork which I have never understood. Maybe someone here can explain it to me. Why would anyone ever want fork as a primitive? It seems to me that what you really want is a combination of fork and exec because 99% of the time you immediately call exec after fork (at least that's what I do 99% of the time when I use fork). If you know that you're going to call exec immediately after fork, then all the issues of dealing with the (potentially large) address space of the parent just evaporate because the child process is just going to immediately discard it all. So why is there not a fork-exec combo? And why has it not replaced fork for 99% of use cases? And as long as I'm asking stupid questions, why would anyone ever use vfork? If the child shares the parent's address space and uses the same stack as the parent, and the parent has to block, how is that different from a function call (other than being more expensive)? None of this makes sense to me.
- duskwuff 5y ago
  > Why would anyone ever want fork as a primitive? fork() without exec() can make sense in the context of a process-per-connection application server (like SSH). I've also used it quite effectively as a threading alternative in some scripting languages. > So why is there not a fork-exec combo? There is; it's called posix_spawn(). Like a lot of POSIX APIs, it's kind of overcomplicated, but it does solve a lot of the problems with fork/exec. > And as long as I'm asking stupid questions, why would anyone ever use vfork? For processes with a very large address space, fork() can be an expensive operation. vfork() avoids that, so long as you can guarantee that it'll immediately be followed by an exec().
  - kazinator 5y ago
    fork with copy-on-write semantics avoids copying the whole address space. It does have to copy some data structures that manage virtual memory and maybe the first level of the paging structure(page directory or whatever).
    - cryptonector 5y ago
      copy-on-write == slow when called from threaded processes with large resident set sizes.
immibis 5y ago
Another option is to allow the parent to create an empty child process, and then make arbitrary system calls and execute code in the child, like a debugger does. In most cases the last "remote system call" would be exec.
- cryptonector 5y ago
  posix_spawn() essentially is like that, or can be, as an implementation detail.
lucideer 5y ago
The good/evil/etc. here seem to be defined exclusively around "performance above all else", and - more specifically - performant primitives over performant application architecture. It strikes me that performance gains associated with sharing address space & stack are similar to many performance gains: trade-offs. So calling them "good" and "evil" when performance is seemingly your sole goal and interest seems a bit forward.
- cryptonector 5y ago
  In my world we often say things like "X is the moral equivalent of Y" where X and Y are just technologies and, clearly, are morally-neutral things. Why do we do this? Well, because it adds emphasis, and a dash of humor. Clearly fork() is neither Good nor Evil. It's morally neutral. It has no moral value whatsoever. But to say "fork() is evil" is to cause the audience to raise their eyebrows -"what, why would you say fork() is evil?!"- and maybe pay attention. Yes, there is the risk that the audience might react dismissively because fork() obviously is morally-neutral, so any claim that it is "evil" must be vacuous or hyperbolic. It's a risk I chose to take. Really, it's a rhetorical device. I think it's pretty standard. I didn't create that device myself -- I've seen it used before and I liked it.
  - lucideer 5y ago
    Morally-neutral does not equate to neutral insofar as I think most technologists consider some tech to be "good" and some to be "bad" in a practical sense. "Good -vs- evil" is obviously hyperbolic - particularly the latter - but outside of morals they still imply a tendency to be technically/practically good or bad in an objective sense. So discounting it as a mere rhetorical device seems overly dismissive.
mrob 5y ago
Fork() is the second worst idea in programming, behind null pointers. Fork() is the reason overcommit exists, which is the reason my web browser crashes if I open too many tabs, and the reason the "safe" Rust programming language leaves software vulnerable to DOS attacks if it uses the standard library. It's a clear example of "worse is worse", and we should have switched to the Microsoft Windows model decades ago. Here's a paper from Microsoft Research supporting this point of view: https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf https://www.microsoft.com/en-us/research/uploads/prod/2019/0...
- nick_ 5y ago
  Interesting take. If you don't mind explaining, what is the MS Windows model in in this context?
  - mrob 5y ago
    You opt into inheriting specific contexts from the parent, instead of copying everything by default: https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessa https://docs.microsoft.com/en-us/windows/win32/api/processth...
    - manwe150 5y ago
      More importantly, all syscalls also take a target process as an argument, making the Windows version both simpler and more powerful than can be done with fork. Spawn is also a lot slower on Windows, but that is an implementation issue.
      - kllrnohj 5y ago
        > Spawn is also a lot slower on Windows, but that is an implementation issue. afaik most of that slowdown is because malware scanners (including Windows Defender) hook spawn to do blocking verification of what to launch. Which is an issue also present on eg. MacOS, and why it's also kinda slow to launch new processes (and can be subject to extreme latencies): https://www.engadget.com/macos-slow-apps-launching-221445977.html https://www.engadget.com/macos-slow-apps-launching-221445977... Which is yes an implementation problem, but also a problem that potentially changes/impacts the design. Like maybe it'd make sense to get a handle to a pre-verified process so that repeated spawns of it don't need to hit that path (for eg. something like Make or Ninja that just spam the same executable over and over and over again). Or the kernel/trusted module needs to in some way be involved & can recognize that an executable was already scanned & doesn't need to be re-scanned.
JoeAltmaier 5y ago
The whole idea of fork is strange - the design pattern of "child process is executing exactly where the parent process is executing" is foreign to me. Don't we want to direct where the child process is executing? Like, when creating a thread? Why is fork() so conceptually orthogonal to that? Is there a good reason? A historical reason? I don't find fork() to be obvious or useful or natural. I work hard to never do it.
- shadowofneptune 5y ago
  fork()–exec() separation indeed exists for historical reasons: https://www.bell-labs.com/usr/dmr/www/hist.html https://www.bell-labs.com/usr/dmr/www/hist.html Search for the phrase "Process control in its modern form was designed and implemented within a couple of days."
- kccqzy 5y ago
  If you want the child to start executing some other code but you have fork(), it's easy to do it yourself by calling that function. But on the other hand, if you do want the child to execute code at the same place as the parent, but a hypothetical fork() asks you to provide a function pointer, it would be a bit more complicated.
- zamalek 5y ago
  It's a leaky abstraction and everything it does can be done manually, and possibly better. It exists purely because, at some point in the past, threads didn't exist. If you design your program without fork, you'll probably end up with a cleaner and faster solution. Some things are best forgotten or never learned in the first place.
  - _flux 5y ago
    Can it though? The beauty of (v)fork(+exec) is that it doesn't need a new interface for configuring the environment in whichever way you want before the other process starts. Instead you get to use the exact same means of modifying the environment to your needs, and once it's done, you can call exec and the new process inherits those things. I mean, just look at the interface of posix_spawn. I grant though that this isn't without its problems (including performance) and IMO e.g. FD_CLOEXEC is one example of how those problems can be patched up. It's like the reverse problem: you have too wide implicit interface in it, and then you need to come up with all these ways to be explicit about some things.
tiffanyh 5y ago
Slightly off topic, how does Erlang handle this because isn’t it know for having extremely fast & cheap process spawning baked in (with isolation).
- __s 5y ago
  m:n threads (aka green threads) https://twitter.com/joeerl/status/1010485913393254401 https://twitter.com/joeerl/status/1010485913393254401
  - tiffanyh 5y ago
    I realize that tweet is from the authority himself but am I mistaken in my understanding … I thought green threads share memory but Erlang processes do NOT share memory, which is what makes Erlang so unique. Did Erlang create a so called “green process”? If so, why can’t this model be implemented in the kernel?
    - masklinn 5y ago
      > I thought green threads share memory but Erlang processes do NOT share memory, which is what makes Erlang so unique. Erlang processes don’t share memory because the language and vm don’t give primitives which let you do it. They all exist within the same address space (e.g. large binaries are reference-counted and stored on a shared heap, excluding clustering obviously). > Did Erlang create a so called “green process”? Yes. > If so, why can’t this model be implemented in the kernel? Because erlang processes are not an antagonistic model, and the language restricts the ability to attack the VM (kinda, I’m sure you could write NIFs to fuck up everything, you just don’t have any reason to as an application developer).
- jerf 5y ago
  In another comment, I observe how Go doesn't even have a binding to fork. Erlang is another example of that. There is no standard library binding to the fork function. If someone were to bash one into a NIF, I have no idea what would happen to the resulting processes, but there's no good that can come of it. (To use Star Trek, think less good and evil Kirk and more "What we got back, didn't live long... fortunately.") Despite the terminology, all Erlang processes are green threads in a single OS process.
evmar 5y ago
In Ninja, which needs to spawn a lot of subprocesses but it otherwise not especially large in memory and which doesn't use threads, we moved from fork to posix_spawn (which is the "I want fork+exec immediately, please do the smartest thing you can" wrapper) because it performed better on OS X and Solaris: https://github.com/ninja-build/ninja/commit/89587196705f54afb904c8f4572e65de7274dd81 https://github.com/ninja-build/ninja/commit/89587196705f54af...
- ridiculous_fish 5y ago
  posix_spawn also outperforms fork on Linux under more recent glibc and musl, which can use vfork under the hood. https://twitter.com/ridiculous_fish/status/1232889390763933698 https://twitter.com/ridiculous_fish/status/12328893907639336...
- xroche 5y ago
  The issue with posix_spawn is that you can't close all descriptors before exec. This is especially an issue as most libraries are still unaware they need to open every single handle with the close-on-exec flag set.
  - cryptonector 5y ago
    Solaris/Illumos has an extension[0] for that. [0] http://src.illumos.org/source/search?project=illumos-gate&full=posix_spawn_file_actions_addclosefrom_np&defs=&refs=&path=&hist=&type=&xrd=&nn=1 [1] https://docs.oracle.com/cd/E36784_01/html/E36874/posix-spawn-file-actions-addclosefrom-np-3c.html
    - kazinator 5y ago
      For implementations which don't have it, you can stuff, into the file_actions, say, 4093 close action entries into the file_actions, targeting descriptors 3 to 4095. This big file_actions object can be cached and re-used for multiple calls to posix_spawn. It won't close descriptor 4096, but that's probably beyond giving a darn in most cases. If you have an application that opens high descriptor numbers, you probably know.
kazinator 5y ago
Concurrently running dupe currently on front page: https://news.ycombinator.com/item?id=30499169 https://news.ycombinator.com/item?id=30499169 :) :) :)
- cryptonector 5y ago
  Ha!
ridiculous_fish 5y ago
Amusingly vfork semantics differ across OSes. This program prints 42 in Linux but 1 on Mac: https://godbolt.org/z/jn7Gaf5Me https://godbolt.org/z/jn7Gaf5Me because on Linux they share address space.
- butterisgood 5y ago
  I am pretty sure Mac OS doesn't COW fork(), and that the address space is copied. At least it was the last time I looked. FreeBSD and Linux both seem to COW. Perhaps there's a reason vfork is different too.
  - ridiculous_fish 5y ago
    My (very possibly wrong) understanding is that xnu does CoW fork but doesn't overcommit, meaning that memory must be reserved (perhaps in swap) in case the pages need to be duplicated. There's other complications relating to inheriting Mach ports and the mach_task <-> BSD process "duality" in xnu, which Linux doesn't have. I'd love for someone to chime in who knows more about how this stuff works.
  - cryptonector 5y ago
    Hopefully someone at Apple will see this post and be convinced to restore vfork() and un-obsolete it.
- cryptonector 5y ago
  Unfortunately there was this paper from the 80s titled "vfork() Considered Dangerous", which led to BSDs removing vfork(), and then later it was re-added because that paper was clearly quite wrong. But the news hasn't quite filtered through to Apple, I guess.
mark_undoio 5y ago
The code I currently work on actually has a use of `clone` with the `CLONE_VM` flag to create something that isn't a thread. Since `CLONE_VM` will share the entire address space with the child (you know, like a thread does!) a very reasonable response would be "WAT?!" What led us here was a need to create an additional thread within an existing process's address space but in a way that was non-disruptive - to the rest of the process it shouldn't really appear to exist. We achieved this by using `CLONE_VM` (and a handful of other flags) to give the new "thread-like" entity access to the whole address space. But, we omitted `CLONE_THREAD`, as if we were making a new process. The new "thread-like" entity would not technically be part of the same thread group but would live in the same address space. We also used two chained `clone()` calls (with the intermediate exiting, like when you daemonise) so that the new "thread-like" wouldn't be a child of the original process. All this existed before I joined, it's just really cool that it works. I've never encountered a such a non-standard use of clone before but it was the right tool for this particular job!
- scottlamb 5y ago
  > What led us here was a need to create an additional thread within an existing process's address space but in a way that was non-disruptive - to the rest of the process it shouldn't really appear to exist. I'm curious to hear more. What's its purpose?
  - kccqzy 5y ago
    Maybe some kind of snapshotting for an in-memory database?
  - mark_undoio 5y ago
    > I'm curious to hear more. What's its purpose? Sure! I'll try to illustrate the general idea, though I'm taking liberties with a few of the details to keep things simple(r). Our software (see https://undo.io https://undo.io) does record and replay (including the full set of Time Travel Debug stuff - executing backwards, etc) of Linux processes. Conceptually that's similar to `rr` (see https://rr-project.org/ https://rr-project.org/) - the differences probably aren't relevant here. We're using `ptrace` as part of monitoring process behaviour (we also have in-process instrumentation). This reflects our origins in building a debugger - but it's also because `ptrace` is just very powerful for monitoring a process / thread. It is a very challenging API to work with, though. One feature / quirk of `ptrace` is that you can't really do anything useful with a traced thread that's currently running - including peeking its memory. So if a program we're recording is just getting along with its day we can't just examine it whenever we want. First choice is just to avoid messing with the process but sometimes we really do need to interact with it. We could just interrupt a thread, use `ptrace` to examine it, then start it up again. But there's a problem - in the corners of Linux kernel behaviour there's a risk that this will have a program-visible side effect. Specifically, you might cause a syscall restart not to happen. So when we're recording a real process we need something that: * acts like a thread in the process - so we can peek / poke its memory, etc via ptrace * is always in a known, quiescent state - so that we can use ptrace on it whenever we want * doesn't impact the behaviour of the process it's "in" - so we don't affect the process we're trying to record * doesn't cause SIGCHLD to be sent to the process we're recording when it does stuff - so we don't affect the process we're trying to record Our solution is double clone + magic flags. There are other points in the solution space (manage without, handle the syscall restarting problem, ...) but this seems to be a pretty good tradeoff. [edit: fixed a typo]
pipeline_peak 5y ago
Your idea good Your idea stupid I’m not woke by any means, idk what it is about low level programming but calling someone’s idea “stupid” is a really shitty thing to say. “He chose to take it personally” is the type of lazy, pseudo-stoic argument I have no interest in reading. Yes I’m having a morning, lol.
- cryptonector 5y ago
  I answered this here: https://news.ycombinator.com/item?id=30504804 https://news.ycombinator.com/item?id=30504804 It's a rhetorical device. I didn't expect this to -years later- become a front-page item on HN. I wrote that to share with certain people. And yes, clone() has some real problems, and if calling it "stupid" pisses off some people, but maybe also leads others to want to improve clone() or create a better alternative, then that's fine. If I'd wanted to write an alternative to Linux I'd probably have had to deal with the very, very fine language that Linus and others use on the Linux kernel mailing lists -- if you don't like my using the word "stupid", then you really shouldn't look there because you're likely to be very disappointed. Indeed, not only would I have to accept colorful language from reviewers there, I'd probably have to employ some such language myself. TL;DR: clone() came from Linux, where "stupid" is the least colorful language you'll find, and me calling it "stupid" is just a rhetorical device.
cryptonector 5y ago
Well, I'm surprised to see this on the front page, let alone as #1. Ask me anything. EDIT: Also, don't miss @NobodyXu's comment on my gist, and don't miss @NobodyXu's aspawn[1]. [0] https://gist.github.com/nicowilliams/a8a07b0fc75df05f684c23c18d7db234?permalink_comment_id=3467980#gistcomment-3467980 [1] https://github.com/NobodyXu/aspawn/
- ahmedalsudani 5y ago
  No questions yet as I am yet to read ... but I can already comment and say grade A title.
  - cryptonector 5y ago
    It's a bit opinionated. It's meant to get a reaction, but also to have meaningful and thought-provoking content, and I think it's correct in the main too. Anyways, hope you and others enjoy it.
    - ahmedalsudani 5y ago
      That was a great read. Thank you for writing it up; I learned quite a few things! Especially appreciated the OS minutiae and opinionated commentary (... and the doc vs reality observation in Linux's vfork). The piece lives up to the great title :)
    - disgruntledphd2 5y ago
      What do you mean by zones/jails and why are they better than containers?
      - cryptonector 5y ago
        Zones -> Solaris/Illumos Zones Jails -> BSD jails They're software VMs. It's a lot like containers, yes. The problem with containers is that the construction toolkit for them is subtractive ("start by cloning my environment, then remove / replace various namespaces"), while the construction toolkit for zones/jails is additive ("start with an empty universe, and add namespaces or share them with the parent"). Constructing containers subtractively means that every time there's a new kind of namespace to virtualize, you have to update all container-creating tools or risk a security vulnerability. Constructing containers additively from an empty universe means that every time there's a new kind of namespace to virtualize, you have to update all container-creating tools or risk not getting sharing that you want (i.e., breakage). I'm placing a higher value on security. Maybe that's a bad choice. It's not like breaking is a good thing -- it might be just as bad as creating a security vulnerability.
throwaway984393 5y ago
I don't think containers should be like jails. Containers should be more like chroots than they are now. Have you ever tried to run a modern X/whatever app with 3D graphics and audio and DBUS and God knows what else in a container and get it to show up on your desktop? It's a fucking nightmare. I spent over a week trying to get 1Password to run in a container. Somebody decided containers had to be "secure", even though they don't actually exist as a single concept and security was never their primary purpose. If instead containers were used only to isolate filesystem dependencies, we could actually pretend containers were like normal applications and treat them with the same lack of security concern that all the rest of our non-containerized programs are. Firecracker is the correct abstraction for isolation: a micro-VM. That is the model you want if you want to run an app securely (not to mention reliably, as it can come with its own kernel, rather than needing you to run a compatible host kernel).
- cryptonector 5y ago
  I... didn't mean that containers have to have a copy of the operating system inside them, systemd and many other things included. I meant only that they should be created in ways like how the BSDs and Illumos do it.
infogulch 5y ago
The dense fog lifts, tree branches part, a ray of light beams down on a pedestal revealing the hidden intentions of the ancients. A plaque states "The operational semantics of the most basic primitives of your operating system are designed to simplify the implementation of shells." You hesitantly lift your eyes to the item presented upon the pedestal, take a pause in respect, then turn away slumped and disappointed but not entirely surprised. As you walk you shake your head trying to evict the after image of a beam of light illuminating a turd.
- deleted 5y ago
  [deleted]
- ckastner 5y ago
  > "The operational semantics of the most basic primitives of your operating system are designed to simplify the implementation of shells." Yes, but why is this characterized as something negative? Isn't that the entire point? Operating systems are there to serve user requests, and shells are an interface between user and OS. Shells simply developed features that users required of them.
  - int_19h 5y ago
    Shells haven't been the primary interface between the user and the OS for decades.
    - kragen 5y ago
      "The primary interface between the user and the OS" is the definition of "shell". That's why the Microsoft Windows process that draws the Start button and filesystem windows is called "the Windows shell".
      - int_19h 5y ago
        I don't think OP meant shell as in the Windows shell, or Linux DEs. I mean, how many of those use fork() even on Linux, or would be easier to implement if they did?
scottlamb 5y ago
> clone() is stupid ... the clone(2) design, or its maintainers, encourages a proliferation of flags, which means one must constantly pay attention to the possible need to add new flags at existing call sites. IMHO a bigger problem [2] in practice with clone is that (according to glibc maintainers) once your program calls it, you can't call any glibc function anymore. [1] Essentially the raw syscall is a tool for the libc implementation to use. The libc implementation hasn't provided a wrapper for programs to use which maintains the libc's internal invariants about things like (IIUC) thread-local storage for errno. The author's aforkx implementation is something that glibc maintainers could (and maybe should) provide, but my understanding is that you can get in trouble by implementing it yourself. [1] https://github.com/rust-lang/rust/issues/89522#issuecomment-946023944 https://github.com/rust-lang/rust/issues/89522#issuecomment-... [2] editing to add: or at least a more concrete expression of the problem. Wouldn't surprise me if they haven't provided this wrapper in part because the proliferation the author mentioned makes it difficult for them to do so.
- jlokier 5y ago
  It's really unfortunate that the sanctioned way to call Linux syscalls directly is via the syscall() function (previously the _syscallN macros), and both of those methods set errno on error, which fails in a clone() thread. If only Glibc provided a syscall_r() or something that returns the raw return value whether it's an error or not. It is possible to make syscall() (and regular libc syscalls like read()) work in a clone() thread. I use this in performance-optimised I/O code in a database engine, so I know it works, but it requires some ugly Glibc-and-architecture-specific things. Doing it portably doesn't seem to be an option.
aylmao 5y ago
Meta comment: Github Gist seems to be great for blogging. Yeah, the UI is not very blog-specific, but it has all the useful features, and then some: markdown, comments, hosting, an index of all posts, some measure of popularity (stars), a very detailed edit history, etc. All without having to pay or setup anything yourself.
- JoshTriplett 5y ago
  Unfortunately, there's no way to turn off comments on a Gist, which makes it not a viable replacement for anyone who doesn't want to spend a lot of time processing and moderating comments.
  - chmaynard 5y ago
    Good point. However, you need a GitHub account to post comments so everyone knows who you are. Your reputation might suffer if you constantly post comments that require moderation.
    - JoshTriplett 5y ago
      This does not, in practice, stop people. Both because it's possible to make throw-away accounts, and because some people don't have a reputation to care about to begin with.
- fomine3 5y ago
  It's great because Chinese GFW won't block.
mattgreenrocks 5y ago
> Long ago, I, like many Unix fans, thought that fork(2) and the fork-exec process spawning model were the greatest thing, and the Windows sucked for only having exec() and _spawn(), the last being a Windows-ism. I appreciate this quite a bit. Vocal Unix proponents tend to believe that anything Unix does is automatically better than Windows, sometimes without even knowing what the Windows analogue is. Programming in both is necessary to have an informed opinion on this subject. The one thing I miss most on Unix: the unified model of HANDLEs that enables you to WaitOnMultipleObjects() with almost any system primitive you could want, such as an event with a socket (blocking I/O + a shutdown notification) in one call. On Unix, a flavor of select() tends to be the base primitive for waiting on things to happen, which means you end up writing adapter code for file descriptors to other resources, or need something like eventfd. Things I don't miss from Windows at all: wchar_t everywhere. :)
- cryptonector 5y ago
  WIN32 got a few things very right: - SIDs - access tokens (like struct cred / cred_t in Unix kernels, but exposed as a first-class type to user-land) - security descriptors (like owner + group mode_t + ACL in Unix land, but as a first-class type) - HANDLEs, as you say - HANDLEs for processes Many other things, Windows got wrong. But the above are far superior to what Unix has to offer.
  - al2o3cr 5y ago
    I'd be curious how many of those derive from NT's VMS roots - for instance: http://lxmi.mi.infn.it/~calcolo/OpenVMS/ssb71/6346/6346p004.htm http://lxmi.mi.infn.it/~calcolo/OpenVMS/ssb71/6346/6346p004....
    - cryptonector 5y ago
      Most of them, as far as I know.
      - p_l 5y ago
ismaildonmez 5y ago
Microsoft Research has a paper about the very same issue (2019): https://www.microsoft.com/en-us/research/publication/a-fork-in-the-road/ https://www.microsoft.com/en-us/research/publication/a-fork-...
- cryptonector 5y ago
  It's a very good paper, yeah. I will link it from the gist.
- georgia_peach 5y ago
  That paper smacks of a Chesterton Fence. They haven't come up with a tested replacement for many of the use cases, i.e.: These designs are not yet general enough to cover all the use-cases outlined above, but perhaps can serve as a starting point... yet bullet #1 in the next paragraph is Deprecate Fork I think this is a case of security guys being upset about fork gumming-up their experiments. I don't really care about their experiments. The security regime for the past 20 years may have bought us a little more security against eastern bloc hackers, but it hasn't done squat to protect us from Apple, Google, & Microsoft! I have never had a virus de-rail my computing life as much as the automatic Windows 10 upgrade. Robert Morris got 400 hours community service for a relatively benign worm. If that's the penalty scale, Redmond should get actual time in the slammer for Cortana, forced Windows Update, and adding telemetry to Calculator.
  - cryptonector 5y ago
    You fail to address any of the substance of their paper, or of my gist (TFA), then go on a rant about unrelated things. The authors of that paper deserve better treatment even if you hate Microsoft.
    - deleted 5y ago
      [deleted]
    - georgia_peach 5y ago
      I did. Chesterton Fence. fork() has been in Unix from the beginning. Taking it out at this point will cause more problems than it solves. Until you have a working Unix distro (kernel AND common userland services) that elegantly covers all of the forkless cases, your paper and their paper are just opinions. Theirs is a formally written one. Yours is a clickbaity one. And casting vfork() as any kind of improvement here is just bonkers. And the rant is totally related: i.e. devs breaking things that worked just fine to begin with for the sake of doctrinal purity. It is usually a false doctrine.
psanford 5y ago
I saw a bug once where an application would get way slower on MacOS after calling fork(). Not just temporarily either; many syscalls would continue to run slowly from the call to fork() until the process exited. Looking on Stack Overflow, I see a few reports of this behavior[0][1]. [0]: https://stackoverflow.com/questions/4411840/memory-access-after-fork-extremely-slow-on-mac-os-x https://stackoverflow.com/questions/4411840/memory-access-af... [1]: https://stackoverflow.com/questions/27932330/why-is-tzset-a-lot-slower-after-forking-on-mac-os-x https://stackoverflow.com/questions/27932330/why-is-tzset-a-...
deleted 5y ago
[deleted]
mywacaday 5y ago
"I won't bother explaining what fork(2) is -- if you're reading this, I assume you know.", If that applied to everything I looked at from HN I'd read precious little.
- cryptonector 5y ago
  I didn't write it for HN. It wasn't a paper to publish in some Computer Science journal. It was just a github gist. If you don't get the subject, it's not for you. I might well write a paper now based on it, and then it might be a good read for you, but I still won't be writing it for you, but for people who are interested in the topic. The intended audience is small, expert on the matter, and probably even more opinionated than I am.
  - mywacaday 5y ago
    I found the article well written and informative even though it's not my area of expertise, I intended my comment as a light hearted reflection of the fact that a lot of articles on HN go over my head but are still worth a read to me, just like your article.
saurik 5y ago
One use case for fork()--which is used extensively on Android--is to build an expensive template process that can then be replicated for later work, which is exactly what people often want for the behavior with virtual machines. I wrote an article on the history of linking and loading optimizations leading up to how Android handles their "zygote" which touches on this behavior. http://www.cydiasubstrate.com/id/727f62ed-69d3-4956-86b2-bc0ffea110c1/ http://www.cydiasubstrate.com/id/727f62ed-69d3-4956-86b2-bc0...
monocasa 5y ago
Hard disagree to most of this. fork(2) makes a lot more sense when you realize its heritage. It came from a land before Unix supported full MMUs. In this model, to still have per process address spaces and preemptive multitasking on what was essentially a PC-DOS level of hardware, the kernel would checkpoint the memory for a process, slurp it all out to dectape or some such, and load in the memory for whatever the scheduler wanted to run next. It's simplicity of being process checkpoint based wasn't a reaction to windows style calls (which wouldn't exist for almost a couple decades), but instead mainframe process spawning abominations like JCL. The idea "you probably want most of what you have so force a checkpoint, copy the checkpoint into a new slot, and continue separately from both checkpoints" was soooo much better than JCL and it's tomes of incantations to do just about anything. vfork(2) is an abomination. Even when the child returns, the parent now has a heavily modified stack if the child didn't immediately exec(). All of those bugs that causes are super fun to chase, lemme tell you. AFAIC, about the only valid use for vfork now is nommu systems where fork() incredibly expensive compared to what is generally expected. clone(2) is great. Start from a checkpoint like fork, but instead of semantically copying everything, optionally share or not based on a bitmask. Share a tgid, virtual address space, and FD table? You just made a thread. Share nothing? You just made a process. It's the most 'mechanism, not policy' way I've seen to do context creation outside of maybe the l4 variants and the exokernels. This isn't an old holdover, this is how threads work today, processes spawned that happen to share resources. Modern archs on linux don't even have a fork(2) syscall; it all happens through clone(2). Even vfork is clone set to share virtual address space and nothing else that fork wouldn't share. Namespaces are a way to opt into not sharing resources that normally fork would share. And I don't see what afork gets you that clone doesn't, except afork isn't as general.
- cryptonector 5y ago
  > vfork(2) is an abomination. Even when the child returns, the parent now has a heavily modified stack if the child didn't immediately exec(). What stack modifications? Sure, the child can scribble over the stack frame, or worse, the child could do things like return -- but you are the author of the code calling vfork() and you know not to do that, so why would that happen? A: It just wouldn't happen. And as to exec() failing, this is why exec calls must be followed with calls to either exec() or _exit(), and this is true even if you use fork() instead of vfork(). I.e.: /* do a bunch of pre-vfork() setup */ ... pid_t pid = vfork(); if (pid == -1) err(1, "Couldn't vfork()"); if (pid == 0) { /* do a bunch of child-side setup */ execve(...); /* oops, ENOENT or something */ _exit(1); } /* the child either exec'ed or exited */ if (waitpid(pid, &status, 0) != pid) err(1, "..."); ... How do you detect if the child exec'ed or exited? Well, you make a pipe before you vfork(), you set its ends to be O_CLOEXEC, then on the child side of vfork() you write one byte into it if the exec call fails. On the parent side you read from the pipe before you reap the child, and if you get EOF then you know the child exec'ed, and if you get one byte then you know the child exited. The one byte could be an errno value. No, really, what you say about vfork() is lore, and very very wrong. That said, vfork() blocks a thread in the parent. The point of my gist was to explain why fork() sucks, why vfork() is much better, and what would be better still. > And I don't see what afork gets you that clone doesn't, except afork isn't as general. afork()/avfork() is not meant to be as general as clone() but to be more performant than vfork() by not blocking a thread on the parent side. clone() needs some improvements. It should be possible to create a container additively. See elsewhere in the comments on this post.
albertzeyer 5y ago
We had the case that some library we were using (OpenBLAS) used pthread_atfork. Unfortunately, the atfork handler behaved buggy in certain situations involving multiple threads and caused a crash. This was annoying because we basically did not need fork at all but just fork+exec (for various other libraries spawning sub processes), where those atfork handlers would not be relevant. Our solution was to override pthread_atfork to ignore any functions, and in case this is not enough, also fork itself to just directly do the syscall without calling the atfork handlers. https://github.com/tensorflow/tensorflow/issues/13802 https://github.com/tensorflow/tensorflow/issues/13802 https://github.com/xianyi/OpenBLAS/issues/240 https://github.com/xianyi/OpenBLAS/issues/240 https://trac.sagemath.org/ticket/22021 https://trac.sagemath.org/ticket/22021 https://bugs.python.org/issue31814 https://bugs.python.org/issue31814 https://stackoverflow.com/questions/46845496/ld-preload-and-linkage https://stackoverflow.com/questions/46845496/ld-preload-and-... https://stackoverflow.com/questions/46810597/forkexec-without-atfork-handlers https://stackoverflow.com/questions/46810597/forkexec-withou...
- cryptonector 5y ago
  posix_spawn() shouldn't call atfork handlers. It's allowed to call them or not call them because implementors can use fork(), which must call them, or they can use vfork(), which must not call them -- or they can make posix_spawn() a proper system call, too, or they can use clone(), or my putative avfork(), or whatever. If you used vfork(), you wouldn't have had this problem. Fork-safety issues arise mainly because of the sharing of resources between the parent and child. pthread_atfork() exists mainly to allow libraries to add a measure of fork-safety by letting them disable things on the child-side of fork() or re-set-up things on the child-side of fork(). For example, a PKCS#11 provider might need to create a new connection to the tokens and re-C_Login() to them (except, since it really can't quite do that, most likely it must render every session inoperable on the child-side). (Indeed, PKCS#11 specifically mandates that on the child-side of fork all sessions must be dead and must not be used.)
- cryptonector 5y ago
  I left a comment on TF #13802.
phendrenad2 5y ago
For those saying to use posix_spawn: What am I supposed to make of the writeup in the posix_spawn manpage though? "...specified by POSIX to provide a standardized method of creating new processes on machines that lack the capability to support the fork(2) system call. These machines are generally small, embedded systems lacking MMU support" Is this why no one uses it? It has this gratuitous opinion piece at the beginning that makes people think it's just for embedded systems and my dad's Amiga?
- cryptonector 5y ago
  Hah, that's pretty funny. Regardless of the motivation as written, the motivation I surmise is: - some systems (e.g., Windows) lack fork() for various reasons - vfork() is baaaad - I know, let's do something like WIN32's spawn() or CreateProcess(), but, like, better The middle item I have good reason to think is very likely. vfork() still has a bad rap from that old "vfork() Considered Dangerous" paper. That paper circulated a lot way back when, and was the reason vfork() was removed from some Unixes for a while (well, it was left as an alias of fork()) before it was eventually re-added. The Open Group participants would been very aware of that paper, and that is almost certainly the reason that POSIX says about vfork(): Conforming applications are recommended not to depend on vfork(), but to use fork() instead. The vfork() function may be withdrawn in a future version. So if fork() can't perform well, and the committee won't recommend the use of vfork(), what shall the committee do? Answer: design and specify posix_spawn(). It's not an unreasonable answer. Though, IMO of course, they should have un-obsoleted vfork().
- xenadu02 5y ago
  That's just some injected opinion, I assume from someone contributing to glibc who doesn't like posix_spawn I guess? In any case it is wrong. Don't assume what is written in man pages is the truth. Some of them have a lot of opinion added. It can be useful to cross-check man pages between systems - they don't always call out non-portable options or behavior. On some kernels posix_spawn is a syscall or specifies flags that make it more efficient than fork+exec. Darwin is one such system, though you can use POSIX_SPAWN_SETEXEC if you still want to replace the current process with a new executable rather than creating a child.
throwawaylinux 5y ago
This avfork implementation is poor. You don't want to make your single threaded programs multi-threaded. I don't really get the big benefit of afork over other existing mechanisms other than handwaving about things being evil. Also, > Linux should have had a thread creation system call -- it would have then saved itself the pain of the first pthread implementation for Linux. Linux should have learned from Solaris/SVR4, where emulation of BSD sockets via libsocket on top of STREAMS proved to be a very long and costly mistake. Emulating one API from another API with impedance mismatches is difficult at best. Linux does have a thread creation system call. It's clone(2). It literally creates new threads of execution with various properties. It does not "emulate" threads, it is threads.
- cryptonector 5y ago
  > You don't want to make your single threaded programs multi-threaded. Correct. I want to spawn processes faster from already-multi-threaded programs.
  - throwawaylinux 5y ago
    You do, but it's not a good implementation for a general API is all I was trying to say. Do you really need an "asynchronous process creation" call? The rationale is that "blocking is bad", but a thread creation system call blocks the caller too until the thread is created. So it's not just "blocking", it's the amount of blocking if anything. Is posix_spawn or vfork+exec really too slow for your case? Then multi-process and multi-threading seems like a reasonable solution. Asynchronous system calls are the exception not the rule in unix. So it wouldn't make sense as a traditional afork(2) system call. You could probably do a posix_spawn for io_uring, but do you really need to?
bergkvist 5y ago
I have to disagree that fork is evil. fork is great because of copy-on-write. I guess my particular use case is not very typical/common though. I'm running powerflow simulations on a power grid model (several GB of memory to store the model). Copy-on-write means I can make small modifications to this model and run simulations in parallel. Thanks to fork/copy-on-write, I can run 32 simulations in parallel, each will small modifications without requiring 32 times as much memory.
- cryptonector 5y ago
  Neat!
userbinator 5y ago
I started with DOS, where spawn() is the norm, so I've always considered the fork()-like behaviour to be unusual yet handy for certain use-cases. Perhaps a system call that offers a combination of the two behaviours should be named spork().
Ericson2314 5y ago
This stuff is still all confused Read http://catern.com/rsys21.pdf http://catern.com/rsys21.pdf What you want is: 1. create "embryonic" unscheduled process 2. Set it up from the parent process, it just lies on the operating table passively. 3. Submit it to the scheduler. This is just....obviously correct. Totally flexible. Totally efficient. Hell, if you really want to fork anything, fork those embryonic process which have no active threads! Much safer and easier to understand! I did not write the paper above, but I did write https://lore.kernel.org/lkml/f8457e20-c3cc-6e56-96a4-3090d7da0cb6@JohnEricson.me/ https://lore.kernel.org/lkml/f8457e20-c3cc-6e56-96a4-3090d7d... https://lists.freebsd.org/archives/freebsd-arch/2022-January/000139.html https://lists.freebsd.org/archives/freebsd-arch/2022-January... I hope I or someone else will have time to make it happen!
- cryptonector 5y ago
  Yes, I like the larval process idea. No doubt it's good.
- IgorPartola 5y ago
  When I was first learning about UNIX and similar OSes I just assumed that this is how things worked because this is the obvious way of doing it. Why would you fork a process, then try to determine which of the two processes you are, then fix whatever the parent process messed up in your global state, and only then execute what you actually wanted to do? That seems insane (I guess until you realize that the main use case is creating /bin/sh).
  - Ericson2314 5y ago
    Me too! But even when writing /bin/sh, I don't see why this would get in the way? I was once told earlier Unix didn't even have fork, but something more purpose-made for shells instead.
- zokier 5y ago
  Sounds a bit like fuchsias launchpad library where you create launchpad object, do all the setup, and then call launchpad_go to actually start the process. Launchpad doesn't allow arbitrary syscalls in the setup, so in that sense it is maybe closer to "spawn" interface but with better ergonomics https://cs.opensource.google/fuchsia/fuchsia/+/main:zircon/system/ulib/launchpad/include/launchpad/launchpad.h https://cs.opensource.google/fuchsia/fuchsia/+/main:zircon/s...
harry8 5y ago
Fork with cow is inefficient. Compared to what? In what dimension? Any numbers on that? Where is the trade-off? To what extent does anyone need to care and on what circumstances?
- xroche 5y ago
  It's inherently inefficient because while the child process does its initialization (pre-exec) stuff, the parent gets page faults for every thread writing into the memory due to COW. This will basically stall the parent and can cause funny issues.
- cryptonector 5y ago
  > Fork with cow is inefficient. > Compared to what? vfork() > Any numbers on that? I added links to the gist, some of which discuss performance in detail. E.g., https://blog.famzah.net/tag/fork-vfork-popen-clone-performance/ https://blog.famzah.net/tag/fork-vfork-popen-clone-performan... and https://bugzilla.redhat.com/show_bug.cgi?id=682922 https://bugzilla.redhat.com/show_bug.cgi?id=682922 But you can just reason about this: - vfork() is O(1) - copying fork() is O(N) where N is the amount of writable memory in the parent's address space - copy-on-write fork() is O(N) where N is the resident set size (RSS) of the parent O(1) beats O(N). And O(N) is just the complexity of fork() for a single-threaded parent process. Now imagine a very busy, threaded, large-RSS process that forks a lot. You get threads and child processes stepping all over each other's CoW mappings, causing lots of page faults and copies. Ok, that is still O(N), but users will feel the added pain of all those page faults and TLB shootdowns.
  - harry8 5y ago
    Ok but you're just repeating "It's inefficient" and not saying in any way for what use is its inefficiency even noticeable. I want to reason about when I would care. You see? The first link didn't even have units on its numbers(!) I assume they're milliseconds. When does that scale become something one would care about at all? Not launching a gui process. Not a shell pipeline. So when is this issue arising at all? What is being done that makes fork inefficiency anything other than academic interest. Must be something, right? Forking webserver?
quietbritishjim 5y ago
Apologies if this is a silly question, but it seems like there's a false dichotomy here: (1) You have separate fork() (etc.) and exec(), so that in the brief window in between you can set all the properties of the new process using APIs that exist anyway for controlling your own process. (2) You have a single call to spawn a new process, but you have a million different options to control every aspect of the new process. Why not do it this other way instead? Perhaps a bit late now but seems like in retrospect it would give the API simplicity of fork+exec without any of the complications. (3) There are two steps to run a new process. The first fully sets up its memory and returns a PID, but doesn't start running it. The second call, unfreeze(), allows it to begin executing code. All the usual APIs that exist anyway for controlling your own process take an extra parameter specifying the PID of a frozen child (or -1 for the current process).
cryptonector 5y ago
Some links I found today researching this: - @famzah'z blog about fork vs vfork vs clone performance: https://blog.famzah.net/tag/fork-vfork-popen-clone-performance/ - A very similar idea to my afork() idea, from 2 years earlier: https://developers.redhat.com/blog/2015/08/19/launching-helper-process-under-memory-and-latency-constraints-pthread_create-and-vfork - misc https://inbox.vuxu.org/tuhs/CAEoi9W6HFL3UcnWkKoqka8Dt16MWskKd6yEJr3HYCcCT9pMTig@mail.gmail.com/T/ https://bugzilla.redhat.com/show_bug.cgi?id=682922 (see attachments)
switch33 5y ago
The problem is clone is more of a start phase after vfork but before fork regardless for github. So it's kind of a bit strange that we call vfork first but that is about templates too. As for templates they need to be in different languages and in different formats for video games consoles, and so many other formats they port systems and games that sort of work digitally to certain things but not playable to certain things too. The other problem is that clone is part of syscall interfaces and part of apis and part of a lot of other things too.
- deleted 5y ago
  [deleted]

379 points by __s 5y ago

The problem with this argument is that the set of programs that just fork() and then exec() is fairly small. Sure, shells are small and do this, but then the article argues that shells are a good use of fork(). In larger programs, you're forking because you need to diverge the work that's going to be done and probably where it's going to be done (maybe you want to create a new pid ns, you need a separate mm because you're going to allocate a bunch, whatever). Maybe the argument is that programs should never do this? I don't buy that. Then there's a lot of string-slinging through exec().

pm215 5y ago
That's backwards from my experience, which is that most users of fork() only do "fork; child does small amount of setup, eg closing file descriptors; exec". Shells are one of the few programs that do serious work in the child, because the POSIX shell semantics surface "create a subshell and do..." to the shell user, and then the natural way to implement that when you're evaluating an expression tree is "fork, and let the child process continue evaluating as a long-lived process continuing to execute as the same shell binary". (Depending on what's in that sub-tree of the expression, it might eventually exec, but it equally might not.) Many years back I worked on an rtos that had no fork(), only a 'spawn new process' primitive (it didn't use an MMU and all processes shared an address space, so fork would have been hard). Most unixy programs were easy to port, because you could just replace the fork-tweak-exec sequence with an appropriate spawn call. The shells (bash, ash I think were the two I looked at) were practically impossible to port -- at any rate, we never found it worth the effort, though I think with a lot of effort and willingness to carry invasive local patches it could have been done.
olliej 5y ago
The vast majority of programs that fork are doing fork() followed almost immediately by exec(), to the extent that on macOS for example a process is only really considered safe for exec() after fork() happens. Pretty much nothing else is considered safe.
- NovemberWhiskey 5y ago
  Yeah; that would be my assumption too. I worked one time on a significant project that benefit from fork() without exec() and it was a monstrous pain - only if you own every single line of code in your project, have centralized resource management, and have no significant library dependencies should you ever consider doing this.
  - olliej 5y ago
    Yeah, you can't depend on pthreads or pthread mutexes (they're not defined as being fork safe). The entirety of Foundation (so presumably anything in Swift) is not fork safe either. To be clear: "not fork safe" in this case means "severely constrained environment": e.g. you can do things liker limits, set up pipes, etc but good luck with much more. I guess morally similar to the restrictions you have in a signal handler, albeit with different restrictions.
kllrnohj 5y ago
> In larger programs, you're forking because you need to diverge the work that's going to be done and probably where it's going to be done That's usually going to be done with clone() instead, no? You'll likely want to fiddle with the various flags for those usages and are unlikely to be happy with what fork() otherwise does.
cryptonector 5y ago
Oh no, there's tons of ProcessBuilder type APIs in Java, Python, and... every major language you can think of. The problems with fork() become very apparent in any Java apps that try to run external programs, especially in apps that have many threads and massive heaps and are very busy.

sys_64738 5y ago

The intent of fork() is to start a new process in its own address space. That *fork() variations that run in the SAME address space are confusing. A use case today for fork() might also be sandboxing apps. Certainly I expect browsers use this approach to spawn unique pages. But generally fork() is very specific from my recollection.

cryptonector 5y ago
> The intent of fork() is to start a new process in its own address space. True! > That *fork() variations that run in the SAME address space are confusing. Why is it confusing? They are distinct and different system calls, with different semantics. They are also sufficiently similar that they are also similarly named. But there's nothing confusing about their semantics. vfork() is not harder to use than fork() -- it's just subtly different. > A use case today for fork() might also be sandboxing apps. Certainly I expect browsers use this approach to spawn unique pages. I wouldn't expect that. Sandboxing is a large and complex topic.*

jph 5y ago

Is it a fair point to implement first with fork() because of memory protection, then optimize by using benchmarks and potentially vfork() for speed? Benchmark areas can look at synchronous locks, copy-on-write memory, stack sharing, etc. What are the good practices of security tradeoffs of fork() vs. vfork() especially in terms of ease of writing correct code? I'd thought that fork() + exec() tends to favor thinking about clearer separation/isolation. For example I've written small daemons using fork() + exec() because it seems safe and easy to do at the start.

GrumpySloth 5y ago
In short, fork() mixes poorly with multi-threaded code (and has some security footguns like needing to explicitly unshare elements of environment which may be sensitive, such as file descriptors (suddenly you need to know all the file descriptors used in the whole program from a single place in code)). Here is a well-written comment about fork() from David Chisnall: <https://lobste.rs/s/cowy6y/fork_road_2019#c_zec42d https://lobste.rs/s/cowy6y/fork_road_2019#c_zec42d> Additionally, the fork()+exec() idiom practically forces OS designers into a corner where they simply have to implement Copy-on-Write for virtual memory pages, or otherwise the whole userspace using this idiom is going to be terribly slow. Without the fork()+exec() idiom you don't need CoW to be efficient.
- kevincox 5y ago
  TL;DR if another thread is holding a lock when you fork that lock will be stuck locked in the child, but that thread that was using that lock no longer exists. So if your multi-threaded program uses malloc you may fork while a global allocation lock is being held and you won't be able to use malloc or free in the child (thread-local caches aside). There are other problems but this is the basic idea. To be fork-safe you need to allow any thread to just disappear (or halt forever) at any point in your program.
  - kazinator 5y ago
    malloc has to guard its locks against fork, probably using pthread_atfork, or some lower level internal API related to that. The problem with pthread_atfork is third party libs. YOU will use it in YOUR code. The C library will correctly use it in its code. But you have no assurance that any other libraries are doing the right things with their locks.

londons_explore 5y ago

I was always disappointed by the performance of fork()/clone(). CompSci class told me it was a very cheap operation, because all the actual memory is copy-on-write, so its a great way to do all kinds of things. But the reality is that duplicating huge page tables, and hundreds of file handles is very slow. Like 10's of milliseconds slow for a big process. And then the process runs slowly for a long time after that because every memory access ends up causing lots of faults and page copying. I think my CompSci class lied to me... it might seem cheap and a neat thing to do, but the reality is there are very few usecases where it makes sense.

smasher164 5y ago
Also, mandating copy-on-write as an implementation strategy is a huge burden to place on the host. Now you’ve made the amount of memory a process is is using unquantifiable.
- immibis 5y ago
  You also mandate a system complex enough to have an MMU.
- vgel 5y ago
  It's not necessarily unquantifiable -- the kernel can count the not-yet-copied pages pessimistically as allocated memory, triggering OOM allocation failures if the amount of potential memory usage is greater than RAM. IIUC, this is how Linux vm.overcommit_memory[1] mode 2 works, if overcommit_ratio = 100. However, if an application is written to assume that it can fork a ton and rely on COW to not trigger OOM, it obviously won't work under mode 2. [1] https://www.kernel.org/doc/Documentation/vm/overcommit-accounting https://www.kernel.org/doc/Documentation/vm/overcommit-accou... > 2 - Don't overcommit. The total address space commit for the system is not permitted to exceed swap + a configurable amount (default is 50%) of physical RAM. > Depending on the amount you use, in most situations this means a process will not be killed while accessing pages but will receive errors on memory allocation as appropriate. > Useful for applications that want to guarantee their memory allocations will be available in the future without having to initialize every page.

lisper 5y ago

There is something about fork which I have never understood. Maybe someone here can explain it to me. Why would anyone ever want fork as a primitive? It seems to me that what you really want is a combination of fork and exec because 99% of the time you immediately call exec after fork (at least that's what I do 99% of the time when I use fork). If you know that you're going to call exec immediately after fork, then all the issues of dealing with the (potentially large) address space of the parent just evaporate because the child process is just going to immediately discard it all. So why is there not a fork-exec combo? And why has it not replaced fork for 99% of use cases? And as long as I'm asking stupid questions, why would anyone ever use vfork? If the child shares the parent's address space and uses the same stack as the parent, and the parent has to block, how is that different from a function call (other than being more expensive)? None of this makes sense to me.

duskwuff 5y ago
> Why would anyone ever want fork as a primitive? fork() without exec() can make sense in the context of a process-per-connection application server (like SSH). I've also used it quite effectively as a threading alternative in some scripting languages. > So why is there not a fork-exec combo? There is; it's called posix_spawn(). Like a lot of POSIX APIs, it's kind of overcomplicated, but it does solve a lot of the problems with fork/exec. > And as long as I'm asking stupid questions, why would anyone ever use vfork? For processes with a very large address space, fork() can be an expensive operation. vfork() avoids that, so long as you can guarantee that it'll immediately be followed by an exec().
- kazinator 5y ago
  fork with copy-on-write semantics avoids copying the whole address space. It does have to copy some data structures that manage virtual memory and maybe the first level of the paging structure(page directory or whatever).
  - cryptonector 5y ago
    copy-on-write == slow when called from threaded processes with large resident set sizes.

immibis 5y ago

Another option is to allow the parent to create an empty child process, and then make arbitrary system calls and execute code in the child, like a debugger does. In most cases the last "remote system call" would be exec.

cryptonector 5y ago
posix_spawn() essentially is like that, or can be, as an implementation detail.

lucideer 5y ago

The good/evil/etc. here seem to be defined exclusively around "performance above all else", and - more specifically - performant primitives over performant application architecture. It strikes me that performance gains associated with sharing address space & stack are similar to many performance gains: trade-offs. So calling them "good" and "evil" when performance is seemingly your sole goal and interest seems a bit forward.

cryptonector 5y ago
In my world we often say things like "X is the moral equivalent of Y" where X and Y are just technologies and, clearly, are morally-neutral things. Why do we do this? Well, because it adds emphasis, and a dash of humor. Clearly fork() is neither Good nor Evil. It's morally neutral. It has no moral value whatsoever. But to say "fork() is evil" is to cause the audience to raise their eyebrows -"what, why would you say fork() is evil?!"- and maybe pay attention. Yes, there is the risk that the audience might react dismissively because fork() obviously is morally-neutral, so any claim that it is "evil" must be vacuous or hyperbolic. It's a risk I chose to take. Really, it's a rhetorical device. I think it's pretty standard. I didn't create that device myself -- I've seen it used before and I liked it.
- lucideer 5y ago
  Morally-neutral does not equate to neutral insofar as I think most technologists consider some tech to be "good" and some to be "bad" in a practical sense. "Good -vs- evil" is obviously hyperbolic - particularly the latter - but outside of morals they still imply a tendency to be technically/practically good or bad in an objective sense. So discounting it as a mere rhetorical device seems overly dismissive.

mrob 5y ago

Fork() is the second worst idea in programming, behind null pointers. Fork() is the reason overcommit exists, which is the reason my web browser crashes if I open too many tabs, and the reason the "safe" Rust programming language leaves software vulnerable to DOS attacks if it uses the standard library. It's a clear example of "worse is worse", and we should have switched to the Microsoft Windows model decades ago. Here's a paper from Microsoft Research supporting this point of view: https://www.microsoft.com/en-us/research/uploads/prod/2019/04/fork-hotos19.pdf https://www.microsoft.com/en-us/research/uploads/prod/2019/0...

nick_ 5y ago
Interesting take. If you don't mind explaining, what is the MS Windows model in in this context?
- mrob 5y ago
  You opt into inheriting specific contexts from the parent, instead of copying everything by default: https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessa https://docs.microsoft.com/en-us/windows/win32/api/processth...
  - manwe150 5y ago
    More importantly, all syscalls also take a target process as an argument, making the Windows version both simpler and more powerful than can be done with fork. Spawn is also a lot slower on Windows, but that is an implementation issue.
    - kllrnohj 5y ago
      > Spawn is also a lot slower on Windows, but that is an implementation issue. afaik most of that slowdown is because malware scanners (including Windows Defender) hook spawn to do blocking verification of what to launch. Which is an issue also present on eg. MacOS, and why it's also kinda slow to launch new processes (and can be subject to extreme latencies): https://www.engadget.com/macos-slow-apps-launching-221445977.html https://www.engadget.com/macos-slow-apps-launching-221445977... Which is yes an implementation problem, but also a problem that potentially changes/impacts the design. Like maybe it'd make sense to get a handle to a pre-verified process so that repeated spawns of it don't need to hit that path (for eg. something like Make or Ninja that just spam the same executable over and over and over again). Or the kernel/trusted module needs to in some way be involved & can recognize that an executable was already scanned & doesn't need to be re-scanned.

JoeAltmaier 5y ago

The whole idea of fork is strange - the design pattern of "child process is executing exactly where the parent process is executing" is foreign to me. Don't we want to direct where the child process is executing? Like, when creating a thread? Why is fork() so conceptually orthogonal to that? Is there a good reason? A historical reason? I don't find fork() to be obvious or useful or natural. I work hard to never do it.

shadowofneptune 5y ago
fork()–exec() separation indeed exists for historical reasons: https://www.bell-labs.com/usr/dmr/www/hist.html https://www.bell-labs.com/usr/dmr/www/hist.html Search for the phrase "Process control in its modern form was designed and implemented within a couple of days."
kccqzy 5y ago
If you want the child to start executing some other code but you have fork(), it's easy to do it yourself by calling that function. But on the other hand, if you do want the child to execute code at the same place as the parent, but a hypothetical fork() asks you to provide a function pointer, it would be a bit more complicated.
zamalek 5y ago
It's a leaky abstraction and everything it does can be done manually, and possibly better. It exists purely because, at some point in the past, threads didn't exist. If you design your program without fork, you'll probably end up with a cleaner and faster solution. Some things are best forgotten or never learned in the first place.
- _flux 5y ago
  Can it though? The beauty of (v)fork(+exec) is that it doesn't need a new interface for configuring the environment in whichever way you want before the other process starts. Instead you get to use the exact same means of modifying the environment to your needs, and once it's done, you can call exec and the new process inherits those things. I mean, just look at the interface of posix_spawn. I grant though that this isn't without its problems (including performance) and IMO e.g. FD_CLOEXEC is one example of how those problems can be patched up. It's like the reverse problem: you have too wide implicit interface in it, and then you need to come up with all these ways to be explicit about some things.

tiffanyh 5y ago

Slightly off topic, how does Erlang handle this because isn’t it know for having extremely fast & cheap process spawning baked in (with isolation).

__s 5y ago
m:n threads (aka green threads) https://twitter.com/joeerl/status/1010485913393254401 https://twitter.com/joeerl/status/1010485913393254401
- tiffanyh 5y ago
  I realize that tweet is from the authority himself but am I mistaken in my understanding … I thought green threads share memory but Erlang processes do NOT share memory, which is what makes Erlang so unique. Did Erlang create a so called “green process”? If so, why can’t this model be implemented in the kernel?
  - masklinn 5y ago
    > I thought green threads share memory but Erlang processes do NOT share memory, which is what makes Erlang so unique. Erlang processes don’t share memory because the language and vm don’t give primitives which let you do it. They all exist within the same address space (e.g. large binaries are reference-counted and stored on a shared heap, excluding clustering obviously). > Did Erlang create a so called “green process”? Yes. > If so, why can’t this model be implemented in the kernel? Because erlang processes are not an antagonistic model, and the language restricts the ability to attack the VM (kinda, I’m sure you could write NIFs to fuck up everything, you just don’t have any reason to as an application developer).
jerf 5y ago
In another comment, I observe how Go doesn't even have a binding to fork. Erlang is another example of that. There is no standard library binding to the fork function. If someone were to bash one into a NIF, I have no idea what would happen to the resulting processes, but there's no good that can come of it. (To use Star Trek, think less good and evil Kirk and more "What we got back, didn't live long... fortunately.") Despite the terminology, all Erlang processes are green threads in a single OS process.

evmar 5y ago

In Ninja, which needs to spawn a lot of subprocesses but it otherwise not especially large in memory and which doesn't use threads, we moved from fork to posix_spawn (which is the "I want fork+exec immediately, please do the smartest thing you can" wrapper) because it performed better on OS X and Solaris: https://github.com/ninja-build/ninja/commit/89587196705f54afb904c8f4572e65de7274dd81 https://github.com/ninja-build/ninja/commit/89587196705f54af...

ridiculous_fish 5y ago
posix_spawn also outperforms fork on Linux under more recent glibc and musl, which can use vfork under the hood. https://twitter.com/ridiculous_fish/status/1232889390763933698 https://twitter.com/ridiculous_fish/status/12328893907639336...
xroche 5y ago
The issue with posix_spawn is that you can't close all descriptors before exec. This is especially an issue as most libraries are still unaware they need to open every single handle with the close-on-exec flag set.
- cryptonector 5y ago
  Solaris/Illumos has an extension[0] for that. [0] http://src.illumos.org/source/search?project=illumos-gate&full=posix_spawn_file_actions_addclosefrom_np&defs=&refs=&path=&hist=&type=&xrd=&nn=1 [1] https://docs.oracle.com/cd/E36784_01/html/E36874/posix-spawn-file-actions-addclosefrom-np-3c.html
  - kazinator 5y ago
    For implementations which don't have it, you can stuff, into the file_actions, say, 4093 close action entries into the file_actions, targeting descriptors 3 to 4095. This big file_actions object can be cached and re-used for multiple calls to posix_spawn. It won't close descriptor 4096, but that's probably beyond giving a darn in most cases. If you have an application that opens high descriptor numbers, you probably know.

kazinator 5y ago

Concurrently running dupe currently on front page: https://news.ycombinator.com/item?id=30499169 https://news.ycombinator.com/item?id=30499169 :) :) :)

cryptonector 5y ago
Ha!

ridiculous_fish 5y ago

Amusingly vfork semantics differ across OSes. This program prints 42 in Linux but 1 on Mac: https://godbolt.org/z/jn7Gaf5Me https://godbolt.org/z/jn7Gaf5Me because on Linux they share address space.

butterisgood 5y ago
I am pretty sure Mac OS doesn't COW fork(), and that the address space is copied. At least it was the last time I looked. FreeBSD and Linux both seem to COW. Perhaps there's a reason vfork is different too.
- ridiculous_fish 5y ago
  My (very possibly wrong) understanding is that xnu does CoW fork but doesn't overcommit, meaning that memory must be reserved (perhaps in swap) in case the pages need to be duplicated. There's other complications relating to inheriting Mach ports and the mach_task <-> BSD process "duality" in xnu, which Linux doesn't have. I'd love for someone to chime in who knows more about how this stuff works.
- cryptonector 5y ago
  Hopefully someone at Apple will see this post and be convinced to restore vfork() and un-obsolete it.
cryptonector 5y ago
Unfortunately there was this paper from the 80s titled "vfork() Considered Dangerous", which led to BSDs removing vfork(), and then later it was re-added because that paper was clearly quite wrong. But the news hasn't quite filtered through to Apple, I guess.

mark_undoio 5y ago

The code I currently work on actually has a use of `clone` with the `CLONE_VM` flag to create something that isn't a thread. Since `CLONE_VM` will share the entire address space with the child (you know, like a thread does!) a very reasonable response would be "WAT?!" What led us here was a need to create an additional thread within an existing process's address space but in a way that was non-disruptive - to the rest of the process it shouldn't really appear to exist. We achieved this by using `CLONE_VM` (and a handful of other flags) to give the new "thread-like" entity access to the whole address space. But, we omitted `CLONE_THREAD`, as if we were making a new process. The new "thread-like" entity would not technically be part of the same thread group but would live in the same address space. We also used two chained `clone()` calls (with the intermediate exiting, like when you daemonise) so that the new "thread-like" wouldn't be a child of the original process. All this existed before I joined, it's just really cool that it works. I've never encountered a such a non-standard use of clone before but it was the right tool for this particular job!

scottlamb 5y ago
> What led us here was a need to create an additional thread within an existing process's address space but in a way that was non-disruptive - to the rest of the process it shouldn't really appear to exist. I'm curious to hear more. What's its purpose?
- kccqzy 5y ago
  Maybe some kind of snapshotting for an in-memory database?
- mark_undoio 5y ago
  > I'm curious to hear more. What's its purpose? Sure! I'll try to illustrate the general idea, though I'm taking liberties with a few of the details to keep things simple(r). Our software (see https://undo.io https://undo.io) does record and replay (including the full set of Time Travel Debug stuff - executing backwards, etc) of Linux processes. Conceptually that's similar to `rr` (see https://rr-project.org/ https://rr-project.org/) - the differences probably aren't relevant here. We're using `ptrace` as part of monitoring process behaviour (we also have in-process instrumentation). This reflects our origins in building a debugger - but it's also because `ptrace` is just very powerful for monitoring a process / thread. It is a very challenging API to work with, though. One feature / quirk of `ptrace` is that you can't really do anything useful with a traced thread that's currently running - including peeking its memory. So if a program we're recording is just getting along with its day we can't just examine it whenever we want. First choice is just to avoid messing with the process but sometimes we really do need to interact with it. We could just interrupt a thread, use `ptrace` to examine it, then start it up again. But there's a problem - in the corners of Linux kernel behaviour there's a risk that this will have a program-visible side effect. Specifically, you might cause a syscall restart not to happen. So when we're recording a real process we need something that: * acts like a thread in the process - so we can peek / poke its memory, etc via ptrace * is always in a known, quiescent state - so that we can use ptrace on it whenever we want * doesn't impact the behaviour of the process it's "in" - so we don't affect the process we're trying to record * doesn't cause SIGCHLD to be sent to the process we're recording when it does stuff - so we don't affect the process we're trying to record Our solution is double clone + magic flags. There are other points in the solution space (manage without, handle the syscall restarting problem, ...) but this seems to be a pretty good tradeoff. [edit: fixed a typo]

pipeline_peak 5y ago

Your idea good Your idea stupid I’m not woke by any means, idk what it is about low level programming but calling someone’s idea “stupid” is a really shitty thing to say. “He chose to take it personally” is the type of lazy, pseudo-stoic argument I have no interest in reading. Yes I’m having a morning, lol.

cryptonector 5y ago
I answered this here: https://news.ycombinator.com/item?id=30504804 https://news.ycombinator.com/item?id=30504804 It's a rhetorical device. I didn't expect this to -years later- become a front-page item on HN. I wrote that to share with certain people. And yes, clone() has some real problems, and if calling it "stupid" pisses off some people, but maybe also leads others to want to improve clone() or create a better alternative, then that's fine. If I'd wanted to write an alternative to Linux I'd probably have had to deal with the very, very fine language that Linus and others use on the Linux kernel mailing lists -- if you don't like my using the word "stupid", then you really shouldn't look there because you're likely to be very disappointed. Indeed, not only would I have to accept colorful language from reviewers there, I'd probably have to employ some such language myself. TL;DR: clone() came from Linux, where "stupid" is the least colorful language you'll find, and me calling it "stupid" is just a rhetorical device.

cryptonector 5y ago

Well, I'm surprised to see this on the front page, let alone as #1. Ask me anything. EDIT: Also, don't miss @NobodyXu's comment on my gist, and don't miss @NobodyXu's aspawn[1]. [0] https://gist.github.com/nicowilliams/a8a07b0fc75df05f684c23c18d7db234?permalink_comment_id=3467980#gistcomment-3467980 [1] https://github.com/NobodyXu/aspawn/

ahmedalsudani 5y ago
No questions yet as I am yet to read ... but I can already comment and say grade A title.
- cryptonector 5y ago
  It's a bit opinionated. It's meant to get a reaction, but also to have meaningful and thought-provoking content, and I think it's correct in the main too. Anyways, hope you and others enjoy it.
  - ahmedalsudani 5y ago
    That was a great read. Thank you for writing it up; I learned quite a few things! Especially appreciated the OS minutiae and opinionated commentary (... and the doc vs reality observation in Linux's vfork). The piece lives up to the great title :)
  - disgruntledphd2 5y ago
    What do you mean by zones/jails and why are they better than containers?
    - cryptonector 5y ago
      Zones -> Solaris/Illumos Zones Jails -> BSD jails They're software VMs. It's a lot like containers, yes. The problem with containers is that the construction toolkit for them is subtractive ("start by cloning my environment, then remove / replace various namespaces"), while the construction toolkit for zones/jails is additive ("start with an empty universe, and add namespaces or share them with the parent"). Constructing containers subtractively means that every time there's a new kind of namespace to virtualize, you have to update all container-creating tools or risk a security vulnerability. Constructing containers additively from an empty universe means that every time there's a new kind of namespace to virtualize, you have to update all container-creating tools or risk not getting sharing that you want (i.e., breakage). I'm placing a higher value on security. Maybe that's a bad choice. It's not like breaking is a good thing -- it might be just as bad as creating a security vulnerability.

throwaway984393 5y ago

I don't think containers should be like jails. Containers should be more like chroots than they are now. Have you ever tried to run a modern X/whatever app with 3D graphics and audio and DBUS and God knows what else in a container and get it to show up on your desktop? It's a fucking nightmare. I spent over a week trying to get 1Password to run in a container. Somebody decided containers had to be "secure", even though they don't actually exist as a single concept and security was never their primary purpose. If instead containers were used only to isolate filesystem dependencies, we could actually pretend containers were like normal applications and treat them with the same lack of security concern that all the rest of our non-containerized programs are. Firecracker is the correct abstraction for isolation: a micro-VM. That is the model you want if you want to run an app securely (not to mention reliably, as it can come with its own kernel, rather than needing you to run a compatible host kernel).

cryptonector 5y ago
I... didn't mean that containers have to have a copy of the operating system inside them, systemd and many other things included. I meant only that they should be created in ways like how the BSDs and Illumos do it.

infogulch 5y ago

The dense fog lifts, tree branches part, a ray of light beams down on a pedestal revealing the hidden intentions of the ancients. A plaque states "The operational semantics of the most basic primitives of your operating system are designed to simplify the implementation of shells." You hesitantly lift your eyes to the item presented upon the pedestal, take a pause in respect, then turn away slumped and disappointed but not entirely surprised. As you walk you shake your head trying to evict the after image of a beam of light illuminating a turd.

deleted 5y ago
[deleted]
ckastner 5y ago
> "The operational semantics of the most basic primitives of your operating system are designed to simplify the implementation of shells." Yes, but why is this characterized as something negative? Isn't that the entire point? Operating systems are there to serve user requests, and shells are an interface between user and OS. Shells simply developed features that users required of them.
- int_19h 5y ago
  Shells haven't been the primary interface between the user and the OS for decades.
  - kragen 5y ago
    "The primary interface between the user and the OS" is the definition of "shell". That's why the Microsoft Windows process that draws the Start button and filesystem windows is called "the Windows shell".
    - int_19h 5y ago
      I don't think OP meant shell as in the Windows shell, or Linux DEs. I mean, how many of those use fork() even on Linux, or would be easier to implement if they did?

scottlamb 5y ago

> clone() is stupid ... the clone(2) design, or its maintainers, encourages a proliferation of flags, which means one must constantly pay attention to the possible need to add new flags at existing call sites. IMHO a bigger problem [2] in practice with clone is that (according to glibc maintainers) once your program calls it, you can't call any glibc function anymore. [1] Essentially the raw syscall is a tool for the libc implementation to use. The libc implementation hasn't provided a wrapper for programs to use which maintains the libc's internal invariants about things like (IIUC) thread-local storage for errno. The author's aforkx implementation is something that glibc maintainers could (and maybe should) provide, but my understanding is that you can get in trouble by implementing it yourself. [1] https://github.com/rust-lang/rust/issues/89522#issuecomment-946023944 https://github.com/rust-lang/rust/issues/89522#issuecomment-... [2] editing to add: or at least a more concrete expression of the problem. Wouldn't surprise me if they haven't provided this wrapper in part because the proliferation the author mentioned makes it difficult for them to do so.

jlokier 5y ago
It's really unfortunate that the sanctioned way to call Linux syscalls directly is via the syscall() function (previously the _syscallN macros), and both of those methods set errno on error, which fails in a clone() thread. If only Glibc provided a syscall_r() or something that returns the raw return value whether it's an error or not. It is possible to make syscall() (and regular libc syscalls like read()) work in a clone() thread. I use this in performance-optimised I/O code in a database engine, so I know it works, but it requires some ugly Glibc-and-architecture-specific things. Doing it portably doesn't seem to be an option.

aylmao 5y ago

Meta comment: Github Gist seems to be great for blogging. Yeah, the UI is not very blog-specific, but it has all the useful features, and then some: markdown, comments, hosting, an index of all posts, some measure of popularity (stars), a very detailed edit history, etc. All without having to pay or setup anything yourself.

JoshTriplett 5y ago
Unfortunately, there's no way to turn off comments on a Gist, which makes it not a viable replacement for anyone who doesn't want to spend a lot of time processing and moderating comments.
- chmaynard 5y ago
  Good point. However, you need a GitHub account to post comments so everyone knows who you are. Your reputation might suffer if you constantly post comments that require moderation.
  - JoshTriplett 5y ago
    This does not, in practice, stop people. Both because it's possible to make throw-away accounts, and because some people don't have a reputation to care about to begin with.
fomine3 5y ago
It's great because Chinese GFW won't block.

mattgreenrocks 5y ago

> Long ago, I, like many Unix fans, thought that fork(2) and the fork-exec process spawning model were the greatest thing, and the Windows sucked for only having exec() and _spawn(), the last being a Windows-ism. I appreciate this quite a bit. Vocal Unix proponents tend to believe that anything Unix does is automatically better than Windows, sometimes without even knowing what the Windows analogue is. Programming in both is necessary to have an informed opinion on this subject. The one thing I miss most on Unix: the unified model of HANDLEs that enables you to WaitOnMultipleObjects() with almost any system primitive you could want, such as an event with a socket (blocking I/O + a shutdown notification) in one call. On Unix, a flavor of select() tends to be the base primitive for waiting on things to happen, which means you end up writing adapter code for file descriptors to other resources, or need something like eventfd. Things I don't miss from Windows at all: wchar_t everywhere. :)

cryptonector 5y ago
WIN32 got a few things very right: - SIDs - access tokens (like struct cred / cred_t in Unix kernels, but exposed as a first-class type to user-land) - security descriptors (like owner + group mode_t + ACL in Unix land, but as a first-class type) - HANDLEs, as you say - HANDLEs for processes Many other things, Windows got wrong. But the above are far superior to what Unix has to offer.
- al2o3cr 5y ago
  I'd be curious how many of those derive from NT's VMS roots - for instance: http://lxmi.mi.infn.it/~calcolo/OpenVMS/ssb71/6346/6346p004.htm http://lxmi.mi.infn.it/~calcolo/OpenVMS/ssb71/6346/6346p004....
  - cryptonector 5y ago
    Most of them, as far as I know.
    - p_l 5y ago

ismaildonmez 5y ago

Microsoft Research has a paper about the very same issue (2019): https://www.microsoft.com/en-us/research/publication/a-fork-in-the-road/ https://www.microsoft.com/en-us/research/publication/a-fork-...

cryptonector 5y ago
It's a very good paper, yeah. I will link it from the gist.
georgia_peach 5y ago
That paper smacks of a Chesterton Fence. They haven't come up with a tested replacement for many of the use cases, i.e.: These designs are not yet general enough to cover all the use-cases outlined above, but perhaps can serve as a starting point... yet bullet #1 in the next paragraph is Deprecate Fork I think this is a case of security guys being upset about fork gumming-up their experiments. I don't really care about their experiments. The security regime for the past 20 years may have bought us a little more security against eastern bloc hackers, but it hasn't done squat to protect us from Apple, Google, & Microsoft! I have never had a virus de-rail my computing life as much as the automatic Windows 10 upgrade. Robert Morris got 400 hours community service for a relatively benign worm. If that's the penalty scale, Redmond should get actual time in the slammer for Cortana, forced Windows Update, and adding telemetry to Calculator.
- cryptonector 5y ago
  You fail to address any of the substance of their paper, or of my gist (TFA), then go on a rant about unrelated things. The authors of that paper deserve better treatment even if you hate Microsoft.
  - deleted 5y ago
    [deleted]
  - georgia_peach 5y ago
    I did. Chesterton Fence. fork() has been in Unix from the beginning. Taking it out at this point will cause more problems than it solves. Until you have a working Unix distro (kernel AND common userland services) that elegantly covers all of the forkless cases, your paper and their paper are just opinions. Theirs is a formally written one. Yours is a clickbaity one. And casting vfork() as any kind of improvement here is just bonkers. And the rant is totally related: i.e. devs breaking things that worked just fine to begin with for the sake of doctrinal purity. It is usually a false doctrine.

psanford 5y ago

I saw a bug once where an application would get way slower on MacOS after calling fork(). Not just temporarily either; many syscalls would continue to run slowly from the call to fork() until the process exited. Looking on Stack Overflow, I see a few reports of this behavior[0][1]. [0]: https://stackoverflow.com/questions/4411840/memory-access-after-fork-extremely-slow-on-mac-os-x https://stackoverflow.com/questions/4411840/memory-access-af... [1]: https://stackoverflow.com/questions/27932330/why-is-tzset-a-lot-slower-after-forking-on-mac-os-x https://stackoverflow.com/questions/27932330/why-is-tzset-a-...

deleted 5y ago

[deleted]

mywacaday 5y ago

"I won't bother explaining what fork(2) is -- if you're reading this, I assume you know.", If that applied to everything I looked at from HN I'd read precious little.

cryptonector 5y ago
I didn't write it for HN. It wasn't a paper to publish in some Computer Science journal. It was just a github gist. If you don't get the subject, it's not for you. I might well write a paper now based on it, and then it might be a good read for you, but I still won't be writing it for you, but for people who are interested in the topic. The intended audience is small, expert on the matter, and probably even more opinionated than I am.
- mywacaday 5y ago
  I found the article well written and informative even though it's not my area of expertise, I intended my comment as a light hearted reflection of the fact that a lot of articles on HN go over my head but are still worth a read to me, just like your article.

saurik 5y ago

One use case for fork()--which is used extensively on Android--is to build an expensive template process that can then be replicated for later work, which is exactly what people often want for the behavior with virtual machines. I wrote an article on the history of linking and loading optimizations leading up to how Android handles their "zygote" which touches on this behavior. http://www.cydiasubstrate.com/id/727f62ed-69d3-4956-86b2-bc0ffea110c1/ http://www.cydiasubstrate.com/id/727f62ed-69d3-4956-86b2-bc0...

monocasa 5y ago

Hard disagree to most of this. fork(2) makes a lot more sense when you realize its heritage. It came from a land before Unix supported full MMUs. In this model, to still have per process address spaces and preemptive multitasking on what was essentially a PC-DOS level of hardware, the kernel would checkpoint the memory for a process, slurp it all out to dectape or some such, and load in the memory for whatever the scheduler wanted to run next. It's simplicity of being process checkpoint based wasn't a reaction to windows style calls (which wouldn't exist for almost a couple decades), but instead mainframe process spawning abominations like JCL. The idea "you probably want most of what you have so force a checkpoint, copy the checkpoint into a new slot, and continue separately from both checkpoints" was soooo much better than JCL and it's tomes of incantations to do just about anything. vfork(2) is an abomination. Even when the child returns, the parent now has a heavily modified stack if the child didn't immediately exec(). All of those bugs that causes are super fun to chase, lemme tell you. AFAIC, about the only valid use for vfork now is nommu systems where fork() incredibly expensive compared to what is generally expected. clone(2) is great. Start from a checkpoint like fork, but instead of semantically copying everything, optionally share or not based on a bitmask. Share a tgid, virtual address space, and FD table? You just made a thread. Share nothing? You just made a process. It's the most 'mechanism, not policy' way I've seen to do context creation outside of maybe the l4 variants and the exokernels. This isn't an old holdover, this is how threads work today, processes spawned that happen to share resources. Modern archs on linux don't even have a fork(2) syscall; it all happens through clone(2). Even vfork is clone set to share virtual address space and nothing else that fork wouldn't share. Namespaces are a way to opt into not sharing resources that normally fork would share. And I don't see what afork gets you that clone doesn't, except afork isn't as general.

cryptonector 5y ago
> vfork(2) is an abomination. Even when the child returns, the parent now has a heavily modified stack if the child didn't immediately exec(). What stack modifications? Sure, the child can scribble over the stack frame, or worse, the child could do things like return -- but you are the author of the code calling vfork() and you know not to do that, so why would that happen? A: It just wouldn't happen. And as to exec() failing, this is why exec calls must be followed with calls to either exec() or _exit(), and this is true even if you use fork() instead of vfork(). I.e.: /* do a bunch of pre-vfork() setup */ ... pid_t pid = vfork(); if (pid == -1) err(1, "Couldn't vfork()"); if (pid == 0) { /* do a bunch of child-side setup */ execve(...); /* oops, ENOENT or something */ _exit(1); } /* the child either exec'ed or exited */ if (waitpid(pid, &status, 0) != pid) err(1, "..."); ... How do you detect if the child exec'ed or exited? Well, you make a pipe before you vfork(), you set its ends to be O_CLOEXEC, then on the child side of vfork() you write one byte into it if the exec call fails. On the parent side you read from the pipe before you reap the child, and if you get EOF then you know the child exec'ed, and if you get one byte then you know the child exited. The one byte could be an errno value. No, really, what you say about vfork() is lore, and very very wrong. That said, vfork() blocks a thread in the parent. The point of my gist was to explain why fork() sucks, why vfork() is much better, and what would be better still. > And I don't see what afork gets you that clone doesn't, except afork isn't as general. afork()/avfork() is not meant to be as general as clone() but to be more performant than vfork() by not blocking a thread on the parent side. clone() needs some improvements. It should be possible to create a container additively. See elsewhere in the comments on this post.

albertzeyer 5y ago

We had the case that some library we were using (OpenBLAS) used pthread_atfork. Unfortunately, the atfork handler behaved buggy in certain situations involving multiple threads and caused a crash. This was annoying because we basically did not need fork at all but just fork+exec (for various other libraries spawning sub processes), where those atfork handlers would not be relevant. Our solution was to override pthread_atfork to ignore any functions, and in case this is not enough, also fork itself to just directly do the syscall without calling the atfork handlers. https://github.com/tensorflow/tensorflow/issues/13802 https://github.com/tensorflow/tensorflow/issues/13802 https://github.com/xianyi/OpenBLAS/issues/240 https://github.com/xianyi/OpenBLAS/issues/240 https://trac.sagemath.org/ticket/22021 https://trac.sagemath.org/ticket/22021 https://bugs.python.org/issue31814 https://bugs.python.org/issue31814 https://stackoverflow.com/questions/46845496/ld-preload-and-linkage https://stackoverflow.com/questions/46845496/ld-preload-and-... https://stackoverflow.com/questions/46810597/forkexec-without-atfork-handlers https://stackoverflow.com/questions/46810597/forkexec-withou...

cryptonector 5y ago
posix_spawn() shouldn't call atfork handlers. It's allowed to call them or not call them because implementors can use fork(), which must call them, or they can use vfork(), which must not call them -- or they can make posix_spawn() a proper system call, too, or they can use clone(), or my putative avfork(), or whatever. If you used vfork(), you wouldn't have had this problem. Fork-safety issues arise mainly because of the sharing of resources between the parent and child. pthread_atfork() exists mainly to allow libraries to add a measure of fork-safety by letting them disable things on the child-side of fork() or re-set-up things on the child-side of fork(). For example, a PKCS#11 provider might need to create a new connection to the tokens and re-C_Login() to them (except, since it really can't quite do that, most likely it must render every session inoperable on the child-side). (Indeed, PKCS#11 specifically mandates that on the child-side of fork all sessions must be dead and must not be used.)
cryptonector 5y ago
I left a comment on TF #13802.

phendrenad2 5y ago

For those saying to use posix_spawn: What am I supposed to make of the writeup in the posix_spawn manpage though? "...specified by POSIX to provide a standardized method of creating new processes on machines that lack the capability to support the fork(2) system call. These machines are generally small, embedded systems lacking MMU support" Is this why no one uses it? It has this gratuitous opinion piece at the beginning that makes people think it's just for embedded systems and my dad's Amiga?

cryptonector 5y ago
Hah, that's pretty funny. Regardless of the motivation as written, the motivation I surmise is: - some systems (e.g., Windows) lack fork() for various reasons - vfork() is baaaad - I know, let's do something like WIN32's spawn() or CreateProcess(), but, like, better The middle item I have good reason to think is very likely. vfork() still has a bad rap from that old "vfork() Considered Dangerous" paper. That paper circulated a lot way back when, and was the reason vfork() was removed from some Unixes for a while (well, it was left as an alias of fork()) before it was eventually re-added. The Open Group participants would been very aware of that paper, and that is almost certainly the reason that POSIX says about vfork(): Conforming applications are recommended not to depend on vfork(), but to use fork() instead. The vfork() function may be withdrawn in a future version. So if fork() can't perform well, and the committee won't recommend the use of vfork(), what shall the committee do? Answer: design and specify posix_spawn(). It's not an unreasonable answer. Though, IMO of course, they should have un-obsoleted vfork().
xenadu02 5y ago
That's just some injected opinion, I assume from someone contributing to glibc who doesn't like posix_spawn I guess? In any case it is wrong. Don't assume what is written in man pages is the truth. Some of them have a lot of opinion added. It can be useful to cross-check man pages between systems - they don't always call out non-portable options or behavior. On some kernels posix_spawn is a syscall or specifies flags that make it more efficient than fork+exec. Darwin is one such system, though you can use POSIX_SPAWN_SETEXEC if you still want to replace the current process with a new executable rather than creating a child.

throwawaylinux 5y ago

This avfork implementation is poor. You don't want to make your single threaded programs multi-threaded. I don't really get the big benefit of afork over other existing mechanisms other than handwaving about things being evil. Also, > Linux should have had a thread creation system call -- it would have then saved itself the pain of the first pthread implementation for Linux. Linux should have learned from Solaris/SVR4, where emulation of BSD sockets via libsocket on top of STREAMS proved to be a very long and costly mistake. Emulating one API from another API with impedance mismatches is difficult at best. Linux does have a thread creation system call. It's clone(2). It literally creates new threads of execution with various properties. It does not "emulate" threads, it is threads.

cryptonector 5y ago
> You don't want to make your single threaded programs multi-threaded. Correct. I want to spawn processes faster from already-multi-threaded programs.
- throwawaylinux 5y ago
  You do, but it's not a good implementation for a general API is all I was trying to say. Do you really need an "asynchronous process creation" call? The rationale is that "blocking is bad", but a thread creation system call blocks the caller too until the thread is created. So it's not just "blocking", it's the amount of blocking if anything. Is posix_spawn or vfork+exec really too slow for your case? Then multi-process and multi-threading seems like a reasonable solution. Asynchronous system calls are the exception not the rule in unix. So it wouldn't make sense as a traditional afork(2) system call. You could probably do a posix_spawn for io_uring, but do you really need to?

bergkvist 5y ago

I have to disagree that fork is evil. fork is great because of copy-on-write. I guess my particular use case is not very typical/common though. I'm running powerflow simulations on a power grid model (several GB of memory to store the model). Copy-on-write means I can make small modifications to this model and run simulations in parallel. Thanks to fork/copy-on-write, I can run 32 simulations in parallel, each will small modifications without requiring 32 times as much memory.

cryptonector 5y ago
Neat!

userbinator 5y ago

I started with DOS, where spawn() is the norm, so I've always considered the fork()-like behaviour to be unusual yet handy for certain use-cases. Perhaps a system call that offers a combination of the two behaviours should be named spork().

Ericson2314 5y ago

This stuff is still all confused Read http://catern.com/rsys21.pdf http://catern.com/rsys21.pdf What you want is: 1. create "embryonic" unscheduled process 2. Set it up from the parent process, it just lies on the operating table passively. 3. Submit it to the scheduler. This is just....obviously correct. Totally flexible. Totally efficient. Hell, if you really want to fork anything, fork those embryonic process which have no active threads! Much safer and easier to understand! I did not write the paper above, but I did write https://lore.kernel.org/lkml/f8457e20-c3cc-6e56-96a4-3090d7da0cb6@JohnEricson.me/ https://lore.kernel.org/lkml/f8457e20-c3cc-6e56-96a4-3090d7d... https://lists.freebsd.org/archives/freebsd-arch/2022-January/000139.html https://lists.freebsd.org/archives/freebsd-arch/2022-January... I hope I or someone else will have time to make it happen!

cryptonector 5y ago
Yes, I like the larval process idea. No doubt it's good.
IgorPartola 5y ago
When I was first learning about UNIX and similar OSes I just assumed that this is how things worked because this is the obvious way of doing it. Why would you fork a process, then try to determine which of the two processes you are, then fix whatever the parent process messed up in your global state, and only then execute what you actually wanted to do? That seems insane (I guess until you realize that the main use case is creating /bin/sh).
- Ericson2314 5y ago
  Me too! But even when writing /bin/sh, I don't see why this would get in the way? I was once told earlier Unix didn't even have fork, but something more purpose-made for shells instead.
zokier 5y ago
Sounds a bit like fuchsias launchpad library where you create launchpad object, do all the setup, and then call launchpad_go to actually start the process. Launchpad doesn't allow arbitrary syscalls in the setup, so in that sense it is maybe closer to "spawn" interface but with better ergonomics https://cs.opensource.google/fuchsia/fuchsia/+/main:zircon/system/ulib/launchpad/include/launchpad/launchpad.h https://cs.opensource.google/fuchsia/fuchsia/+/main:zircon/s...

harry8 5y ago

Fork with cow is inefficient. Compared to what? In what dimension? Any numbers on that? Where is the trade-off? To what extent does anyone need to care and on what circumstances?

xroche 5y ago
It's inherently inefficient because while the child process does its initialization (pre-exec) stuff, the parent gets page faults for every thread writing into the memory due to COW. This will basically stall the parent and can cause funny issues.
cryptonector 5y ago
> Fork with cow is inefficient. > Compared to what? vfork() > Any numbers on that? I added links to the gist, some of which discuss performance in detail. E.g., https://blog.famzah.net/tag/fork-vfork-popen-clone-performance/ https://blog.famzah.net/tag/fork-vfork-popen-clone-performan... and https://bugzilla.redhat.com/show_bug.cgi?id=682922 https://bugzilla.redhat.com/show_bug.cgi?id=682922 But you can just reason about this: - vfork() is O(1) - copying fork() is O(N) where N is the amount of writable memory in the parent's address space - copy-on-write fork() is O(N) where N is the resident set size (RSS) of the parent O(1) beats O(N). And O(N) is just the complexity of fork() for a single-threaded parent process. Now imagine a very busy, threaded, large-RSS process that forks a lot. You get threads and child processes stepping all over each other's CoW mappings, causing lots of page faults and copies. Ok, that is still O(N), but users will feel the added pain of all those page faults and TLB shootdowns.
- harry8 5y ago
  Ok but you're just repeating "It's inefficient" and not saying in any way for what use is its inefficiency even noticeable. I want to reason about when I would care. You see? The first link didn't even have units on its numbers(!) I assume they're milliseconds. When does that scale become something one would care about at all? Not launching a gui process. Not a shell pipeline. So when is this issue arising at all? What is being done that makes fork inefficiency anything other than academic interest. Must be something, right? Forking webserver?

quietbritishjim 5y ago

Apologies if this is a silly question, but it seems like there's a false dichotomy here: (1) You have separate fork() (etc.) and exec(), so that in the brief window in between you can set all the properties of the new process using APIs that exist anyway for controlling your own process. (2) You have a single call to spawn a new process, but you have a million different options to control every aspect of the new process. Why not do it this other way instead? Perhaps a bit late now but seems like in retrospect it would give the API simplicity of fork+exec without any of the complications. (3) There are two steps to run a new process. The first fully sets up its memory and returns a PID, but doesn't start running it. The second call, unfreeze(), allows it to begin executing code. All the usual APIs that exist anyway for controlling your own process take an extra parameter specifying the PID of a frozen child (or -1 for the current process).

cryptonector 5y ago

Some links I found today researching this: - @famzah'z blog about fork vs vfork vs clone performance: https://blog.famzah.net/tag/fork-vfork-popen-clone-performance/ - A very similar idea to my afork() idea, from 2 years earlier: https://developers.redhat.com/blog/2015/08/19/launching-helper-process-under-memory-and-latency-constraints-pthread_create-and-vfork - misc https://inbox.vuxu.org/tuhs/CAEoi9W6HFL3UcnWkKoqka8Dt16MWskKd6yEJr3HYCcCT9pMTig@mail.gmail.com/T/ https://bugzilla.redhat.com/show_bug.cgi?id=682922 (see attachments)

switch33 5y ago

The problem is clone is more of a start phase after vfork but before fork regardless for github. So it's kind of a bit strange that we call vfork first but that is about templates too. As for templates they need to be in different languages and in different formats for video games consoles, and so many other formats they port systems and games that sort of work digitally to certain things but not playable to certain things too. The other problem is that clone is part of syscall interfaces and part of apis and part of a lot of other things too.

deleted 5y ago
[deleted]