3 ms·

Solution: don't use copy-on-write?

viraptor 5y ago
That would kill performance of many machines. Some wouldn't even notice (for example hosts dedicated to a Go webserver), some would get absolutely trashed (PHP pre-fork would likely be an extreme case). Additionally your baseline memory usage for each application is a full in-memory copy of every linked library. Say hello to 10+MB hello-world. (edit: actually just 3MB on my machine) Also all stacks would have to get reserved and cleared ahead of time, so thread spawning time grows ~5x. Basically unless you want to run single-process unikernels, our systems and software really isn't designed or ready for non-cow world.
- jcranmer 5y ago
  It's not that bad. The real problem is fork(), which is just a bad syscall in general (note how the single most common desktop OS doesn't have any equivalent to fork and seems to get along just fine). You can still allow mmap in a CoW-less world (you'll have to prevent concurrent file modification, which--again--a certain common operating system does!). Memory reservation is completely orthogonal to CoW: lazy page table initialization isn't the same thing as copy-on-write.
  - j16sdiz 5y ago
    Windows have PAGE_WRITECOPY and PAGE_EXECUTE_WRITECOPY. Those are essential for efficient shared library loading.
  - jabl 5y ago
    This paper argues that fork+exec is the wrong primitive for launching new processes: https://www.microsoft.com/en-us/research/publication/a-fork-in-the-road/ https://www.microsoft.com/en-us/research/publication/a-fork-... posix_spawn can take care of many common cases today.
    - zxzax 5y ago
      Unfortunately, on Linux there is no posix_spawn syscall. It's implemented in glibc as a wrapper around clone + exec.
      - geofft 5y ago
        Yes, but it's a wrapper around clone(CLONE_VFORK | CLONE_VM) (aka vfork) + exec, which makes a big difference. CLONE_VFORK means that the parent process is suspended until the child either exits or execs, and CLONE_VM means that no copy-on-write happens - the child has full read/write access to the original memory space until it execs (or exits). In the child, posix_spawn is careful to work within temporary memory so it doesn't overwrite anything that the parent process might care about, including global variables etc. Then once exec succeeds (or fails, and the process exits), posix_spawn in the parent cleans up that temporary memory. This means that posix_spawn doesn't require any copy-on-write support from the kernel. Or in other words, there's no posix_spawn syscall and there shouldn't be because it's a library function, but there is a vfork syscall.
        zxzax 5y ago
        I generally agree with the comments in the research paper about clone + vfork, most applications seem to not use it at all (either directly or indirectly through posix_spawn) for those reasons.
- eyberg 5y ago
  You are correct that there is an awful lot of software that came out of the 90s that was explicitly written for forking/multiple processes (we're talking 25-30 years now) but that only reflects the type of commodity computers we had then - pre SMP, pre commercialized virtualization, pre-cloud, etc. Threads back then weren't in a useable state either. Fast forward to today and we're finally starting to see the tide turn. I can go spin up a 384 thread instance on gcp right now. We have very popular languages like go and rust. In the case of Go they've made multi-threading easily accessible to many developers to the point that many people don't even think about it. Yes, there's a lot of cruft out there but we can build for the future.
  - toast0 5y ago
    If you don't actually need shared anything but config (memory / fds / task pooling), pre-fork is still better than threading. 384 threads sitting on one fd table is going to bottleneck on open (and maybe close) and anything else that is a lock per process.
    - eyberg 5y ago
      I'd disagree here. You can create a thread pool that matches your pre-fork env as well and it will still be vastly more performant especially in those environments that require more memory - which is many applications today.
rurban 5y ago
Always look at the hammer first. The solution is not use mmunmap() of course. But there are still security cases left

That would kill performance of many machines. Some wouldn't even notice (for example hosts dedicated to a Go webserver), some would get absolutely trashed (PHP pre-fork would likely be an extreme case). Additionally your baseline memory usage for each application is a full in-memory copy of every linked library. Say hello to 10+MB hello-world. (edit: actually just 3MB on my machine) Also all stacks would have to get reserved and cleared ahead of time, so thread spawning time grows ~5x. Basically unless you want to run single-process unikernels, our systems and software really isn't designed or ready for non-cow world.

jcranmer 5y ago
It's not that bad. The real problem is fork(), which is just a bad syscall in general (note how the single most common desktop OS doesn't have any equivalent to fork and seems to get along just fine). You can still allow mmap in a CoW-less world (you'll have to prevent concurrent file modification, which--again--a certain common operating system does!). Memory reservation is completely orthogonal to CoW: lazy page table initialization isn't the same thing as copy-on-write.
- j16sdiz 5y ago
  Windows have PAGE_WRITECOPY and PAGE_EXECUTE_WRITECOPY. Those are essential for efficient shared library loading.
- jabl 5y ago
  This paper argues that fork+exec is the wrong primitive for launching new processes: https://www.microsoft.com/en-us/research/publication/a-fork-in-the-road/ https://www.microsoft.com/en-us/research/publication/a-fork-... posix_spawn can take care of many common cases today.
  - zxzax 5y ago
    Unfortunately, on Linux there is no posix_spawn syscall. It's implemented in glibc as a wrapper around clone + exec.
    - geofft 5y ago
      Yes, but it's a wrapper around clone(CLONE_VFORK | CLONE_VM) (aka vfork) + exec, which makes a big difference. CLONE_VFORK means that the parent process is suspended until the child either exits or execs, and CLONE_VM means that no copy-on-write happens - the child has full read/write access to the original memory space until it execs (or exits). In the child, posix_spawn is careful to work within temporary memory so it doesn't overwrite anything that the parent process might care about, including global variables etc. Then once exec succeeds (or fails, and the process exits), posix_spawn in the parent cleans up that temporary memory. This means that posix_spawn doesn't require any copy-on-write support from the kernel. Or in other words, there's no posix_spawn syscall and there shouldn't be because it's a library function, but there is a vfork syscall.
      - zxzax 5y ago
        I generally agree with the comments in the research paper about clone + vfork, most applications seem to not use it at all (either directly or indirectly through posix_spawn) for those reasons.
eyberg 5y ago
You are correct that there is an awful lot of software that came out of the 90s that was explicitly written for forking/multiple processes (we're talking 25-30 years now) but that only reflects the type of commodity computers we had then - pre SMP, pre commercialized virtualization, pre-cloud, etc. Threads back then weren't in a useable state either. Fast forward to today and we're finally starting to see the tide turn. I can go spin up a 384 thread instance on gcp right now. We have very popular languages like go and rust. In the case of Go they've made multi-threading easily accessible to many developers to the point that many people don't even think about it. Yes, there's a lot of cruft out there but we can build for the future.
- toast0 5y ago
  If you don't actually need shared anything but config (memory / fds / task pooling), pre-fork is still better than threading. 384 threads sitting on one fd table is going to bottleneck on open (and maybe close) and anything else that is a lock per process.
  - eyberg 5y ago
    I'd disagree here. You can create a thread pool that matches your pre-fork env as well and it will still be vastly more performant especially in those environments that require more memory - which is many applications today.

rurban 5y ago

Always look at the hammer first. The solution is not use mmunmap() of course. But there are still security cases left