10 ms·

It's important to remember the biggest reason why overcommit exists on Linux and macOS: fork(). When a process forks, the vast majority of the child process's m

It's important to remember the biggest reason why overcommit exists on Linux and macOS: fork(). When a process forks, the vast majority of the child process's memory is safely shared with the parent process, due to copy-on-write. But a strict accounting would say that the total memory usage of the system has doubled, which is too conservative in most cases. Since forking is so common on Unix, overcommit ends up being necessary pretty quickly. Otherwise, fork/exec would stop working in any process using more than half of the system memory.

secondcoming 5y ago
Even the linux docs for it suggest to turn it on if you're working with sparse arrays, there's no mention of fork() https://www.kernel.org/doc/Documentation/vm/overcommit-accounting https://www.kernel.org/doc/Documentation/vm/overcommit-accou...
convolvatron 5y ago
wouldn't you just account the COW pages against the parent until they are copied? kicking the can down the road means there isn't any longer a reasonable correction (failing the allocation), but instead we get to drive around randomly trying to find something to kill. this is particularly annoying if you are running a service. there is no hope for it to recover - for example by flushing a cache. instead the OS looks around - sees this fat process just sitting there, and .. good news, we have plenty of memory now.
- pjc50 5y ago
  But what happens when the kernel needs to copy one and has run out of memory? You still get a random process killed. (I note that Windows has a different approach, with "reserve" vs "commit", but nobody regards that as a preferential reason for using Windows as a server OS)
  - magicalhippo 5y ago
    > But what happens when the kernel needs to copy one and has run out of memory? Don't allow it. Fork the process with read only pages except for the ranges passed to fork(). Count read-write pages as used memory by the child process. If the forked process wants to write to a page that's read-only it'll have to do a system call to turn it read-write. That call can then fail if there's not enough free memory to copy the pages. Of course, kinda hard to fix this now...
    - enedil 5y ago
      That also means that if you used seccomp to restrict mprotect syscall, the child process may never ever write again, which is an unfortunate design choice.
      - magicalhippo 5y ago
        Seccomp can check for the PROT_EXEC flag though, and let the others pass, no?
    - toast0 5y ago
      You'd presumably need the parent to also do a system call to mark pages read-write after forking, and that would need to include the stack pages, and that sounds like lots of fun; especially since sometimes library functions fork (although most of that is to fork/exec and more specific apis for that exist now). Not that it isn't loads of fun anyway, but woe betide thee who forks and threads and makes the other threads have a read-only stack. Probably you'd get people just marking as read-write and returning in sigsegv handlers, which I'm sure has great security properties... OTOH, at least there's an opportunity to deny the remap in the handler and get a decent crashdump from the program or a sliver of hope for managing the situation.
    - tsimionescu 5y ago
      > Don't allow it. Fork the process with read only pages except for the ranges passed to fork(). This doesn't work because of the horrible fork/exec design. If I am a huge process and I want to run `ls`, I will first have to clone myself using fork(), which may trigger an OOM, even if the first action of my clone would have been exec(), ignoring all of that memory. I am always surprised that no one has added a sane 'spawn process' primitive to replace fork/exec. Especially since fork() without exec() only really works in single-threaded processes.
- secondcoming 5y ago
  Indeed. It's totally dumb that the OS is allowed lie to you when you've asked for some resources. And because of this behaviour people no longer check for success from malloc(), etc, because of laziness. It's a bad situation.
  - alexgartrell 5y ago
    Allocated-but-unavailable is a totally reasonable part of the memory hierarchy. Main Memory => zswap (compressed memory) => swap In this case, the pages may be logically allocated or not -- the assurance is that the data will be the value you expect it to be when it becomes resident. Should those pages be uninitialized, the "Swapped" state is really just "Remember that this thing was all zeros." We could do computing your way, but it'd be phenomenally more expensive. I know this because every thing we introduce to the hierarchy in practice makes computing phenomenally less expensive.
    - cturner 5y ago
      "We could do computing your way, but it'd be phenomenally more expensive" It must be viable - Windows prevents overcommit. But it has slow child-process-creation (edit: previously said "forking"), and this steers development towards native threads which is its own set of problems. I had never previously joined the dots on the point pcwalton makes at the top of this thread. It is a dramatic trade-off.
      - pjc50 5y ago
        Windows does not have fork(), apart from the hidden ZwCreateProcess; https://news.ycombinator.com/item?id=9653975 https://news.ycombinator.com/item?id=9653975 For ages cygwin had to emulate it manually by hand-copying the process.
      - throwawaylinux 5y ago
        Linux can prevent overcommit. And that's not the reason for much faster process creation than Windows as far as I know.
      - 5y ago
- trhway 5y ago
  >there is no hope for it to recover - for example by flushing a cache. instead the OS looks around - sees this fat process just sitting there, and .. good news, we have plenty of memory now. Overcommit was godsent in the times of expensive memory and when people used virtual memory on disk (so it will spill low use memory pages there instead of the kill). Of course these days with abundance of cheap memory and people not configuring virtual memory any more we get the situation you describe.
  - angry_octet 5y ago
    Virtual memory is always on... If you actually hit swap the system effectively deadlocks anyway because the swap daemon is too dumb by being too fair. Persistent memory might change this but it needs so much work.
    - trhway 5y ago
      it was a throwback down the memory lane and the years before my first Linux in 1995 i had been dealing with Windows where configuration of swap was "configuring virtual memory", so i automatically used that term. > If you actually hit swap the system effectively deadlocks anyway that depends. In many cases in the past the options would be either with swap and thus slow or pretty much not at all. And again these days there is so much memory that there is always a way to avoid the swapping. Though i've met funny situations in recent years like when a several terabyte sized database process would get killed by the OOMKiller on a machine with overcommit left on and no swap configured.
      - angry_octet 5y ago
        OOM kill exists because swap is not being useful. A process failure is something you can deal with, livelock isn't. If you have an algorithm that works out of core then you'll never hit swap because you maintain the working set below memory limits.
- pcwalton 5y ago
  > wouldn't you just account the COW pages against the parent until they are copied? That's what Linux does. But how do you return ENOMEM when the copy does happen and now the system is out of memory? Memory writes don't return error codes. The best you could do is send a signal, which is exactly what the OOM killer does.
  - deleted 5y ago
    [deleted]
  - lokedhs 5y ago
    That's why you ensure you have enough swap to cover the worst case scenario. You as swap not because you need to actually use it, but on order to be able to guarantee there is enough memory available if the worst case scenario happens. In normal circumstances, swap should never really be utilised.
    - vbezhenar 5y ago
      > In normal circumstances, swap should never really be utilised. I don't agree. There're plenty of dormant virtual memory pages which will never be used. Keeping them in RAM is wasting precious resources.
    - Hendrikto 5y ago
      > You as swap not because you need to actually use it […] In normal circumstances, swap should never really be utilised. Swap is actually there so anonymous pages can be evicted, and is often used long before there is memory contention, during normal operation. Not having swap means that only file-backed pages can be evicted. During memory contention, this can cause thrashing. During "normal" operation, it degrades performance. Swap only being used when memory runs out, as kind of "emergency RAM", is a very widespread misunderstanding.
wahern 5y ago
> Otherwise, fork/exec would stop working in any process using more than half of the system memory. Somehow Solaris manages just fine. And don't forget that swap memory exists. Ironically, using overcommit without swap is asking for trouble on Linux. Overcommit or no overcommit, the Linux VM and page buffer systems are designed with the expectation of swap.
- pjc50 5y ago
  How does Solaris handle this? Or Darwin?
  - wahern 5y ago
    Solaris has strict memory accounting; fork will fail if the system can't guarantee space for all non-shared anonymous memory. macOS has overcommit (all BSDs do to some extent, at least for fork), but it also automatically creates swap space so you rarely encounter issues one way or another in practice. fork and malloc can also fail in Linux even with overcommit enabled (rlimits, but also OOM killer racing with I/O page dirtying triggering best-effort timeout), so Linux buys you a little convenience at the cost of making it impossibly difficult to actually guarantee behavior when it matters most.
  - scottlamb 5y ago
    I don't use Solaris, but I see they have vfork [1] which should be unaffected. Likewise posix_spawn{,p}. (I think those are often implemented in userspace via vfork; unsure if there's a separate system call on Solaris.) [1] https://docs.oracle.com/cd/E26505_01/html/816-5167/vfork-2.html https://docs.oracle.com/cd/E26505_01/html/816-5167/vfork-2.h...
- joosters 5y ago
  Solaris does have this problem! If you have a huge program running, fork()img it can fail on Solaris, even if you only want to just exec a tiny program. The key way of avoiding this is to ensure you have lots and lots of swap space.
- _3u10 5y ago
dekhn 5y ago
The biggest reason overcommit exists is because it allows the system to operate more efficiently. The reality is most applications touch only some of the pages they allocate, and it's silly for the system to fail a malloc. Often times other expensive cleanup activities can be deferred (you don't really want to drop a handy directory entry cache just so an app can be sure it got physically backed memory for its request, 99.99% of the time). IIUC Linux was really the first OS to make overcommit so prominent. Most systems were a lot more conservative.
- joosters 5y ago
  In theory, a system without overcommit can run just as efficiently, IF you have reserved huge amounts of swap space. As long as swap is available, the OS can do all the same COW and efficiency tricks. It’s not the physically backed memory is the limiting factor, it’s RAM+swap
  - dekhn 5y ago
    no, the kernel maintains other allocated objects (dentry, inode caches) that it can't swap out. Under memory pressure, those get dropped before application pages. See https://unix.stackexchange.com/questions/17936/setting-proc-sys-vm-drop-caches-to-clear-cache https://unix.stackexchange.com/questions/17936/setting-proc-... and https://unix.stackexchange.com/questions/111893/how-long-do-dentries-stay-in-the-dcache https://unix.stackexchange.com/questions/111893/how-long-do-... I've found the linux memory system to be far too complicated to understand for quite some time, compared to what's documented in, for example, The Design and Implementation of The FreeBSD Operating System, for a more comprehensible system.
    - joosters 5y ago
      Those objects must exist with and without overcommit, I don’t understand why they must make one less efficient than the other. (I’m talking in general here - not Linux specifically)
      - 5y ago
jiveturkey 5y ago
Most processes that fork know that they are going to fork. Therefore they can pre-fork very early, so as to commit the memory when they only have a bare minimum allocated. Your typical daemon does this anyway. Some other type of process like an interpreter that can subshell out doesn't know how big the allocation is going to get, would have to pre-fork early on. In this way, you wouldn't "need" overcommit and the Linux horror of OOM. Well, perhaps you don't need it so badly. Programs that use sparse arrays without mmap() probably need overcommit or lots of swap.
shoo 5y ago
I first learned about this while trying to understand processes being killed by OOM in production. We had python 2.x batch jobs being executed by long-running python worker processes -- some of the arbitrary application code in some of the arbitrary batch jobs would occasionally want to execute some command line tool in a new process, and to create the new process under the hood python's subprocess library would fork/exec, and if the parent worker process had already accumulated a large virtual memory footprint, linux's approximate memory accounting heuristics would kick in during the attempted "fork", decide that we were obviously going to run out of physical memory, and kill the process. We didn't actually want to fork anything and share gigabytes of virtual memory with the child process, we wanted to spawn an almost entirely independent process to do something and report results, but that got implemented under the hood by fork. Spawning processes is one area where Windows is more elegant than linux: windows offers spawn. Apparently macos and solaris implement a posix_spawn that avoid the complications of fork/exec. linux offers posix_spawn, apparently which may may or may not call fork under the hood depending on which libc you're using. If libc implements posix_spawn by calling fork then you're back in the same mess with linux heuristic memory accounting and overcommit. E.g. old versions of glibc will fork when you posix_spawn, newer versions of glibc may vfork . musl apparently will always vfork. It looks like cpython's subprocess.Popen was patched in python 3.8 to detect some cases where posix_spawn can be used -- it reads as if it will only kick in on linux if it detects a sufficiently new version of glibc: https://github.com/python/cpython/blob/main/Lib/subprocess.py#L647-L695 https://github.com/python/cpython/blob/main/Lib/subprocess.p... edit: Python 3.10 now supports using vfork for linux inside subprocess: https://bugs.python.org/issue35823 https://bugs.python.org/issue35823 docker run --rm -it --entrypoint=/bin/sh python:3.9-alpine # apk add strace # strace python -c "import subprocess; subprocess.run(['ls', '-l'])" 2>&1 >/dev/null | grep fork fork() = 88 docker run --rm -it --entrypoint=/bin/sh python:3.10-alpine # apk add strace # strace python -c "import subprocess; subprocess.run(['ls', '-l'])" 2>&1 >/dev/null | grep fork vfork() = 15 edit 2: here's a similar tale from go, replacing use of fork in fork/exec: https://github.com/golang/go/issues/5838 https://github.com/golang/go/issues/5838 https://go-review.googlesource.com/c/go/+/37439/ https://go-review.googlesource.com/c/go/+/37439/ https://about.gitlab.com/blog/2018/01/23/how-a-fix-in-go-19-sped-up-our-gitaly-service-by-30x/ https://about.gitlab.com/blog/2018/01/23/how-a-fix-in-go-19-...

secondcoming 5y ago

Even the linux docs for it suggest to turn it on if you're working with sparse arrays, there's no mention of fork() https://www.kernel.org/doc/Documentation/vm/overcommit-accounting https://www.kernel.org/doc/Documentation/vm/overcommit-accou...

convolvatron 5y ago

wouldn't you just account the COW pages against the parent until they are copied? kicking the can down the road means there isn't any longer a reasonable correction (failing the allocation), but instead we get to drive around randomly trying to find something to kill. this is particularly annoying if you are running a service. there is no hope for it to recover - for example by flushing a cache. instead the OS looks around - sees this fat process just sitting there, and .. good news, we have plenty of memory now.

pjc50 5y ago
But what happens when the kernel needs to copy one and has run out of memory? You still get a random process killed. (I note that Windows has a different approach, with "reserve" vs "commit", but nobody regards that as a preferential reason for using Windows as a server OS)
- magicalhippo 5y ago
  > But what happens when the kernel needs to copy one and has run out of memory? Don't allow it. Fork the process with read only pages except for the ranges passed to fork(). Count read-write pages as used memory by the child process. If the forked process wants to write to a page that's read-only it'll have to do a system call to turn it read-write. That call can then fail if there's not enough free memory to copy the pages. Of course, kinda hard to fix this now...
  - enedil 5y ago
    That also means that if you used seccomp to restrict mprotect syscall, the child process may never ever write again, which is an unfortunate design choice.
    - magicalhippo 5y ago
      Seccomp can check for the PROT_EXEC flag though, and let the others pass, no?
  - toast0 5y ago
    You'd presumably need the parent to also do a system call to mark pages read-write after forking, and that would need to include the stack pages, and that sounds like lots of fun; especially since sometimes library functions fork (although most of that is to fork/exec and more specific apis for that exist now). Not that it isn't loads of fun anyway, but woe betide thee who forks and threads and makes the other threads have a read-only stack. Probably you'd get people just marking as read-write and returning in sigsegv handlers, which I'm sure has great security properties... OTOH, at least there's an opportunity to deny the remap in the handler and get a decent crashdump from the program or a sliver of hope for managing the situation.
  - tsimionescu 5y ago
    > Don't allow it. Fork the process with read only pages except for the ranges passed to fork(). This doesn't work because of the horrible fork/exec design. If I am a huge process and I want to run `ls`, I will first have to clone myself using fork(), which may trigger an OOM, even if the first action of my clone would have been exec(), ignoring all of that memory. I am always surprised that no one has added a sane 'spawn process' primitive to replace fork/exec. Especially since fork() without exec() only really works in single-threaded processes.
secondcoming 5y ago
Indeed. It's totally dumb that the OS is allowed lie to you when you've asked for some resources. And because of this behaviour people no longer check for success from malloc(), etc, because of laziness. It's a bad situation.
- alexgartrell 5y ago
  Allocated-but-unavailable is a totally reasonable part of the memory hierarchy. Main Memory => zswap (compressed memory) => swap In this case, the pages may be logically allocated or not -- the assurance is that the data will be the value you expect it to be when it becomes resident. Should those pages be uninitialized, the "Swapped" state is really just "Remember that this thing was all zeros." We could do computing your way, but it'd be phenomenally more expensive. I know this because every thing we introduce to the hierarchy in practice makes computing phenomenally less expensive.
  - cturner 5y ago
    "We could do computing your way, but it'd be phenomenally more expensive" It must be viable - Windows prevents overcommit. But it has slow child-process-creation (edit: previously said "forking"), and this steers development towards native threads which is its own set of problems. I had never previously joined the dots on the point pcwalton makes at the top of this thread. It is a dramatic trade-off.
    - pjc50 5y ago
      Windows does not have fork(), apart from the hidden ZwCreateProcess; https://news.ycombinator.com/item?id=9653975 https://news.ycombinator.com/item?id=9653975 For ages cygwin had to emulate it manually by hand-copying the process.
    - throwawaylinux 5y ago
      Linux can prevent overcommit. And that's not the reason for much faster process creation than Windows as far as I know.
    - 5y ago
trhway 5y ago
>there is no hope for it to recover - for example by flushing a cache. instead the OS looks around - sees this fat process just sitting there, and .. good news, we have plenty of memory now. Overcommit was godsent in the times of expensive memory and when people used virtual memory on disk (so it will spill low use memory pages there instead of the kill). Of course these days with abundance of cheap memory and people not configuring virtual memory any more we get the situation you describe.
- angry_octet 5y ago
  Virtual memory is always on... If you actually hit swap the system effectively deadlocks anyway because the swap daemon is too dumb by being too fair. Persistent memory might change this but it needs so much work.
  - trhway 5y ago
    it was a throwback down the memory lane and the years before my first Linux in 1995 i had been dealing with Windows where configuration of swap was "configuring virtual memory", so i automatically used that term. > If you actually hit swap the system effectively deadlocks anyway that depends. In many cases in the past the options would be either with swap and thus slow or pretty much not at all. And again these days there is so much memory that there is always a way to avoid the swapping. Though i've met funny situations in recent years like when a several terabyte sized database process would get killed by the OOMKiller on a machine with overcommit left on and no swap configured.
    - angry_octet 5y ago
      OOM kill exists because swap is not being useful. A process failure is something you can deal with, livelock isn't. If you have an algorithm that works out of core then you'll never hit swap because you maintain the working set below memory limits.
pcwalton 5y ago
> wouldn't you just account the COW pages against the parent until they are copied? That's what Linux does. But how do you return ENOMEM when the copy does happen and now the system is out of memory? Memory writes don't return error codes. The best you could do is send a signal, which is exactly what the OOM killer does.
- deleted 5y ago
  [deleted]
- lokedhs 5y ago
  That's why you ensure you have enough swap to cover the worst case scenario. You as swap not because you need to actually use it, but on order to be able to guarantee there is enough memory available if the worst case scenario happens. In normal circumstances, swap should never really be utilised.
  - vbezhenar 5y ago
    > In normal circumstances, swap should never really be utilised. I don't agree. There're plenty of dormant virtual memory pages which will never be used. Keeping them in RAM is wasting precious resources.
  - Hendrikto 5y ago
    > You as swap not because you need to actually use it […] In normal circumstances, swap should never really be utilised. Swap is actually there so anonymous pages can be evicted, and is often used long before there is memory contention, during normal operation. Not having swap means that only file-backed pages can be evicted. During memory contention, this can cause thrashing. During "normal" operation, it degrades performance. Swap only being used when memory runs out, as kind of "emergency RAM", is a very widespread misunderstanding.

wahern 5y ago

> Otherwise, fork/exec would stop working in any process using more than half of the system memory. Somehow Solaris manages just fine. And don't forget that swap memory exists. Ironically, using overcommit without swap is asking for trouble on Linux. Overcommit or no overcommit, the Linux VM and page buffer systems are designed with the expectation of swap.

pjc50 5y ago
How does Solaris handle this? Or Darwin?
- wahern 5y ago
  Solaris has strict memory accounting; fork will fail if the system can't guarantee space for all non-shared anonymous memory. macOS has overcommit (all BSDs do to some extent, at least for fork), but it also automatically creates swap space so you rarely encounter issues one way or another in practice. fork and malloc can also fail in Linux even with overcommit enabled (rlimits, but also OOM killer racing with I/O page dirtying triggering best-effort timeout), so Linux buys you a little convenience at the cost of making it impossibly difficult to actually guarantee behavior when it matters most.
- scottlamb 5y ago
  I don't use Solaris, but I see they have vfork [1] which should be unaffected. Likewise posix_spawn{,p}. (I think those are often implemented in userspace via vfork; unsure if there's a separate system call on Solaris.) [1] https://docs.oracle.com/cd/E26505_01/html/816-5167/vfork-2.html https://docs.oracle.com/cd/E26505_01/html/816-5167/vfork-2.h...
joosters 5y ago
Solaris does have this problem! If you have a huge program running, fork()img it can fail on Solaris, even if you only want to just exec a tiny program. The key way of avoiding this is to ensure you have lots and lots of swap space.
_3u10 5y ago

dekhn 5y ago

The biggest reason overcommit exists is because it allows the system to operate more efficiently. The reality is most applications touch only some of the pages they allocate, and it's silly for the system to fail a malloc. Often times other expensive cleanup activities can be deferred (you don't really want to drop a handy directory entry cache just so an app can be sure it got physically backed memory for its request, 99.99% of the time). IIUC Linux was really the first OS to make overcommit so prominent. Most systems were a lot more conservative.

joosters 5y ago
In theory, a system without overcommit can run just as efficiently, IF you have reserved huge amounts of swap space. As long as swap is available, the OS can do all the same COW and efficiency tricks. It’s not the physically backed memory is the limiting factor, it’s RAM+swap
- dekhn 5y ago
  no, the kernel maintains other allocated objects (dentry, inode caches) that it can't swap out. Under memory pressure, those get dropped before application pages. See https://unix.stackexchange.com/questions/17936/setting-proc-sys-vm-drop-caches-to-clear-cache https://unix.stackexchange.com/questions/17936/setting-proc-... and https://unix.stackexchange.com/questions/111893/how-long-do-dentries-stay-in-the-dcache https://unix.stackexchange.com/questions/111893/how-long-do-... I've found the linux memory system to be far too complicated to understand for quite some time, compared to what's documented in, for example, The Design and Implementation of The FreeBSD Operating System, for a more comprehensible system.
  - joosters 5y ago
    Those objects must exist with and without overcommit, I don’t understand why they must make one less efficient than the other. (I’m talking in general here - not Linux specifically)
    - 5y ago

jiveturkey 5y ago

Most processes that fork know that they are going to fork. Therefore they can pre-fork very early, so as to commit the memory when they only have a bare minimum allocated. Your typical daemon does this anyway. Some other type of process like an interpreter that can subshell out doesn't know how big the allocation is going to get, would have to pre-fork early on. In this way, you wouldn't "need" overcommit and the Linux horror of OOM. Well, perhaps you don't need it so badly. Programs that use sparse arrays without mmap() probably need overcommit or lots of swap.

shoo 5y ago

I first learned about this while trying to understand processes being killed by OOM in production. We had python 2.x batch jobs being executed by long-running python worker processes -- some of the arbitrary application code in some of the arbitrary batch jobs would occasionally want to execute some command line tool in a new process, and to create the new process under the hood python's subprocess library would fork/exec, and if the parent worker process had already accumulated a large virtual memory footprint, linux's approximate memory accounting heuristics would kick in during the attempted "fork", decide that we were obviously going to run out of physical memory, and kill the process. We didn't actually want to fork anything and share gigabytes of virtual memory with the child process, we wanted to spawn an almost entirely independent process to do something and report results, but that got implemented under the hood by fork. Spawning processes is one area where Windows is more elegant than linux: windows offers spawn. Apparently macos and solaris implement a posix_spawn that avoid the complications of fork/exec. linux offers posix_spawn, apparently which may may or may not call fork under the hood depending on which libc you're using. If libc implements posix_spawn by calling fork then you're back in the same mess with linux heuristic memory accounting and overcommit. E.g. old versions of glibc will fork when you posix_spawn, newer versions of glibc may vfork . musl apparently will always vfork. It looks like cpython's subprocess.Popen was patched in python 3.8 to detect some cases where posix_spawn can be used -- it reads as if it will only kick in on linux if it detects a sufficiently new version of glibc: https://github.com/python/cpython/blob/main/Lib/subprocess.py#L647-L695 https://github.com/python/cpython/blob/main/Lib/subprocess.p... edit: Python 3.10 now supports using vfork for linux inside subprocess: https://bugs.python.org/issue35823 https://bugs.python.org/issue35823 docker run --rm -it --entrypoint=/bin/sh python:3.9-alpine # apk add strace # strace python -c "import subprocess; subprocess.run(['ls', '-l'])" 2>&1 >/dev/null | grep fork fork() = 88 docker run --rm -it --entrypoint=/bin/sh python:3.10-alpine # apk add strace # strace python -c "import subprocess; subprocess.run(['ls', '-l'])" 2>&1 >/dev/null | grep fork vfork() = 15 edit 2: here's a similar tale from go, replacing use of fork in fork/exec: https://github.com/golang/go/issues/5838 https://github.com/golang/go/issues/5838 https://go-review.googlesource.com/c/go/+/37439/ https://go-review.googlesource.com/c/go/+/37439/ https://about.gitlab.com/blog/2018/01/23/how-a-fix-in-go-19-sped-up-our-gitaly-service-by-30x/ https://about.gitlab.com/blog/2018/01/23/how-a-fix-in-go-19-...