3 ms·

Is basic fork/exec from a large process still slow or have newer apis fixed that?

pengaru 4y ago
vfork() / CLONE_VFORK / posix_spawn()
anfilt 4y ago
On linux fork is pretty fast since it's copy on write. Exec or anything similar is gonna be slower. For example you want to exec some program in the elf format. Well exec is likely to lead these things: One is likely got to load the file from non-volatile storage. Then parse the elf file and place each section in the correct place in memory, setup other memory maps, and stack ect.. Not just that though then run time linker probably has to run loading and linking each .so file, also patching code with the correct relocations as needed. Loading a new process is a lot more expensive where as fork can reuse a lot of that work since it's just a clone of the process at the point fork was called and was already done.
- evilotto 4y ago
  Unless something has changed in the past decade, fork on linux is really slow when you're forking a large multi-GB process because even though the memory is COW all the kernel page tables still need to be copied. This is one reason why large servers will have a separate "launcher" process that is forked early on rather than forking themselves. (the other reason of course is threads).
  - anfilt 4y ago
    Well that is needed to support isolated address spaces. It might be possible to share some the of the page tables safely. Still copying page tables is a lot faster than copying many gigs of data around, but for most that don't have massive page tables this is pretty fast. You can also use things like MADV_HUGEPAGE to make the page tables smaller. Now if this is a problem and the application does not need address space isolation. clone() with the CLONE_VM flag will get rid of this problem, and clone() is nice in the fact you can get more granularity/control. This control basically ranging between most of the continuum between thread and separate process depending on your needs.
- pjmlp 4y ago
  The problem with UNIX, is the same as Web browsers, in theory POSIX should be the same everywhere, in practice not really. So whatever Linux does better, or any other UNIX for that matter, might be a pool of surprises somewhere else.

vfork() / CLONE_VFORK / posix_spawn()

On linux fork is pretty fast since it's copy on write. Exec or anything similar is gonna be slower. For example you want to exec some program in the elf format. Well exec is likely to lead these things: One is likely got to load the file from non-volatile storage. Then parse the elf file and place each section in the correct place in memory, setup other memory maps, and stack ect.. Not just that though then run time linker probably has to run loading and linking each .so file, also patching code with the correct relocations as needed. Loading a new process is a lot more expensive where as fork can reuse a lot of that work since it's just a clone of the process at the point fork was called and was already done.

evilotto 4y ago
Unless something has changed in the past decade, fork on linux is really slow when you're forking a large multi-GB process because even though the memory is COW all the kernel page tables still need to be copied. This is one reason why large servers will have a separate "launcher" process that is forked early on rather than forking themselves. (the other reason of course is threads).
- anfilt 4y ago
  Well that is needed to support isolated address spaces. It might be possible to share some the of the page tables safely. Still copying page tables is a lot faster than copying many gigs of data around, but for most that don't have massive page tables this is pretty fast. You can also use things like MADV_HUGEPAGE to make the page tables smaller. Now if this is a problem and the application does not need address space isolation. clone() with the CLONE_VM flag will get rid of this problem, and clone() is nice in the fact you can get more granularity/control. This control basically ranging between most of the continuum between thread and separate process depending on your needs.
pjmlp 4y ago
The problem with UNIX, is the same as Web browsers, in theory POSIX should be the same everywhere, in practice not really. So whatever Linux does better, or any other UNIX for that matter, might be a pool of surprises somewhere else.