3 ms·
malloc has to guard its locks against fork, probably using pthread_atfork, or some lower level internal API related to that. The problem with pthread_atfork is
by kazinator 5y ago
malloc has to guard its locks against fork, probably using pthread_atfork, or some lower level internal API related to that.
The problem with pthread_atfork is third party libs.
YOU will use it in YOUR code. The C library will correctly use it in its code. But you have no assurance that any other libraries are doing the right things with their locks.
- dgrunwald 5y agoYour "third party libs" includes system libraries like libdl. We had a Python process using both threads (for stuff like background downloads, where the GIL doesn't hurt) and multiprocessing (for CPU-intensive work), and found that on Linux, the child process sometimes deadlocks in libdl (which Python uses to import extension modules). The fix was to use `multiprocessing.set_start_method('spawn')` so that Python doesn't use fork().
- kazinator 5y agolibdl is a component of glibc; that needs to be debugged.
- kllrnohj 5y agoThe more stuff that piles on using pthread_atfork then also contribute to fork() being unnecessarily slow for the specific combination of fork+exec.
- kazinator 5y agoRight, and so POSIX "fixed" that by standardizing posix_spawn. Thus fork is now mainly for those scenarios in which exec is not called, plus traditional coding that is portable to old systems.
- mark_undoio 5y agoAlso if, for any reason, you end up doing a `fork()` syscall directly rather than via libc you'll still have a problem as appropriate cleanup won't happen. Of course, the best answer to that is usually going to be "don't do that"!
- cryptonector 5y ago> But you have no assurance that any other libraries are doing the right things with their locks. I mean, if they're broken, fix them or get upstream to fix them.