4 ms·

Less memory usage, and - hopefully - without all the quirks that crop up with multiprocessing. Off the top of my head: subprocesses don't always want to die alo

Less memory usage, and - hopefully - without all the quirks that crop up with multiprocessing. Off the top of my head: subprocesses don't always want to die along with the main process; error conditions can cause the underlying IPC layer to end up in a permanently stalled state.

frutiger 7y ago
I think you can avoid these kinds of issues if you use multiprocessing.Pool as a context manager.
loeg 7y ago
It avoids some quirks but also introduces new quirks multiprocessing doesn't have, like broken C modules (including parts of the core interpreter and stdlib) that have global state, rather than per-interpreter state. There's a huge ecosystem of Python libraries in the world and most have been able to more or less ignore the distinction between per-interpreter state and global state prior to this proposal. (Not true if you actually used the C API to embed many interpreters in a process, but most people don't do that.)
- btown 7y ago
  Is it possible to load an instance of a native library per interpreter? Give each one its own memory space?
  - pas 7y ago
    That requires making the native parts of the stdlib position independent (relocatable). I don't know what obstacles are there currently, but basically you'd need to have a runtime linker that gives a reference to the loaded object/module to access the symbols, but these are usually handled by the C compiler and linker. So it's doable, just a lot of effort.
    - loeg 7y ago
      Theoretically, if this proposal lands, all of the Python stdlib will have to be fixed not to have global state — the main concern would be 3rd party C libraries, I think.
  - loeg 7y ago
    I don't think you can do this with traditional dynamic linkers — the separate memory space is not so difficult (requires relocatable libraries, -fPIC, which is usually already enabled on ASLR systems) — but you would want to be able to load the same .so twice without symbol naming conflicts. I don't think most dynamic linkers (ld-linux / rtld-elf) support that. I could be mistaken, I am not very familiar with any implementation. There is nothing preventing you from adding this support to an existing dynamic linker and using it for your program, though.
    - nybble41 7y ago
      > but you would want to be able to load the same .so twice without symbol naming conflicts This is supported in ld-linux by using the dlmopen() glibc function and distinct namespaces. Loading the same .so file multiple times is one of the use cases explicitly mentioned in the manual page[1]. [1]: http://man7.org/linux/man-pages/man3/dlmopen.3.html#NOTES http://man7.org/linux/man-pages/man3/dlmopen.3.html#NOTES
      - loeg 7y ago
        Thanks, I wasn't aware of that! One caveat seems to be: > The glibc implementation supports a maximum of 16 namespaces.

frutiger 7y ago

I think you can avoid these kinds of issues if you use multiprocessing.Pool as a context manager.

loeg 7y ago

It avoids some quirks but also introduces new quirks multiprocessing doesn't have, like broken C modules (including parts of the core interpreter and stdlib) that have global state, rather than per-interpreter state. There's a huge ecosystem of Python libraries in the world and most have been able to more or less ignore the distinction between per-interpreter state and global state prior to this proposal. (Not true if you actually used the C API to embed many interpreters in a process, but most people don't do that.)

btown 7y ago
Is it possible to load an instance of a native library per interpreter? Give each one its own memory space?
- pas 7y ago
  That requires making the native parts of the stdlib position independent (relocatable). I don't know what obstacles are there currently, but basically you'd need to have a runtime linker that gives a reference to the loaded object/module to access the symbols, but these are usually handled by the C compiler and linker. So it's doable, just a lot of effort.
  - loeg 7y ago
    Theoretically, if this proposal lands, all of the Python stdlib will have to be fixed not to have global state — the main concern would be 3rd party C libraries, I think.
- loeg 7y ago
  I don't think you can do this with traditional dynamic linkers — the separate memory space is not so difficult (requires relocatable libraries, -fPIC, which is usually already enabled on ASLR systems) — but you would want to be able to load the same .so twice without symbol naming conflicts. I don't think most dynamic linkers (ld-linux / rtld-elf) support that. I could be mistaken, I am not very familiar with any implementation. There is nothing preventing you from adding this support to an existing dynamic linker and using it for your program, though.
  - nybble41 7y ago
    > but you would want to be able to load the same .so twice without symbol naming conflicts This is supported in ld-linux by using the dlmopen() glibc function and distinct namespaces. Loading the same .so file multiple times is one of the use cases explicitly mentioned in the manual page[1]. [1]: http://man7.org/linux/man-pages/man3/dlmopen.3.html#NOTES http://man7.org/linux/man-pages/man3/dlmopen.3.html#NOTES
    - loeg 7y ago
      Thanks, I wasn't aware of that! One caveat seems to be: > The glibc implementation supports a maximum of 16 namespaces.