4 ms·

> doing that means the GIL doesn't get released so that thread will prevent any further network I/O while it's running, the same as would happen in an async fra

by Znafon 6y ago

> doing that means the GIL doesn't get released so that thread will prevent any further network I/O while it's running, the same as would happen in an async framework if a worker did a lot of CPU work. So Python doesn't really let you realize the advantages of threads in this context. I don't think that's true, the GIL get released for many computing intensive or IO bound tasks in Python, for example when reading from a socket the GIL gets released at https://github.com/python/cpython/blob/e822e37946f27c09953bb5733acf3b07c2db690f/Modules/socketmodule.c#L899 https://github.com/python/cpython/blob/e822e37946f27c09953bb...

pdonis 6y ago
> the GIL get released for many computing intensive or IO bound tasks in Python I/O bound, yes, since that requires system calls, and system calls (reading from a socket is an example of a system call) release the GIL. Computing intensive, no. Code that is doing a CPU intensive computation but makes no system calls will never release the GIL.
- jashmatthews 6y ago
  The GIL gets released whenever it gets released. C extensions like zlib release the GIL while (de)compressing requests/responses. https://docs.python.org/3/c-api/init.html#releasing-the-gil-from-extension-code https://docs.python.org/3/c-api/init.html#releasing-the-gil-...
  - pdonis 6y ago
    > C extensions like zlib release the GIL while (de)compressing requests/responses Yes, you're right, my statement was too broad.
- Znafon 6y ago
  > Computing intensive, no. Code that is doing a CPU intensive computation but makes no system calls will never release the GIL. Any code that does not involve Python objects can release the GIL, no matter whether it makes system call or not. For example, NumPy the most popular scientific computation package in Python, on which many other popular packages like Pandas are based, releases the GIL when doing operation on matrix. This is documented at https://numpy.org/doc/stable/reference/internals.code-explanations.html#function-call https://numpy.org/doc/stable/reference/internals.code-explan...: > If NPY_ALLOW_THREADS is defined during compilation, then as long as no object arrays are involved, the Python Global Interpreter Lock (GIL) is released prior to calling the loops. It is re-acquired if necessary to handle error conditions. To do so it uses the same macro used by the socket module when doing system calls: https://github.com/numpy/numpy/blob/18a6e3e505ee416ddfc617f3e9afdff5a031c2c2/numpy/core/include/numpy/ndarraytypes.h#L959 https://github.com/numpy/numpy/blob/18a6e3e505ee416ddfc617f3...
  - pdonis 6y ago
    > Any code that does not involve Python objects And does not involve running Python bytecode. Yes, numpy and other packages that provide C extensions do this when they are doing computations that don't require running Python bytecode. > no matter whether it makes system call or not Yes, you're right, my statement was too broad.
colinmhayes 6y ago
GIL gets released on blocking calls. CPU intensive is synonymous with non-blocking calls.

by Znafon 6y ago

pdonis 6y ago

> the GIL get released for many computing intensive or IO bound tasks in Python I/O bound, yes, since that requires system calls, and system calls (reading from a socket is an example of a system call) release the GIL. Computing intensive, no. Code that is doing a CPU intensive computation but makes no system calls will never release the GIL.

jashmatthews 6y ago
The GIL gets released whenever it gets released. C extensions like zlib release the GIL while (de)compressing requests/responses. https://docs.python.org/3/c-api/init.html#releasing-the-gil-from-extension-code https://docs.python.org/3/c-api/init.html#releasing-the-gil-...
- pdonis 6y ago
  > C extensions like zlib release the GIL while (de)compressing requests/responses Yes, you're right, my statement was too broad.
Znafon 6y ago
> Computing intensive, no. Code that is doing a CPU intensive computation but makes no system calls will never release the GIL. Any code that does not involve Python objects can release the GIL, no matter whether it makes system call or not. For example, NumPy the most popular scientific computation package in Python, on which many other popular packages like Pandas are based, releases the GIL when doing operation on matrix. This is documented at https://numpy.org/doc/stable/reference/internals.code-explanations.html#function-call https://numpy.org/doc/stable/reference/internals.code-explan...: > If NPY_ALLOW_THREADS is defined during compilation, then as long as no object arrays are involved, the Python Global Interpreter Lock (GIL) is released prior to calling the loops. It is re-acquired if necessary to handle error conditions. To do so it uses the same macro used by the socket module when doing system calls: https://github.com/numpy/numpy/blob/18a6e3e505ee416ddfc617f3e9afdff5a031c2c2/numpy/core/include/numpy/ndarraytypes.h#L959 https://github.com/numpy/numpy/blob/18a6e3e505ee416ddfc617f3...
- pdonis 6y ago
  > Any code that does not involve Python objects And does not involve running Python bytecode. Yes, numpy and other packages that provide C extensions do this when they are doing computations that don't require running Python bytecode. > no matter whether it makes system call or not Yes, you're right, my statement was too broad.

colinmhayes 6y ago

GIL gets released on blocking calls. CPU intensive is synonymous with non-blocking calls.