8 ms·

Many people don't know about Python's GIL https://wiki.python.org/moin/GlobalInterpreterLock https://wiki.python.org/moin/GlobalInterpreterLock That's the reas

Many people don't know about Python's GIL https://wiki.python.org/moin/GlobalInterpreterLock https://wiki.python.org/moin/GlobalInterpreterLock That's the reason why you need to go multi-process if you want to reach a similar level of concurrency in Python as multi-thread in C++. And that surely adds a lot of complexity. As a very practical example of this, TensorFlow has a dedicated page with advice on how to make the Python part that reads the files from disk less slow. Think about that: The bottleneck for training a highly advanced AI with millions of parameters is in the 20 lines of Python code to read in a binary file... https://www.tensorflow.org/guide/data_performance https://www.tensorflow.org/guide/data_performance

lewis1028282 7y ago
Not really IO is always going to be slow compared to the CPU. The Python interpreter will be blocked 90% of that time waiting for IO anyway.
- SaxonRobber 7y ago
  Percentages are misleading, what matters is the actual time it takes. Even small gains will have an effect as you increase the traffic and add additional stages to the system. Small delays accumulate and can have a surprising effect on queue sizes and latency.
- postpawl 7y ago
  Exactly, and I’m surprised none of the other comments mentioned this so far. Often web app endpoints are bottlenecked by IO because they’re spending most of their time talking to a database or cache server. Python is probably not the right tool for a CPU intensive endpoint that needs to serve up hundreds of thousands of requests per minute and can’t be cached. There are a lot of ways to handle the IO intensive scenario in python: * Threading - Works with python libraries written in C, but now you need to add locks to your code to prevent race conditions. Not good for CPU heavy work because it’s switching context every ~100 instructions. * Gevent - Requires minimal code changes, but it usually blocks when running code from python libraries written in C. This uses event loop style concurrency and automatically patches internal python libraries to switch context when IO occurs. This means less time spent unnecessarily switching context, so it can scale better than python’s threading. * multiprocessing - Better for CPU intensive work, but requires more memory than other solutions. You don’t need to worry about race conditions since the processes are separate. * asyncio - Requires code changes and using compatible libraries. It does event loop style concurrency that allows you to specify specifically when to switch context. Like with gevent, this means less time spent unnecessarily switching context. * ...And there are probably a bunch of other ways I’m missing. I’ve heard that this sort of stuff is simpler in Go.
  - ummonk 7y ago
    From what I’ve seen in production environments, Python has an uncanny ability to take tasks that should be IO- / server- bound and make them CPU-bound.
    - postpawl 7y ago
      For sure, but often that’s a sign that you’re doing it wrong. Maybe those aggregations should be done in the database, maybe it should be cached, maybe you should do that in bulk with one request, maybe it should run in the background anyway, etc.
      - celticmusic 7y ago
        or maybe you should consider using C++, C#, Java, et al instead?
      - pnako 7y ago
        i.e. "maybe do it anywhere else than in Python"
  - brobdingnagians 7y ago
    Sometimes the language is the reason it is IO bound. A good example of this and discussion is in the link below. The heap of abstractions can cause it to be IO bound, or the CPU usage can bound the IO. https://erikengbrecht.blogspot.com/2008/06/what-does-io-bound-really-mean.html https://erikengbrecht.blogspot.com/2008/06/what-does-io-boun...
celticmusic 7y ago
Years ago I had someone on reddit arguing with me about whether or not the GIL existed and affected Matz Ruby (this was 10+ years ago). This person was actively arguing with me about what it was. It's one of many battle stories for why I pretty much just assume everyone on reddit is stupid unless shown otherwise.
avip 7y ago
Many other people "know" about the GIL, to the extent of believing there's no point using threads in python "because of the GIL". I had a funny such experience lately in a job interview. I told the interviewer his misconception could be falsified with ~10 LOC summing a list with 2 threads.
- muxator 7y ago
  Could you expand on this please?
  - fooker 7y ago
    You could just call raw pthread functions, or a library which does that.
    - rowanG077 7y ago
      But then you are not writing portable python anymore.
- mlthoughts2018 7y ago
  How did you ensure those threads would actually execute concurrently? If any operations were GIL-bound (which is extremely common, even if not intentional, by reliance on bytecode instructions dealing with CPython API under the hood, like attribute lookups or iteration special methods), then execution is probably interleaved serially and constrained by the interpreter’s GIL allocation, and slower than just summing serially. I’ve seen a lot of people who are cocksure they know some contrarian “actually you can use threads” trivia about Python and just naively use the threading module or naively use the newish ThreadPoolExecutor stuff not realizing that no, in fact, it’s not somehow magically always GIL-avoiding to do so.
  - rdtsc 7y ago
    > How did you ensure those threads would actually execute concurrently? By having multiple concurrent IO requests from multiple threads? Can probably log detailed timestamps to see if IO requests were happening in parallel.
mlthoughts2018 7y ago
I think you misunderstand the training of many types of large statistical models. It’s almost always I/O bound (getting batches of data into memory or shipped to GPU) and the CPU bound part is intensely optimized SIMD matrix algebra operations. The hard part is always that you have more data than can fit into memory, and need complex prefetch solutions to parallelize getting the next batch of data while one batch’s numerical linear algebra computation is in progress. Many libraries handle this transparently for the user in an extension module, such as Keras & PyTorch. I think it’s misleading to act incredulous that this is the bottleneck ... it doesn’t have anything to do with Python, GIL, etc. (and there are easy-to-maintain solutions for this in Python).
est 7y ago
Many people don't know about every game I played written in C++ can utilize only one CPU core.

lewis1028282 7y ago

Not really IO is always going to be slow compared to the CPU. The Python interpreter will be blocked 90% of that time waiting for IO anyway.

SaxonRobber 7y ago
Percentages are misleading, what matters is the actual time it takes. Even small gains will have an effect as you increase the traffic and add additional stages to the system. Small delays accumulate and can have a surprising effect on queue sizes and latency.
postpawl 7y ago
Exactly, and I’m surprised none of the other comments mentioned this so far. Often web app endpoints are bottlenecked by IO because they’re spending most of their time talking to a database or cache server. Python is probably not the right tool for a CPU intensive endpoint that needs to serve up hundreds of thousands of requests per minute and can’t be cached. There are a lot of ways to handle the IO intensive scenario in python: * Threading - Works with python libraries written in C, but now you need to add locks to your code to prevent race conditions. Not good for CPU heavy work because it’s switching context every ~100 instructions. * Gevent - Requires minimal code changes, but it usually blocks when running code from python libraries written in C. This uses event loop style concurrency and automatically patches internal python libraries to switch context when IO occurs. This means less time spent unnecessarily switching context, so it can scale better than python’s threading. * multiprocessing - Better for CPU intensive work, but requires more memory than other solutions. You don’t need to worry about race conditions since the processes are separate. * asyncio - Requires code changes and using compatible libraries. It does event loop style concurrency that allows you to specify specifically when to switch context. Like with gevent, this means less time spent unnecessarily switching context. * ...And there are probably a bunch of other ways I’m missing. I’ve heard that this sort of stuff is simpler in Go.
- ummonk 7y ago
  From what I’ve seen in production environments, Python has an uncanny ability to take tasks that should be IO- / server- bound and make them CPU-bound.
  - postpawl 7y ago
    For sure, but often that’s a sign that you’re doing it wrong. Maybe those aggregations should be done in the database, maybe it should be cached, maybe you should do that in bulk with one request, maybe it should run in the background anyway, etc.
    - celticmusic 7y ago
      or maybe you should consider using C++, C#, Java, et al instead?
    - pnako 7y ago
      i.e. "maybe do it anywhere else than in Python"
- brobdingnagians 7y ago
  Sometimes the language is the reason it is IO bound. A good example of this and discussion is in the link below. The heap of abstractions can cause it to be IO bound, or the CPU usage can bound the IO. https://erikengbrecht.blogspot.com/2008/06/what-does-io-bound-really-mean.html https://erikengbrecht.blogspot.com/2008/06/what-does-io-boun...

celticmusic 7y ago

Years ago I had someone on reddit arguing with me about whether or not the GIL existed and affected Matz Ruby (this was 10+ years ago). This person was actively arguing with me about what it was. It's one of many battle stories for why I pretty much just assume everyone on reddit is stupid unless shown otherwise.

avip 7y ago

Many other people "know" about the GIL, to the extent of believing there's no point using threads in python "because of the GIL". I had a funny such experience lately in a job interview. I told the interviewer his misconception could be falsified with ~10 LOC summing a list with 2 threads.

muxator 7y ago
Could you expand on this please?
- fooker 7y ago
  You could just call raw pthread functions, or a library which does that.
  - rowanG077 7y ago
    But then you are not writing portable python anymore.
mlthoughts2018 7y ago
How did you ensure those threads would actually execute concurrently? If any operations were GIL-bound (which is extremely common, even if not intentional, by reliance on bytecode instructions dealing with CPython API under the hood, like attribute lookups or iteration special methods), then execution is probably interleaved serially and constrained by the interpreter’s GIL allocation, and slower than just summing serially. I’ve seen a lot of people who are cocksure they know some contrarian “actually you can use threads” trivia about Python and just naively use the threading module or naively use the newish ThreadPoolExecutor stuff not realizing that no, in fact, it’s not somehow magically always GIL-avoiding to do so.
- rdtsc 7y ago
  > How did you ensure those threads would actually execute concurrently? By having multiple concurrent IO requests from multiple threads? Can probably log detailed timestamps to see if IO requests were happening in parallel.

mlthoughts2018 7y ago

I think you misunderstand the training of many types of large statistical models. It’s almost always I/O bound (getting batches of data into memory or shipped to GPU) and the CPU bound part is intensely optimized SIMD matrix algebra operations. The hard part is always that you have more data than can fit into memory, and need complex prefetch solutions to parallelize getting the next batch of data while one batch’s numerical linear algebra computation is in progress. Many libraries handle this transparently for the user in an extension module, such as Keras & PyTorch. I think it’s misleading to act incredulous that this is the bottleneck ... it doesn’t have anything to do with Python, GIL, etc. (and there are easy-to-maintain solutions for this in Python).

est 7y ago

Many people don't know about every game I played written in C++ can utilize only one CPU core.