3 ms·
It's far more common to have Python programs be I/O bound, and when they are CPU bound it's often not due to GIL contention (remember that C extensions can drop
by acdha 7y ago
It's far more common to have Python programs be I/O bound, and when they are CPU bound it's often not due to GIL contention (remember that C extensions can drop the GIL before doing something lengthy). It would be nice if the GIL was gone but a fair fraction of Python developers would not notice much difference.
- pdonis 7y ago> It's far more common to have Python programs be I/O bound But part of the reason for that is that Python programmers know there's no point in trying to run multiple CPU-bound tasks in the same Python process, so they don't try. > C extensions can drop the GIL before doing something lengthy Yes, but they're still limited in what they can do--as soon as they call back into a Python bytecode they're GIL-bound again. And if you depend on C extensions any time you need CPU bound concurrent tasks, you're giving up a lot of the advantages of using Python in the first place.
- acdha 7y agoIt’s part of the reason but not all or, I suspect, even most: the number of things which people need to compute with multi threading but without much I/O is not an especially large fraction of what people use pure Python for. If you need raw CPU speed for arbitrary code, it’s not the language most people would pick. The exceptions also tend to have existing high-quality extensions (crypto, compression, image processing, etc.) so while it’s technically true that you’re giving up Python most people aren’t doing that personally - they’re just calling Pillow or numpy - or they’re using it for a tiny fraction of the total program. Frequently this ends up being the same speed or even faster than using other languages because most people are either using the same C libraries or learning just how many optimizations their simple implementation lacked. Again, it’s not hard to come up with things where the GIL is inarguably a bottleneck but it comes up a lot more in debates than real-life in my experience.
- andreareina 7y agoI'm one of those with embarrassingly-parallel cpu-bound workloads. The multiprocessing module works, but the extra bookkeeping and plumbing over an actually-parallel-multithreading implementation is a pain in the butt. That said, the multithreading speedup is both faster and easier than porting to another language.
- acdha 7y agoDefinitely - I used to support a computational lab and have worked on enough other CPU-bound problems to have plenty of things which I wouldn’t recommend Python for. I just think that as a field we’re predisposed to focus on that as the kind of work Real Programmers™️ do when most groups are limited by their ability to implement and maintain business logic long before they hit the wall on what Python can do.
- nateberkopec 7y agoDoes I/O release the GIL in Python?
- pansa2 7y agoYes, generally.