5 ms·

Curious: At any point, is it explained why the Global Interpreter Lock is necessary? If so, I'll spend the time to watch.

m_mueller 10y ago
It's not necessary, it was a design choice that made sense back in the 90ies. In a multithreaded environment you can lock at a fine grained level or on a coarse grained level - or you can crash, but let's ignore that as an option. Python chose coarse grained, giving up parallel interpreter computations, but gaining a lot of thread sync overhead. All attempts so far to remove the GIL have resulted in a (usually much) slower interpreter, but the latest attempt shows some promise and it's thinkable (but not guaranteed) that in a few years there will be an official GIL-less cPython.
- syllogism 10y ago
  Removing Python's GIL will never make much sense. Not today and not in future. If you need CPU-fast code and would bother to multi-thread, it's much more worth it to write the code in Cython. If your code is CPU bound and you're using native Python, you're going to be making a tonne of heap allocations and pointer dereferences. This will be very slow. If you implement the relevant stuff in Cython, even without using multi-threading you'll likely see 10x performance improvement, and can often see up to 100x. Removing the GIL makes Python worse at the stuff it's good at, for questionable improvements in the areas Python is really terrible. This is not a good trade.
  - Skunkleton 10y ago
    What if you were trying to thread an non-processor bound task?
    - sp332 10y ago
      Do you mean waiting for I/O? It's already possible to do I/O asynchronously or with separate locks. The interpreter lock only applies to the interpreter.
- hermitdev 10y ago
  Do you happen to have any papers about the current efforts to remove the GIL? I love Python, and use it a lot for ETL type work, but if threading worked well, I could/would possibly use it for far more purposes.
  - brianwawok 10y ago
    Can you give an example where the GIL is really holding you back? Because with multiprocessing and greenlets, 99.99% of concurrency problems are trivilially solved by current Cython.
    - deleted 10y ago
      [deleted]
    - mrits 10y ago
      2 TB hash join
      - doubleunplussed 10y ago
        Isn't that up to the RDBMS whether than's multithreaded or not? Unless the RDBMS is implemented in Python, CPython doesn't force extension code to be single-threaded. Just Python bytecode.
        foo101 10y ago
        That's pretty much the point, isn't it? If I need true multithreading, then I am forced to write an extension in C or offload the multithreading work to another process (such as RDBMS in your case). It would have been nice if true multithreading was possible in Python itself. It would immediately make Python more useful in a variety of scenarios where splitting the work into multiple processes is not optimal or more convoluted.
        m_mueller 10y ago
        Exactly, see my own post a few branches up.
- xapata 10y ago
  It still makes sense today, for the trade-off you described. Also because getting around the Gil is easy.
makmanalp 10y ago
The first 5 minutes of this covers that in general: https://www.youtube.com/watch?v=P3AyI_u66Bw https://www.youtube.com/watch?v=P3AyI_u66Bw

m_mueller 10y ago

It's not necessary, it was a design choice that made sense back in the 90ies. In a multithreaded environment you can lock at a fine grained level or on a coarse grained level - or you can crash, but let's ignore that as an option. Python chose coarse grained, giving up parallel interpreter computations, but gaining a lot of thread sync overhead. All attempts so far to remove the GIL have resulted in a (usually much) slower interpreter, but the latest attempt shows some promise and it's thinkable (but not guaranteed) that in a few years there will be an official GIL-less cPython.

syllogism 10y ago
Removing Python's GIL will never make much sense. Not today and not in future. If you need CPU-fast code and would bother to multi-thread, it's much more worth it to write the code in Cython. If your code is CPU bound and you're using native Python, you're going to be making a tonne of heap allocations and pointer dereferences. This will be very slow. If you implement the relevant stuff in Cython, even without using multi-threading you'll likely see 10x performance improvement, and can often see up to 100x. Removing the GIL makes Python worse at the stuff it's good at, for questionable improvements in the areas Python is really terrible. This is not a good trade.
- Skunkleton 10y ago
  What if you were trying to thread an non-processor bound task?
  - sp332 10y ago
    Do you mean waiting for I/O? It's already possible to do I/O asynchronously or with separate locks. The interpreter lock only applies to the interpreter.
hermitdev 10y ago
Do you happen to have any papers about the current efforts to remove the GIL? I love Python, and use it a lot for ETL type work, but if threading worked well, I could/would possibly use it for far more purposes.
- brianwawok 10y ago
  Can you give an example where the GIL is really holding you back? Because with multiprocessing and greenlets, 99.99% of concurrency problems are trivilially solved by current Cython.
  - deleted 10y ago
    [deleted]
  - mrits 10y ago
    2 TB hash join
    - doubleunplussed 10y ago
      Isn't that up to the RDBMS whether than's multithreaded or not? Unless the RDBMS is implemented in Python, CPython doesn't force extension code to be single-threaded. Just Python bytecode.
      - foo101 10y ago
        That's pretty much the point, isn't it? If I need true multithreading, then I am forced to write an extension in C or offload the multithreading work to another process (such as RDBMS in your case). It would have been nice if true multithreading was possible in Python itself. It would immediately make Python more useful in a variety of scenarios where splitting the work into multiple processes is not optimal or more convoluted.
        m_mueller 10y ago
        Exactly, see my own post a few branches up.
xapata 10y ago
It still makes sense today, for the trade-off you described. Also because getting around the Gil is easy.

makmanalp 10y ago

The first 5 minutes of this covers that in general: https://www.youtube.com/watch?v=P3AyI_u66Bw https://www.youtube.com/watch?v=P3AyI_u66Bw