4 ms·

For those who find GIL impacts their developments, can you post some examples? I'm curious as to what they are.

• jillesvangurp 5y ago
  Well many developers end up using different languages for this reason. E.g. Go and node.js are popular and both do asynchronous stuff effortlessly (even though they are single threaded typically). And if you use Jython (python on the jvm), you can actually do threads properly as the Gil is not a thing there: https://stackoverflow.com/questions/1120354/does-jython-have-the-gil https://stackoverflow.com/questions/1120354/does-jython-have... Likewise, Iron Python also has no Gil apparently. Both are running quite a bit behind PythonC at this point unfortunately. For properly multi threaded stuff, people use more higher level languages like Erlang, Scala, Java, Kotlin, Rust, C++, etc. In general batch processing data is both IO and CPU intensive (e.g. parsing, regular expressions) and a common use-case for Python. You could say, data science is actually the primary use case driving its popularity. Python is effectively single threaded for that stuff. Using multiple processes fixes the problem well enough though. Conveniently the module for using processes looks very similar to the one for using threads, so switching is easy once you figure out why shit doesn't scale at all (been there done that). But using processes also adds some complexity. The good thing is that it makes doing why the Gil is needed (sharing data between threads/processes) a lot harder/impossible. The bad news is that it's hard. Typically people use queues and databases for this. ETL systems like Airflow tend to use multiple processes and queues instead of threads as using python threads is kind of pointlessly futile (from a making things faster point of view). However, with Airflow a pattern that you see a lot is that python process workers are typically not used and instead it is configured to use one of several cloud based schedulers (e.g. kubernetes, ECS, etc) to schedule work. Basically, airflow is just used to orchestrates work where the actual work happens elsewhere and may or may not involve python code. A lot of the performance critical stuff is native anyway.
  • doctordesh 5y ago
    > ... Go and node.js are popular and both do asynchronous stuff effortlessly (even though they are single threaded typically) I know that I'm being very pedantic here, but Go is not a "typically single threaded" language. Go has as fine control over it's threads (goroutines) as C does over it's threads. Go is very capable of true parallellism.
• malkia 5y ago
  I don't know the details, but Guido has written Mondrian, a code review system for use at Google - https://www.niallkennedy.com/blog/2006/11/google-mondrian.html https://www.niallkennedy.com/blog/2006/11/google-mondrian.ht... - it was later replaced by something else, in a different language.
  • malkia 5y ago
    Some history is captured here - https://www.gerritcodereview.com/about.html https://www.gerritcodereview.com/about.html (note, this is for the open source version of the tool, rietveld, from which gerrit roots are)
    • malkia 5y ago
      Sorry, for replying myself, but wanted to rely on outside sources - https://en.wikipedia.org/wiki/Gerrit_(software)#History https://en.wikipedia.org/wiki/Gerrit_(software)#History So it got from Python -> Java
  • zo1 5y ago
    Was it specifically replaced because of the GIL?
    • malkia 5y ago
      I don't know, but from what I've heard Java just scales much better, and has better tooling/monitoring/etc. for running on cloud.