4 ms·

in my opinion, getting rid of the GIL is the one thing that Python can do to stave off Julia's eventual dominance of numeric computing. Not indefinitely, but by

in my opinion, getting rid of the GIL is the one thing that Python can do to stave off Julia's eventual dominance of numeric computing. Not indefinitely, but by at least 5 years. There are still many cases when data scientists don't have a C extension and need to run pure Python functions in parallel, and the GIL makes this an order of magnitude harder (and sometimes slower). I think the core Python devs have gotten rather too conservative in the language's old age.

Areading314 4y ago
wouldn't removing the GIL cause all sorts of sudden race conditions and regressions in existing python code? I think there is a valid argument to keeping the GIL just for simplicity's sake
- bratao 4y ago
  From my understanding, race conditions already exists today as the GIL is made to protect the internal Python structures. It only made it appears more infrequently.
  - masklinn 4y ago
    > It only made it appears more infrequently. It makes some operations atomic which is quite relevant.
    - MrJohz 4y ago
      Fewer than people expect. For example, the += operator looks atomic, but under the hood, it's made up of multiple python bytecode instructions, so the GIL won't apply here.
      - masklinn 4y ago
        > For example, the += operator looks atomic, but under the hood, it's made up of multiple python bytecode instructions, so the GIL won't apply here. Sure, but any call to a native function which doesn’t release the gil (which is most of them at least for the builtins) is currently atomic so something like dict.setdefault.
        masklinn 4y ago
        And note that this is mentioned in the faq.
        dontlaugh 4y ago
        Those would have to remain atomic by taking explicit more granular locks.
        masklinn 4y ago
        Obviously? But the point is currently they do work, they are thread-safe and they are correct in the strictest sense of the word, per-FAQ: https://docs.python.org/3/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe https://docs.python.org/3/faq/library.html#what-kinds-of-glo... So as the original commenter notes: > wouldn't removing the GIL cause all sorts of sudden race conditions and regressions in existing python code?
shadowofneptune 4y ago
I have never used Julia before. What about it makes seems so inevitable?
- evgen 4y ago
  It isn't inevitable, but some people promoting Julia like to pretend that there is some sort of competition for mindshare happening. There are three basic markets for numerical computing: hobbyist, academic, and commercial. The hobbyists will always remain in Python because it is good enough and there is little benefit to learning a new language for casual numerical computing (this also applies to undergrad-level academics, which is closer to hobbyists than deep academics.) In academics it is possible that Julia will replace R for a lot of use cases, but it is unlikely to make much progress in displacing Python outside of math-heavy fields: a biologist or chemist will stay with Python because of the ecosystem and its applications outside of pure numerical processing code. In the commercial world the race is already over and Python won, it will continue to grow in this role due to simple inertia and because for cases where numerical computation speed actually matters a company can hire people to write the code in something even faster and more efficient than Julia.
  - noiwillnot 4y ago
    > Is possible that Julia will replace R for a lot of use cases R is a interactive statistical programming language that acts as a frontend for more performance languages. AFAIK interactivity is not the strongest points of Julia at the moment.
    - MontyCarloHall 4y ago
      >interactivity is not the strongest points of Julia at the moment Not sure what you mean. Julia has the exact same notebook environment (Jupyter) as R and Python. Fun fact: the “Ju” in Jupyter stands for “Julia” (the “pyt” and “r” stand for what you think).
      - noiwillnot 4y ago
dagss 4y ago
For numerical computation compiled code is 10x-1000x faster; even without the GIL Python cannot compete without dropping to C/Fortran (which typically releases the GIL anyway)
dannyz 4y ago
In terms of numerical computing I don't think this is going to be a battle of core language speed, Python has already lost here. The question is going to be what are the advantages of a language designed with a focus on number crunching vs a multi-purpose language with some number crunching extensions. I mostly do scientific computing but I really enjoy the toolchain of an extremely popular multi-purpose programming language (Python) + C extensions where necessary for speed. The reason is that many tasks these days are not just pure number crunching. If I need to start up a quick web server, do some web site scraping, basically anything can be done easily from Python. It's unlikely Julia will ever be able to catch-up in these areas.

Areading314 4y ago

wouldn't removing the GIL cause all sorts of sudden race conditions and regressions in existing python code? I think there is a valid argument to keeping the GIL just for simplicity's sake

bratao 4y ago
From my understanding, race conditions already exists today as the GIL is made to protect the internal Python structures. It only made it appears more infrequently.
- masklinn 4y ago
  > It only made it appears more infrequently. It makes some operations atomic which is quite relevant.
  - MrJohz 4y ago
    Fewer than people expect. For example, the += operator looks atomic, but under the hood, it's made up of multiple python bytecode instructions, so the GIL won't apply here.
    - masklinn 4y ago
      > For example, the += operator looks atomic, but under the hood, it's made up of multiple python bytecode instructions, so the GIL won't apply here. Sure, but any call to a native function which doesn’t release the gil (which is most of them at least for the builtins) is currently atomic so something like dict.setdefault.
      - masklinn 4y ago
        And note that this is mentioned in the faq.
      - dontlaugh 4y ago
        Those would have to remain atomic by taking explicit more granular locks.
        masklinn 4y ago
        Obviously? But the point is currently they do work, they are thread-safe and they are correct in the strictest sense of the word, per-FAQ: https://docs.python.org/3/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe https://docs.python.org/3/faq/library.html#what-kinds-of-glo... So as the original commenter notes: > wouldn't removing the GIL cause all sorts of sudden race conditions and regressions in existing python code?

shadowofneptune 4y ago

I have never used Julia before. What about it makes seems so inevitable?

evgen 4y ago
It isn't inevitable, but some people promoting Julia like to pretend that there is some sort of competition for mindshare happening. There are three basic markets for numerical computing: hobbyist, academic, and commercial. The hobbyists will always remain in Python because it is good enough and there is little benefit to learning a new language for casual numerical computing (this also applies to undergrad-level academics, which is closer to hobbyists than deep academics.) In academics it is possible that Julia will replace R for a lot of use cases, but it is unlikely to make much progress in displacing Python outside of math-heavy fields: a biologist or chemist will stay with Python because of the ecosystem and its applications outside of pure numerical processing code. In the commercial world the race is already over and Python won, it will continue to grow in this role due to simple inertia and because for cases where numerical computation speed actually matters a company can hire people to write the code in something even faster and more efficient than Julia.
- noiwillnot 4y ago
  > Is possible that Julia will replace R for a lot of use cases R is a interactive statistical programming language that acts as a frontend for more performance languages. AFAIK interactivity is not the strongest points of Julia at the moment.
  - MontyCarloHall 4y ago
    >interactivity is not the strongest points of Julia at the moment Not sure what you mean. Julia has the exact same notebook environment (Jupyter) as R and Python. Fun fact: the “Ju” in Jupyter stands for “Julia” (the “pyt” and “r” stand for what you think).
    - noiwillnot 4y ago

dagss 4y ago

For numerical computation compiled code is 10x-1000x faster; even without the GIL Python cannot compete without dropping to C/Fortran (which typically releases the GIL anyway)

dannyz 4y ago

In terms of numerical computing I don't think this is going to be a battle of core language speed, Python has already lost here. The question is going to be what are the advantages of a language designed with a focus on number crunching vs a multi-purpose language with some number crunching extensions. I mostly do scientific computing but I really enjoy the toolchain of an extremely popular multi-purpose programming language (Python) + C extensions where necessary for speed. The reason is that many tasks these days are not just pure number crunching. If I need to start up a quick web server, do some web site scraping, basically anything can be done easily from Python. It's unlikely Julia will ever be able to catch-up in these areas.