6 ms·

> your average python dev can just ignore it if they want to. Oh, so naive... All the mutation code in Python which "worked" because Python didn't really have

> your average python dev can just ignore it if they want to. Oh, so naive... All the mutation code in Python which "worked" because Python didn't really have any real concurrency. Add to it -- there's no real plan about what to do with Python concurrency. Removing GIL is only one "half" of the problem, you need to give developers some sort of a framework to use to deal with concurrency. Python's threads are extremely underdeveloped and dangerous to use. Python doesn't even have anything like "synchronized" from the Java world. So, all synchronization requires dealing with locks, mutexes, condition variables... Most Python programs today didn't bother to deal with threads because they didn't confer enough benefits to be worth using. So, "automatically" parallelizing Python code, as in allowing it to run in actual threads is going to bring about lots and lots of bugs in trivial code written by people with no clue about concurrency.

quietbritishjim 3y ago
> So, all synchronization requires dealing with locks, mutexes, condition variables... As always, by far the best way to interact between threads is to use thread-safe queues (AKA message passing). Luckily, Python has one of those [1]. No complicated synchronisation needed. [1] https://docs.python.org/3/library/queue.html https://docs.python.org/3/library/queue.html
- crabbone 3y ago
  That's just completely missing the point of threads... but that wouldn't be the first nor the hundreds stupid thing found in Python's documentation. The reason to want threads is to be able to share memory. That's literally why they were created. If you are sending messages instead of sharing memory, you don't need threads. You need something like Erlang processes. The problem is that people who wrote Python never had a plan. They sucked and still suck as programmers. So... they knew there are threads. And it was easy to write a bunch of wrappers around pthreads. And that's what they did. And then they realized they don't know how to deal with concurrency, so they found a simple way out -- GIL. The whole history of Python is the history of choosing the easy but wrong. And it's probably the only consistent thing about the language.
  - quietbritishjim 3y ago
    The object going onto the queue is the shared memory. The queue itself is essentially a fancy type of lock. Yes you could use multiple processes, but you have the extra expense of serialisation, or you could use shared memory but you'd have to administer that and you'd still have the expense of context switches. And inevitably, yes, there is some actual shared state like a logger and modules that you've loaded, which again would be a pain in multiple processes. You can call a Python thread plus a queue an Erlang process if you like, or say that I should use Erlang processes instead. But the fact is, the Python version works perfectly well for many problems. It does all the things that you typically need if threads: shares state (via the queues), let's you concurrently use the CPU (via C libraries that release the GIL – but no GIL would be even better), and writing blocking IO if you wish. Not missing the point at all. The developers of Python didn't "suck as programmers" and it doesn't help your point to claim they do. Guido choose to use the GIL because he was OK with multiple threads but not at the expense of single thread performance, and no one showed any solution to that that beats the GIL – until now. (Personally I think the trade off was wrong, and a small hit to single threaded performance would have been with it. But that's different from being ignorant to the fact there was an actual reason.)
gjulianm 3y ago
Which code is automatically going to run in threads? As you say, basically nobody uses Python threads. So even enabling no-gil, nothing is going to change because sequential code will still be sequential.
- woodrowbarlow 3y ago
  any existing async/await code.
  - gjulianm 3y ago
    async/await code already runs in threads, so that's not really a change.
    - quietbritishjim 3y ago
      What do you mean it already runs in threads? It does so if you specify it with run_in_executor [1], or if you run multiple event loops at once, but it doesn't automatically. [1] https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor https://docs.python.org/3/library/asyncio-eventloop.html#asy...
      - gjulianm 3y ago
        Woops, yeah, you're right. I thought the default executor was a threadpool, my mistake. However, in that case, I assume that the default executor will not change to multithreaded when no-gil comes.
        quietbritishjim 3y ago
        There isn't an executor at all, at least not in the concurrent.futures.Executor sense. It just runs in the thread where you call asyncio.run.
  - crabbone 3y ago
    Async is for asynchronous I/O, i.e. such I/O that is only possible through file descriptors that support something like epoll() (i.e. network sockets). It's a thing completely separate from threads, where no code is supposed to run concurrently. The idea of this feature is that a program may schedule a bunch of I/O operations and then wait for their completion instead of scheduling I/O operations one at a time. As of now, this is an obsolete mechanism of dealing with I/O as now we have uring_io. But, truth be told, it never really worked well... I mean, if you knew what you were doing, you could have taken advantage of this feature, but it was never in shape to be library-grade multi-purpose functionality. Python made a stupid bet on it and encoded it into the language through async / await keywords. But if this was the only stupid thing Python has done in its history, flying cars and hoverboards would probably be an integral part of our daily lives.

quietbritishjim 3y ago

> So, all synchronization requires dealing with locks, mutexes, condition variables... As always, by far the best way to interact between threads is to use thread-safe queues (AKA message passing). Luckily, Python has one of those [1]. No complicated synchronisation needed. [1] https://docs.python.org/3/library/queue.html https://docs.python.org/3/library/queue.html

crabbone 3y ago
That's just completely missing the point of threads... but that wouldn't be the first nor the hundreds stupid thing found in Python's documentation. The reason to want threads is to be able to share memory. That's literally why they were created. If you are sending messages instead of sharing memory, you don't need threads. You need something like Erlang processes. The problem is that people who wrote Python never had a plan. They sucked and still suck as programmers. So... they knew there are threads. And it was easy to write a bunch of wrappers around pthreads. And that's what they did. And then they realized they don't know how to deal with concurrency, so they found a simple way out -- GIL. The whole history of Python is the history of choosing the easy but wrong. And it's probably the only consistent thing about the language.
- quietbritishjim 3y ago
  The object going onto the queue is the shared memory. The queue itself is essentially a fancy type of lock. Yes you could use multiple processes, but you have the extra expense of serialisation, or you could use shared memory but you'd have to administer that and you'd still have the expense of context switches. And inevitably, yes, there is some actual shared state like a logger and modules that you've loaded, which again would be a pain in multiple processes. You can call a Python thread plus a queue an Erlang process if you like, or say that I should use Erlang processes instead. But the fact is, the Python version works perfectly well for many problems. It does all the things that you typically need if threads: shares state (via the queues), let's you concurrently use the CPU (via C libraries that release the GIL – but no GIL would be even better), and writing blocking IO if you wish. Not missing the point at all. The developers of Python didn't "suck as programmers" and it doesn't help your point to claim they do. Guido choose to use the GIL because he was OK with multiple threads but not at the expense of single thread performance, and no one showed any solution to that that beats the GIL – until now. (Personally I think the trade off was wrong, and a small hit to single threaded performance would have been with it. But that's different from being ignorant to the fact there was an actual reason.)

gjulianm 3y ago

Which code is automatically going to run in threads? As you say, basically nobody uses Python threads. So even enabling no-gil, nothing is going to change because sequential code will still be sequential.

woodrowbarlow 3y ago
any existing async/await code.
- gjulianm 3y ago
  async/await code already runs in threads, so that's not really a change.
  - quietbritishjim 3y ago
    What do you mean it already runs in threads? It does so if you specify it with run_in_executor [1], or if you run multiple event loops at once, but it doesn't automatically. [1] https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor https://docs.python.org/3/library/asyncio-eventloop.html#asy...
    - gjulianm 3y ago
      Woops, yeah, you're right. I thought the default executor was a threadpool, my mistake. However, in that case, I assume that the default executor will not change to multithreaded when no-gil comes.
      - quietbritishjim 3y ago
        There isn't an executor at all, at least not in the concurrent.futures.Executor sense. It just runs in the thread where you call asyncio.run.
- crabbone 3y ago
  Async is for asynchronous I/O, i.e. such I/O that is only possible through file descriptors that support something like epoll() (i.e. network sockets). It's a thing completely separate from threads, where no code is supposed to run concurrently. The idea of this feature is that a program may schedule a bunch of I/O operations and then wait for their completion instead of scheduling I/O operations one at a time. As of now, this is an obsolete mechanism of dealing with I/O as now we have uring_io. But, truth be told, it never really worked well... I mean, if you knew what you were doing, you could have taken advantage of this feature, but it was never in shape to be library-grade multi-purpose functionality. Python made a stupid bet on it and encoded it into the language through async / await keywords. But if this was the only stupid thing Python has done in its history, flying cars and hoverboards would probably be an integral part of our daily lives.