3 ms·

Hi, I'm the author. Please point out the nonsense :) But honestly, what I'm trying to advocate here is that if we remove the problem of what's executing your t

by mdomans 10y ago

Hi, I'm the author. Please point out the nonsense :) But honestly, what I'm trying to advocate here is that if we remove the problem of what's executing your tasks and the problem of implicit locking, we can stop worrying about GIL. A very good implementation of that is how Grand Central Dispatch works.

danpalmer 10y ago
I was under the impression that GCD used green threads, which means it's sharing the same address space, and ultimately would be subject to the GIL? When you're memory bound, sharing the address space is important, and Celery might not work well at all.
- mdomans 10y ago
  Depends on how much memory we're talking about. In regard to GCD - AFAIK it uses classic posix threads. In regard to GIL, if you can indeed guarantee that each task is atomic and lockless, you can use the same approach pyparralel uses
  - danpalmer 10y ago
    I believe GCD actually multiplexes green threads across a pool of posix threads, which wouldn't work for Python because of the GIL.
    - richardwhiuk 10y ago
      Correct. GCD has dispatch_queues, and each block on the dispatch queue will be executed on one of a pool of posix threads.
      - mdomans 10y ago
        That's correct, but looking at the Python internals it seems that working around the GIL is possible if having this pattern exposed to end programmer.
martius 10y ago
Many examples have been commented already, but here are some of them: > In the last 10 years I've never met a case where someone pointed out that "this code needs to be concurrent, but single threaded". > Going back to concurrency - have I ever been constrained by the number of threads I can use? No, not even once. Well, in the past 10 years, people have been working on the famous "c10k problem", about how a server software can handle a large amount of concurrent connections, since the model 1 thread/connection doesn't scale (the general purpose kernel scheduler will eat most of the CPU). This is a real world problem. > Do people use coroutines? Yes they do, even if you claim they don't. I honestly don't know on which basis you can claim they don't. > For me, this sums up the whole GIL and couroutines/asyncio debate. I think the main problem we have at the core of Python is that it's heavily inspired by C. But I think in this case it's Python's weakness. I can't see a link between the GIL, C and coroutines. One thing though, using an IO multiplexing framework when doing batch processing is probably suboptimal as using independent processes or queues will have the benefits you point: easier to read and understand, and less contention (kudos to my friend iksaif for the highlight). However, celery is a solution for some use cases, but will not work for many of them. For a similar queue-batch-work problem, executors can be a tool which fits someone needs better than celery (https://docs.python.org/3/library/concurrent.futures.html https://docs.python.org/3/library/concurrent.futures.html). > I think that if we want to support full concurrency in Python, this is the way to go. Introduce primitives for fully lockless paradigm using queues and enable programmers to define queues. Queues introduce head of line blocking (hence latency). You need several queues to handle independent and concurrent events in a timely fashion. This is why you need to multiplex. > I think we never really wanted concurrent code, as programmers. I definitely never wanted coroutines and never needed to multiplex I/O. Yes again, we do. That's how an interactive system works, that's how your computer works: one giant event loop handling concurrency between tasks and hardware inputs. (update: words about concurrent.future executors)

by mdomans 10y ago

danpalmer 10y ago

I was under the impression that GCD used green threads, which means it's sharing the same address space, and ultimately would be subject to the GIL? When you're memory bound, sharing the address space is important, and Celery might not work well at all.

mdomans 10y ago
Depends on how much memory we're talking about. In regard to GCD - AFAIK it uses classic posix threads. In regard to GIL, if you can indeed guarantee that each task is atomic and lockless, you can use the same approach pyparralel uses
- danpalmer 10y ago
  I believe GCD actually multiplexes green threads across a pool of posix threads, which wouldn't work for Python because of the GIL.
  - richardwhiuk 10y ago
    Correct. GCD has dispatch_queues, and each block on the dispatch queue will be executed on one of a pool of posix threads.
    - mdomans 10y ago
      That's correct, but looking at the Python internals it seems that working around the GIL is possible if having this pattern exposed to end programmer.

martius 10y ago

Many examples have been commented already, but here are some of them: > In the last 10 years I've never met a case where someone pointed out that "this code needs to be concurrent, but single threaded". > Going back to concurrency - have I ever been constrained by the number of threads I can use? No, not even once. Well, in the past 10 years, people have been working on the famous "c10k problem", about how a server software can handle a large amount of concurrent connections, since the model 1 thread/connection doesn't scale (the general purpose kernel scheduler will eat most of the CPU). This is a real world problem. > Do people use coroutines? Yes they do, even if you claim they don't. I honestly don't know on which basis you can claim they don't. > For me, this sums up the whole GIL and couroutines/asyncio debate. I think the main problem we have at the core of Python is that it's heavily inspired by C. But I think in this case it's Python's weakness. I can't see a link between the GIL, C and coroutines. One thing though, using an IO multiplexing framework when doing batch processing is probably suboptimal as using independent processes or queues will have the benefits you point: easier to read and understand, and less contention (kudos to my friend iksaif for the highlight). However, celery is a solution for some use cases, but will not work for many of them. For a similar queue-batch-work problem, executors can be a tool which fits someone needs better than celery (https://docs.python.org/3/library/concurrent.futures.html https://docs.python.org/3/library/concurrent.futures.html). > I think that if we want to support full concurrency in Python, this is the way to go. Introduce primitives for fully lockless paradigm using queues and enable programmers to define queues. Queues introduce head of line blocking (hence latency). You need several queues to handle independent and concurrent events in a timely fashion. This is why you need to multiplex. > I think we never really wanted concurrent code, as programmers. I definitely never wanted coroutines and never needed to multiplex I/O. Yes again, we do. That's how an interactive system works, that's how your computer works: one giant event loop handling concurrency between tasks and hardware inputs. (update: words about concurrent.future executors)