3 ms·
Many examples have been commented already, but here are some of them: > In the last 10 years I've never met a case where someone pointed out that "this code ne
by martius 10y ago
Many examples have been commented already, but here are some of them:
> In the last 10 years I've never met a case where someone pointed out that "this code needs to be concurrent, but single threaded".
> Going back to concurrency - have I ever been constrained by the number of threads I can use? No, not even once.
Well, in the past 10 years, people have been working on the famous "c10k problem", about how a server software can handle a large amount of concurrent connections, since the model 1 thread/connection doesn't scale (the general purpose kernel scheduler will eat most of the CPU).
This is a real world problem.
> Do people use coroutines?
Yes they do, even if you claim they don't. I honestly don't know on which basis you can claim they don't.
> For me, this sums up the whole GIL and couroutines/asyncio debate. I think the main problem we have at the core of Python is that it's heavily inspired by C. But I think in this case it's Python's weakness.
I can't see a link between the GIL, C and coroutines.
One thing though, using an IO multiplexing framework when doing batch processing is probably suboptimal as using independent processes or queues will have the benefits you point: easier to read and understand, and less contention (kudos to my friend iksaif for the highlight). However, celery is a solution for some use cases, but will not work for many of them. For a similar queue-batch-work problem, executors can be a tool which fits someone needs better than celery (https://docs.python.org/3/library/concurrent.futures.html https://docs.python.org/3/library/concurrent.futures.html).
> I think that if we want to support full concurrency in Python, this is the way to go. Introduce primitives for fully lockless paradigm using queues and enable programmers to define queues.
Queues introduce head of line blocking (hence latency). You need several queues to handle independent and concurrent events in a timely fashion. This is why you need to multiplex.
> I think we never really wanted concurrent code, as programmers. I definitely never wanted coroutines and never needed to multiplex I/O.
Yes again, we do. That's how an interactive system works, that's how your computer works: one giant event loop handling concurrency between tasks and hardware inputs.
(update: words about concurrent.future executors)