46 ms·
> That is exactly the problem with asyncio, the minute you want to do something a little more cpu heavy it blocks. Isn't that by design? Sending CPU-heavy task
by joconde 5y ago
> That is exactly the problem with asyncio, the minute you want to do something a little more cpu heavy it blocks.
Isn't that by design? Sending CPU-heavy tasks to the default executor with asyncio.to_thread() works perfectly for me.
Or maybe this doesn't scale in your use-case, because of the thread count limit? (Although in that case, I don't think you'd be better off without asyncio.)
- JimDabell 5y agoWon’t CPU-heavy tasks still make everything choke because of the GIL? You want to send CPU-heavy tasks to a different process instead of a different thread, don’t you?
- joconde 5y agoIn my case no, because I heavily use numpy, PyTorch, and other libraries that use C++ extensions and release the GIL during long operations. I easily use 24 CPU cores fully by submitting to one thread pool. If you use pure Python for long tasks, you're right that a ProcessPoolExecutor would be better.
- ris 5y agoIt depends what you mean by "choke". I think the parents consider "choke" to mean "will focus on one long-running task at the expense of any other task that needs to run". The GIL means that multiple threads won't be able to run python code in parallel, but the design should fairly share the available processing time between all threads that want to run.
- joconde 5y agoIn my case, I'm lucky to be using computation libraries that release the GIL.
- pbalau 5y agoMaybe the IO part of asyncio is confusing people...