3 ms·
Part of the problem - to me at least - is that in practice, concurrency with python is pretty much only heavy (slow) multiprocessing.queues. It's what multiproc
by uniqueuid 4y ago
Part of the problem - to me at least - is that in practice, concurrency with python is pretty much only heavy (slow) multiprocessing.queues. It's what multiprocessing.pool and the corresponding executors use.
At the same time, my experience (along with that of many others) has been that these queues are not exactly robust - there seem to be many horrible edge cases where queues and/or processes get stuck and it's really hard to have a robust all-python app that recovers from e.g. a dead child (!). Ask ML frameworks, for example, how they handle spawning while holding GPU contexts and the like. It's a nightmare.
So if we had a single robust, elegant and half-way fast abstraction for IPC, that would solve a lot of the existing issues. And it needs to be bi-directional of course.