Even in the python async/await world, you can get preemptable multitasking by adding in a few ProcessPoolExecutor threads. I've found this to be extremely effec

Even in the python async/await world, you can get preemptable multitasking by adding in a few ProcessPoolExecutor threads. I've found this to be extremely effective in a web crawler: the main thread is 100% cooperative and does all of the network I/O on a single core, while cpu-hungry activities like webpage parsing are done by ProcessPoolExecutor processes on other cores. The code is pretty and I can fully use an entire server with what is conceptually a single crawler. asyncio.create_subprocess_shell() is another way to get a preemptable thing in the Python async/await world.