There are many packages that can do that, like Vax [1] and Dask [2]. I don't know exactly your workflow. But the concurrency in python is limited to multiproces

There are many packages that can do that, like Vax [1] and Dask [2]. I don't know exactly your workflow. But the concurrency in python is limited to multiprocessing, which is much expensive than threads which usually a typical streaming parallelizer will use outside python world. [1] https://vaex.io/docs/index.html https://vaex.io/docs/index.html [2] https://docs.dask.org/en/latest/ https://docs.dask.org/en/latest/