Dask is really good for scientists, when I worked with bioinformaticians it was way easier to get them to use dask for out of memory processing than spark. In p

Dask is really good for scientists, when I worked with bioinformaticians it was way easier to get them to use dask for out of memory processing than spark. In particular dask bag offers a great amount of flexibility for ETL use cases. The problem with dask for me though (as a user of dask and prefect) is that I was never able to get the throughput of pySpark out of dask. For dask I can reach 100 mb/s on my laptop while pyspark can each 260 mb/s on my laptop for the same workload (cleaning and restructuring). This was around 2 years ago that I tested that I’m curious what the community has experienced now. Also for most use cases your data isn’t really big data and dask is much simpler and fast enough which is why I reach for dask first.