The tie breaker here really is kubernetes. Most likely your company's infrastructure is run on k8s. As a data scientist you do not get control over that. Dask

The tie breaker here really is kubernetes. Most likely your company's infrastructure is run on k8s. As a data scientist you do not get control over that. Dask natively integrates with Kubernetes. That's why I see a lot of people moving away even from Apache Spark (which is generally used through its inbuilt scheduler YARN) and towards Dask. Second reason is that the dask-ml project is building seamless compatibility for higher order ML algorithms (sklearn,etc) on top of Dask. Not just Numpy/Pandas