Having used both ray, dask, and writing custom threads, my personal view is that while there are advantages I wouldn’t want to use any of these unless absolutel

Having used both ray, dask, and writing custom threads, my personal view is that while there are advantages I wouldn’t want to use any of these unless absolutely necessary. My personal approach for most of these tasks are to try to break down the problem to be as asynchronous as possible. Then you can create threads. The nice thing about dask is really the way you can effectively use it on pandas dataframe; Without any large overhead. Having said that, we opted to write our own parallelization for this library: https://github.com/capitalone/DataProfiler https://github.com/capitalone/DataProfiler As opposed to using the dask frame. Effectively, the dataframe processing wasn’t the bottleneck and easier to maintain the threading ourselves given the particular approaches taken. That said, if I was working with large pandas dataframes, id likely use dask. For large datasets which couldn’t be stored in memory of use ray.io