3 ms·
Has anyone here had positive experiences with Dask? I chose it as an alternative to manual batching, where a dataset couldn't fit into memory. Anecdotally, it
by teetertater 6y ago
Has anyone here had positive experiences with Dask?
I chose it as an alternative to manual batching, where a dataset couldn't fit into memory. Anecdotally, it was great when it worked, but caused a lot of random bugs, and I spent more time fixing them than the time it saved in the first place. (This was mid-2019 so maybe it has improved since then)
- kevlar1818 6y agoI generally share your criticism, but my overall experience has been positive. Dask indeed requires some fiddling, and it's API and documentation has some blind spots, but I've successfully used it to scale up some Pandas code to run on dataframes measuring in the tens of GB. Being able to transfer my knowledge of Pandas and NumPy so readily makes Dask's speed bumps worth it.
- teetertater 6y agoThanks for sharing your experience!