3 ms·
We haven't seen any corruptions yet but sure there are some strange things that aren't well documented. One thing that hit us was that parquet could only been r
by riedel 5y ago
We haven't seen any corruptions yet but sure there are some strange things that aren't well documented. One thing that hit us was that parquet could only been read using dask if you use the same backend. If you start with thousands of files partitioning is IMHO tricky because using dask naively will create task graphs that can be multiple gigabytes large. We also had a lot of dead lock situations. This is where intuitivity shoots you into the foot because your average data scientist has no clue what to do.
Fortunately many dead lock have been fixed in one of the latest releases. Other than that I have been really happy and thankful that dask exists. One othe major use cases is, however, that my colleagues use it as a simple joblib backed for our HPC cluster. Even if it sounds stupid: dask is the easiest way to get data scientists to use distributed computing resources (Disclaimer: I got active in the dask jobqueue project to get stuff fixed to run on HTCondor)