3 ms·
Nice post but they quickly disregard Dask, don't explain why, don't test it, even exclude it from the benchmark they quote. I don't know if is the better answer
by dweinus 23d ago
Nice post but they quickly disregard Dask, don't explain why, don't test it, even exclude it from the benchmark they quote. I don't know if is the better answer, but it seems worth testing if you want a balance approachable + scalable. That's kind the thing Dask was meant to do.
- x0xMaximus 23d agoWe run Dask distributed for contents in what their chart covers (10TB+ with 4+ physical hosts) and it's a problem that can't really be discussed without well defined drive read or network latency limitations (if non-local storage). We've found defining recordsize, raid and other filesystem patterns to have an equal or larger impact in itself. Pandas is great because it allows us to validate on a sampling or smaller time window dataset, and then the same engineers can use a very similar implementation for Dask distributed through their scheduler tool. While throwing hardware at problems isn't always a reasonable solution, it's trivialized to discuss this without IOPS or other read limitations.. and the nvme enterprise market is rapidly developing making this change quickly.