3 ms·

can you expand or give more context on this ?

• datadrivenangel 2y ago
  Dask is designed for scaling to multiple compute nodes, and is often slower than vanilla pandas on a single machine because of the overhead. If your data is larger than memory though, at a certain point Dask is faster than pandas (which will often just fail), and DuckDB (Which is super fast for in-memory data, and still quite fast for out of memory data). Rule of thumb these days though is that if your data isn't in the terabytes per run, a single processing node is probably best.