3 ms·
Dask groupby example: https://examples.dask.org/dataframes/02-groupby.html https://examples.dask.org/dataframes/02-groupby.html > Generally speaking, Dask.data
by westurner 7y ago
Dask groupby example:
https://examples.dask.org/dataframes/02-groupby.html https://examples.dask.org/dataframes/02-groupby.html
> Generally speaking, Dask.dataframe groupby-aggregations are roughly same performance as Pandas groupby-aggregations, just more scalable.
The dask.distributed scheduler can also run on one high-RAM instance (with threads or processes)
https://docs.dask.org/en/latest/setup.html https://docs.dask.org/en/latest/setup.html
Pandas docs > Ecosystem > Out-of-core:
https://pandas.pydata.org/pandas-docs/stable/ecosystem.html#out-of-core https://pandas.pydata.org/pandas-docs/stable/ecosystem.html#...
Reading from Parquet into Apache Arrow is much faster than CSV because the data can just be directly loaded into RAM.
https://ursalabs.org/blog/2019-10-columnar-perf/ https://ursalabs.org/blog/2019-10-columnar-perf/
If you have GPU instances, cuDF has a Pandas-like API on top of Apache Arrow.
https://github.com/rapidsai/cudf https://github.com/rapidsai/cudf
> Built based on the Apache Arrow columnar memory format, cuDF is a GPU DataFrame library for loading, joining, aggregating, filtering, and otherwise manipulating data.
> cuDF provides a pandas-like API that will be familiar to data engineers & data scientists, so they can use it to easily accelerate their workflows without going into the details of CUDA programming.
Dask-ML makes scalable scikit-learn, XGBoost, TensorFlow really easy.
https://dask-ml.readthedocs.io/en/latest/ https://dask-ml.readthedocs.io/en/latest/
... re: the OT: While it's possible to write C++ code that's really fast, it's generally inflexible, expensive to develop, and dangerous for devs with experience in their respective domains of experience to write. Much saner to put a Python API on top and optimize that during compilation.
There are a few C++ frameworks in the top quartile of the TechEmpower framework benchmarks.
https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/
Hardware/hosting is relatively cheap. Developers and memory vulnerabilities aren't.
- hnews_account_1 7y agoUnfortunately, I'm not sure what's wrong with dask, but it doesn't work properly on my cluster. I tested it on an exceedingly simple operation - find all unique values in a very big column (5 billion rows, but I know for a fact that there are only 500-502 unique values in there). With a 100 workers, it still failed. Now this is an embarrassingly parallel operation that can be implemented trivially. So I'm not sure if there's a problem with my cluster or if dask just does not work with slurm clusters very well.
- westurner 7y agohttps://docs.dask.org/en/latest/setup/hpc.html https://docs.dask.org/en/latest/setup/hpc.html says dask-jobqueue handles "PBS, SLURM, LSF, SGE and other resource managers" "Dask on HPC, what works and what doesn't" https://github.com/dask/dask-blog/issues/5 https://github.com/dask/dask-blog/issues/5 Maybe you should spend some time developing a job visualization system for end users from scratch, for end users with lots of C, JS, and HTML experience https://jobqueue.dask.org/en/latest/interactive.html https://jobqueue.dask.org/en/latest/interactive.html
- hnews_account_1 7y agoYes, I think the library still isn't ready for non HPC experts to use without tinkering. I don't have that level of expertise. I'm a data person who can handle working tools.