3 ms·
> With frameworks like Python pandas, you always end up having to manually partition your data if it doesn’t fit in memory. "Pandas Docs > Pandas Ecosystem > O
by westurner 7y ago
> With frameworks like Python pandas, you always end up having to manually partition your data if it doesn’t fit in memory.
"Pandas Docs > Pandas Ecosystem > Out of Core" lists a number of solutions for working with datasets that don't fit into RAM: Blaze, Dask, Dask-ML (dask-distributed; Scikit-Learn, XGBoost, TensorFlow), Koalas, Odo, Ray, Vaex
https://pandas-docs.github.io/pandas-docs-travis/ecosystem.html#out-of-core https://pandas-docs.github.io/pandas-docs-travis/ecosystem.h...
The dask API is very similar to the pandas API.
Are there any plans for ROOT to gain support for Apache Parquet, and/or Apache Arrow zero-copy reads and SIMD support, and/or https://RAPIDS.ai https://RAPIDS.ai (Arrow, numba, Dask, pandas, scikit-learn, XGboost, spark, CUDA-X GPU acceleration, HPC)?
https://arrow.apache.org/ https://arrow.apache.org/