5 ms·
Does anybody here know dataframe systems that are able to handle file sizes bigger than the available RAM? Is polars able to handle this? I am only aware of dis
by pvitz 5y ago
Does anybody here know dataframe systems that are able to handle file sizes bigger than the available RAM? Is polars able to handle this? I am only aware of disk.frame (diskframe.com), but don't know how well it performs.
- Fiahil 5y agoYou either stream them, or use bigger VMs.
- alexisread 5y agoI believe Vaex can do this, in addition to GPU processing and reading direct from s3. https://github.com/vaexio/vaex https://github.com/vaexio/vaex
- pvitz 5y agoTo you and all the other sibling comments: Thanks a lot! Exactly what I have been looking for! With regard to Vaex, I would really be interested in an independent benchmark comparing it to dask, spark, data.table etc. However, I have seen in the comments that others also can't find that.
- chrisaycock 5y agoThe H20 benchmarks cover Dataframe operations: https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/ It has pandas, dask, Spark, data.table, Polars, etc. Sadly, Vaex is currently missing from this suite.
- Matumio 5y agoFor Python there is Dask: https://docs.dask.org/en/stable/dataframe.html https://docs.dask.org/en/stable/dataframe.html
- cmollis 5y agospark dataframe api..
- KptMarchewa 5y agoApache Spark.
- VHRanger 5y agoVaex