4 ms·
Thank you! I was planning to try using DataFusion to query large parquet files without retrieving them from s3 with https://github.com/datafusion-contrib/dataf
by Kinrany 4y ago
Thank you!
I was planning to try using DataFusion to query large parquet files without retrieving them from s3 with https://github.com/datafusion-contrib/datafusion-objectstore-s3 https://github.com/datafusion-contrib/datafusion-objectstore.... I guess I should expect that that won't be necessary :)
- jmartin2683 4y agoDatafusion has better support for object_store api (you can just register the store and pass it s3 uris). With polars, you have to have the file on the local file system (which I’ve found to be a huge performance win, cacheing them anyway).