I've used Parquet here to filter/map 400 Million Twitter geotagged tweets [1]. The main advantage was that it automatically partitions data so it can be streame

I've used Parquet here to filter/map 400 Million Twitter geotagged tweets [1]. The main advantage was that it automatically partitions data so it can be streamed and chunked during processing. [1]: https://ad.vgiscience.org/twitter-global-preview/00_Twitter_datashader.html https://ad.vgiscience.org/twitter-global-preview/00_Twitter_...