4 ms·
DuckDB supports partial reading of Parquet files (also via HTTPS and S3) [1], so it can limit the scans to the required columns in the Parquet file. It can also
by szarnyasg 2y ago
DuckDB supports partial reading of Parquet files (also via HTTPS and S3) [1], so it can limit the scans to the required columns in the Parquet file. It can also perform filter pushdown, so querying data in S3 can be quite efficient.
Disclaimer: I work at DuckDB Labs.
[1] https://duckdb.org/docs/data/parquet/overview#partial-reading https://duckdb.org/docs/data/parquet/overview#partial-readin...