4 ms·
Parquet files are also /can also be split into different row groups. By ordering the contents by whatever you're most likely to filter on you can cut down on ac
by IanCal 3y ago
Parquet files are also /can also be split into different row groups. By ordering the contents by whatever you're most likely to filter on you can cut down on accessed data even more.
E.g if you filter on creation time order the data by that. Then when you query it, duckdb/similar should read the metadata for each row groups and ignore any that cannot contain the data you want. Ordering by user id or country, etc. are common other ones. Order by the biggest group first (so country then timestamp if you often filter on both).
You can tune the size of the row groups depending on your access pattern.