3 ms·
Yes, this works for S3 files as well. The trick it's using is the HTTP Range header, which lets a client request eg bytes 4500-4900 of an HTTP file. Most stat
by simonw 3y ago
Yes, this works for S3 files as well.
The trick it's using is the HTTP Range header, which lets a client request eg bytes 4500-4900 of an HTTP file.
Most static file hosting platforms - S3, GCS, nginx, Apache etc - support Range headers. They're most commonly used for streaming video and audio.
The Parquet file format is designed with this in mind. You can read metadata at the start of the file and use it to figure out which ranges to fetch. Columns are grouped together, so sum() against a column can be handled by fetching a subset of the file.
- mosselman 3y agoThank you for explaining. I never really looked into this to understand it and because of that it felt like magic, which is always in indicator that you just don’t understand something. I was going to add “in tech”, but it is an indicator of that with anything in life.