3 ms·
It doesn't seem to be the same thing though. Like duckdb doesn't seem to use s3 select. The difference is that duckdb seems to be doing the filtering here. If y
by crop_rotation 4y ago
It doesn't seem to be the same thing though. Like duckdb doesn't seem to use s3 select. The difference is that duckdb seems to be doing the filtering here. If you use s3 select s3 will give you result of the sql query. Would be good to know if any existing solution does it.
> select * from a parquet file wherever it resides.
I am sure any tool can do this. The point was to push the query to the source if you are issuing tons of them.
- deleted 4y ago[deleted]
- simonw 4y agoDuckDB pulls a clever trick with SQL and Parquet where it uses the HTTP range header to fetch back only the data it needs to answer the query. The design of Parquet supports this. If you run `select sum(cost) from sales` and the cost column is all in the same byte range, DuckDB will need to fetch just the header that tells it the offset of that cost column and then just the bytes that form the cost column - saving a huge amount of bandwidth. Running `select * from sales` will still have to scan over a much larger section of the file though.
- crop_rotation 4y agoUnless the file is very small, the column would most likely not be in the same byte range. Duckdb also can't filter on multiple columns without fetching both. Being able to fully preselect data at the source is much more powerful than the parquet tricks duckdub uses. Don't get me wrong, the duckdb impl is good and generic and will support any parquet store, but it is not as powerful as S3 select.
- tomnipotent 4y ago> the cost column is all in the same byte range Parquet is hybrid, so you have random chunks of contiguous bytes scattered throughout the file.