Here is some other interesting and related duckdb SQL that you all might find helpful. inspect the parquet metadata: SELECT * FROM

Here is some other interesting and related duckdb SQL that you all might find helpful. inspect the parquet metadata: SELECT * FROM parquet_metadata('https://huggingface.co/datasets/vivym/midjourney-messages/resolve/main/data/000000.parquet'); If the data was on blob storage you could use a glob instead of a generator: SELECT sum(*) AS TOTAL_SIZE FROM read_parquet('https://huggingface.co/datasets/vivym/midjourney-messages/resolve/main/data/*.parquet'); You can use the huggingface API to list and then read with duckdb: SELECT concat('https://huggingface.co/datasets/vivym/midjourney-messages/resolve/main/', path) as parquet_file FROM read_json_auto('https://huggingface.co/api/datasets/vivym/midjourney-messages/tree/main/data'); So this means we can combine the list files and read files SQL into a single statement!!! Error: Binder Error: Table function cannot contain subqueries :( No Want the query in a more concise, reusable form? CREATE MACRO GET_MJ_TOTAL_SIZE(num_of_files) AS TABLE ( SELECT SUM(size) AS size FROM read_parquet( list_transform( generate_series(0, num_of_files), n -> 'https://huggingface.co/datasets/vivym/midjourney-messages/resolve/main/data/' || format('{:06d}', n) || '.parquet' ) ) ); You can simply query table function: SELECT * FROM GET_MJ_TOTAL_SIZE(55); You don't need to run nettop: EXPLAIN ANALYZE SELECT * FROM GET_MJ_TOTAL_SIZE(55); ┌─────────────────────────────────────┐ │┌───────────────────────────────────┐│ ││ HTTP Stats: ││ ││ ││ ││ in: 295.5MB ││ ││ out: 0 bytes ││ ││ #HEAD: 55 ││ ││ #GET: 166 ││ ││ #PUT: 0 ││ ││ #POST: 0 ││ │└───────────────────────────────────┘│ └─────────────────────────────────────┘