5 ms·
Wouldn't that be pretty slow? :) Why not store it on the EC2 instance directly? Or locally on your computer? 1 TB is .. not a lot of data .. I rent a server wi
by gizzlon 2y ago
Wouldn't that be pretty slow? :) Why not store it on the EC2 instance directly? Or locally on your computer?
1 TB is .. not a lot of data .. I rent a server with 15 TB for ~50$ per month and buying a new 2TB disk is less than 100$.
- szarnyasg 2y agoDuckDB supports partial reading of Parquet files (also via HTTPS and S3) [1], so it can limit the scans to the required columns in the Parquet file. It can also perform filter pushdown, so querying data in S3 can be quite efficient. Disclaimer: I work at DuckDB Labs. [1] https://duckdb.org/docs/data/parquet/overview#partial-reading https://duckdb.org/docs/data/parquet/overview#partial-readin...
- nomilk 2y ago> Why not store it on the EC2 instance directly? Since the data in S3 receives updates every few hours, querying it directly ensures queries are on up-to-date data, whereas creating a copy of the dataset on the EC2 would necessitate periodically checking for updates and moving the deltas across to the EC2 (not insurmountable; but complexity worth avoiding if possible).
- gizzlon 2y agoah, I see. I though you decided t put it there (also, what the sibling said makes it a better idea)