3 ms·
Thanks! I hadn't seen anyone do it this way before with a very large, partitioned dataset, but it works shockingly well as long as you're not trying to `SELECT
by dfsnow 2y ago
Thanks! I hadn't seen anyone do it this way before with a very large, partitioned dataset, but it works shockingly well as long as you're not trying to `SELECT *` the entire table. Props to the DuckDB folks.
Eventually I plan to add some thin R and Python wrapper packages around the DuckDB calls just to make it easier for researchers.
- simonw 2y agoI blogged a few more notes here: https://simonwillison.net/2025/Mar/17/opentimes/ https://simonwillison.net/2025/Mar/17/opentimes/
- kalendos 2y agoNice! I know a couple of projects that have been using this pattern. - https://bsky.app/profile/jakthom.bsky.social/post/3lbarcvzrck24 https://bsky.app/profile/jakthom.bsky.social/post/3lbarcvzrc... - https://bsky.app/profile/jakthom.bsky.social/post/3lb4y65z24k2q https://bsky.app/profile/jakthom.bsky.social/post/3lb4y65z24... - https://skyfirehose.com https://skyfirehose.com Love this distribution pattern. Users can go to the Parquet files or attach to your "curated views" on a small DuckDB database file.