3 ms·
There was neat post https://news.ycombinator.com/item?id=27016630 https://news.ycombinator.com/item?id=27016630 a while ago about about using sqlite on static p
by obeliskora 5y ago
There was neat post https://news.ycombinator.com/item?id=27016630 https://news.ycombinator.com/item?id=27016630 a while ago about about using sqlite on static pages with large datasets that wouldn't have to be loaded entirely. Does duckdb do something similar with arrow/parquet files or its own format?
- ankoh 5y agoYes we do! DuckDB-Wasm can read files using HTTP range requests very similar to the sql.js-httpvfs from phiresky. The blog post contains a few examples how this can be used, for example, to partially query Parquet files over the network. E.g. just visit shell.duckdb.org and enter: select * from 'https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parquet https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parqu...' limit 10;
- 1egg0myegg0 5y agoNIIIICE! Data twitter was pretty excited about that cool SQLite trick - now you can turn it up a notch!
- tomrod 5y agoIs data twitter == #datatwitter, like Econ Twitter is #econtwitter? If so, I have another cool community to follow!
- obeliskora 5y agoThat's really neat! Can you control the cache too?
- ankoh 5y agoDuckDB-Wasm uses a traditional buffer manager and evicts pages using a combination of FIFO + LRU (to distinguish sequential scans from hot pages like the Parquet metadata).
- obeliskora 5y agoIt would be really cool to load duckdb files too. sql.js-httpvfs seems convenient because it works on everything in database so you don't have to create indexes, or setup keys and constraints in the client.
- ankoh 5y agoI agree! DuckDB-Wasm can already open DuckDB database files in the browser the very same way.