7 ms·
This looks fantastic! I’ve been waiting for something like this. My only gripe is that data loaders don’t seem to support Parquet files, which is really annoyi
by orf 3y ago
This looks fantastic! I’ve been waiting for something like this.
My only gripe is that data loaders don’t seem to support Parquet files, which is really annoying.
There’s an interesting possibility here where you can have large datasets in Parquet, exposed via HTTP, whilst being generated at build time with all the benefits that gives you (being able to read only specific columns, filtering via row group statistics etc). Not dissimilar to the “SQLite over http” WASM demo I guess.
Because right now I need to take my large, nicely compressed dataset and export it as either a CSV or a zip file? And then the browser needs the entire thing, even if I’m just viewing a subset of the data? Which is much bigger and much slower than it needs to be.
- simonw 3y agoThere's a parquet example here: https://github.com/observablehq/framework/blob/main/examples/api/docs/index.md?plain=1 https://github.com/observablehq/framework/blob/main/examples... Using data from here: https://github.com/observablehq/framework/tree/main/examples/api/docs/data https://github.com/observablehq/framework/tree/main/examples... Rendered version here: https://observablehq.com/framework/examples/api/ https://observablehq.com/framework/examples/api/ If you want to run a Data loader that outputs to parquet there are plenty of ways to do that - I would suggest a Bash or Python script that wraps DuckDB.
- mbostock 3y agoWe do support it. (And use it!) Please see: https://observablehq.com/framework/lib/arrow#apache-parquet https://observablehq.com/framework/lib/arrow#apache-parquet
- orf 3y agoAhh, amazing! That’s not entirely clear from the data loading docs[1], which when I read it seemed only focused around CSV and JSON. 1. https://observablehq.com/framework/loaders https://observablehq.com/framework/loaders