3 ms·
This is a relatively similar architecture to that we are building at Evidence.dev (open-source data viz framework) Architecture: https://evidence.dev/blog/why-
by archiewood 3y ago
This is a relatively similar architecture to that we are building at Evidence.dev (open-source data viz framework)
Architecture: https://evidence.dev/blog/why-we-built-usql/ https://evidence.dev/blog/why-we-built-usql/
1. Query SQL databases, APIs, or local data (eg CSV)
2. Compile all the data sources into Parquet files
3. DuckDB-WASM in the browser that allows you to aggregate across sources
4. Users write code in DuckDB SQL and Markdown, enriched with viz components (built in Svelte)
Some things that we have learned in the process:
- It can be pretty performant up to about 20M rows of data in the parquet files,
- Above a certain level, for speed, it's helpful to sort your data in your parquet files to take advantage of DuckDB's predicate pushdown
- It's helpful to map DB types into a smaller set of Arrow types when you convert to Parquet - otherwise you have to consider a lot of different cases in the browser when you render in JS
- DuckDB-WASM still has some rough edges, though is improving fast
- tobilg 3y agoI love Evidence.dev, it’s a great tool!
- archiewood 3y agoWe're really excited to see all the stuff built on DuckDB-WASM too! I love how fast your workbench runs, and how effortlessly it renders 60k rows in the browser Some UX feedback, if you're open to it: - It was initially unintiutive for me that I needed to highlight a whole SQL statement to get Ctrl-Enter to run it. Also a prompt that this was the correct shortcut would help! - I dragged in a CSV file, and the name was too long to show up in your table explorer, so I couldn't tell what the table name had been called (it turned out I needed to `select * from table_name.csv` - the csv postfix was unexpected - The CSV file had headers, but they were not auto detected - would be good to be able to configure this
- tobilg 3y agoThanks for the feedback! Regarding the run query shortcut, if you open the page, there‘s a comment banner on the top of the editor that mentions it. It‘s not the first time that I hear/read that people overlooked it though. What would be a more intuitive way from your POV? I‘ll look into the horizontal scrolling/CSV header issues, thanks!
- nicoritschel 3y agoHi Tobi; Hi Archie, we all know each other from Twitter. Check out what Postico does; it just highlights the current statement under the cursor by default and that's what gets run by default. Can't resist imposing this subtle SQL UX on everyone because it's my favorite (for like 10 years now).
- tobilg 3y agoThanks Nico, a small world :-) I‘ll look into this as fallback method. I don’t like Monaco Editor‘s handling of selections and cursors, it’s quite complicated, but this should be possible to implement…
- archiewood 3y agoAha I did not notice that prompt. By default my brain is trained to ignore things that are commented out in code I think I would execute the SQL query that the cursor was within. Ie anything until the next ;
- tobilg 3y agoThanks, I‘ll look into that!
- deleted 3y ago[deleted]