4 ms·

No, I just went straight to sqlite. What is duckdb?

cess11 10mo ago
It is very similar to SQLite in that it can run in-process and store its data as a file. It's different in that it is tailored to analytics, among other things storage is columnar, and it can run off some common data analytics file formats.
fsiefken 10mo ago
DuckDB is an open-source column-oriented Relational Database Management System (RDBMS). It's designed to provide high performance on complex queries against large databases in embedded configuration. It has transparent compression built-in and has support for natural language queries. https://buckenhofer.com/2025/11/agentic-ai-with-duckdb-and-smolagents-natural-language-queries-for-analytics/ https://buckenhofer.com/2025/11/agentic-ai-with-duckdb-and-s... "DICT FSST (Dictionary FSST) represents a hybrid compression technique that combines the benefits of Dictionary Encoding with the string-level compression capabilities of FSST. This approach was implemented and integrated into DuckDB as part of ongoing efforts to optimize string storage and processing performance." https://homepages.cwi.nl/~boncz/msc/2025-YanLannaAlexandre.pdf https://homepages.cwi.nl/~boncz/msc/2025-YanLannaAlexandre.p...
simonw 10mo ago
One interesting feature of DuckDB is that it can run queries against HTTP ranges of a static file hosted via HTTPS, and there's an official WebAssembly build of it that can do that same trick. So you can dump e.g. all of Hacker News in a single multi-GB Parquet file somewhere and build a client-side JavaScript application that can run queries against that without having to fetch the whole thing. You can run searches on https://lil.law.harvard.edu/data-gov-archive/ https://lil.law.harvard.edu/data-gov-archive/ and watch the network panel to see DuckDB in action.
- keepamovin 10mo ago
  In that case, then using duckdb might be even more performant than using what we’re doing here. It would be an interesting experiment to add the duckdb hackend
1vuio0pswjnm7 10mo ago
"What is duckdb?" duckdb is a 45M dynamically-linked binary (amd64) sqlite3 1.7M static binary (amd64) DuckDB is a 6yr-old project SQLite is a 25yr-old project
- 1vuio0pswjnm7 10mo ago
  I like SQLite

cess11 10mo ago

It is very similar to SQLite in that it can run in-process and store its data as a file. It's different in that it is tailored to analytics, among other things storage is columnar, and it can run off some common data analytics file formats.

fsiefken 10mo ago

DuckDB is an open-source column-oriented Relational Database Management System (RDBMS). It's designed to provide high performance on complex queries against large databases in embedded configuration. It has transparent compression built-in and has support for natural language queries. https://buckenhofer.com/2025/11/agentic-ai-with-duckdb-and-smolagents-natural-language-queries-for-analytics/ https://buckenhofer.com/2025/11/agentic-ai-with-duckdb-and-s... "DICT FSST (Dictionary FSST) represents a hybrid compression technique that combines the benefits of Dictionary Encoding with the string-level compression capabilities of FSST. This approach was implemented and integrated into DuckDB as part of ongoing efforts to optimize string storage and processing performance." https://homepages.cwi.nl/~boncz/msc/2025-YanLannaAlexandre.pdf https://homepages.cwi.nl/~boncz/msc/2025-YanLannaAlexandre.p...

simonw 10mo ago

One interesting feature of DuckDB is that it can run queries against HTTP ranges of a static file hosted via HTTPS, and there's an official WebAssembly build of it that can do that same trick. So you can dump e.g. all of Hacker News in a single multi-GB Parquet file somewhere and build a client-side JavaScript application that can run queries against that without having to fetch the whole thing. You can run searches on https://lil.law.harvard.edu/data-gov-archive/ https://lil.law.harvard.edu/data-gov-archive/ and watch the network panel to see DuckDB in action.

keepamovin 10mo ago
In that case, then using duckdb might be even more performant than using what we’re doing here. It would be an interesting experiment to add the duckdb hackend

1vuio0pswjnm7 10mo ago

"What is duckdb?" duckdb is a 45M dynamically-linked binary (amd64) sqlite3 1.7M static binary (amd64) DuckDB is a 6yr-old project SQLite is a 25yr-old project

1vuio0pswjnm7 10mo ago
I like SQLite