5 ms·
I spent a while looking at this today. It's really interesting. It's not based on SQLite at all (except for borrowing the SQLite shell implementation) but it l
by simonw 6y ago
I spent a while looking at this today. It's really interesting.
It's not based on SQLite at all (except for borrowing the SQLite shell implementation) but it looks very much like SQLite, in particular:
- It's designed to work as an embedded library, eliminating the network overhead you usually get when talking to a database
- Each database is a single file on disk
- It ships as an "amalgamation" build - a single giant C++ file (SQLite is a single giant C file)
Impressively, if you run "pip install duckdb" it Just Works - you can then "import duckdb" and start using it, with an interface that looks very similar to the sqlite3 module that ships with Python.
The key reason this exists is that it's a column store, with vectorized operations across columns - making it ideal for analytical workloads. This blog entry has some benchmarks that illustrate how well it works in that regard: https://uwekorn.com/2019/10/19/taking-duckdb-for-a-spin.html https://uwekorn.com/2019/10/19/taking-duckdb-for-a-spin.html
It's also backed up by some strong computer science. It's by the academic researchers behind MonetDB and includes implementations of a bunch of interesting papers: https://duckdb.org/docs/why_duckdb#standing-on-the-shoulders-of-giants https://duckdb.org/docs/why_duckdb#standing-on-the-shoulders...
It's a really interesting piece of software, and unlike many other "new databases" it feels like it fills a very genuine gap in my toolbox.
- haggy 6y agoI really appreciate the breakdown here but this comment smells like a giant plant for DuckDB.
- gregsadetsky 6y agoI don’t think that it’s reasonable to assume that Simon would risk his reputation by coming here to astroturf.
- Smerity 6y agoA giant plant as in someone from their team planning this out and then planting this as a comment..? simonw wrote Datasette[1] which makes extensive use of SQLite and acts as a viewer, allowing you to create interactive websites and APIs from arbitrary databases. He'd be a very long term plant and it instead seems far more likely he's interested in the possibilities of DuckDB compatibility within the context of Datasette and other similar projects. [1]: https://github.com/simonw/datasette https://github.com/simonw/datasette
- simonw 6y agoI actually started looking at it today because someone was asking about it on the Datasette discussions board: https://github.com/simonw/datasette/discussions/968 https://github.com/simonw/datasette/discussions/968
- Abishek_Muthian 6y ago> - Each database is a single file on disk Are these architecture independent? i.e. If I create the database on x86_64 and move it to ARM64; would it work seamlessly?
- mytherin 6y agoOne of the authors here - portability of the storage is indeed one of our goals. We test that the same file can be loaded everywhere on different systems, including between ARM and x86. I should mention the storage is still very much a work-in-progress. We are actively working on redesigning the storage to add compression and other extra features, meaning the storage format can change and be incompatible between different DuckDB versions. We plan to stabilize the storage format with V1.0, after which every subsequent DuckDB version should be able to read database files created by previous versions.
- Abishek_Muthian 6y agoThanks for the confirmation on portability, I didn't see Go support in the home page; Is there an official Go driver support planned as well?
- mytherin 6y agoNeither Hannes nor me are very experienced with Go, so we are not the most suitable to do that. We do very much welcome external contributions, and someone did make a WIP go driver [1] in the past. We also have a sqlite3 compatible C API that can be used to use DuckDB. This is actually how we use the sqlite3 shell: we directly use the sqlite shell but bind it to the duckdb library using this API. It might be possible to use the Go SQLite driver in a similar manner to talk to DuckDB. [1] https://github.com/marcboeker/go-duckdb https://github.com/marcboeker/go-duckdb
- sgt 6y agoAnd Dart bindings? That could be really interesting for Flutter developers.
- 6y ago
- tmpz22 6y agoHow might one go about using this in a language that’s not supported in the documentation such as golang?
- mytherin 6y agoGenerally you would need to write bindings for the system. Experimental DuckDB bindings for Go do exist [1], however, they are rather old and might not work anymore. We also have JDBC support, which might help for languages that know how to use that protocol. ODBC support is not implemented yet, but also planned. We also have a SQLite-compatible C API [2], that can potentially be used to use an existing SQLite driver that uses the SQLite C API. [1] https://github.com/marcboeker/go-duckdb https://github.com/marcboeker/go-duckdb [2] https://github.com/cwida/duckdb/blob/master/tools/sqlite3_api_wrapper/sqlite3_api_wrapper.cpp https://github.com/cwida/duckdb/blob/master/tools/sqlite3_ap...
- gijzelaerr 6y agoAnd MonetDB took some inspiration back from DuckDB to develop an embedded version also: https://github.com/MonetDBSolutions/MonetDBe-Python/ https://github.com/MonetDBSolutions/MonetDBe-Python/ Disclaimer: i'm working on this.
- Firerouge 6y agoCould you elaborate on some of the similarities and differences between DuckDB and MonetDB?
- mkersten 6y agoDisclaimer: i am working on MonetDB and I established the CWI database architectures group DuckDB is designed as an experimental system after heavy exposure to the technniques deployed in MonetDB (open-source), Hyper, and Vectorwise. The properties of the embedded version of MonetDB can be found here https://monetdbe.readthedocs.io/en/latest/introduction.html# https://monetdbe.readthedocs.io/en/latest/introduction.html# Some difference between MonetDB and DuckDB can be found here https://monetdbe.readthedocs.io/en/latest/migrations.html https://monetdbe.readthedocs.io/en/latest/migrations.html and the blogpost mentioned above is covered in https://twitter.com/MonetDB/status/1282412295235280901?s=20 https://twitter.com/MonetDB/status/1282412295235280901?s=20
- deleted 6y ago[deleted]
- brian_herman 6y agoYes, their testing folder is 26mb while their source code of the database is only 5.6mb. It looks like they took the same approach to testing as sqlite also.
- webmobdev 6y agoIf everything you said is accurate, somebody should make another pitch for WebSQL ( https://en.wikipedia.org/wiki/Web_SQL_Database https://en.wikipedia.org/wiki/Web_SQL_Database ) with a custom version of DuckDb! Every browser maker was interested in implementing it but the W3C couldn't go ahead with it because everyone chose to implement it using SQLite, where as W3C required more than one db back-end implementation to move forward.
- wffurr 6y agoHaving sqlite included in all browsers with standard bindings would have been great.
- jansommer 6y agoIt would! But it would also be a security nightmare. Exploit SQLite and you could use it in Safari, Chrome and Firefox - everybody could be affected.
- andyfleming 6y agoHow would that be different from someone editing something in window.localStorage?
- jansommer 6y agoIf that's all you need, it wouldn't make sense to include SQLite in the browser. If you're going to include a relational database you might as well use it
- CRConrad 6y agoI think that meant: How would that be more of a security nightmare than some malicious site editing your browser's whatever.localstorage?
- hajile 6y ago