3 ms·

Anyone looked at DuckDB [1]? Like SQLite, it's an embedded relational database with a C API. In fact, it has a wrapper which presents the same API as SQLite [2]

by twic 6y ago

Anyone looked at DuckDB [1]? Like SQLite, it's an embedded relational database with a C API. In fact, it has a wrapper which presents the same API as SQLite [2]. But it's optimised for analytical rather than transactional work - a few huge queries rather than lots of little reads and writes. [1] https://duckdb.org/ https://duckdb.org/ [2] https://github.com/cwida/duckdb/tree/master/tools/sqlite3_api_wrapper https://github.com/cwida/duckdb/tree/master/tools/sqlite3_ap...

iagovar 6y ago
So I guess I can use SQLite connectors or GUI to play with it? I was just looking for this. Not a dev so I'm a bit dumb with this stuff, but I was definitely looking for pivoting stuff and such, which requires a lot of extra steps with sqlite.
mytherin 6y ago
DuckDB offers a SQLite compatible interface; we use it extensively ourselves as it allows us to use the SQLite shell directly. The internal design of DuckDB, however, is different and not directly related to SQLite. As you mentioned, it is entirely optimized for analytical workloads whereas SQLite is optimized for point-queries and point-updates. The main things we share is that we also have a single-file storage format and offer an amalgamation (i.e. single-source file compilation, duckdb.cpp and duckdb.hpp).
st1ck 6y ago
For analytical workloads, DuckDB may be better than SQLite, but it's just incomparably slower than e.g. ClickHouse (which is admittedly not embedded).
- mytherin 6y ago
  The main reason for that is that DuckDB is currently single-threaded, we are currently actively working on parallelism for DuckDB which will resolve this :)
  - st1ck 6y ago
    I think there are also tons of other optimizations currently missing in DuckDB, vectorization probably the most important.
    - mytherin 6y ago
      Very true, DuckDB is very young and we have so far focused on correctness over performance. As a result we are still missing several optimizations that more mature systems do have, for example, proper cardinality estimation. We are working on adding those as well.

by twic 6y ago

iagovar 6y ago

So I guess I can use SQLite connectors or GUI to play with it? I was just looking for this. Not a dev so I'm a bit dumb with this stuff, but I was definitely looking for pivoting stuff and such, which requires a lot of extra steps with sqlite.

mytherin 6y ago

DuckDB offers a SQLite compatible interface; we use it extensively ourselves as it allows us to use the SQLite shell directly. The internal design of DuckDB, however, is different and not directly related to SQLite. As you mentioned, it is entirely optimized for analytical workloads whereas SQLite is optimized for point-queries and point-updates. The main things we share is that we also have a single-file storage format and offer an amalgamation (i.e. single-source file compilation, duckdb.cpp and duckdb.hpp).

st1ck 6y ago

For analytical workloads, DuckDB may be better than SQLite, but it's just incomparably slower than e.g. ClickHouse (which is admittedly not embedded).

mytherin 6y ago
The main reason for that is that DuckDB is currently single-threaded, we are currently actively working on parallelism for DuckDB which will resolve this :)
- st1ck 6y ago
  I think there are also tons of other optimizations currently missing in DuckDB, vectorization probably the most important.
  - mytherin 6y ago
    Very true, DuckDB is very young and we have so far focused on correctness over performance. As a result we are still missing several optimizations that more mature systems do have, for example, proper cardinality estimation. We are working on adding those as well.