5 ms·

Given SQLite is already distributed on basically every OS,is amazingly rock solid, I’d ask instead any advantage to using Apache Arrow/duckdb? I don’t have grea

• jitl 2y ago
  DuckDB can have massive performance advantage over SQLite for things like aggregates on wide tables or or filters on unindexed columns, with advantage growing with the size of the table. We tried a few head to heads between the two and while we ultimately went with SQLite, we may go back and double write everything to DuckDB to get better performance for those kinds of queries on the same data set. Duckdb can also read / import SQLite natively so we could ingest lazily from sqlite disk -> duckdb. Biggest issue with duckdb is some memory leaks and crashes for the nodejs driver. Seems not production ready.
  • datadrivenangel 2y ago
    They did just release DuckDB version 1.0, so it should be getting much more reliable and stable.
    • jitl 2y ago
      They said 1.0 means “we aren’t going to break the disk format backwards compatibility anymore” other than that, it’s just a number and doesn’t have any power to make the software more reliable. It really depends on their priorities how much they’ll actually stabilize my use-case. I’ll probably check back in a year and see if things improved.