lnkuiper

4 ms·

1.
by lnkuiper 3y ago
Stabilizing storage is high priority for us! You won’t notice anything new while using DuckDB, but we’ve actually rewritten our entire serialization framework in this release. This was a big step towards stable storage
2.
by lnkuiper 3y ago
We plan to update our results on ClickBench at some point, as we’ve made some improvements there in this release. I’d expect performance to be similar
3.
by lnkuiper 4y ago
This is very true. DuckDB does not support JSON because it’s a good tabular format, but because JSON is ubiquitous, and there are many use cases where querying JSON dumps for analytics is useful.
4.
by lnkuiper 4y ago
DuckDB will use multiple threads for reading the same file. Each thread will read different parts of the file, but the output will be in the order that the file came in due to DuckDB’s order preserving parallelism.
5.
by lnkuiper 5y ago
There are plenty of TopN sorts in TPC-DS, but not many (if any) regular sorts, so no. There are order-dependent window functions in there though, which did show a speed up.
6.
by lnkuiper 5y ago
You are right, we could probably re-run SQLite purely in memory, but only because macos dynamically allocates additional swap. However, I would not expect much improved performance, because I do not believe that SQLite has a different sorti
7.
by lnkuiper 5y ago
Exactly. Even when it’s inlined, having a comparator with if/else is much, much slower than a single memcmp
8.
by lnkuiper 5y ago
Customer fits in memory, whereas catalog_sales does not. We chose to remove SQLite from the results because it was so much slower. The plots are much less readable when they are stretched out by something that is slower by an order of magni
9.
by lnkuiper 5y ago
I'm not sure if I understand, but we did not use a memory map (mmap), but rather blocks of memory that are explicitly (un)loaded by the buffer manager. The M1 + SSD performs really well here. We tried to an external sort experiment on
10.
by lnkuiper 5y ago
We wanted to use the same setup for all experiments, so we had to choose for an on-disk DB for SQLite, because TPC-DS SF100 catalog_sales does not fit in 16GB memory.