pdet

3 ms·

1. 1.▲
   by pdet 1mo ago

   DuckDB uses a speculative parallel CSV parsing technique. The basic idea is that the parser speculates about the state the CSV parser is in at a random byte (e.g., whether it is inside a quoted field) and tries to figure out where the next
2. 2.▲
   by pdet 1mo ago

   I've run quite a few benchmarks on that as well, on a few different machines, and oversubscribing ASYNC threads demonstrated very little performance downside. In the end, the memory governor also keeps these threads "in check&quot
3. 3.▲
   by pdet 1mo ago

   The main reason I decided to use a beefier machine is that it gives me flexibility when benchmarking, without the need to set up different environments. The CSV data, for example, is >80 GB. We can also “scale down the machine” for exper
4. 4.▲
   Asynchronous I/O in DuckDB: Work, Thread, Work(duckdb.org)

   281 points by pdet 2mo ago|31 comments
5. 5.▲
   by pdet 5mo ago

   I think that Quack will become the primary option for a DuckLake catalog in the future, for several reasons. To list a few: 1. No type mismatches for inlining. If you use a non-DuckDB catalog, many types do not have a 1:1 mapping, which int
6. 6.▲
   DuckDB’s Vectorized, Zero-Copy Scalar Python UDFs(duckdb.org)

   4 points by pdet 3y ago|0 comments