4 ms·

I really like the DuckDB philosophy, and the maintainers were really responsive and helpful with my docs questions. Didn't end up using it because the compresse

by mastax 5y ago

I really like the DuckDB philosophy, and the maintainers were really responsive and helpful with my docs questions. Didn't end up using it because the compressed database files were much larger than SQLite for my data (weird use case, admittedly), and the .NET libraries were poor. I look forward to trying it out again, though.

wenc 5y ago
> the compressed database files were much larger than SQLite for my data Just curious, what format did you use? If it's Parquet, it does column compression via Snappy by default (which more optimized for speed than size). You can change the compression to ZSTD or GZIP for higher compression ratios. https://duckdb.org/docs/data/parquet https://duckdb.org/docs/data/parquet
- mastax 5y ago
  I tried all of the available compression formats in parquet, natively in duckdb, and compressing raw duckdb files as a whole. IIRC just GZIPing SQLite files beat them all. The SQLite files were much smaller uncompressed (~220mb vs 800mb) which was probably most of the difference.
  - pletnes 5y ago
    Can you still query a gzipped sqlite file? What’s the sense in gzipping it if not?
    - floydnoel 5y ago
      Do other formats (Snappy?) allow this?
      - benglish11 5y ago
        You can query snappy parquet files directly.
      - wenc 5y ago
        You can also query gzip and zstd parquet files directly.
ignoramous 5y ago
> ...the .NET libraries were poor. DuckDB's Node bindings leave a lot to be desired, as well. WASM bindings have a different API surface, too. From my reading of the code and docs, it looks like duckdb 3p bindings for anything other than python is in dire need fit and polish.
- ankoh 5y ago
  We're still on a journey to explore what APIs work best with JavaScript but the differences between WASM and Node are on purpose. DuckDB-Wasm has isolated wasm heap memory and runs in a separate web worker. That's why we serialize everything as Arrow IPC buffer and pass the ArrayBuffer through the workers message API as transferrable. On Node.js, we can interact with DuckDB much more easily and don't want to pay the price for the IPC stream every time. The truth is, we tried quite a few different APIs and this turned out the be a good trade off between efficiency and convenience. But I'm happy to discuss this further if you have any suggestions.

by mastax 5y ago

wenc 5y ago

> the compressed database files were much larger than SQLite for my data Just curious, what format did you use? If it's Parquet, it does column compression via Snappy by default (which more optimized for speed than size). You can change the compression to ZSTD or GZIP for higher compression ratios. https://duckdb.org/docs/data/parquet https://duckdb.org/docs/data/parquet

mastax 5y ago
I tried all of the available compression formats in parquet, natively in duckdb, and compressing raw duckdb files as a whole. IIRC just GZIPing SQLite files beat them all. The SQLite files were much smaller uncompressed (~220mb vs 800mb) which was probably most of the difference.
- pletnes 5y ago
  Can you still query a gzipped sqlite file? What’s the sense in gzipping it if not?
  - floydnoel 5y ago
    Do other formats (Snappy?) allow this?
    - benglish11 5y ago
      You can query snappy parquet files directly.
    - wenc 5y ago
      You can also query gzip and zstd parquet files directly.

ignoramous 5y ago

> ...the .NET libraries were poor. DuckDB's Node bindings leave a lot to be desired, as well. WASM bindings have a different API surface, too. From my reading of the code and docs, it looks like duckdb 3p bindings for anything other than python is in dire need fit and polish.

ankoh 5y ago
We're still on a journey to explore what APIs work best with JavaScript but the differences between WASM and Node are on purpose. DuckDB-Wasm has isolated wasm heap memory and runs in a separate web worker. That's why we serialize everything as Arrow IPC buffer and pass the ArrayBuffer through the workers message API as transferrable. On Node.js, we can interact with DuckDB much more easily and don't want to pay the price for the IPC stream every time. The truth is, we tried quite a few different APIs and this turned out the be a good trade off between efficiency and convenience. But I'm happy to discuss this further if you have any suggestions.