3 ms·
> the compressed database files were much larger than SQLite for my data Just curious, what format did you use? If it's Parquet, it does column compression vi
by wenc 5y ago
> the compressed database files were much larger than SQLite for my data
Just curious, what format did you use?
If it's Parquet, it does column compression via Snappy by default (which more optimized for speed than size).
You can change the compression to ZSTD or GZIP for higher compression ratios.
https://duckdb.org/docs/data/parquet https://duckdb.org/docs/data/parquet
- mastax 5y agoI tried all of the available compression formats in parquet, natively in duckdb, and compressing raw duckdb files as a whole. IIRC just GZIPing SQLite files beat them all. The SQLite files were much smaller uncompressed (~220mb vs 800mb) which was probably most of the difference.
- pletnes 5y agoCan you still query a gzipped sqlite file? What’s the sense in gzipping it if not?
- floydnoel 5y agoDo other formats (Snappy?) allow this?
- benglish11 5y agoYou can query snappy parquet files directly.
- wenc 5y agoYou can also query gzip and zstd parquet files directly.