4 ms·
In combination with SQLite to allow fast random access (query) and compression: https://github.com/phiresky/sqlite-zstd https://github.com/phiresky/sqlite-zstd
by punnerud 4y ago
In combination with SQLite to allow fast random access (query) and compression:
https://github.com/phiresky/sqlite-zstd https://github.com/phiresky/sqlite-zstd
- dang 4y agoDiscussed a few months ago: Show HN: Reduce SQLite database size by up to 80% with transparent compression - https://news.ycombinator.com/item?id=32303762 https://news.ycombinator.com/item?id=32303762 - Aug 2022 (90 comments) sqlite-zstd: Transparent dictionary-based row-level compression for SQLite - https://news.ycombinator.com/item?id=32297193 https://news.ycombinator.com/item?id=32297193 - July 2022 (1 comment)
- rkagerer 4y agoLooks awesome and I don't want to detract. But a few things from the documentation to be aware of: - Q: Is this library production ready? A: I wouldn't trust it with my data (yet). Make sure you have backups of everything. - usual zstd checksum is stripped from compressed data - columns you want to index can't be compressed - works best with rows containing less than ~30kB of data
- deleted 4y ago[deleted]
- charcircuit 4y agoFrom the readme Zstandard is currently deployed within Facebook. It is used continuously to compress large amounts of data in multiple formats and use cases. Zstandard is considered safe for production environments.
- asveikau 4y agoThat's a statement about the block compression algorithm. The part that is claimed as not production ready is the third party sqlite integration. Skimming the code I can see why. It does some pretty gnarly things involving altering the schema of all your tables. Not to say it doesn't work or isn't good, but, it seems like it really does a lot of intrusive, potentially failure prone stuff.