4 ms·
Show HN: Crystal Unified – Search compressed archives without decompressing
Crystal Unified lets you grep through compressed archives (700MB logs, etc.) in milliseconds without extracting.
Other features:
- Binary delta patching for firmware/OTA updates
- Block-level random access
- 10x+ compression on logs
- Genomic data: 4:1 on 2-bit encoded DNA, 111,000:1 with reference compression
- Parallel compression, 1GB/s+ decompression
https://github.com/powerhubinc/crystal-unified-public https://github.com/powerhubinc/crystal-unified-public
- theamk 10mo agoProtip: if what's shown in README is your "Traditional workflow" # Traditional workflow: decompress, grep, wait, clean up zstd -d huge_logs.zst && grep "error" huge_logs && rm huge_logs STOP NOW! There is never need to decompress files to disk, every compressor supports streaming decompression just fine: # Traditional workflow: decompress + grep zstd -dc huge_logs.zst | grep "error" (and as a bonus, no need to install any third-party tools)
- danebl 10mo agoYou are right, that was poor wording. If we compare apples to apples: Traditional: must decompress entire file (even when streaming) - zstd -dc huge_logs.zst | grep "error" # 709 MB decompressed through memory Crystal: indexed search, skip to matches - cuz search huge_logs.cuz "error" # Only decompress matching blocks Thanks for pointing this up. We will update the readme file.