3 ms·
Wow, that zip file of JSON is 6.5MB, but when you decompress it you get a 382.MB CvrExport.json file! I guess repetitive JSON compresses REALLY well. I tried
by simonw 4y ago
Wow, that zip file of JSON is 6.5MB, but when you decompress it you get a 382.MB CvrExport.json file!
I guess repetitive JSON compresses REALLY well.
I tried playing with it in SQLite but that CvrExport JSON needs some extra work to break it up into separate tables for it to be readable.
Here's a demo that shows the other tables but ignores the 300MB+ one: https://lite.datasette.io/?sql=https://gist.githubusercontent.com/simonw/0e6901974a14ab7d56c2746a04d72c8c/raw/49f6d766d242425ce42c3dee646b48396f481f29/database.sql https://lite.datasette.io/?sql=https://gist.githubuserconten...
- dlbucci 4y agoA year ago, I was making a game for JS13K, where the goal is to make a game that fits in a zip file under 13,312 bytes. I had a two spots that were using the exact same copy-pasted chunk of code, so I figured I'd refactor it into a function to save space, but it actually _increased_ the size of the final zip when I did so. Turns out zip files compress repetitions of the same exact text _very_ well.
- dekhn 4y agoYes, the algorithm zip uses in this case uses backrefs "LZ77 algorithms achieve compression by replacing repeated occurrences of data with references to a single copy of that data existing earlier in the uncompressed data stream. A match is encoded by a pair of numbers called a length-distance pair, which is equivalent to the statement "each of the next length characters is equal to the characters exactly distance characters behind it in the uncompressed stream". (The distance is sometimes called the offset instead.)" This is a different technique from entropy coding, which gets its improvements by allocating fewer bits for more frequent symbols. But most modern compresses uses a mix, for example gzip uses DEFLATE, which is a combination of literal backrefs and dynamic and static Huffman tables. and Yes, JSON is usually absurdly compressible.
- yyyk 4y ago>Wow, that zip file of JSON is 6.5MB, but when you decompress it you get a 382.MB CvrExport.json file! Not only is it so small, but it's easy to improve while remaining compatible with zip: Infozip -9 -> 5693Kb zopfli -> 5437Kb