4 ms·

I wonder how much smaller it could get with some compression. You could probably encode "This website hijacks the scrollbar and I don't like it" comments into j

by yread 9mo ago

I wonder how much smaller it could get with some compression. You could probably encode "This website hijacks the scrollbar and I don't like it" comments into just a few bits.

hamburglar 9mo ago
It might be a neat experiment to use ai to produce canonicalized paraphrasings of HN arguments so they could be compared directly and compress well.
jacquesm 9mo ago
That's at least 45%, then you can leave out all of my comments and you're left with only 5!
Rendello 9mo ago
The hard-coded dictionary wouldn't be much stranger than Brotli's: https://news.ycombinator.com/item?id=27160590 https://news.ycombinator.com/item?id=27160590
- maxbond 9mo ago
  You can use a BPE variant like SentencePiece to identify these patterns rather than hard coding them.
rossant 9mo ago
Guilty.
deleted 9mo ago
[deleted]
keepamovin 9mo ago
Dear it is already compressed using G zip – nine for every SQLlight shard and manifest 22 GB is uncompressed and compressed the entire things about 9 GB