3 ms·
Just to give a comparisson, this website can compress to about 15% size, the current state of the art compressing English text (for a large chunk of wikipedia)
by CJefferson 6y ago
Just to give a comparisson, this website can compress to about 15% size, the current state of the art compressing English text (for a large chunk of wikipedia) is about the same http://prize.hutter1.net/ http://prize.hutter1.net/ (15%).
The hutter prize has the advantage it gets more input, but the disadvantage that you must include the uncompression program (GPT2 is huge, so you wouldn't want to include it).
- nullc 6y agoIt's closer to 11% with the large model. I assume it would be the hutter prize leader if the prizes' challenge were large enough to amortize the model.
- CJefferson 6y agoThe entire of wikipedia's English text is only about 15GB, and GPT-2 is 5GB -- so it couldn't "win". Also, other techniques might get better as size increases.
- xamuel 6y agoEven worse, the Hutter challenge is based on just a 1GB fragment of (one particular snapshot of) English Wikipedia. Further highlighting what has been said elsewhere in this thread, that as far as compression goes this is basically just a variation on compression-via-giant-dictionary
- nullc 6y agoThus the "if the prizes' challenge were large enough". :)
- moultano 6y agoI'd bet that a shallow transformer that was distilled on gpt2, possibly with quantized weights, would easily win. GPT2 isn't optimized for size at all, but there are tons of ways to do so.