4 ms·
Impressive! But an order of a magnitude slower then non-LLM compression techniques. I try to compare with the results from the top 9 of this enwik8 compression
by fsiefken 3y ago
Impressive! But an order of a magnitude slower then non-LLM compression techniques.
I try to compare with the results from the top 9 of this enwik8 compression test by Matt Mahoney:
https://www.mattmahoney.net/dc/text.html https://www.mattmahoney.net/dc/text.html
The durilca compressor by Dmitry Shkarin (what happened to him?) has been the fastest compressor in the top since it's debut in 2006.
The original size, enwik8, is 10^8 bytes. The compressed size is the compressend enwik8 plus the size of a zip archive containing the decompressor.
The bpb value of rwkv_430M on enwik8 is 0.948 bpb, the 7B models will be lower, perhaps around 0.800. So if my calculations are correct, the LLM's can perform 50% better then the best performing conventional compressors excluding the zipped LLM decompressor code (I am unsure about the size).
The bpb ratio for each existing program can be calculated as:
bpb ratio=(Original size (enwik9)Total size (enwik9+prog) )×8
nncp v3.1:
Given compressed size is not provided, we cannot calculate bpb for nncp v3.1 using enwik8.
cmix v19:
bpb=(14,837,987+223,485108)×8bpb=(10814,837,987+223,485 )×8
bpb≈1.205
tensorflow-compress v4:
bpb=(15,905,037+55,283108)×8bpb=(10815,905,037+55,283 )×8
bpb≈1.272
cmix-hp 10 Jun 2021:
Given compressed size is not provided, we cannot calculate bpb for cmix-hp using enwik8.
fast-cmix:
Given compressed size is not provided, we cannot calculate bpb for fast-cmix using enwik8.
starlit 31 May 2021:
bpb=(15,215,107108)×8bpb=(10815,215,107 )×8
bpb≈1.217
phda9 1.8:
bpb=(15,010,414+42,944108)×8bpb=(10815,010,414+42,944 )×8
bpb≈1.205
paq8px_v206fix1:
bpb=(15,849,084+402,949108)×8bpb=(10815,849,084+402,949 )×8
bpb≈1.265
durilca'kingsize:
bpb=(16,209,167+407,477108)×8bpb=(10816,209,167+407,477 )×8
bpb≈1.317
- londons_explore 3y ago> The bpb value of rwkv_430M on enwik8 is 0.948 bpb This is an unfair test, because the rwkv_430M model was almost certainly trained on wikipedia. Testing using training data is a big no-no in the ML world - you get unrealistically good results. To test this properly, it needs to be run on a bit of newly written text which is nowhere on the internet. Obviously writing 100 MB of human written text which is not already on the internet is rather a big task...
- etiam 3y agoNewly declassified material maybe? Or just newly digitized?
- ndriscoll 3y agoIt's fair because the model itself is included in the compressed size. You're not trying to extrapolate predictions. It makes perfect sense for compression "at rest" applications to intentionally overfit on the data you're trying to compress if that gives a smaller total size(model ++ compressed text). If you were going to standardize on a foundational model that gets delivered to billions of devices to be used in network transfers (where you just transfer the compressed text and assume the other side already has the model), then it makes sense to optimize for generalizability. In that case it would make sense to exclude the model itself from the compressed size. Basically like how brotli gets to "cheat" by including a 120KB pre-defined dictionary. Edit: Actually that can't be right. The numbers on https://www.mattmahoney.net/dc/text.html https://www.mattmahoney.net/dc/text.html use that methodology, but the numbers in OP here can't be, since a zip compressed version of rwkv_430M is ~650 MB, so obviously it's not being included or Alice in Wonderland would have an abysmal ratio.