2 ms·
> The bpb value of rwkv_430M on enwik8 is 0.948 bpb This is an unfair test, because the rwkv_430M model was almost certainly trained on wikipedia. Testing us
by londons_explore 3y ago
> The bpb value of rwkv_430M on enwik8 is 0.948 bpb
This is an unfair test, because the rwkv_430M model was almost certainly trained on wikipedia. Testing using training data is a big no-no in the ML world - you get unrealistically good results.
To test this properly, it needs to be run on a bit of newly written text which is nowhere on the internet. Obviously writing 100 MB of human written text which is not already on the internet is rather a big task...
- etiam 3y agoNewly declassified material maybe? Or just newly digitized?
- ndriscoll 3y agoIt's fair because the model itself is included in the compressed size. You're not trying to extrapolate predictions. It makes perfect sense for compression "at rest" applications to intentionally overfit on the data you're trying to compress if that gives a smaller total size(model ++ compressed text). If you were going to standardize on a foundational model that gets delivered to billions of devices to be used in network transfers (where you just transfer the compressed text and assume the other side already has the model), then it makes sense to optimize for generalizability. In that case it would make sense to exclude the model itself from the compressed size. Basically like how brotli gets to "cheat" by including a 120KB pre-defined dictionary. Edit: Actually that can't be right. The numbers on https://www.mattmahoney.net/dc/text.html https://www.mattmahoney.net/dc/text.html use that methodology, but the numbers in OP here can't be, since a zip compressed version of rwkv_430M is ~650 MB, so obviously it's not being included or Alice in Wonderland would have an abysmal ratio.