2 ms·
The deep connections between compression and prediction are not always obvious to those not in the field. To illustrate just how much they are the same, here i
by Der_Einzige 4y ago
The deep connections between compression and prediction are not always obvious to those not in the field.
To illustrate just how much they are the same, here is (at one point SOTA) lossless text compression with GPT-2
https://bellard.org/libnc/gpt2tc.html https://bellard.org/libnc/gpt2tc.html
- mjburgess 4y agoWell.. I think they're over-stated because of the current wave of AI basically only having naïve compression as its tool. Is the concept `addition` a compression of the space `(Int, Int, Int)` ? If you want to say it is, OK for some definition of compression. But that compression isnt "mere" in the modern AI sense, it's "exponentially dense". In that my concept `addition` can generate arbitrarily large amounts of that decompressed space, which is infinite in size. There's a kind of trick played in the marketing here: since NNs compress, and since learning "can be seen as compression", NNs learn... no, because NNs aren't "exponentially dense", they're "exponentially large" -- I'd claim, the opposite of learning!