3 ms·
> That means we won't get a winner based on GPT3 anytime soon. 350+ GB of weights is a lot to overcome :) You're right, and in-there lays the crucial differenc
by visarga 5y ago
> That means we won't get a winner based on GPT3 anytime soon. 350+ GB of weights is a lot to overcome :)
You're right, and in-there lays the crucial difference between compression and language modelling. Models are concerned with having a good representation for both past and future distributions, while compressors only care about the past. Models support many tasks while compressors are just for input replication.
- vintermann 5y agoNo, I don't quite agree. All compressors have a model, at least the general purpose ones. If you're writing self-extracting compression for your 8-bit demo, maybe you only care about the data you have right in front of you... But for anything general purpose, from DEFLATE to paq, there's an implicit, fixed model of future data in there (even if we sometimes prefer to see it as an "adaptive" model). A good illustration of this is that you can quite easily use a general purpose compression algorithm for classification, and get surprisingly good performance.