3 ms·
Another interesting facet is that, according to some schools of thought, compression and AI are equivalent problems: The better you understand something, the le
by MauranKilom 3y ago
Another interesting facet is that, according to some schools of thought, compression and AI are equivalent problems: The better you understand something, the less you need to memorize to reproduce it.
https://en.wikipedia.org/wiki/Hutter_Prize https://en.wikipedia.org/wiki/Hutter_Prize (and https://en.wikipedia.org/wiki/AIXI https://en.wikipedia.org/wiki/AIXI)
Of course, large language models are (by definition) currently going the other direction, but it remains to be seen whether that leads to artificial intelligence (whatever that ends up meaning).
- brookst 3y agoMaybe a naive question: are LLMs really going the other way? My intuition is that the model weights are much smaller than the information encoded in them.
- floobertoober 3y ago> Of course, large language models are (by definition) currently going the other direction ... How so? Aren't the networks' weights orders of magnitude smaller than the training data?
- govg 3y agoI interpreted that statement as saying the current practice is to make LLMs larger and larger (so they effectively memorize more and more data) to make them more powerful, but from the perspective of information theory, if models were powerful and "understanding", then models could stay the same size and become more and more powerful as they get increasingly better at compressing the available information. I am not sure if this interpretation was what was meant though.
- deleted 3y ago[deleted]
- karpierz 3y agoI believe the parent poster's point is: LLMs are more effective when they use more memory, meaning the less they are forced to compress the training data, the better they perform.
- MauranKilom 3y agoBut they don't losslessly recreate the training data.
- ta93754829 3y agoOnce ChatGPT was trained on "the internet of 2021", how big is the remaining model? Is ChatGPT effectively a compressed version of the internet?