2 ms·
This is a good point, it happens because newer models are able to use larger and larger context - long range correlations in text. This amounts to finding more
by Isamu 3y ago
This is a good point, it happens because newer models are able to use larger and larger context - long range correlations in text. This amounts to finding more redundancy in the source text, driving down the estimated entropy.
Language models are related to text compression, except that the compression is lossy, not trying to be able to recreate the source text exactly but rather to be able to recreate something that is statistically similar.
- User23 3y agoHow is that not just K-complexity? A larger model is a larger interpreter. The entropy is just moving from the stream to the state machine.