3 ms·

What's funny about this is as we get better at text prediction, we revise our entropy estimations downward. The last I heard was 2016, at around 6 bits/word of

by kqr 4y ago

What's funny about this is as we get better at text prediction, we revise our entropy estimations downward. The last I heard was 2016, at around 6 bits/word of EU-produced English. I'm sure it's lower now.

deleted 4y ago
[deleted]
madcaptenor 4y ago
Will ChatGPT drag this downward even more?
- kqr 4y ago
  I couldn't resist asking it about the word entropy of its training corpus and quite predictably it wasn't able to understand my question and instead answered similar questions. However, it stuck quite firmly to a figure of 10--12 bits per word! I'm fairly sure that refers to the vocabulary entropy and not word entropy of English.
  - fooker 4y ago
    This is like forcing a human to answer questions about brain anatomy. You’ll be lucky to get anything reasonable at all.
- zw123456 4y ago
  I was wondering the same thing. Even the prediction as I type this in is pretty good. It seems like LLM's could be even better.
- canjobear 4y ago
  Yes, the GPT's are literally trained to minimize (cross-)entropy of text. You can see the bits-per-byte of text for lots of models here: https://crfm.stanford.edu/helm/v0.2.0/?group=targeted_evaluations https://crfm.stanford.edu/helm/v0.2.0/?group=targeted_evalua...
Isamu 4y ago
This is a good point, it happens because newer models are able to use larger and larger context - long range correlations in text. This amounts to finding more redundancy in the source text, driving down the estimated entropy. Language models are related to text compression, except that the compression is lossy, not trying to be able to recreate the source text exactly but rather to be able to recreate something that is statistically similar.
- User23 4y ago
  How is that not just K-complexity? A larger model is a larger interpreter. The entropy is just moving from the stream to the state machine.
canjobear 4y ago
For what it's worth, GPT-2 got PTB perplexity of around 20, which translates to about 4.3 bits/word. The later language models are presumably better but they report bits-per-character or bits-per-token which doesn't convert straightforwardly into bits per word.
- kqr 4y ago
  For a layman, can you elucidate whether this is about the entropy of its output or input?
  - canjobear 4y ago
    Kind of neither, but it's more like the entropy of its output than its input. LLM's can be used to generate text but fundamentally what they do is calculate probabilities of a token given previous tokens. Given a context like "I saw a", the LLM gives a probability to each token in its vocabulary within that context. It's really a probability distribution p(token | context). You can sample from that distribution to generate text, or you can use it to get the probability of some fixed text that you feed into it, which is the probability that the model would have generated that text (at temperature 1). The PTB perplexity numbers are based on the probabilities assigned to some text from the Wall Street Journal that used to be used as a standard test set in language modeling. It's 2 to the power of the average log probability of each word in the text under the model. So perplexity of 20 corresponds to around 4.3 bits of log probability per word, or another way of thinking of it is that the model will guess the correct following word in 1/20 cases. The reason I say this isn't exactly the output entropy is that, since perplexity is evaluated on a fixed text, it might not correspond to the kinds of texts that the model would generate purely on its own. In other words when you're looking at some fixed txt, the contexts you get in p(token | context) might not be anything like a context that the LM would generate itself. This was a problem people pointed out a few years ago---back then it looked like LM generations were somewhat higher-entropy than what you measured looking at a fixed text. I don't know if the modern huge LMs have this same problem.
htag 4y ago
It seems plausible that the entry of written English is decreasing over time. As it becomes cheaper to 'publish' the median effort spent in writing is decreasing.

by kqr 4y ago

deleted 4y ago

[deleted]

madcaptenor 4y ago

Will ChatGPT drag this downward even more?

kqr 4y ago
I couldn't resist asking it about the word entropy of its training corpus and quite predictably it wasn't able to understand my question and instead answered similar questions. However, it stuck quite firmly to a figure of 10--12 bits per word! I'm fairly sure that refers to the vocabulary entropy and not word entropy of English.
- fooker 4y ago
  This is like forcing a human to answer questions about brain anatomy. You’ll be lucky to get anything reasonable at all.
zw123456 4y ago
I was wondering the same thing. Even the prediction as I type this in is pretty good. It seems like LLM's could be even better.
canjobear 4y ago
Yes, the GPT's are literally trained to minimize (cross-)entropy of text. You can see the bits-per-byte of text for lots of models here: https://crfm.stanford.edu/helm/v0.2.0/?group=targeted_evaluations https://crfm.stanford.edu/helm/v0.2.0/?group=targeted_evalua...

Isamu 4y ago

This is a good point, it happens because newer models are able to use larger and larger context - long range correlations in text. This amounts to finding more redundancy in the source text, driving down the estimated entropy. Language models are related to text compression, except that the compression is lossy, not trying to be able to recreate the source text exactly but rather to be able to recreate something that is statistically similar.

User23 4y ago
How is that not just K-complexity? A larger model is a larger interpreter. The entropy is just moving from the stream to the state machine.

canjobear 4y ago

For what it's worth, GPT-2 got PTB perplexity of around 20, which translates to about 4.3 bits/word. The later language models are presumably better but they report bits-per-character or bits-per-token which doesn't convert straightforwardly into bits per word.

kqr 4y ago
For a layman, can you elucidate whether this is about the entropy of its output or input?
- canjobear 4y ago
  Kind of neither, but it's more like the entropy of its output than its input. LLM's can be used to generate text but fundamentally what they do is calculate probabilities of a token given previous tokens. Given a context like "I saw a", the LLM gives a probability to each token in its vocabulary within that context. It's really a probability distribution p(token | context). You can sample from that distribution to generate text, or you can use it to get the probability of some fixed text that you feed into it, which is the probability that the model would have generated that text (at temperature 1). The PTB perplexity numbers are based on the probabilities assigned to some text from the Wall Street Journal that used to be used as a standard test set in language modeling. It's 2 to the power of the average log probability of each word in the text under the model. So perplexity of 20 corresponds to around 4.3 bits of log probability per word, or another way of thinking of it is that the model will guess the correct following word in 1/20 cases. The reason I say this isn't exactly the output entropy is that, since perplexity is evaluated on a fixed text, it might not correspond to the kinds of texts that the model would generate purely on its own. In other words when you're looking at some fixed txt, the contexts you get in p(token | context) might not be anything like a context that the LM would generate itself. This was a problem people pointed out a few years ago---back then it looked like LM generations were somewhat higher-entropy than what you measured looking at a fixed text. I don't know if the modern huge LMs have this same problem.

htag 4y ago

It seems plausible that the entry of written English is decreasing over time. As it becomes cheaper to 'publish' the median effort spent in writing is decreasing.