4 ms·

For what it's worth, GPT-2 got PTB perplexity of around 20, which translates to about 4.3 bits/word. The later language models are presumably better but they re

For what it's worth, GPT-2 got PTB perplexity of around 20, which translates to about 4.3 bits/word. The later language models are presumably better but they report bits-per-character or bits-per-token which doesn't convert straightforwardly into bits per word.

kqr 3y ago
For a layman, can you elucidate whether this is about the entropy of its output or input?
- canjobear 3y ago
  Kind of neither, but it's more like the entropy of its output than its input. LLM's can be used to generate text but fundamentally what they do is calculate probabilities of a token given previous tokens. Given a context like "I saw a", the LLM gives a probability to each token in its vocabulary within that context. It's really a probability distribution p(token | context). You can sample from that distribution to generate text, or you can use it to get the probability of some fixed text that you feed into it, which is the probability that the model would have generated that text (at temperature 1). The PTB perplexity numbers are based on the probabilities assigned to some text from the Wall Street Journal that used to be used as a standard test set in language modeling. It's 2 to the power of the average log probability of each word in the text under the model. So perplexity of 20 corresponds to around 4.3 bits of log probability per word, or another way of thinking of it is that the model will guess the correct following word in 1/20 cases. The reason I say this isn't exactly the output entropy is that, since perplexity is evaluated on a fixed text, it might not correspond to the kinds of texts that the model would generate purely on its own. In other words when you're looking at some fixed txt, the contexts you get in p(token | context) might not be anything like a context that the LM would generate itself. This was a problem people pointed out a few years ago---back then it looked like LM generations were somewhat higher-entropy than what you measured looking at a fixed text. I don't know if the modern huge LMs have this same problem.

kqr 3y ago

For a layman, can you elucidate whether this is about the entropy of its output or input?

canjobear 3y ago
Kind of neither, but it's more like the entropy of its output than its input. LLM's can be used to generate text but fundamentally what they do is calculate probabilities of a token given previous tokens. Given a context like "I saw a", the LLM gives a probability to each token in its vocabulary within that context. It's really a probability distribution p(token | context). You can sample from that distribution to generate text, or you can use it to get the probability of some fixed text that you feed into it, which is the probability that the model would have generated that text (at temperature 1). The PTB perplexity numbers are based on the probabilities assigned to some text from the Wall Street Journal that used to be used as a standard test set in language modeling. It's 2 to the power of the average log probability of each word in the text under the model. So perplexity of 20 corresponds to around 4.3 bits of log probability per word, or another way of thinking of it is that the model will guess the correct following word in 1/20 cases. The reason I say this isn't exactly the output entropy is that, since perplexity is evaluated on a fixed text, it might not correspond to the kinds of texts that the model would generate purely on its own. In other words when you're looking at some fixed txt, the contexts you get in p(token | context) might not be anything like a context that the LM would generate itself. This was a problem people pointed out a few years ago---back then it looked like LM generations were somewhat higher-entropy than what you measured looking at a fixed text. I don't know if the modern huge LMs have this same problem.