3 ms·

All text generated by current language models is defective. For reasons that are not fully understood, text language models emit is less interesting than text e

All text generated by current language models is defective. For reasons that are not fully understood, text language models emit is less interesting than text emitted by humans. Unlike humans, they can not score well on both coherence, as in grammatically correct sentences, and interest, as in "having something to say". They are however good at "coherent rambling". Since most language models work by using its outputs as its inputs, this means that the emitted text becomes worse the more words that are emitted. Suppose you want your model to emit a novel. The first 30 words might come out ok. The next 30 are so-so. Eventually, after a few hundred words, the model will begin to ramble and will get stuck in loops: "then she went to visit her mother and she went to visit her mother and she vent to visit her mother and she ..." All language models suffer from this problem; Markov models, RNNs, Transformers, GPT-2/3... In fact, one could argue that GPT-3 has "only" decreased the deterioration speed. Markov models deteriorate after a dozen words while GPT-3 can keep going for perhaps a hundred words before it begins to ramble. However, the statistical properties of the text GPT-3 generates are different from what a human emits (and what those "statistical properties" are is unknown - it appears that capturing what makes human prose human-like in a mathematical formula is very difficult) from the very first word so you can't use GPT-3-generated text to train GPT-4. The paper "The Curious Case of Neural Text Degeneration" describes the problem in more detail and with math. A major breakthrough in language modeling may come if someone is able to invent a model that doesn't degenerate. One whose thousandth word is as human-like as its first word.

minimaxir 6y ago
That's a No True Scotsman way of looking at language models. GPT-3 or any Transformer model doesn't need to be innovative to be successful. But that's why it's important to run experiments and build test cases. In some cases, degeneration can actually be a feature: https://twitter.com/dril_gpt2 https://twitter.com/dril_gpt2
wpietri 6y ago
Great points. I suspect a non-degenerative model is orders of magnitude harder, in that good long prose requires much deeper complexity. For example, I was just reading this excellent long piece from Rebecca Traister: https://nymag.com/intelligencer/article/andrew-cuomo-misconduct-allegations.html https://nymag.com/intelligencer/article/andrew-cuomo-miscond... Even if a language model had all the raw facts needed to write a long piece, there's a ton of structure here. She develops ideas, builds themes, assembles a thesis. She is not just going somewhere, but carefully taking us along with her. That requires modeling not just textual relationships, but taking conceptual relationships and expressing them in ways that require deep awareness of how the audience processes information. So I'd be willing to bet a true non-degenerative model requires an increasingly complex and accurate theory of mind.
Der_Einzige 6y ago
The solution proposed in that paper, nucleolus sampling, effectively solves this problem though...
- skybrian 6y ago
  It’s an odd hack, though. Adding a bit of randomness results in a more interesting exploration of Borge’s Library of Babel, but it seems weird to assign meaning to the word choice?
- bjourne 6y ago
  Nucleus sampling can increase the rambling horizon for some datasets and models by some small percentage. A top-k sampled model might start rambling after 50 words and a nucleus sampled model perhaps can keep going until 60 words. But even with nucleus sampling, the text will eventually degenerate.
sprobertson 6y ago
> For reasons that are not fully understood, text language models emit is less interesting than text emitted by humans. What's not understood? As you describe, every standard language model is statistically predicting the next word given the previous words, ad infinitum. GPT3 has more parameters to build a more precise "thought" vector, but the goal is still only to produce the next word.
- drdeca 6y ago
  Suppose you had an impossibly perfect conditional probability distribution over "text that a person would write". I realize there may be some issues with that being a well-defined concept. If you don't think it is well defined (and perhaps it isn't), please momentarily pretend that that is something that is/can-be well defined. In that case, for any property of a text, such as "the text gets ramble-y and basically doesn't have any real point", the probability that a randomly selected text produced by a person has the property, and the probability that a text sampled from that distribution, would be equal. Could text be sampled from such a distribution on a word by word (or token by token) basis? Yes. Let's see why: For any event (or property), there are countably many texts which satisfy the property. So, it can be expressed as a countable disjoint union of sets which each have as their only element one of the texts which have the property. So, the probability of a text sampled from the distribution having the property, is equal to the sum of the probabilities of each of the texts which have the property. The probability of each text is equal to the probability of "the next token is [the last word of the text]" given [all the tokens before the last one], multiplied by the probability of the text [all of the tokens before the last one]. And then recurse on that, in order to break the probability of each text down into products of conditional probabilities of (next token | previous tokens) . Hm, actually, I think there's an issue with what I said. It would work if all texts people wrote had the same length, or if all of them continued indefinitely, but that's not the case. Maybe that can be patched by including a conditional probability for an "end of text" token. Actually, yes, that seems likely. Say that the probability of the text is the probability of (that is all of the text | [the text] is some initial segment of the text), multiplied by the probability of ([the text] is some initial segment of the text). and then the probability of ([an initial segment] is some initial segment of the text) is the probability of ([the last token of the initial segment] is the next token in an initial segment of the text | [the tokens before the last one of the segment] is an initial segment of the text ) times the probability of ([the tokens before the last one of the segment] is an initial segment of the text). Ok, yeah. So, in theory, if you allow arbitrary conditional probabilities for the next token (or an end-of-text token) conditioned on all the previous tokens, you can produce any probability distribution over texts. Therefore, "predicting the next word given the prior words" isn't any limitation on what probability distributions over text can be produced. (though perhaps it could drastically increase the cost of producing certain probability distributions, and perhaps in such a way that makes it practically infeasible to produce in such a way.)
mannykannot 6y ago
That paper includes this observation: Why is human-written text not the most probable text? We conjecture that this is an intrinsic property of human language... Grice’s Maxims of Communication (Grice,1975) show that people optimize against stating the obvious. I don't think that should be at all surprising, if one purpose of speech is to convey information: predictability is the converse of being informational. Furthermore, to be informative, it is not sufficient for a passage to avoid predictability; it also needs to be pertinent.

minimaxir 6y ago

That's a No True Scotsman way of looking at language models. GPT-3 or any Transformer model doesn't need to be innovative to be successful. But that's why it's important to run experiments and build test cases. In some cases, degeneration can actually be a feature: https://twitter.com/dril_gpt2 https://twitter.com/dril_gpt2

wpietri 6y ago

Great points. I suspect a non-degenerative model is orders of magnitude harder, in that good long prose requires much deeper complexity. For example, I was just reading this excellent long piece from Rebecca Traister: https://nymag.com/intelligencer/article/andrew-cuomo-misconduct-allegations.html https://nymag.com/intelligencer/article/andrew-cuomo-miscond... Even if a language model had all the raw facts needed to write a long piece, there's a ton of structure here. She develops ideas, builds themes, assembles a thesis. She is not just going somewhere, but carefully taking us along with her. That requires modeling not just textual relationships, but taking conceptual relationships and expressing them in ways that require deep awareness of how the audience processes information. So I'd be willing to bet a true non-degenerative model requires an increasingly complex and accurate theory of mind.

Der_Einzige 6y ago

The solution proposed in that paper, nucleolus sampling, effectively solves this problem though...

skybrian 6y ago
It’s an odd hack, though. Adding a bit of randomness results in a more interesting exploration of Borge’s Library of Babel, but it seems weird to assign meaning to the word choice?
bjourne 6y ago
Nucleus sampling can increase the rambling horizon for some datasets and models by some small percentage. A top-k sampled model might start rambling after 50 words and a nucleus sampled model perhaps can keep going until 60 words. But even with nucleus sampling, the text will eventually degenerate.

sprobertson 6y ago

> For reasons that are not fully understood, text language models emit is less interesting than text emitted by humans. What's not understood? As you describe, every standard language model is statistically predicting the next word given the previous words, ad infinitum. GPT3 has more parameters to build a more precise "thought" vector, but the goal is still only to produce the next word.

drdeca 6y ago
Suppose you had an impossibly perfect conditional probability distribution over "text that a person would write". I realize there may be some issues with that being a well-defined concept. If you don't think it is well defined (and perhaps it isn't), please momentarily pretend that that is something that is/can-be well defined. In that case, for any property of a text, such as "the text gets ramble-y and basically doesn't have any real point", the probability that a randomly selected text produced by a person has the property, and the probability that a text sampled from that distribution, would be equal. Could text be sampled from such a distribution on a word by word (or token by token) basis? Yes. Let's see why: For any event (or property), there are countably many texts which satisfy the property. So, it can be expressed as a countable disjoint union of sets which each have as their only element one of the texts which have the property. So, the probability of a text sampled from the distribution having the property, is equal to the sum of the probabilities of each of the texts which have the property. The probability of each text is equal to the probability of "the next token is [the last word of the text]" given [all the tokens before the last one], multiplied by the probability of the text [all of the tokens before the last one]. And then recurse on that, in order to break the probability of each text down into products of conditional probabilities of (next token | previous tokens) . Hm, actually, I think there's an issue with what I said. It would work if all texts people wrote had the same length, or if all of them continued indefinitely, but that's not the case. Maybe that can be patched by including a conditional probability for an "end of text" token. Actually, yes, that seems likely. Say that the probability of the text is the probability of (that is all of the text | [the text] is some initial segment of the text), multiplied by the probability of ([the text] is some initial segment of the text). and then the probability of ([an initial segment] is some initial segment of the text) is the probability of ([the last token of the initial segment] is the next token in an initial segment of the text | [the tokens before the last one of the segment] is an initial segment of the text ) times the probability of ([the tokens before the last one of the segment] is an initial segment of the text). Ok, yeah. So, in theory, if you allow arbitrary conditional probabilities for the next token (or an end-of-text token) conditioned on all the previous tokens, you can produce any probability distribution over texts. Therefore, "predicting the next word given the prior words" isn't any limitation on what probability distributions over text can be produced. (though perhaps it could drastically increase the cost of producing certain probability distributions, and perhaps in such a way that makes it practically infeasible to produce in such a way.)

mannykannot 6y ago

That paper includes this observation: Why is human-written text not the most probable text? We conjecture that this is an intrinsic property of human language... Grice’s Maxims of Communication (Grice,1975) show that people optimize against stating the obvious. I don't think that should be at all surprising, if one purpose of speech is to convey information: predictability is the converse of being informational. Furthermore, to be informative, it is not sufficient for a passage to avoid predictability; it also needs to be pertinent.