5 ms·

Somewhat related, but I can't stop thinking about what's going to happen when GPT-4 reads articles like this. Articles like these that discuss GPT-3's output,

Somewhat related, but I can't stop thinking about what's going to happen when GPT-4 reads articles like this. Articles like these that discuss GPT-3's output, or comments and articles directly generated by GPT-3, are going to be fed into GPT-4 as part of a theoretical WebText3 dataset. Will that help or hurt GPT-4's learning? What effect will it have? Theoretically you can feed GPT-3's responses back into itself and ask it to introspect about them; whether it was right or wrong and to comment on why it gave the response it did. But I doubt GPT-3 is particularly good at self introspection. GPT-3 was trained before GPT-3 existed, obviously. So it was never trained on articles analyzing the output of an AI of its caliber. But GPT-4 is going to be trained on a corpus filled with people analyzing GPT-3's outputs, like this article. We would expect GPT-4 to be able to write an article like this. So it should be theoretically possible to give GPT-4 its own output, and then ask it to provide introspection, and for that introspection to be insightful. EDIT: Follow up thought. It's almost as if the internet is being filled with a training corpus on GPT-3's failings. Every fact that GPT-3 failed to learn from WebText2 is now going to be repeated, alongside the correct answer, in WebText3. Humans are globally working, unknowingly, to build a curated dataset by which GPT-4 can learn from GPT-3's mistakes.

mrfusion 6y ago
Somewhere in it’s billions of parameters it will have a sense of self. A model of itself. That’s pretty mind boggling.
bjourne 6y ago
All text generated by current language models is defective. For reasons that are not fully understood, text language models emit is less interesting than text emitted by humans. Unlike humans, they can not score well on both coherence, as in grammatically correct sentences, and interest, as in "having something to say". They are however good at "coherent rambling". Since most language models work by using its outputs as its inputs, this means that the emitted text becomes worse the more words that are emitted. Suppose you want your model to emit a novel. The first 30 words might come out ok. The next 30 are so-so. Eventually, after a few hundred words, the model will begin to ramble and will get stuck in loops: "then she went to visit her mother and she went to visit her mother and she vent to visit her mother and she ..." All language models suffer from this problem; Markov models, RNNs, Transformers, GPT-2/3... In fact, one could argue that GPT-3 has "only" decreased the deterioration speed. Markov models deteriorate after a dozen words while GPT-3 can keep going for perhaps a hundred words before it begins to ramble. However, the statistical properties of the text GPT-3 generates are different from what a human emits (and what those "statistical properties" are is unknown - it appears that capturing what makes human prose human-like in a mathematical formula is very difficult) from the very first word so you can't use GPT-3-generated text to train GPT-4. The paper "The Curious Case of Neural Text Degeneration" describes the problem in more detail and with math. A major breakthrough in language modeling may come if someone is able to invent a model that doesn't degenerate. One whose thousandth word is as human-like as its first word.
- minimaxir 6y ago
  That's a No True Scotsman way of looking at language models. GPT-3 or any Transformer model doesn't need to be innovative to be successful. But that's why it's important to run experiments and build test cases. In some cases, degeneration can actually be a feature: https://twitter.com/dril_gpt2 https://twitter.com/dril_gpt2
- wpietri 6y ago
  Great points. I suspect a non-degenerative model is orders of magnitude harder, in that good long prose requires much deeper complexity. For example, I was just reading this excellent long piece from Rebecca Traister: https://nymag.com/intelligencer/article/andrew-cuomo-misconduct-allegations.html https://nymag.com/intelligencer/article/andrew-cuomo-miscond... Even if a language model had all the raw facts needed to write a long piece, there's a ton of structure here. She develops ideas, builds themes, assembles a thesis. She is not just going somewhere, but carefully taking us along with her. That requires modeling not just textual relationships, but taking conceptual relationships and expressing them in ways that require deep awareness of how the audience processes information. So I'd be willing to bet a true non-degenerative model requires an increasingly complex and accurate theory of mind.
- Der_Einzige 6y ago
  The solution proposed in that paper, nucleolus sampling, effectively solves this problem though...
  - skybrian 6y ago
    It’s an odd hack, though. Adding a bit of randomness results in a more interesting exploration of Borge’s Library of Babel, but it seems weird to assign meaning to the word choice?
  - bjourne 6y ago
    Nucleus sampling can increase the rambling horizon for some datasets and models by some small percentage. A top-k sampled model might start rambling after 50 words and a nucleus sampled model perhaps can keep going until 60 words. But even with nucleus sampling, the text will eventually degenerate.
knuthsat 6y ago
The whole point of a model is to take advantage of data with more information. If GPT-3 output sucks for predicting human output then it will be ignored by the model during descent. Otherwise the learning algorithm is defective.
notahacker 6y ago
The first part of your statement feels like it's anthropomorphising GPT-4. GPT-4s response to learning that GPT-3 is bad at word play will to store the concept of GPT-3 being bad at word play to recall if prompted about GPT-3 in future, not to swot up on word play. If you want GPT-4 to be better at word play you'll need a corpus of puns, rhymes and higher level concept association and a lot of training on replicating them and when to recognise a prompt might require them.
tsimionescu 6y ago
Your ascribing way too much intelligence to a very simplistic text prediction algorithm. Neither GPT-3 nor GPT-4 (assuming it will just be a bigger GPT-3) have any concept of identity that could influence their predictions in the way you seem to assume. If GPT-4 would be trained on text produced by GPT-3 and on text that analyzes that, what GPT-4 would learn is how to sound like GPT-3 and like detractors or admirers of GPT-3. While impressive achievements, GPT-2/3/4/X are just models that predict what the most likely next word is based on a huge corpus of text. There is no model of the world in there, no objects, no identities. Just probabilities of glyphs in certain orders, without any meaning attached.
- haolez 6y ago
  I mostly agree with you, but I wonder what's the distance between what you described and our actual intelligence.
  - tsimionescu 6y ago
    I don't think there is any kind of similarity between a text prediction model and human (or animal) intelligence. At most we maybe have a subsystem that could work somewhat similarly. It's very clear though that human intelligence has a model of the world being made up of agents which interact with objects. Words and sentences refer to entities in this model in our head, and our use of language essentially consists of describing instances of this model, with various purposes. Text prediction works exclusively on text, no model of the world, and it has a single purpose: to produce the text that would be most likely to follow the prompt based on its training. This isn't just a theoretical difference. GPT-3 couldn't be used to, for example, be presented with a situation and asked to respond to it in such a way that it achieves some goal.
johbjo 6y ago
One intuition about the large GPT-models is that they are learning "facts" like they are learning grammar rules. It's just that their capacity is so immensely large as to learn facts as part of the language model.

mrfusion 6y ago

Somewhere in it’s billions of parameters it will have a sense of self. A model of itself. That’s pretty mind boggling.

bjourne 6y ago

All text generated by current language models is defective. For reasons that are not fully understood, text language models emit is less interesting than text emitted by humans. Unlike humans, they can not score well on both coherence, as in grammatically correct sentences, and interest, as in "having something to say". They are however good at "coherent rambling". Since most language models work by using its outputs as its inputs, this means that the emitted text becomes worse the more words that are emitted. Suppose you want your model to emit a novel. The first 30 words might come out ok. The next 30 are so-so. Eventually, after a few hundred words, the model will begin to ramble and will get stuck in loops: "then she went to visit her mother and she went to visit her mother and she vent to visit her mother and she ..." All language models suffer from this problem; Markov models, RNNs, Transformers, GPT-2/3... In fact, one could argue that GPT-3 has "only" decreased the deterioration speed. Markov models deteriorate after a dozen words while GPT-3 can keep going for perhaps a hundred words before it begins to ramble. However, the statistical properties of the text GPT-3 generates are different from what a human emits (and what those "statistical properties" are is unknown - it appears that capturing what makes human prose human-like in a mathematical formula is very difficult) from the very first word so you can't use GPT-3-generated text to train GPT-4. The paper "The Curious Case of Neural Text Degeneration" describes the problem in more detail and with math. A major breakthrough in language modeling may come if someone is able to invent a model that doesn't degenerate. One whose thousandth word is as human-like as its first word.

minimaxir 6y ago
That's a No True Scotsman way of looking at language models. GPT-3 or any Transformer model doesn't need to be innovative to be successful. But that's why it's important to run experiments and build test cases. In some cases, degeneration can actually be a feature: https://twitter.com/dril_gpt2 https://twitter.com/dril_gpt2
wpietri 6y ago
Great points. I suspect a non-degenerative model is orders of magnitude harder, in that good long prose requires much deeper complexity. For example, I was just reading this excellent long piece from Rebecca Traister: https://nymag.com/intelligencer/article/andrew-cuomo-misconduct-allegations.html https://nymag.com/intelligencer/article/andrew-cuomo-miscond... Even if a language model had all the raw facts needed to write a long piece, there's a ton of structure here. She develops ideas, builds themes, assembles a thesis. She is not just going somewhere, but carefully taking us along with her. That requires modeling not just textual relationships, but taking conceptual relationships and expressing them in ways that require deep awareness of how the audience processes information. So I'd be willing to bet a true non-degenerative model requires an increasingly complex and accurate theory of mind.
Der_Einzige 6y ago
The solution proposed in that paper, nucleolus sampling, effectively solves this problem though...
- skybrian 6y ago
  It’s an odd hack, though. Adding a bit of randomness results in a more interesting exploration of Borge’s Library of Babel, but it seems weird to assign meaning to the word choice?
- bjourne 6y ago
  Nucleus sampling can increase the rambling horizon for some datasets and models by some small percentage. A top-k sampled model might start rambling after 50 words and a nucleus sampled model perhaps can keep going until 60 words. But even with nucleus sampling, the text will eventually degenerate.

knuthsat 6y ago

The whole point of a model is to take advantage of data with more information. If GPT-3 output sucks for predicting human output then it will be ignored by the model during descent. Otherwise the learning algorithm is defective.

notahacker 6y ago

The first part of your statement feels like it's anthropomorphising GPT-4. GPT-4s response to learning that GPT-3 is bad at word play will to store the concept of GPT-3 being bad at word play to recall if prompted about GPT-3 in future, not to swot up on word play. If you want GPT-4 to be better at word play you'll need a corpus of puns, rhymes and higher level concept association and a lot of training on replicating them and when to recognise a prompt might require them.

tsimionescu 6y ago

Your ascribing way too much intelligence to a very simplistic text prediction algorithm. Neither GPT-3 nor GPT-4 (assuming it will just be a bigger GPT-3) have any concept of identity that could influence their predictions in the way you seem to assume. If GPT-4 would be trained on text produced by GPT-3 and on text that analyzes that, what GPT-4 would learn is how to sound like GPT-3 and like detractors or admirers of GPT-3. While impressive achievements, GPT-2/3/4/X are just models that predict what the most likely next word is based on a huge corpus of text. There is no model of the world in there, no objects, no identities. Just probabilities of glyphs in certain orders, without any meaning attached.

haolez 6y ago
I mostly agree with you, but I wonder what's the distance between what you described and our actual intelligence.
- tsimionescu 6y ago
  I don't think there is any kind of similarity between a text prediction model and human (or animal) intelligence. At most we maybe have a subsystem that could work somewhat similarly. It's very clear though that human intelligence has a model of the world being made up of agents which interact with objects. Words and sentences refer to entities in this model in our head, and our use of language essentially consists of describing instances of this model, with various purposes. Text prediction works exclusively on text, no model of the world, and it has a single purpose: to produce the text that would be most likely to follow the prompt based on its training. This isn't just a theoretical difference. GPT-3 couldn't be used to, for example, be presented with a situation and asked to respond to it in such a way that it achieves some goal.

johbjo 6y ago

One intuition about the large GPT-models is that they are learning "facts" like they are learning grammar rules. It's just that their capacity is so immensely large as to learn facts as part of the language model.