3 ms·

GPT is a language model, meaning it tries to model language - output something that would be acceptable as text by a human reader. In a similar way how your scr

GPT is a language model, meaning it tries to model language - output something that would be acceptable as text by a human reader. In a similar way how your screen-saver might mimic water. But just how your screen saver isn't useful to study the properties of water, these language models are not useful to come up with new ideas. In fact, they are even biased against making new discoveries. Anything new is, by definition, outside the model. If a model were to return a new thing (not found in the training set) it would be down-weighted as being in the risk of not looking like an acceptable text (i.e. - was never seen before). Not unlike how a new behaviour of water (say under certain interesting conditions) would be outside the capabilities of your screen-saver, which only tries to mimic the existing and known properties of water. As a practical example - try asking it to make an original joke. It will either fail, or give you a joke, written down word for word, from somewhere else on the internet. By the time of GPT6 these problems might be overcome, but then it would be more than a language model.

ad404b8a372f2b9 4y ago
You conflate natural language and ideas, creating new ideas does not require creating a new statistical model of language. There is plenty of evidence that GPT can indeed create entirely novel ideas, it can do logical inference, imagine scenarios that don't fit its world model, create entirely new pieces of code. The fact that a Python program is restricted to Python syntax (language model) and libraries (world model), does not prevent it from being Turing complete.
- kkoncevicius 4y ago
  > You conflate natural language and ideas, creating new ideas does not require creating a new statistical model of language. The way I see it, currently, the correct logical inferences made by a model are a by-product of it trying to output acceptable text. That is to say - they are one of the properties, that it has learned, that makes the text acceptable. So it can say that 2+2 is 4 and, of course, even more complex statements like that, but it's based on modelling language, not modelling what is under the language. > There is plenty of evidence that GPT can indeed create entirely novel ideas, it can do logical inference, imagine scenarios that don't fit its world model, create entirely new code. I am certainly open to being convinced otherwise, maybe you can give me the best example of a new idea created by GPT?
  - ad404b8a372f2b9 4y ago
    Language is a world model, outputting acceptable text is outputting text that fits what's underneath the language: the meaning of the terms and their relationships. As far as I can tell that's how we do it too, I have no reason to believe we need a special mechanism for logic. You should just ask ChatGPT, it will be more convincing than anything I say, I've literally had it make up dozens of things. I had it suggest a niche type system for a novel language to program variational models, and elaborate on top of it given a specific set of constraints. Yesterday I was asking him to make up a new U.I element to replace the timeline in motion graphics software, he proposed a series of bubbles with each bubble representing an effect and its radius representing the length of the effect and changing color when the effect is being played. When it had just come out, I had him imagine the effects of reversing specific principles of lung physiology and how it would affect others. I've had it apply complex functions like ROT13 on entirely original text, encoding numbers in different bases and so on. If it can execute code on unseen data and perform logical inference on entirely imagined scenarios that don't fit its world model, that's as much evidence of creating a new idea as you can get.
    - fauigerzigerk 4y ago
      >Language is a world model, outputting acceptable text is outputting text that fits what's underneath the language: the meaning of the terms and their relationships. As far as I can tell that's how we do it too, I have no reason to believe we need a special mechanism for logic. So why did we invent mathematical notation, programming languages, diagrams and all sorts of formal semantics? Clearly it's because natural language does have some characteristics that make it less suitable for some tasks. It may be partly down to our cognitive limitations. But I think the most important issue is the way in which natural language changes. It all happens in a distributed and implicit (on the meta level) way, i.e. without anyone writing a specification of how new expressions relate to meaning. Having said that, there is no reason to assume that the principles of LLMs are unsuitable to learning formal languages and formal semantics. The question is whether LLMs are the best algorithm for generating and testing formal hypotheses though. They may just turn out to be rather mediocre at this. It's definitely worth trying though and I'm sure this is being done already.
nopinsight 4y ago
Check out "ACT-1: Transformer for Actions" as an example. There is at least 1 other public paper on how to implement a tool-using (language) model. Keep in mind that these are first generation systems. If the past 10 years is any indicator, the progress might be unexpectedly fast even for most experts in the field. These tools may not be independently operational in the physical world yet in 5 years. (Who can say what will happen in 10 years as the arms race is happening now). But if there are humans working with them, their impact can be vast.
maze-le 4y ago
New ideas, concepts or styles often come up by remixing old ones -- I wouldn't dismiss it entirely. True, ChatGPT does not have a context or world-model to generate these ideas, and in most cases they will be utter nonsense. But in some cases there just might me a connection between concepts we haven't seen before that have merit to generate a truly new idea.
- alright_scowl 4y ago
  > But in some cases there just might me a connection between concepts we haven't seen before that have merit to generate a truly new idea. But it's important to notice that a language model cannot reason to synthesize new information by making this connection. It can only connect two ideas if those ideas were already connected in the training data. To put in another way - ChatGPT is a new powerful way to organize existing information. And that's not to be dismissive. Natural Language Processing is a hard problem, and ChatGPT gracefully parses through and generates natural language, giving out mostly correct answers at the same time. But the quality of information it gives improves with my skill to ask it good questions. Not different from a internet search engine that gives you better answers if you now how to make better search queries.
visarga 4y ago
> If a model were to return a new thing (not found in the training set) it would be down-weighted LLMs can also learn by reinforcement learning and evolutionary techniques. They don't do just next token prediction, and they can even generate their own training data. Evolution through Large Models https://arxiv.org/abs/2206.08896 https://arxiv.org/abs/2206.08896

ad404b8a372f2b9 4y ago

You conflate natural language and ideas, creating new ideas does not require creating a new statistical model of language. There is plenty of evidence that GPT can indeed create entirely novel ideas, it can do logical inference, imagine scenarios that don't fit its world model, create entirely new pieces of code. The fact that a Python program is restricted to Python syntax (language model) and libraries (world model), does not prevent it from being Turing complete.

kkoncevicius 4y ago
> You conflate natural language and ideas, creating new ideas does not require creating a new statistical model of language. The way I see it, currently, the correct logical inferences made by a model are a by-product of it trying to output acceptable text. That is to say - they are one of the properties, that it has learned, that makes the text acceptable. So it can say that 2+2 is 4 and, of course, even more complex statements like that, but it's based on modelling language, not modelling what is under the language. > There is plenty of evidence that GPT can indeed create entirely novel ideas, it can do logical inference, imagine scenarios that don't fit its world model, create entirely new code. I am certainly open to being convinced otherwise, maybe you can give me the best example of a new idea created by GPT?
- ad404b8a372f2b9 4y ago
  Language is a world model, outputting acceptable text is outputting text that fits what's underneath the language: the meaning of the terms and their relationships. As far as I can tell that's how we do it too, I have no reason to believe we need a special mechanism for logic. You should just ask ChatGPT, it will be more convincing than anything I say, I've literally had it make up dozens of things. I had it suggest a niche type system for a novel language to program variational models, and elaborate on top of it given a specific set of constraints. Yesterday I was asking him to make up a new U.I element to replace the timeline in motion graphics software, he proposed a series of bubbles with each bubble representing an effect and its radius representing the length of the effect and changing color when the effect is being played. When it had just come out, I had him imagine the effects of reversing specific principles of lung physiology and how it would affect others. I've had it apply complex functions like ROT13 on entirely original text, encoding numbers in different bases and so on. If it can execute code on unseen data and perform logical inference on entirely imagined scenarios that don't fit its world model, that's as much evidence of creating a new idea as you can get.
  - fauigerzigerk 4y ago
    >Language is a world model, outputting acceptable text is outputting text that fits what's underneath the language: the meaning of the terms and their relationships. As far as I can tell that's how we do it too, I have no reason to believe we need a special mechanism for logic. So why did we invent mathematical notation, programming languages, diagrams and all sorts of formal semantics? Clearly it's because natural language does have some characteristics that make it less suitable for some tasks. It may be partly down to our cognitive limitations. But I think the most important issue is the way in which natural language changes. It all happens in a distributed and implicit (on the meta level) way, i.e. without anyone writing a specification of how new expressions relate to meaning. Having said that, there is no reason to assume that the principles of LLMs are unsuitable to learning formal languages and formal semantics. The question is whether LLMs are the best algorithm for generating and testing formal hypotheses though. They may just turn out to be rather mediocre at this. It's definitely worth trying though and I'm sure this is being done already.

nopinsight 4y ago

Check out "ACT-1: Transformer for Actions" as an example. There is at least 1 other public paper on how to implement a tool-using (language) model. Keep in mind that these are first generation systems. If the past 10 years is any indicator, the progress might be unexpectedly fast even for most experts in the field. These tools may not be independently operational in the physical world yet in 5 years. (Who can say what will happen in 10 years as the arms race is happening now). But if there are humans working with them, their impact can be vast.

maze-le 4y ago

New ideas, concepts or styles often come up by remixing old ones -- I wouldn't dismiss it entirely. True, ChatGPT does not have a context or world-model to generate these ideas, and in most cases they will be utter nonsense. But in some cases there just might me a connection between concepts we haven't seen before that have merit to generate a truly new idea.

alright_scowl 4y ago
> But in some cases there just might me a connection between concepts we haven't seen before that have merit to generate a truly new idea. But it's important to notice that a language model cannot reason to synthesize new information by making this connection. It can only connect two ideas if those ideas were already connected in the training data. To put in another way - ChatGPT is a new powerful way to organize existing information. And that's not to be dismissive. Natural Language Processing is a hard problem, and ChatGPT gracefully parses through and generates natural language, giving out mostly correct answers at the same time. But the quality of information it gives improves with my skill to ask it good questions. Not different from a internet search engine that gives you better answers if you now how to make better search queries.

visarga 4y ago

> If a model were to return a new thing (not found in the training set) it would be down-weighted LLMs can also learn by reinforcement learning and evolutionary techniques. They don't do just next token prediction, and they can even generate their own training data. Evolution through Large Models https://arxiv.org/abs/2206.08896 https://arxiv.org/abs/2206.08896