8 ms·

A lot of HNers were so adamant that the LLMs understand absolutely nothing and that these models are just predicting the next most likely word. I think with th

A lot of HNers were so adamant that the LLMs understand absolutely nothing and that these models are just predicting the next most likely word. I think with this paper it becomes clear that the adamant denial was just human bias talking. LLMs crossed a certain line here. You would think people would be amazed or in awe or react in fear at technological break through a and they often are. The weird part for LLMs was the delusion. Many people were in denial that it was any sort of breakthrough.

YeGoblynQueenne 3y ago
This is a preprint that was just uploaded to arxiv two days ago. Don't be hasty and assume that it settles any matter at all. Many such claims have been made before and many counter-claims also. There is still a lively debate on the subject and it will be some time before there is agreement. More generally, any scholarly article is a claim, and should never be read as automatically true. That's something to keep in mind.
- circuit10 3y ago
  I thought it was completely obvious that if it can predict things about the world as well as it can it has some form of world model, even if inaccurate. I’m surprised that anyone would argue that a language model doesn’t model things
  - YeGoblynQueenne 3y ago
    Yes, I see what you mean. It's frustrating but the word "model" is severely overused in Computer Science and AI and it can cause a lot of confusion. Briefly, a "world model" is a theory possessed by an autonomous agent that describes the entities that exist in the world and how they interact with each other and with the agent, and that the agent can use to make decisions. This is the sense in which "model" is used when people talk about "model based" approaches to AI (such as planning, for instance, which is "the model based approach to autonomous behaviour"). A "language model" on the other hand is a statistical model of the text in a corpus. A statistical model is really a set of events, and their probabilities. The "events" in a statistical language model are usually word collocations, where a collocation is a word A found near a word B, for some measure of "near". If you've heard about "word embeddings", that's a model of word collocations in text corpora. So the claim that LLMs "have a model" is about LLMs being autonomous agents with a world-model, while "Large Language Model" is about statistical modelling. There's no reason why a statistical language model should have a world model.
    - circuit10 3y ago
      Since language often describes the world, I think a good language model must include a world model
      - YeGoblynQueenne 3y ago
        Well, this has been claimed often, namely by some of the people who have developed statistical language models [1] but it's really not obvious how that should work. Where would a language model find the world model? Where would it store it? And why would it even need it? Obviously I don't know how the human linguistic ability works but it's clear that for us, text, words, language, isn't carrying around with it a representation of the entire world. So where would a language model find it in its training text? For example if I say "I sat by the sea today", there's no representation of the sea in the word "sea", rather, the representation is inside my head and yours, and the word "sea" is only used to point to that representation [2]. Why would it ever be possible to derive the representation just from the pointer? Like I say, it's really not obvious and I don't think anyone has given a convincing explanation - it's usually words all the way down when people try. In any case, it seems that humans had most of the capabilities conferred by a world model (which we share with other animals) before we had language (which we do not - share with other animals), so it seems likely that a world model must be developed first, before linguistic ability can be built on top of it. But that's a conjecture, of course. I guess maybe, in principle, it should be possible to treat all of human cognition as some kind of hidden (latent) variable and then train a model to represent it only from its expression in text, but we don't really have any kind of modelling technique with that power. That would be a truly omnipotent model. So powerful, indeed, that it wouldn't need to do that in the first place- it would already be a kind of machine god. ______________ [1] e.g. I believe Tomáš Mikolov, one of the creators of word2vec, has claimed something like that but I'm not sure if I have a source for my own claim; more recently Ilya Sutskever, of OpenAI, has said words to that effect- I have a link somewhere I think, if you are curious. [2] That's obvious because the word "sea" doesn't mean anything to someone who doesn't know English, but they'll probably have a word for the sea in their language, so they'll have the concept, without needing a specific word for it.
andrewguenther 3y ago
I think LLMs are presenting some uncomfortable philosophical questions for people about how our own brains work and admitting that there is any kind of "intelligence" (even if very basic) in an LLM is an admission that our own brains may work in a similar manner.
- YeGoblynQueenne 3y ago
  For me, one of the most interesting things that have come out of LLMs is the confirmation that humans are very bad at reasoning and, consequently, its' a very bad idea to try and make machines that "think like humans", because that way we'll only make machines with none of the advantages of machines and all the disadvantages of computers. For instance -I'm not trying to be mean and I'm certainly not blaming you in particular, because I've seen this very often- but the reasoning that because LLMs can generate language, and humans can generate language not only LLMs are somehow like humans but also humans are like LLMs is not sound. For example, walls have ears, cats have ears, therefore walls are like cats and cats are like walls. That doesn't work because walls' ears are not like cats' ears and even if they were, that still wouldn't make walls cats and cats walls, it would just make them both entities with ears.
  - famouswaffles 3y ago
    >because that way we'll only make machines with none of the advantages of machines and all the disadvantages of computers. There is no evidence, basically none whatsoever that general "perfect logical reasoning" is a thing that actually exists in the real world. None. No animal we've observed does it. Humans certainly don't do it. The only realm this idea actually works is Fiction. and this was not like for a lack of trying. Some of the greatest minds worked on this for decades and some people still don't seem to get it. Logic doesn't scale. They break at real world relationships. Logic systems are that guy in the stands yelling that he could've made the shot, while he's not even on the field.
    - YeezyMode 3y ago
      Would you say that this in itself is due to how incomplete human reasoning is in the first place? That as a result, our ideas of logic and what perfect logic looks like are bound to fail? Or are you saying that the purest mathematical representation of logic cannot scale to a point where they can model and predict real world relationships successfully?
anigbrowl 3y ago
Indeed. I was offering examples of how you could bootstrap cognitive processes at the beginning of the year and people were just sticking their fingers in their ears. A lot of people exercise their intellectual, economic, or political freedoms by rejecting others' efforts to gain the same things.
- SiempreViernes 3y ago
  Who is the subject in the last bit of your last sentence? More pointedly: are you claiming that OpenAI is doing LLM research to gain political freedoms?
  - anigbrowl 3y ago
    No, the second sentence is a general observation on a widespread human behavior pattern. I don't feel like expanding it into an essay right now.
- OneDonOne 3y ago
  >A lot of people exercise their intellectual, economic, or political freedoms by rejecting others' efforts to gain the same things. Could you please explain what the above has to do with a Large Languange Model?
  - anigbrowl 3y ago
    I'm talking about how many people react to any startling new information or idea. The advent of LLMs is just a relevant example.
    - OneDonOne 3y ago
      No new ideas have been advanced in anything reported here. A state machine change state is all that is going on here. What I'd like to know is why should anyone care about this preprint? Nothing has been stated on how this software will improve the life of the average human being. LK-99 at least would have had lead.somewhere if it worked.
fipar 3y ago
My only experience so far is with chatgpt, and I am not an expert in AI, or even just in LLMs. With those disclaimers, my interactions inform my opinion that the LLM behind chatgpt has no internal world model. It shows no understanding of basic facts and makes very silly mistakes very easily. I have my bias, like anyone else, but in the case of AI in particular, I should say that I don't think there's anything especially magical or sacred about conscience and the human brain (or the brain of other animals, for that matter), and I'm sure it must be possible to arrive at different forms of intelligence starting with the "inanimate" building blocks of hardware and software, but what I've seen so far in LLMs doesn't make me support your statement that they've crossed a certain line. In fact, I'm very much let down by the experience and I'm afraid once the hype goes down, we may be in for another AI winter. As I said at the start of the comment, I'm well aware I'm not a subject-matter expert. I'm open to being wrong. I just wanted to point out that not all of us unimpressed are in denial. I'd love to be in awe at an AI breakthrough, and I kind of feel I will be in awe sometime during my lifetime, but LLMs are not (yet?) that for me. Edit: s/are not in denial/are in denial/
- ilaksh 3y ago
  Have you used GPT-4?
  - fipar 3y ago
    No, I've only used GPT-3 so far. Should I be excited about GPT-4? :) Examples of things I'd file as silly mistakes: - mentioning non-existing settings when asking it about a particular software. - responding to roughly "give me a way to list all rds clusters that are running a version that will reach EOL within the next 12 months" (which I know is almost impossible to do without scraping, as the EOLs are not returned by the API, but it was worth trying) with a query that compares the version with < 12.0. It did acknowledge the error when I asked "won't that just compare the version number" and, interestingly, it then did add to the response that the API does not provide EOL for versions, but the first response was IMHO bad in a very basic way. - providing wrong steps in an answer to "give me steps to do X", wrong in a way that would cause the process to go very bad, and then when I point out the problematic step, it responds with a seemingly random alternate step that is also wrong. I guess this part is the one that is less bad, because it loosely reminds me of people who are bad at reasoning and just give back random answers to a logic or math question. I must say it is surprisingly good (though it will still make up facts at times) when answering questions about elisp though. Paraphrasing Hofstadter, I'd say if I'm asked about the intelligence of chatgpt using gpt-3 I'd say it's slightly better than a thermostat or a mosquito, but not much more. Furthermore, my (admittedly not very valuable to others) gut feeling is this is not the path to get to what I'd consider an intelligent being, mostly because in my interactions, it makes lots of bad mistakes, and none of the good ones (here I'm using good and bad as qualifiers for mistakes based on my experiences as a parent, and as a mentor to junior peers, where a good mistake is something that lets you know they're on the right path even if they haven't arrived at the right answer/place yet). Try asking it about analogies, for example ("what's a good analogy to explain X to someone new to it"), the results I've gotten are underwhelming, when not just plain wrong. Edit: s/fall/file/ Formatting.
tayo42 3y ago
Seems like the hyperbolic conclusion to make? The paper doesn't make that claim. Simple ml models can take data and find patterns, like separate red and blue, or xy coordinates. Those are a small amount of dimensions and easy to reason about. This seems like the model found on its own another dimension to segregate data in deeper layers by generalizing and using context when learning. Its cool it can do that, but it still to me seems like drawing a best fit line just with higher dimensions. I'm sure someone will correct me if they think I am way off.
Sharlin 3y ago
It's the Chinese Room argument all over again. People hear "predicting the next token" and all they can imagine is some sort of a statistical database lookup, ignoring the fact that when you have a huge corpus of data with incredibly complex internal correlations and all that data also happens to correlate with some unknown external thing, it's almost certain that a powerful learner will end up modeling that external thing if and when doing so will cause a quantum leap in prediction performance! A model that includes the external-thing hypothesis will almost certainly be simpler, ceteris paribus, than a model that doesn't.
- esafak 3y ago
  That is what a model is; a parsimonious explanation of observations that admits extrapolation.
  - Sharlin 3y ago
    Exactly!
gamblor956 3y ago
If you actually read the paper, the analysis absolutely does not support the claim that LLMs are capable of maintaining a spatial or chronological internal world model. The LLMs simply activated when locations or times were mentioned, with no understanding of the relationship between different places or different times beyond the fact that different places and times had different tags. Or in other words, no internal model of the world.
- nyrikki 3y ago
  The lack of generally accepted definition for knowledge and intelligence makes these conversations difficult but the above post is important to remember. The paper has nothing to do with the common sense problem. While it is a preprint, I do wonder how useful this will be to improve generalization. Termination analysis on linear polynomials is undecidable over the naturals without many constraints and is undecidable ofer the rationals. If you think about how sstrees or even rtrees work, this isn't surprising to me that space and time are useful for compression which is required for learnability. But I think people also see 'linear' and think that means easy. When it can mean serious restrictions on how it can be used. Hard to explain the connection I have in mind without serious math but here is a paper that may cause some issues with trying to use these internal structures outside of the learnability implications. https://arxiv.org/abs/2307.14805 https://arxiv.org/abs/2307.14805
School-Cotton 3y ago
> I think with this paper it becomes clear that the adamant denial was just human bias talking It already becomes clear after playing with ChatGPT-4 for five minutes. But a lot of people still refuse to accept it.
OneDonOne 3y ago
A bigger Chinese Room - is still a Chinese Room. We already went through this with ELIZA 60 years ago
- deleted 3y ago
  [deleted]

YeGoblynQueenne 3y ago

This is a preprint that was just uploaded to arxiv two days ago. Don't be hasty and assume that it settles any matter at all. Many such claims have been made before and many counter-claims also. There is still a lively debate on the subject and it will be some time before there is agreement. More generally, any scholarly article is a claim, and should never be read as automatically true. That's something to keep in mind.

circuit10 3y ago
I thought it was completely obvious that if it can predict things about the world as well as it can it has some form of world model, even if inaccurate. I’m surprised that anyone would argue that a language model doesn’t model things
- YeGoblynQueenne 3y ago
  Yes, I see what you mean. It's frustrating but the word "model" is severely overused in Computer Science and AI and it can cause a lot of confusion. Briefly, a "world model" is a theory possessed by an autonomous agent that describes the entities that exist in the world and how they interact with each other and with the agent, and that the agent can use to make decisions. This is the sense in which "model" is used when people talk about "model based" approaches to AI (such as planning, for instance, which is "the model based approach to autonomous behaviour"). A "language model" on the other hand is a statistical model of the text in a corpus. A statistical model is really a set of events, and their probabilities. The "events" in a statistical language model are usually word collocations, where a collocation is a word A found near a word B, for some measure of "near". If you've heard about "word embeddings", that's a model of word collocations in text corpora. So the claim that LLMs "have a model" is about LLMs being autonomous agents with a world-model, while "Large Language Model" is about statistical modelling. There's no reason why a statistical language model should have a world model.
  - circuit10 3y ago
    Since language often describes the world, I think a good language model must include a world model
    - YeGoblynQueenne 3y ago
      Well, this has been claimed often, namely by some of the people who have developed statistical language models [1] but it's really not obvious how that should work. Where would a language model find the world model? Where would it store it? And why would it even need it? Obviously I don't know how the human linguistic ability works but it's clear that for us, text, words, language, isn't carrying around with it a representation of the entire world. So where would a language model find it in its training text? For example if I say "I sat by the sea today", there's no representation of the sea in the word "sea", rather, the representation is inside my head and yours, and the word "sea" is only used to point to that representation [2]. Why would it ever be possible to derive the representation just from the pointer? Like I say, it's really not obvious and I don't think anyone has given a convincing explanation - it's usually words all the way down when people try. In any case, it seems that humans had most of the capabilities conferred by a world model (which we share with other animals) before we had language (which we do not - share with other animals), so it seems likely that a world model must be developed first, before linguistic ability can be built on top of it. But that's a conjecture, of course. I guess maybe, in principle, it should be possible to treat all of human cognition as some kind of hidden (latent) variable and then train a model to represent it only from its expression in text, but we don't really have any kind of modelling technique with that power. That would be a truly omnipotent model. So powerful, indeed, that it wouldn't need to do that in the first place- it would already be a kind of machine god. ______________ [1] e.g. I believe Tomáš Mikolov, one of the creators of word2vec, has claimed something like that but I'm not sure if I have a source for my own claim; more recently Ilya Sutskever, of OpenAI, has said words to that effect- I have a link somewhere I think, if you are curious. [2] That's obvious because the word "sea" doesn't mean anything to someone who doesn't know English, but they'll probably have a word for the sea in their language, so they'll have the concept, without needing a specific word for it.

andrewguenther 3y ago

I think LLMs are presenting some uncomfortable philosophical questions for people about how our own brains work and admitting that there is any kind of "intelligence" (even if very basic) in an LLM is an admission that our own brains may work in a similar manner.

YeGoblynQueenne 3y ago
For me, one of the most interesting things that have come out of LLMs is the confirmation that humans are very bad at reasoning and, consequently, its' a very bad idea to try and make machines that "think like humans", because that way we'll only make machines with none of the advantages of machines and all the disadvantages of computers. For instance -I'm not trying to be mean and I'm certainly not blaming you in particular, because I've seen this very often- but the reasoning that because LLMs can generate language, and humans can generate language not only LLMs are somehow like humans but also humans are like LLMs is not sound. For example, walls have ears, cats have ears, therefore walls are like cats and cats are like walls. That doesn't work because walls' ears are not like cats' ears and even if they were, that still wouldn't make walls cats and cats walls, it would just make them both entities with ears.
- famouswaffles 3y ago
  >because that way we'll only make machines with none of the advantages of machines and all the disadvantages of computers. There is no evidence, basically none whatsoever that general "perfect logical reasoning" is a thing that actually exists in the real world. None. No animal we've observed does it. Humans certainly don't do it. The only realm this idea actually works is Fiction. and this was not like for a lack of trying. Some of the greatest minds worked on this for decades and some people still don't seem to get it. Logic doesn't scale. They break at real world relationships. Logic systems are that guy in the stands yelling that he could've made the shot, while he's not even on the field.
  - YeezyMode 3y ago
    Would you say that this in itself is due to how incomplete human reasoning is in the first place? That as a result, our ideas of logic and what perfect logic looks like are bound to fail? Or are you saying that the purest mathematical representation of logic cannot scale to a point where they can model and predict real world relationships successfully?

anigbrowl 3y ago

Indeed. I was offering examples of how you could bootstrap cognitive processes at the beginning of the year and people were just sticking their fingers in their ears. A lot of people exercise their intellectual, economic, or political freedoms by rejecting others' efforts to gain the same things.

SiempreViernes 3y ago
Who is the subject in the last bit of your last sentence? More pointedly: are you claiming that OpenAI is doing LLM research to gain political freedoms?
- anigbrowl 3y ago
  No, the second sentence is a general observation on a widespread human behavior pattern. I don't feel like expanding it into an essay right now.
OneDonOne 3y ago
>A lot of people exercise their intellectual, economic, or political freedoms by rejecting others' efforts to gain the same things. Could you please explain what the above has to do with a Large Languange Model?
- anigbrowl 3y ago
  I'm talking about how many people react to any startling new information or idea. The advent of LLMs is just a relevant example.
  - OneDonOne 3y ago
    No new ideas have been advanced in anything reported here. A state machine change state is all that is going on here. What I'd like to know is why should anyone care about this preprint? Nothing has been stated on how this software will improve the life of the average human being. LK-99 at least would have had lead.somewhere if it worked.

fipar 3y ago

My only experience so far is with chatgpt, and I am not an expert in AI, or even just in LLMs. With those disclaimers, my interactions inform my opinion that the LLM behind chatgpt has no internal world model. It shows no understanding of basic facts and makes very silly mistakes very easily. I have my bias, like anyone else, but in the case of AI in particular, I should say that I don't think there's anything especially magical or sacred about conscience and the human brain (or the brain of other animals, for that matter), and I'm sure it must be possible to arrive at different forms of intelligence starting with the "inanimate" building blocks of hardware and software, but what I've seen so far in LLMs doesn't make me support your statement that they've crossed a certain line. In fact, I'm very much let down by the experience and I'm afraid once the hype goes down, we may be in for another AI winter. As I said at the start of the comment, I'm well aware I'm not a subject-matter expert. I'm open to being wrong. I just wanted to point out that not all of us unimpressed are in denial. I'd love to be in awe at an AI breakthrough, and I kind of feel I will be in awe sometime during my lifetime, but LLMs are not (yet?) that for me. Edit: s/are not in denial/are in denial/

ilaksh 3y ago
Have you used GPT-4?
- fipar 3y ago
  No, I've only used GPT-3 so far. Should I be excited about GPT-4? :) Examples of things I'd file as silly mistakes: - mentioning non-existing settings when asking it about a particular software. - responding to roughly "give me a way to list all rds clusters that are running a version that will reach EOL within the next 12 months" (which I know is almost impossible to do without scraping, as the EOLs are not returned by the API, but it was worth trying) with a query that compares the version with < 12.0. It did acknowledge the error when I asked "won't that just compare the version number" and, interestingly, it then did add to the response that the API does not provide EOL for versions, but the first response was IMHO bad in a very basic way. - providing wrong steps in an answer to "give me steps to do X", wrong in a way that would cause the process to go very bad, and then when I point out the problematic step, it responds with a seemingly random alternate step that is also wrong. I guess this part is the one that is less bad, because it loosely reminds me of people who are bad at reasoning and just give back random answers to a logic or math question. I must say it is surprisingly good (though it will still make up facts at times) when answering questions about elisp though. Paraphrasing Hofstadter, I'd say if I'm asked about the intelligence of chatgpt using gpt-3 I'd say it's slightly better than a thermostat or a mosquito, but not much more. Furthermore, my (admittedly not very valuable to others) gut feeling is this is not the path to get to what I'd consider an intelligent being, mostly because in my interactions, it makes lots of bad mistakes, and none of the good ones (here I'm using good and bad as qualifiers for mistakes based on my experiences as a parent, and as a mentor to junior peers, where a good mistake is something that lets you know they're on the right path even if they haven't arrived at the right answer/place yet). Try asking it about analogies, for example ("what's a good analogy to explain X to someone new to it"), the results I've gotten are underwhelming, when not just plain wrong. Edit: s/fall/file/ Formatting.

tayo42 3y ago

Seems like the hyperbolic conclusion to make? The paper doesn't make that claim. Simple ml models can take data and find patterns, like separate red and blue, or xy coordinates. Those are a small amount of dimensions and easy to reason about. This seems like the model found on its own another dimension to segregate data in deeper layers by generalizing and using context when learning. Its cool it can do that, but it still to me seems like drawing a best fit line just with higher dimensions. I'm sure someone will correct me if they think I am way off.

Sharlin 3y ago

It's the Chinese Room argument all over again. People hear "predicting the next token" and all they can imagine is some sort of a statistical database lookup, ignoring the fact that when you have a huge corpus of data with incredibly complex internal correlations and all that data also happens to correlate with some unknown external thing, it's almost certain that a powerful learner will end up modeling that external thing if and when doing so will cause a quantum leap in prediction performance! A model that includes the external-thing hypothesis will almost certainly be simpler, ceteris paribus, than a model that doesn't.

esafak 3y ago
That is what a model is; a parsimonious explanation of observations that admits extrapolation.
- Sharlin 3y ago
  Exactly!

gamblor956 3y ago

If you actually read the paper, the analysis absolutely does not support the claim that LLMs are capable of maintaining a spatial or chronological internal world model. The LLMs simply activated when locations or times were mentioned, with no understanding of the relationship between different places or different times beyond the fact that different places and times had different tags. Or in other words, no internal model of the world.

nyrikki 3y ago
The lack of generally accepted definition for knowledge and intelligence makes these conversations difficult but the above post is important to remember. The paper has nothing to do with the common sense problem. While it is a preprint, I do wonder how useful this will be to improve generalization. Termination analysis on linear polynomials is undecidable over the naturals without many constraints and is undecidable ofer the rationals. If you think about how sstrees or even rtrees work, this isn't surprising to me that space and time are useful for compression which is required for learnability. But I think people also see 'linear' and think that means easy. When it can mean serious restrictions on how it can be used. Hard to explain the connection I have in mind without serious math but here is a paper that may cause some issues with trying to use these internal structures outside of the learnability implications. https://arxiv.org/abs/2307.14805 https://arxiv.org/abs/2307.14805

School-Cotton 3y ago

> I think with this paper it becomes clear that the adamant denial was just human bias talking It already becomes clear after playing with ChatGPT-4 for five minutes. But a lot of people still refuse to accept it.

OneDonOne 3y ago

A bigger Chinese Room - is still a Chinese Room. We already went through this with ELIZA 60 years ago

deleted 3y ago
[deleted]