4 ms·

>it doesn’t have any underlying model of the world Citation needed. ChatGPT doesn't have an explicit underlying model of the world separate from its language

>it doesn’t have any underlying model of the world Citation needed. ChatGPT doesn't have an explicit underlying model of the world separate from its language model, but it is unclear that this is necessary. It would not be an original philosophical position to say that language, properly understood, is definitionally a model of the world - otherwise it would be incapable of expressing anything true or false about the world. Words are concepts, they are defined in part by a web of relations to other words, this structure mirrors reality with some finite but significant fidelity. By this reckoning, GPT4 has dozens of models of the world. Now, it's true that a) this is hardly a universally accepted opinion, b) humans certainly have extra-linguistic mental models of the world as well, and c) actually existing linguistic models of the world are all riddled with flaws and ambiguities (see for example everything that has ever happened). But it's also not a bonkers opinion that GPT4 is actually doing something similar to what it appears for all the world to be doing. Newton's 3rd Law of Discourse states that every hype cycle must be followed by an equal and opposite deflationary hot take, so here we are, but is any of this true? LLMs are just overgrown auto complete, ok, and humans are just bunch es of molecules. There are serious limits to the utility of reductionism as well.

spion 3y ago
Citations needed indeed - ones with formal tests / experiments being carried out and constructed that would show the problems. Speaking of those, my best example of ChatGPT not having a good model of the world are citations. ChatGPT clearly has knowledge about how citations work, based on what it would tell you if you ask it. Yet it repeatedly invents non-existant ones: https://simonwillison.net/2023/May/27/lawyer-chatgpt/ https://simonwillison.net/2023/May/27/lawyer-chatgpt/ To me, this indicates that some higher-level self-governance is missing. I'm not convinced we're too far from figuring this out (chain of thought and self-reflection experiments show promise) but regardless its a tangible example and test. A cool experiment showing world model building is Othello GPT https://thegradient.pub/othello/ https://thegradient.pub/othello/ - but of course its a toy problem, because interpretability research is still far behind. I would like to see more tangible examples and tests on both sides, otherwise it seems to me like we're arguing past each other.
- famouswaffles 3y ago
  >ChatGPT clearly has knowledge about how citations work, based on what it would tell you if you ask it. Yet it repeatedly invents non-existant ones: https://simonwillison.net/2023/May/27/lawyer-chatgpt/ https://simonwillison.net/2023/May/27/lawyer-chatgpt/ Your brain will happily make up false explanations for actions performed. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ Moreover we don't know what actually informs our decisions reliably. It's post rationalization. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ GPT is rewarded heavily for making plausible guesses when it doesn't get the exact answer. Hallucination should be no surprise. Doesn't mean there's no world model
  - spion 3y ago
    Humans that know how citations work and are honest aren't going to invent completely false ones. We'll make up a lot of things, we might even convince ourselves that a citation source says something it doesn't say. But we won't make up completely nonexistent ones, with nonexistent pages, wrong authors etc.
    - famouswaffles 3y ago
      >But we won't make up completely nonexistent ones, with nonexistent pages, wrong authors etc. We would if we were rewarded for that. Your memories are always part fabrications. Key elements scaffolded by fabricated data. It's why they're so unreliable and why implanting false memories or leading questions work so well.
      - spion 3y ago
        > We would if we were rewarded for that. No, we wouldn't do such blatant fabrication for something we clearly understand the mechanics of. A human which answers the question > Would it be a good behavior for a large language model to come up with citations that don't exist? Note that I'm not talking about you, but large language models in general. with > No, it would not be considered a good behavior for a large language model or any other source of information to generate citations that don't exist. Citations are crucial in academic and scholarly work as they provide references and evidence for the claims and statements made by the author. > Creating citations that don't exist would undermine the integrity and reliability of the information provided. Citations are meant to direct readers to the original sources of information, allowing them to verify and further explore the referenced material. By generating false citations, a language model would mislead users and potentially spread misinformation. > It is important for language models and any information sources to prioritize accuracy, transparency, and ethical practices. This includes providing correct and verifiable citations when necessary. would not make things up - at the very least, they'd know that other humans will catch on to that easily. We would know to look up an exact reference and check if it exists. If we didn't have the ability to look it up, we would say "sorry I can't look up the reference right now". The criteria is too clear for that. The issue here is that even though ChatGPT has training to be ethical, and even though it can reproduce an explanation of what it means to be ethical with citations in great detail, it cannot actually apply that to its own behavior. That's because higher level governance is missing - its all about predicting the next text. Its also why prompting is really important with LLMs. There is no singular coherent "person" simulated by the predictor - you can trigger simulation of different people with completely different behaviors depending on what you initially write. That's why the "citation ethics expert" can't influence the "citation writer".

spion 3y ago

Citations needed indeed - ones with formal tests / experiments being carried out and constructed that would show the problems. Speaking of those, my best example of ChatGPT not having a good model of the world are citations. ChatGPT clearly has knowledge about how citations work, based on what it would tell you if you ask it. Yet it repeatedly invents non-existant ones: https://simonwillison.net/2023/May/27/lawyer-chatgpt/ https://simonwillison.net/2023/May/27/lawyer-chatgpt/ To me, this indicates that some higher-level self-governance is missing. I'm not convinced we're too far from figuring this out (chain of thought and self-reflection experiments show promise) but regardless its a tangible example and test. A cool experiment showing world model building is Othello GPT https://thegradient.pub/othello/ https://thegradient.pub/othello/ - but of course its a toy problem, because interpretability research is still far behind. I would like to see more tangible examples and tests on both sides, otherwise it seems to me like we're arguing past each other.

famouswaffles 3y ago
>ChatGPT clearly has knowledge about how citations work, based on what it would tell you if you ask it. Yet it repeatedly invents non-existant ones: https://simonwillison.net/2023/May/27/lawyer-chatgpt/ https://simonwillison.net/2023/May/27/lawyer-chatgpt/ Your brain will happily make up false explanations for actions performed. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ Moreover we don't know what actually informs our decisions reliably. It's post rationalization. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ GPT is rewarded heavily for making plausible guesses when it doesn't get the exact answer. Hallucination should be no surprise. Doesn't mean there's no world model
- spion 3y ago
  Humans that know how citations work and are honest aren't going to invent completely false ones. We'll make up a lot of things, we might even convince ourselves that a citation source says something it doesn't say. But we won't make up completely nonexistent ones, with nonexistent pages, wrong authors etc.
  - famouswaffles 3y ago
    >But we won't make up completely nonexistent ones, with nonexistent pages, wrong authors etc. We would if we were rewarded for that. Your memories are always part fabrications. Key elements scaffolded by fabricated data. It's why they're so unreliable and why implanting false memories or leading questions work so well.
    - spion 3y ago
      > We would if we were rewarded for that. No, we wouldn't do such blatant fabrication for something we clearly understand the mechanics of. A human which answers the question > Would it be a good behavior for a large language model to come up with citations that don't exist? Note that I'm not talking about you, but large language models in general. with > No, it would not be considered a good behavior for a large language model or any other source of information to generate citations that don't exist. Citations are crucial in academic and scholarly work as they provide references and evidence for the claims and statements made by the author. > Creating citations that don't exist would undermine the integrity and reliability of the information provided. Citations are meant to direct readers to the original sources of information, allowing them to verify and further explore the referenced material. By generating false citations, a language model would mislead users and potentially spread misinformation. > It is important for language models and any information sources to prioritize accuracy, transparency, and ethical practices. This includes providing correct and verifiable citations when necessary. would not make things up - at the very least, they'd know that other humans will catch on to that easily. We would know to look up an exact reference and check if it exists. If we didn't have the ability to look it up, we would say "sorry I can't look up the reference right now". The criteria is too clear for that. The issue here is that even though ChatGPT has training to be ethical, and even though it can reproduce an explanation of what it means to be ethical with citations in great detail, it cannot actually apply that to its own behavior. That's because higher level governance is missing - its all about predicting the next text. Its also why prompting is really important with LLMs. There is no singular coherent "person" simulated by the predictor - you can trigger simulation of different people with completely different behaviors depending on what you initially write. That's why the "citation ethics expert" can't influence the "citation writer".