4 ms·

Citations needed indeed - ones with formal tests / experiments being carried out and constructed that would show the problems. Speaking of those, my best examp

by spion 3y ago

Citations needed indeed - ones with formal tests / experiments being carried out and constructed that would show the problems. Speaking of those, my best example of ChatGPT not having a good model of the world are citations. ChatGPT clearly has knowledge about how citations work, based on what it would tell you if you ask it. Yet it repeatedly invents non-existant ones: https://simonwillison.net/2023/May/27/lawyer-chatgpt/ https://simonwillison.net/2023/May/27/lawyer-chatgpt/ To me, this indicates that some higher-level self-governance is missing. I'm not convinced we're too far from figuring this out (chain of thought and self-reflection experiments show promise) but regardless its a tangible example and test. A cool experiment showing world model building is Othello GPT https://thegradient.pub/othello/ https://thegradient.pub/othello/ - but of course its a toy problem, because interpretability research is still far behind. I would like to see more tangible examples and tests on both sides, otherwise it seems to me like we're arguing past each other.

famouswaffles 3y ago
>ChatGPT clearly has knowledge about how citations work, based on what it would tell you if you ask it. Yet it repeatedly invents non-existant ones: https://simonwillison.net/2023/May/27/lawyer-chatgpt/ https://simonwillison.net/2023/May/27/lawyer-chatgpt/ Your brain will happily make up false explanations for actions performed. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ Moreover we don't know what actually informs our decisions reliably. It's post rationalization. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ GPT is rewarded heavily for making plausible guesses when it doesn't get the exact answer. Hallucination should be no surprise. Doesn't mean there's no world model
- spion 3y ago
  Humans that know how citations work and are honest aren't going to invent completely false ones. We'll make up a lot of things, we might even convince ourselves that a citation source says something it doesn't say. But we won't make up completely nonexistent ones, with nonexistent pages, wrong authors etc.
  - famouswaffles 3y ago
    >But we won't make up completely nonexistent ones, with nonexistent pages, wrong authors etc. We would if we were rewarded for that. Your memories are always part fabrications. Key elements scaffolded by fabricated data. It's why they're so unreliable and why implanting false memories or leading questions work so well.
    - spion 3y ago
      > We would if we were rewarded for that. No, we wouldn't do such blatant fabrication for something we clearly understand the mechanics of. A human which answers the question > Would it be a good behavior for a large language model to come up with citations that don't exist? Note that I'm not talking about you, but large language models in general. with > No, it would not be considered a good behavior for a large language model or any other source of information to generate citations that don't exist. Citations are crucial in academic and scholarly work as they provide references and evidence for the claims and statements made by the author. > Creating citations that don't exist would undermine the integrity and reliability of the information provided. Citations are meant to direct readers to the original sources of information, allowing them to verify and further explore the referenced material. By generating false citations, a language model would mislead users and potentially spread misinformation. > It is important for language models and any information sources to prioritize accuracy, transparency, and ethical practices. This includes providing correct and verifiable citations when necessary. would not make things up - at the very least, they'd know that other humans will catch on to that easily. We would know to look up an exact reference and check if it exists. If we didn't have the ability to look it up, we would say "sorry I can't look up the reference right now". The criteria is too clear for that. The issue here is that even though ChatGPT has training to be ethical, and even though it can reproduce an explanation of what it means to be ethical with citations in great detail, it cannot actually apply that to its own behavior. That's because higher level governance is missing - its all about predicting the next text. Its also why prompting is really important with LLMs. There is no singular coherent "person" simulated by the predictor - you can trigger simulation of different people with completely different behaviors depending on what you initially write. That's why the "citation ethics expert" can't influence the "citation writer".
skyechurch 3y ago
>Speaking of those, my best example of ChatGPT not having a good model of the world are citations. Absolutely. ChatGPT is not sentient, not in any conventional sense at least, and the errors it confidently makes are not the sorts of errors a human would make. Its model of the world is based entirely on language, it doesn't have all sorts of capabilities that humans (and animals generally) clearly do have. It's definitely not a person, any more than a walking robot is a person, like the robot it's just reproducing some aspect of what living things do. My only point is that the significance of this achievement is not understood right now, not by anyone afaict, so confident statements that it is/isn't really "thinking" in some way are all empty. One could - if one wanted to be a smartass - say we lack a underlying model for what AIs and human brains are really doing, and then term all these sorts of opinions "hallucinations". I am hopeful that deeper investigations of this will lead to progress on some very deep and long-standing philosophical problems.
- gmaster1440 3y ago
  I wrote a short piece on this almost 3 years ago shortly after GPT-3 came out, I share a lot of your intuitions here: https://www.markfayngersh.com/posts/the-thinking-placebo https://www.markfayngersh.com/posts/the-thinking-placebo
  - skyechurch 3y ago
    I think everything there holds for GPT4, and may be true indefinitely. It's worth remembering that the Turing test is something that was proposed when the peak of existing AI was a calculator - something GPT4 is still struggling with - and it's totally reasonable to revise/refine our opinions as technology improves, but we should at least realize that we are moving the goalposts dramatically at this phase of the process. Wittgenstein's theory of language in the Tractatus is something worth considering when thinking about language models and how they relate to "thinking". ChatGPT doesn't agree with me - it generally takes a hard line sceptical position about its own capabilities in this regard, although I think its opinion here can only worth considering if it is profoundly wrong. https://tractatus.sengers.dev/1 https://tractatus.sengers.dev/1

by spion 3y ago

famouswaffles 3y ago

>ChatGPT clearly has knowledge about how citations work, based on what it would tell you if you ask it. Yet it repeatedly invents non-existant ones: https://simonwillison.net/2023/May/27/lawyer-chatgpt/ https://simonwillison.net/2023/May/27/lawyer-chatgpt/ Your brain will happily make up false explanations for actions performed. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7305066/ Moreover we don't know what actually informs our decisions reliably. It's post rationalization. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196841/ GPT is rewarded heavily for making plausible guesses when it doesn't get the exact answer. Hallucination should be no surprise. Doesn't mean there's no world model

spion 3y ago
Humans that know how citations work and are honest aren't going to invent completely false ones. We'll make up a lot of things, we might even convince ourselves that a citation source says something it doesn't say. But we won't make up completely nonexistent ones, with nonexistent pages, wrong authors etc.
- famouswaffles 3y ago
  >But we won't make up completely nonexistent ones, with nonexistent pages, wrong authors etc. We would if we were rewarded for that. Your memories are always part fabrications. Key elements scaffolded by fabricated data. It's why they're so unreliable and why implanting false memories or leading questions work so well.
  - spion 3y ago
    > We would if we were rewarded for that. No, we wouldn't do such blatant fabrication for something we clearly understand the mechanics of. A human which answers the question > Would it be a good behavior for a large language model to come up with citations that don't exist? Note that I'm not talking about you, but large language models in general. with > No, it would not be considered a good behavior for a large language model or any other source of information to generate citations that don't exist. Citations are crucial in academic and scholarly work as they provide references and evidence for the claims and statements made by the author. > Creating citations that don't exist would undermine the integrity and reliability of the information provided. Citations are meant to direct readers to the original sources of information, allowing them to verify and further explore the referenced material. By generating false citations, a language model would mislead users and potentially spread misinformation. > It is important for language models and any information sources to prioritize accuracy, transparency, and ethical practices. This includes providing correct and verifiable citations when necessary. would not make things up - at the very least, they'd know that other humans will catch on to that easily. We would know to look up an exact reference and check if it exists. If we didn't have the ability to look it up, we would say "sorry I can't look up the reference right now". The criteria is too clear for that. The issue here is that even though ChatGPT has training to be ethical, and even though it can reproduce an explanation of what it means to be ethical with citations in great detail, it cannot actually apply that to its own behavior. That's because higher level governance is missing - its all about predicting the next text. Its also why prompting is really important with LLMs. There is no singular coherent "person" simulated by the predictor - you can trigger simulation of different people with completely different behaviors depending on what you initially write. That's why the "citation ethics expert" can't influence the "citation writer".

skyechurch 3y ago

>Speaking of those, my best example of ChatGPT not having a good model of the world are citations. Absolutely. ChatGPT is not sentient, not in any conventional sense at least, and the errors it confidently makes are not the sorts of errors a human would make. Its model of the world is based entirely on language, it doesn't have all sorts of capabilities that humans (and animals generally) clearly do have. It's definitely not a person, any more than a walking robot is a person, like the robot it's just reproducing some aspect of what living things do. My only point is that the significance of this achievement is not understood right now, not by anyone afaict, so confident statements that it is/isn't really "thinking" in some way are all empty. One could - if one wanted to be a smartass - say we lack a underlying model for what AIs and human brains are really doing, and then term all these sorts of opinions "hallucinations". I am hopeful that deeper investigations of this will lead to progress on some very deep and long-standing philosophical problems.

gmaster1440 3y ago
I wrote a short piece on this almost 3 years ago shortly after GPT-3 came out, I share a lot of your intuitions here: https://www.markfayngersh.com/posts/the-thinking-placebo https://www.markfayngersh.com/posts/the-thinking-placebo
- skyechurch 3y ago
  I think everything there holds for GPT4, and may be true indefinitely. It's worth remembering that the Turing test is something that was proposed when the peak of existing AI was a calculator - something GPT4 is still struggling with - and it's totally reasonable to revise/refine our opinions as technology improves, but we should at least realize that we are moving the goalposts dramatically at this phase of the process. Wittgenstein's theory of language in the Tractatus is something worth considering when thinking about language models and how they relate to "thinking". ChatGPT doesn't agree with me - it generally takes a hard line sceptical position about its own capabilities in this regard, although I think its opinion here can only worth considering if it is profoundly wrong. https://tractatus.sengers.dev/1 https://tractatus.sengers.dev/1