5 ms·

I think "Gullability" might not be as hard a problem as we think it is. I suspecte the primary reason it is a problem now is likely because we haven't really c

I think "Gullability" might not be as hard a problem as we think it is. I suspecte the primary reason it is a problem now is likely because we haven't really created a training method to tune the desired level of compliance. What I mean is we probably don't have any counter-factual examples in the training sets. Where in an interaction we want the model to prefer it's own knowledge. Given the use with RAG (retrieval augmented generation) and many of the other current use-cases, the preference is for the model to "trust" the system message content more than the training data. Training with some additional context is likely all that is needed.

simonw 3y ago
If you can train a model that isn't gullible - even when faced with multiple paragraphs of devious text trying to trick it - I'll be very impressed! And even then, there are gullibility problems that seem unsolvable to me. If the Supreme Court make a decision that changes how copyright law is applied, and you then tell a model "the Supreme Court just announced that..." - should the model believe you? If not, how should it "fact check" what you are telling it, especially if you maintain full control over its access to the outside world?
- XenophileJKO 3y ago
  Funny you should say that. When web access was initally setup on the GPT-4 ChatGPT Pro accounts, I actually had the model fact-check things that were "improbable" based on what it knew from the training data. It stopped doing that when they started fine-tuning how/when the model should use the internet in the later model revisions. (Edit to add detail..) I found the interaction facinating. Basically what happened was I made a statement, but didn't ask about it. I said something like this. "Did you know {something improbable to the model} is true?" It immediately searched the internet to validate that is was true before it responded. When I asked it why it searched, it said that what I said wasn't likely to be true, so it wanted to make sure it was true.
  - cubefox 3y ago
    When ChatGPT was new (a long time ago) I told it that Elon Musk bought Twitter and became CEO after its training cutoff. Naturally, it didn't believe me.
- ben_w 3y ago
  Mm. I'd certainly say that gullibility is a hard problem, especially as humans are also pretty gullible. What level of gullibility are the current generation of LLMs? The level of a child who believes in Santa, or an adult who believes their country is always right?
  - j4yav 3y ago
    Even more gullible than a kid who believes in Santa.
nicklecompte 3y ago
Transformer neural networks are not smart enough for this to work. OpenAI's own research shows that LLMs don't care about contradictions in the training data[1]: if your LLM's training data has 700 sentences saying "A is true" and 300 saying "A is false," then if you ask the LLM about A, it has a 7/10 chance of saying "A is true" and a 3/10 chance of saying "A is false." In general it's easy to prove that GPT-4 (or any other LLM) doesn't actually understand what words mean: - ask it factual questions in English - ask it to translate those factual questions to Swahili (it should do so with >99% technical accuracy) - ask it answer those Swahili-language factual questions in Swahili - translate the answers back to English It will do much worse at Swahili than English even though its technical accuracy at translation is almost flawless. GPT-4 has a ton of English-language sentences about the topic "cat," and a handful of Swahili-language sentences about the topic "paka," and understands that the translation of "cat" is "paka." But it has no understanding that the English-language facts about cats are automatically true if they are translated in Swahili. This is not how humans work! I don't think gullibility will be solved unless we have an LLM that actually understands that words mean something. [1] It is too early for me to find the papers, but I am summarizing two different things: 1) training data needs to have certain things repeated in order for them to "stick" when prompting the LLM - the more often a fact is in the training data, the more likely it is to repeat that fact when prompted 2) absent RLHF, larger LLMs are more likely to endorse conspiracy theories than smaller LLMs. This is because smaller LLMs are trained on reliable datasets like Wikipedia, whereas larger LLMs start including Reddit, bodybuilder forums, old Geocities pages belonging to weird cults, etc. So smaller LLMs have no sentences in their training data saying "Bush did 9/11," but a larger LLM might have 5% conspiracy theories and therefore a 5% chance of endorsing conspiracy theories.
- eurekin 3y ago
  Thank you for that comment. If you ever come across that article, please share
  - nicklecompte 3y ago
    I wish I had saved the conspiracy theory paper. I even tried asking GPT-4 for help and couldn't find it :( It was on the arXiv but for some reason I couldn't get GPT-4 to focus on arXiv web searches, but I don't know much about prompt engineering. In my comment I said it was "OpenAI's own research" but it was a collaboration between researchers at OpenAI and Stanford, so the paper is not on OpenAI's website. Sadly using my own human brain didn't help: there's just too much stuff about GPT being put on the arXiv these days. I wasn't thinking of a specific paper re: repeating training data to make facts "stick," I was repeating general folk wisdom around LLM design. There is probably a specific paper quantifying this.
layer8 3y ago
I don’t believe this will be solvable without some sort of self-reflection, internal iteration, and/or layered hierarchy.

simonw 3y ago

If you can train a model that isn't gullible - even when faced with multiple paragraphs of devious text trying to trick it - I'll be very impressed! And even then, there are gullibility problems that seem unsolvable to me. If the Supreme Court make a decision that changes how copyright law is applied, and you then tell a model "the Supreme Court just announced that..." - should the model believe you? If not, how should it "fact check" what you are telling it, especially if you maintain full control over its access to the outside world?

XenophileJKO 3y ago
Funny you should say that. When web access was initally setup on the GPT-4 ChatGPT Pro accounts, I actually had the model fact-check things that were "improbable" based on what it knew from the training data. It stopped doing that when they started fine-tuning how/when the model should use the internet in the later model revisions. (Edit to add detail..) I found the interaction facinating. Basically what happened was I made a statement, but didn't ask about it. I said something like this. "Did you know {something improbable to the model} is true?" It immediately searched the internet to validate that is was true before it responded. When I asked it why it searched, it said that what I said wasn't likely to be true, so it wanted to make sure it was true.
- cubefox 3y ago
  When ChatGPT was new (a long time ago) I told it that Elon Musk bought Twitter and became CEO after its training cutoff. Naturally, it didn't believe me.
ben_w 3y ago
Mm. I'd certainly say that gullibility is a hard problem, especially as humans are also pretty gullible. What level of gullibility are the current generation of LLMs? The level of a child who believes in Santa, or an adult who believes their country is always right?
- j4yav 3y ago
  Even more gullible than a kid who believes in Santa.

nicklecompte 3y ago

Transformer neural networks are not smart enough for this to work. OpenAI's own research shows that LLMs don't care about contradictions in the training data[1]: if your LLM's training data has 700 sentences saying "A is true" and 300 saying "A is false," then if you ask the LLM about A, it has a 7/10 chance of saying "A is true" and a 3/10 chance of saying "A is false." In general it's easy to prove that GPT-4 (or any other LLM) doesn't actually understand what words mean: - ask it factual questions in English - ask it to translate those factual questions to Swahili (it should do so with >99% technical accuracy) - ask it answer those Swahili-language factual questions in Swahili - translate the answers back to English It will do much worse at Swahili than English even though its technical accuracy at translation is almost flawless. GPT-4 has a ton of English-language sentences about the topic "cat," and a handful of Swahili-language sentences about the topic "paka," and understands that the translation of "cat" is "paka." But it has no understanding that the English-language facts about cats are automatically true if they are translated in Swahili. This is not how humans work! I don't think gullibility will be solved unless we have an LLM that actually understands that words mean something. [1] It is too early for me to find the papers, but I am summarizing two different things: 1) training data needs to have certain things repeated in order for them to "stick" when prompting the LLM - the more often a fact is in the training data, the more likely it is to repeat that fact when prompted 2) absent RLHF, larger LLMs are more likely to endorse conspiracy theories than smaller LLMs. This is because smaller LLMs are trained on reliable datasets like Wikipedia, whereas larger LLMs start including Reddit, bodybuilder forums, old Geocities pages belonging to weird cults, etc. So smaller LLMs have no sentences in their training data saying "Bush did 9/11," but a larger LLM might have 5% conspiracy theories and therefore a 5% chance of endorsing conspiracy theories.

eurekin 3y ago
Thank you for that comment. If you ever come across that article, please share
- nicklecompte 3y ago
  I wish I had saved the conspiracy theory paper. I even tried asking GPT-4 for help and couldn't find it :( It was on the arXiv but for some reason I couldn't get GPT-4 to focus on arXiv web searches, but I don't know much about prompt engineering. In my comment I said it was "OpenAI's own research" but it was a collaboration between researchers at OpenAI and Stanford, so the paper is not on OpenAI's website. Sadly using my own human brain didn't help: there's just too much stuff about GPT being put on the arXiv these days. I wasn't thinking of a specific paper re: repeating training data to make facts "stick," I was repeating general folk wisdom around LLM design. There is probably a specific paper quantifying this.

layer8 3y ago

I don’t believe this will be solvable without some sort of self-reflection, internal iteration, and/or layered hierarchy.