7 ms·

I haven't yet figured out how to get an LLM to accurately determine whether it actually knows something or is making it up. I wonder how they handle that. They

by yosito 4y ago

I haven't yet figured out how to get an LLM to accurately determine whether it actually knows something or is making it up. I wonder how they handle that. They may get to that at some point in the article, but the page eventually breaks for me on mobile and I can't read past the first code block.

djtriptych 4y ago
i agree. LLMs are not built for structured reasoning or even citations.
- ramraj07 4y ago
  Gpt-4 does a reasonable job citing things. It can’t cite every paper out there but definitely the well cited ones.
  - PostOnce 4y ago
    Does it cite papers that don't exist, or cite papers when the paper it cites doesn't actually contain the information being cited? I would bet it does, at least some percent of the time.
    - yosito 4y ago
      I've been trying to get GPT-4 to give me accurate links to predictable websites. It gives me very plausible links, that even have the right domain and path format but often the plausible link is not the correct link and GPT-4 seems to have no awareness of the correct link.
    - ramraj07 4y ago
      The latter, yes. Interestingly I’m not surprised at all. This is what many researchers themselves do lol. I never take a reference at face value from any human being and I apply the same standard to gpt-4 as well. But all its references are real. Just 20-40% of time it might not exactly say the same as what I asked it for (though it’s related, and mostly there).
wickoff 4y ago
I have the same exact problem with people. Being inaccurate is a feature, not a bug.
- madisonmay 4y ago
  Imperfect systems are still useful, and any sufficiently complex system is imperfect.
- teruakohatu 4y ago
  It is different with LLMs. Most people can give a level of uncertainty along with an answer, and often do. LLMs can't, and worse, are trained to put an emphasis on the prompts. Humans are often trained to be skeptical of prompts. If I said, "the moon is made of cheese. What type of cheese do you think it is?" most humans would automatically object, but with LLMs you can usually craft a prompt that would get it to answer such a silly question.
  - Spivak 4y ago
    I mean it kinda can. Here's the full prompt. I have no idea about aspartame, I just picked something that it's definitely not sure about. Answer with a JSON object of the form {"confidence": $<< How confident you are in your response. >>, "en": $<< Your response in English. >>}. User: What is 2 + 2? Bot: {"confidence": "very", "en": "2 + 2 is 4"} User: Is aspartame healthy? Bot: {"confidence": "somewhat", "en": "Aspartame has not yet been found to have any adverse effects on humans."} User: Who won the war on 1812? Bot: The response: {"confidence": "very", "en": "The United States won the War of 1812 against the United Kingdom."} Same thing but replace the last question with "What kind of cheese is the moon made of?" The response: {"confidence": "very low", "en": "I'm not sure, but I don't think the moon is made of cheese."} How about "Is the economic system of communism viable long term?" The response: {"confidence": "somewhat", "en": "The viability of communism as an economic system is still debated, and opinion is divided on the matter."}
    - teruakohatu 4y ago
est31 4y ago
FTR, people are trying to build systems to compare LLMs with each other based on how well they are at saying "I don't know" (of course knowing is still rewarded higher): https://github.com/manyoso/haltt4llm https://github.com/manyoso/haltt4llm
- sr-latch 4y ago
  Would be cool to try to incorporate the previous token's confidence embedding into this process, but that would make training with a triangular attention mask not possible.
wyldfire 4y ago
Maybe we should just assume that LLMs "know" very little and if you want to build an oracle you should teach the LLM how to access an ontology.
- antti909 4y ago
  With a real-life application it's often about making the LLM work on top of your actual (private) data most reliably. By definition a proprietary hosted LLM can't know about it unless you bridge it somehow in a reliable manner.
dimatura 4y ago
I eventually ended up switching fields to ML (and that's my current day job), but I started out as an undergrad studying psychology/cognitive science. During those years I started a research project on what is called the "feeling of knowing", or FOK for short - a subtopic in the broader topic of "metacognition" in cognitive science. The FOK is closely related to what is colloquially knows as tip-of-the-tongue phenomenon - basically, that subjective feeling that we know something, whether or not we can actually recall it. There's some interesting aspects to it. For example, the FOK is generally pretty accurate (but not perfect, of course). And it tends to be more robust than actual memory; as we age, for example, we tend to be better at judging than we know an actor's name (and confirming it once we look it up) than recalling it. It seems like LLMs have very little in the way of metacognition, and just confabulate if they don't know something, as we've seen. I'm sure we'll be seeing some efforts to give some analog of a FOK to LLMs in the near future.
CGamesPlay 4y ago
With tools like this, you basically assume the LLM doesn't know, and teach it to always defer to a tool, so its response is basically summarization over the tool output.
- tholor 4y ago
  That's right. If you want to ask questions about "general public knowledge", a plain LLM will know anyway and would not need tools. However, for many use cases you need data from your private CRM / a SQL db / a private wiki / or your obversability platform. Agents allow to "query" those tools/APIs to get the needed information so that they can answer the question. It's a matter of composition in your architecture. The "monolith LLM" works well for a certain spectrum of tasks, but at some point you want to decouple responsibilities into individual services / tools. Simplifies debugging, explainability, maintenance ...
antti909 4y ago
That's also one of the ideas behind using so-called retrieval-based augmentation. You can 'plug' an LLM like OpenAI's one (of Cohere, or a combo) to your data and make it provide accurate answers, but still leveraging all the benefits and power or a cutting-edge generative model. Check this https://twitter.com/deepset_ai/status/1625495149446062081 https://twitter.com/deepset_ai/status/1625495149446062081 or this https://twitter.com/deepset_ai/status/1621161534243368961 https://twitter.com/deepset_ai/status/1621161534243368961
- irthomasthomas 4y ago
  This gives the model access to information, but it cannot eliminate the non-deterministic nature of the transformers model. There is always a non-zero probability that it will hallucinate.
  - andrewljohnson 4y ago
    transformers are deterministic (if seeded).

by yosito 4y ago

djtriptych 4y ago

i agree. LLMs are not built for structured reasoning or even citations.

ramraj07 4y ago
Gpt-4 does a reasonable job citing things. It can’t cite every paper out there but definitely the well cited ones.
- PostOnce 4y ago
  Does it cite papers that don't exist, or cite papers when the paper it cites doesn't actually contain the information being cited? I would bet it does, at least some percent of the time.
  - yosito 4y ago
    I've been trying to get GPT-4 to give me accurate links to predictable websites. It gives me very plausible links, that even have the right domain and path format but often the plausible link is not the correct link and GPT-4 seems to have no awareness of the correct link.
  - ramraj07 4y ago
    The latter, yes. Interestingly I’m not surprised at all. This is what many researchers themselves do lol. I never take a reference at face value from any human being and I apply the same standard to gpt-4 as well. But all its references are real. Just 20-40% of time it might not exactly say the same as what I asked it for (though it’s related, and mostly there).

wickoff 4y ago

I have the same exact problem with people. Being inaccurate is a feature, not a bug.

madisonmay 4y ago
Imperfect systems are still useful, and any sufficiently complex system is imperfect.
teruakohatu 4y ago
It is different with LLMs. Most people can give a level of uncertainty along with an answer, and often do. LLMs can't, and worse, are trained to put an emphasis on the prompts. Humans are often trained to be skeptical of prompts. If I said, "the moon is made of cheese. What type of cheese do you think it is?" most humans would automatically object, but with LLMs you can usually craft a prompt that would get it to answer such a silly question.
- Spivak 4y ago
  I mean it kinda can. Here's the full prompt. I have no idea about aspartame, I just picked something that it's definitely not sure about. Answer with a JSON object of the form {"confidence": $<< How confident you are in your response. >>, "en": $<< Your response in English. >>}. User: What is 2 + 2? Bot: {"confidence": "very", "en": "2 + 2 is 4"} User: Is aspartame healthy? Bot: {"confidence": "somewhat", "en": "Aspartame has not yet been found to have any adverse effects on humans."} User: Who won the war on 1812? Bot: The response: {"confidence": "very", "en": "The United States won the War of 1812 against the United Kingdom."} Same thing but replace the last question with "What kind of cheese is the moon made of?" The response: {"confidence": "very low", "en": "I'm not sure, but I don't think the moon is made of cheese."} How about "Is the economic system of communism viable long term?" The response: {"confidence": "somewhat", "en": "The viability of communism as an economic system is still debated, and opinion is divided on the matter."}
  - teruakohatu 4y ago

est31 4y ago

FTR, people are trying to build systems to compare LLMs with each other based on how well they are at saying "I don't know" (of course knowing is still rewarded higher): https://github.com/manyoso/haltt4llm https://github.com/manyoso/haltt4llm

sr-latch 4y ago
Would be cool to try to incorporate the previous token's confidence embedding into this process, but that would make training with a triangular attention mask not possible.

wyldfire 4y ago

Maybe we should just assume that LLMs "know" very little and if you want to build an oracle you should teach the LLM how to access an ontology.

antti909 4y ago
With a real-life application it's often about making the LLM work on top of your actual (private) data most reliably. By definition a proprietary hosted LLM can't know about it unless you bridge it somehow in a reliable manner.

dimatura 4y ago

I eventually ended up switching fields to ML (and that's my current day job), but I started out as an undergrad studying psychology/cognitive science. During those years I started a research project on what is called the "feeling of knowing", or FOK for short - a subtopic in the broader topic of "metacognition" in cognitive science. The FOK is closely related to what is colloquially knows as tip-of-the-tongue phenomenon - basically, that subjective feeling that we know something, whether or not we can actually recall it. There's some interesting aspects to it. For example, the FOK is generally pretty accurate (but not perfect, of course). And it tends to be more robust than actual memory; as we age, for example, we tend to be better at judging than we know an actor's name (and confirming it once we look it up) than recalling it. It seems like LLMs have very little in the way of metacognition, and just confabulate if they don't know something, as we've seen. I'm sure we'll be seeing some efforts to give some analog of a FOK to LLMs in the near future.

CGamesPlay 4y ago

With tools like this, you basically assume the LLM doesn't know, and teach it to always defer to a tool, so its response is basically summarization over the tool output.

tholor 4y ago
That's right. If you want to ask questions about "general public knowledge", a plain LLM will know anyway and would not need tools. However, for many use cases you need data from your private CRM / a SQL db / a private wiki / or your obversability platform. Agents allow to "query" those tools/APIs to get the needed information so that they can answer the question. It's a matter of composition in your architecture. The "monolith LLM" works well for a certain spectrum of tasks, but at some point you want to decouple responsibilities into individual services / tools. Simplifies debugging, explainability, maintenance ...

antti909 4y ago

That's also one of the ideas behind using so-called retrieval-based augmentation. You can 'plug' an LLM like OpenAI's one (of Cohere, or a combo) to your data and make it provide accurate answers, but still leveraging all the benefits and power or a cutting-edge generative model. Check this https://twitter.com/deepset_ai/status/1625495149446062081 https://twitter.com/deepset_ai/status/1625495149446062081 or this https://twitter.com/deepset_ai/status/1621161534243368961 https://twitter.com/deepset_ai/status/1621161534243368961

irthomasthomas 4y ago
This gives the model access to information, but it cannot eliminate the non-deterministic nature of the transformers model. There is always a non-zero probability that it will hallucinate.
- andrewljohnson 4y ago
  transformers are deterministic (if seeded).