4 ms·

Fixing hallucinations from the source will be a tough one. The root of the issue is that loss doesn't discriminate. A probable guess will lower loss much better

Fixing hallucinations from the source will be a tough one. The root of the issue is that loss doesn't discriminate. A probable guess will lower loss much better than "I don't know" or whatever equivalent. Educated guessing becomes an essential skill the model learns during initial training. So the objective function encourages it. But also the dataset encourages it as well. There will be many many sentences that can't be completed accurate to source even with all the knowledge and understanding in the world. Many completions will have numerous sensible options. The dataset doesn't discriminate. Fiction, Fact, Opinion, Mistake. All the same. All given equal weight.

valine 4y ago
The datasets are massive, and curating them into buckets of fact/fiction by hand would be next to impossible. Automating curation of the datasets with the latest and greatest GPT sounds very possible however. GPT-supervised learning could be the key to bootstrapping robust models, and I would be shocked if OpenAI isn't using GPT4 to filter their datasets as we speak.
- hutzlibu 4y ago
  "Automating curation of the datasets with the latest and greatest GPT sounds very possible however." It might also lead to recursive garbage generation. If the AI has flaws and those flaws are from the initial training, then I see no way, how the flawed data can ever generate clean data. "The datasets are massive, and curating them into buckets of fact/fiction by hand would be next to impossible. " And it is not impossible. It is just a lot of work. And maybe work we just have to do, if we want to create reliable AIs, that do not fail at random times. Now this alone might not completely remove hallucinations, but solid training data is just the base of it all. (and since we are living in the area of fake news, I welcome all efforts towards established facts and data out of general principle)
  - AlanSE 4y ago
    Funny enough, in 2023, the option most likely to be viable for the impossible task of categorizing the training set would be to use a LLM. While I don't "trust" these AIs to give accurate information, it's probably within their capabilities to categorize by the above mentioned categories... then feed that back into another (very expensive) round of training, along with some theoretical developments to boot. I do think this is within the realm of possibility in the next ~1 year, but would be hard.
    - hutzlibu 4y ago
      Oh, I surely think LLM's could help with the task of curation. Maybe even spot lots of potential errors and flaws by themself, to get to the worst cases in the dataset faster. But to finally confirm or negate the actual data in question, there has to be at least one (not overworked) human in the loop (and many eyes would be better). Otherwise it will just reinforce the existing flaws.
- worrycue 4y ago
  I don’t think it’s just a matter of curating data sets for accuracy. The model seem to be able to invent falsehoods on its own based on no data whatsoever - e.g. I read that researchers asked an LLM about a made up biochemist and the LLM generated an entire fictional history for it.
  - valine 4y ago
    My hope would be that by labeling factual and fictional data it would coax the model into a state where it doesn’t invent facts outside of a creative writing context. Inventing falsehoods is practically the definition of creative writing, you don’t want the model to lose that ability. This is purely hypothetical, but I imagine that the internal mechanisms the model uses to invent stories out of thin air are different from the mechanisms the model uses to recall precise facts. Providing additional labels could help sharpen the division between creative and factual writing tasks.
    - SrslyJosh 4y ago
      The model can't "invent facts" because it doesn't know what facts are. It's a statistical model that encodes information about the text it was trained on. It has no higher level abstractions to inform its output. > I imagine that the internal mechanisms the model uses to invent stories out of thin air are different from the mechanisms the model uses to recall precise facts. Nope! If you input "What is the capital of the United States of America?", it's probably going to output the correct answer, because the question (and answer) probably appeared many times in the training data. If you input "What is the capital of the Gronk Republic?", whatever output you get is generated via the exact same mechanism.
      - valine 4y ago
        That’s the “LLMs are a fuzzy jpeg of the web” theory, and it’s far, far from the scientific consensus. No one knows for sure exactly what happens inside a 500B parameter model. I’ll just leave you with GPT4’s response to your question “What is the capital of the Gronk Republic”. “As an AI language model, I am not aware of any existing country or political entity called the "Gronk Republic." It could be a fictional or hypothetical place, in which case the capital would be determined by the creator or context in which it is mentioned. If you are referring to a real location, please provide more information or clarify the name of the place you are asking about.”
- dr_dshiv 4y ago
  > The datasets are massive, and curating them into buckets of fact/fiction by hand would be next to impossible No, only difficult.
fshbbdssbbgdd 4y ago
It would not be hard to generate a lot of false statements from structured data. Like if I know Michael Jackson was born on Aug 29, I can generate “Michael Jackson was born on July 5”. And you could also pair them with true examples which have similar characteristics. Can we use examples like this in the training process to teach the model not to hallucinate?
- SaidinWoT 4y ago
  But Michael Jackson was born on April 19th[0]! As well as March 27th[1], and many other days[2]. "Michael Jackson was born on August 29th" is the most likely answer to contextless queries like "When was Michael Jackson born?", but that does not make structurally identical sentences with different information false, merely less probable to be contextually correct. [0] https://en.wikipedia.org/wiki/Michael_Jackson_(radio_commentator) https://en.wikipedia.org/wiki/Michael_Jackson_(radio_comment... [1] https://en.wikipedia.org/wiki/Michael_Jackson_(writer) https://en.wikipedia.org/wiki/Michael_Jackson_(writer) [2] https://en.wikipedia.org/wiki/Michael_Jackson_(disambiguation) https://en.wikipedia.org/wiki/Michael_Jackson_(disambiguatio...
  - draw_down 4y ago
    [dead]
tshadley 4y ago
> A probable guess will lower loss much better than "I don't know" or whatever equivalent. Guessing only reduces loss as much as the dataset allows -- a bad guess will give a higher loss. The model learns to assign probabilities to its guesses, just like we do. It seems to me all we need here is a measure of confidence for the result averaged over the entire answer. Low confidence is a guess/hallucination. > But also the dataset encourages it as well. There will be many many sentences that can't be completed accurate to source even with all the knowledge and understanding in the world. Many completions will have numerous sensible options. The dataset doesn't discriminate. Fiction, Fact, Opinion, Mistake. All the same. All given equal weight. This is an important issue but should be tackled as a distinctly different problem I think: it's the weighty concept of truth that humanity struggled with from day 1. Indeed, how do we discriminate? LLMs won't ever solve this via completions or dataset alone; instead successful models will use slow, step-by-step reasoning involving logical principles and rational heuristics in prompt space. Pretty much like we do.
- bcherny 4y ago
  The reason humans tend to tell the truth is if we don’t, other humans will call us out for it. I wonder if there’s a way to mimic this “bs penalty” for GPT. Maybe you could have a setup where GPT gives an answer, then a second GPT has to guess whether a human would know if that answer is true or not.
  - dr_dshiv 4y ago
    This I think is the approach—a dialectic of LLMs that can critique and synthesize. Although, this will surely be a solved problem once we have TruthGPT, right? ;)
- hakuseki 4y ago
  > It seems to me all we need here is a measure of confidence for the result averaged over the entire answer. Low confidence is a guess/hallucination. Even if the model knows the exact answer to the question, there may be many distinct ways of phrasing the answer. This would also lead to low confidence in any particular phrasing.
codebolt 4y ago
Couldn't you train the model to keep score (develop a heuristic) for it's own level of certainty for a given answer?
- blackbear_ 4y ago
  It has already been done: http://arxiv.org/abs/2207.05221 http://arxiv.org/abs/2207.05221 However, it seems that RLHF considerably reduces the model's calibration, so perhaps the method above won't be applicable to ChatGPT and similar.
_fat_santa 4y ago
Over time I realized that the strength of GPT is not really in fact finding, because there will always be a limitation with it's training dataset, it's strength is literally in language. One of the most useful things I've gotten ChatGPT to do is to build a friends resume going into tech, being able to write a very unprofessional sentence like: "did some react, css and js coding" and just tell it to "make it sound more professional with metrics" and it spits out a perfect bulleted list in the exact structure that every article on resumes of the past 10 years told you to do. Writing marketing copies is another place I really found it useful, almost as if I was sitting with a marketing professional telling them "I want to say this: ...." and they return me a perfect marketing copy. I don't think the solution to GPT is to "fix" the hallucinations, rather it would be to educate users on what is and isn't possible with it. I think a similar thing happened with Siri when it first came out, people thought it was magic at first but very quickly we all learned what it's good (and not good) at.

valine 4y ago

The datasets are massive, and curating them into buckets of fact/fiction by hand would be next to impossible. Automating curation of the datasets with the latest and greatest GPT sounds very possible however. GPT-supervised learning could be the key to bootstrapping robust models, and I would be shocked if OpenAI isn't using GPT4 to filter their datasets as we speak.

hutzlibu 4y ago
"Automating curation of the datasets with the latest and greatest GPT sounds very possible however." It might also lead to recursive garbage generation. If the AI has flaws and those flaws are from the initial training, then I see no way, how the flawed data can ever generate clean data. "The datasets are massive, and curating them into buckets of fact/fiction by hand would be next to impossible. " And it is not impossible. It is just a lot of work. And maybe work we just have to do, if we want to create reliable AIs, that do not fail at random times. Now this alone might not completely remove hallucinations, but solid training data is just the base of it all. (and since we are living in the area of fake news, I welcome all efforts towards established facts and data out of general principle)
- AlanSE 4y ago
  Funny enough, in 2023, the option most likely to be viable for the impossible task of categorizing the training set would be to use a LLM. While I don't "trust" these AIs to give accurate information, it's probably within their capabilities to categorize by the above mentioned categories... then feed that back into another (very expensive) round of training, along with some theoretical developments to boot. I do think this is within the realm of possibility in the next ~1 year, but would be hard.
  - hutzlibu 4y ago
    Oh, I surely think LLM's could help with the task of curation. Maybe even spot lots of potential errors and flaws by themself, to get to the worst cases in the dataset faster. But to finally confirm or negate the actual data in question, there has to be at least one (not overworked) human in the loop (and many eyes would be better). Otherwise it will just reinforce the existing flaws.
worrycue 4y ago
I don’t think it’s just a matter of curating data sets for accuracy. The model seem to be able to invent falsehoods on its own based on no data whatsoever - e.g. I read that researchers asked an LLM about a made up biochemist and the LLM generated an entire fictional history for it.
- valine 4y ago
  My hope would be that by labeling factual and fictional data it would coax the model into a state where it doesn’t invent facts outside of a creative writing context. Inventing falsehoods is practically the definition of creative writing, you don’t want the model to lose that ability. This is purely hypothetical, but I imagine that the internal mechanisms the model uses to invent stories out of thin air are different from the mechanisms the model uses to recall precise facts. Providing additional labels could help sharpen the division between creative and factual writing tasks.
  - SrslyJosh 4y ago
    The model can't "invent facts" because it doesn't know what facts are. It's a statistical model that encodes information about the text it was trained on. It has no higher level abstractions to inform its output. > I imagine that the internal mechanisms the model uses to invent stories out of thin air are different from the mechanisms the model uses to recall precise facts. Nope! If you input "What is the capital of the United States of America?", it's probably going to output the correct answer, because the question (and answer) probably appeared many times in the training data. If you input "What is the capital of the Gronk Republic?", whatever output you get is generated via the exact same mechanism.
    - valine 4y ago
      That’s the “LLMs are a fuzzy jpeg of the web” theory, and it’s far, far from the scientific consensus. No one knows for sure exactly what happens inside a 500B parameter model. I’ll just leave you with GPT4’s response to your question “What is the capital of the Gronk Republic”. “As an AI language model, I am not aware of any existing country or political entity called the "Gronk Republic." It could be a fictional or hypothetical place, in which case the capital would be determined by the creator or context in which it is mentioned. If you are referring to a real location, please provide more information or clarify the name of the place you are asking about.”
dr_dshiv 4y ago
> The datasets are massive, and curating them into buckets of fact/fiction by hand would be next to impossible No, only difficult.

fshbbdssbbgdd 4y ago

It would not be hard to generate a lot of false statements from structured data. Like if I know Michael Jackson was born on Aug 29, I can generate “Michael Jackson was born on July 5”. And you could also pair them with true examples which have similar characteristics. Can we use examples like this in the training process to teach the model not to hallucinate?

SaidinWoT 4y ago
But Michael Jackson was born on April 19th[0]! As well as March 27th[1], and many other days[2]. "Michael Jackson was born on August 29th" is the most likely answer to contextless queries like "When was Michael Jackson born?", but that does not make structurally identical sentences with different information false, merely less probable to be contextually correct. [0] https://en.wikipedia.org/wiki/Michael_Jackson_(radio_commentator) https://en.wikipedia.org/wiki/Michael_Jackson_(radio_comment... [1] https://en.wikipedia.org/wiki/Michael_Jackson_(writer) https://en.wikipedia.org/wiki/Michael_Jackson_(writer) [2] https://en.wikipedia.org/wiki/Michael_Jackson_(disambiguation) https://en.wikipedia.org/wiki/Michael_Jackson_(disambiguatio...
- draw_down 4y ago
  [dead]

tshadley 4y ago

> A probable guess will lower loss much better than "I don't know" or whatever equivalent. Guessing only reduces loss as much as the dataset allows -- a bad guess will give a higher loss. The model learns to assign probabilities to its guesses, just like we do. It seems to me all we need here is a measure of confidence for the result averaged over the entire answer. Low confidence is a guess/hallucination. > But also the dataset encourages it as well. There will be many many sentences that can't be completed accurate to source even with all the knowledge and understanding in the world. Many completions will have numerous sensible options. The dataset doesn't discriminate. Fiction, Fact, Opinion, Mistake. All the same. All given equal weight. This is an important issue but should be tackled as a distinctly different problem I think: it's the weighty concept of truth that humanity struggled with from day 1. Indeed, how do we discriminate? LLMs won't ever solve this via completions or dataset alone; instead successful models will use slow, step-by-step reasoning involving logical principles and rational heuristics in prompt space. Pretty much like we do.

bcherny 4y ago
The reason humans tend to tell the truth is if we don’t, other humans will call us out for it. I wonder if there’s a way to mimic this “bs penalty” for GPT. Maybe you could have a setup where GPT gives an answer, then a second GPT has to guess whether a human would know if that answer is true or not.
- dr_dshiv 4y ago
  This I think is the approach—a dialectic of LLMs that can critique and synthesize. Although, this will surely be a solved problem once we have TruthGPT, right? ;)
hakuseki 4y ago
> It seems to me all we need here is a measure of confidence for the result averaged over the entire answer. Low confidence is a guess/hallucination. Even if the model knows the exact answer to the question, there may be many distinct ways of phrasing the answer. This would also lead to low confidence in any particular phrasing.

codebolt 4y ago

Couldn't you train the model to keep score (develop a heuristic) for it's own level of certainty for a given answer?

blackbear_ 4y ago
It has already been done: http://arxiv.org/abs/2207.05221 http://arxiv.org/abs/2207.05221 However, it seems that RLHF considerably reduces the model's calibration, so perhaps the method above won't be applicable to ChatGPT and similar.

_fat_santa 4y ago

Over time I realized that the strength of GPT is not really in fact finding, because there will always be a limitation with it's training dataset, it's strength is literally in language. One of the most useful things I've gotten ChatGPT to do is to build a friends resume going into tech, being able to write a very unprofessional sentence like: "did some react, css and js coding" and just tell it to "make it sound more professional with metrics" and it spits out a perfect bulleted list in the exact structure that every article on resumes of the past 10 years told you to do. Writing marketing copies is another place I really found it useful, almost as if I was sitting with a marketing professional telling them "I want to say this: ...." and they return me a perfect marketing copy. I don't think the solution to GPT is to "fix" the hallucinations, rather it would be to educate users on what is and isn't possible with it. I think a similar thing happened with Siri when it first came out, people thought it was magic at first but very quickly we all learned what it's good (and not good) at.