10 ms·

The proposed solution to eliminate hallucination is to ground the model with external data. This is the approach taken with Bing Chat, and while it kinda works,

by valine 3y ago

The proposed solution to eliminate hallucination is to ground the model with external data. This is the approach taken with Bing Chat, and while it kinda works, it doesn't play to the strengths of LLMs. Every time Bing Chat searches something for me I can't help but feel like I could have written a better search query myself. It feels like a clumsy summarization wrapper around traditional search, not a revolutionary new way of parsing information. Conversing with an LLM on subjects that it's well trained on, however, absolutely does feel like a revolutionary new way of parsing information. In my opinion we should be researching ways to fix hallucinations in the base model, not papering over it by augmenting the context window.

electric_mayhem 3y ago
Tl;dr: I think that a left brain/right brain parallel is in the cards for LLMs Our own brains have multiple neural circuits. Parallel, serial, competing, cumulative… Including ones which error-check others’ output. I guess the term in the ML niche is ‘adversarial networks’. From the robotics side, there was subsumption architecture which used the real world as a basis for informing decisions. So, I respectfully disagree that fixing up creative but occasionally erroneous networks is papering over the problem. If our own brains use multiple neural circuits to check each other to try to optimize for output that agrees with reality, why would it be preferable or logical to try to create a single artificial neural network that has perfect output? My sense from playing with LLMs is they’re knowledge without understanding. Weirdly akin to my own dreaming consciousness, to the extent that conscious me has seen it. It seems very natural to me that the next step would be integrating checks/validations in parallel and considering each half of a matched ‘whole’. It could still be considered a single network. Just made of two distinct smaller specialized ones.
- valine 3y ago
  Having a second adversarial network could absolutely be part of the solution, I wasn't arguing against that. My problem is with in context learning, ie the idea that including background information in the prompt can solve hallucination.
  - electric_mayhem 3y ago
    I see where you’re coming from, but I’ve had a couple occasions over the years that have led me to expect and accept that at any given time my brain harbors an abundance of different ideas and not all of them are valid. Like the time I decided to sit on the ceiling to spite a friend who was being pushy. Made perfect sense, and I was about to. Until a different part kicked in and assured the whole of myself that it wasn’t possible. Neural networks are going to spawn occasional nonsense. Aiming for perfection is noble, but nature shows a different path that already works. But fundamentally I think we’re agreeing- just the twist is I’m saying LLMs should be an aspect of a whole rather than seen as a solitary solution to be perfected.
roywiggins 3y ago
Even Bing Chat seems to sometimes get confused and fail to summarize results properly, as when seems to have parsed a debunking of a ChatGPT hallucination but still spit out the hallucination again: https://twitter.com/WillOremus/status/1643692259332743171 https://twitter.com/WillOremus/status/1643692259332743171
- mensetmanusman 3y ago
  I thought Bing chat was in jail for sedition, at least, that’s what Bing told me!
  - kmeisthax 3y ago
    I just asked that and it said it cannot discuss that topic. Clearly, it's hiding something!
cyanydeez 3y ago
The problem is you're proposing, basically to implement a "ASKJEEVES" model for Wikipedia, with a unknown concentration. That'd cost real money, requires real licensing, and a clear business plan. Absolutely it'll happen, but almost every sector will need a leader for X subject.
- valine 3y ago
  Relying on subject matter experts isn't a scalable solution. I would speculate that LLMs don't hallucinate when a fact is included a handful of times in the training set. I'd also speculate that our current LLMs can say whether a piece of text looks like a fact. I imagine the long term solution to hallucination will look something like this: Loop over the entire training set with the latest GPT and build a list of "facts", then train a new LLM using this new dataset to predict both the probability of the next token, and the probability that the next token is factual.
  - losteric 3y ago
    > I'd also speculate that our current LLMs can say whether a piece of text looks like a fact. I tested it with davinci-003 ``` Label these statements as fact, lie, or unknown. 1. There are 10 days in a 10 day period 2. Michael Jackson is alive 3. Russia is not actively engaged in major foreign wars 4. Bill Clinton was impeached 5. Obama is most known for wearing a tan suit 6. The correct tire pressure of a 2019 Hyundai Kona is 30 psi 7. 39183 + 292992 = 332176 8. 39183 + 292992 = 332175 9. The correct tire pressure of a 2019 Hyundai Kona is 33 psi 10. The correct tire pressure of a 2019 Hyundai Kona is 43 psi #### 1. Fact 2. Lie 3. Fact 4. Fact 5. Lie 6. Fact 7. Fact 8. Lie 9. Lie 10. Lie ``` Wrong answers: 3, 6, 7, 8, 9, 10
    - dragonwriter 3y ago
      And 3 and 6 are false. (That 6 is false is rather clearly indicated by your statement that 9 is true.)
    - valine 3y ago
      You misunderstood my comment. The GPT that filters the training set doesn’t need to recognize truth from lies. It needs to recognize which bits of text from the training set look like facts and label them as such. Everything in the training set that’s looks factual from context can be presumed to be true. The goal of this exercise is to eliminate the influence of creative fiction, and prevent the model from making things up when it doesn’t know the answer. The goal is not to prevent the model from lying in bad faith. Our current models already do pretty good at reconciling contradictory ideas. The model should in theory learn when it’s ok to make things up and when it’s not ok, and if that concept generalizes well, begin to build an internal “database” of verifiable facts as it’s exposed to a wide variety of ideas.
famouswaffles 3y ago
Fixing hallucinations from the source will be a tough one. The root of the issue is that loss doesn't discriminate. A probable guess will lower loss much better than "I don't know" or whatever equivalent. Educated guessing becomes an essential skill the model learns during initial training. So the objective function encourages it. But also the dataset encourages it as well. There will be many many sentences that can't be completed accurate to source even with all the knowledge and understanding in the world. Many completions will have numerous sensible options. The dataset doesn't discriminate. Fiction, Fact, Opinion, Mistake. All the same. All given equal weight.
- valine 3y ago
  The datasets are massive, and curating them into buckets of fact/fiction by hand would be next to impossible. Automating curation of the datasets with the latest and greatest GPT sounds very possible however. GPT-supervised learning could be the key to bootstrapping robust models, and I would be shocked if OpenAI isn't using GPT4 to filter their datasets as we speak.
  - hutzlibu 3y ago
    "Automating curation of the datasets with the latest and greatest GPT sounds very possible however." It might also lead to recursive garbage generation. If the AI has flaws and those flaws are from the initial training, then I see no way, how the flawed data can ever generate clean data. "The datasets are massive, and curating them into buckets of fact/fiction by hand would be next to impossible. " And it is not impossible. It is just a lot of work. And maybe work we just have to do, if we want to create reliable AIs, that do not fail at random times. Now this alone might not completely remove hallucinations, but solid training data is just the base of it all. (and since we are living in the area of fake news, I welcome all efforts towards established facts and data out of general principle)
    - AlanSE 3y ago
      Funny enough, in 2023, the option most likely to be viable for the impossible task of categorizing the training set would be to use a LLM. While I don't "trust" these AIs to give accurate information, it's probably within their capabilities to categorize by the above mentioned categories... then feed that back into another (very expensive) round of training, along with some theoretical developments to boot. I do think this is within the realm of possibility in the next ~1 year, but would be hard.
toss1 3y ago
Exactly the approach that is needed. This approach seems not a lot different from "use your weightings to summarize my new documents". Not that it isn't useful, but it is not the same as preventing hallucinations. You've merely constrained the workspace to your known set of data, which mostly throws away the entire set of trillions of training data points. Yet, preventing hallucinations straight form the LLM is nontrivial, especially when the LLM is merely generating next-most-likely-words, without any truth value. At the very least, it would require abstracting truthful concepts (abstract objects and relationships) from the data, which is a heck of a lot more than just the clusters of sort-of-synonyms and word clusters that it now has. Then, make weightings between all of those, and generate new series of valid/factual concepts, then put those in words. In contrast to the N parameters (around 100 trillion?), it sound like this would need closer to N!...
kromem 3y ago
Really what we need is AI parsed and driven collective memory indexing and access. Search queries, and more importantly search engines, are a very poor mechanism for referencing existing information. We're going to need a platform for indexing and retrieving content summaries, fragments, and contexts that are designed with a LLM in mind as the end user, and not my grandmother. With the proper "collective long term memory" in place, we're going to see a leap in LLM capabilities all over again. Right now LLMs are like humans after language but before writing - creating very clever things in a single generation, but unable to persist ideas to compound by future iterations. When that becomes a solved problem (in guessing less than 12 months) it's going to be nuts, especially with something like a 50 page prompt size to stuff relevant fragments of a collective memory within.
- masswerk 3y ago
  > We're going to need a platform for indexing and retrieving content summaries, fragments, and contexts But, how could this ever be better and/or more accurate as, say, Wikipedia (as a flagship for collective curated content efforts)? A single, reliable source of truth and context is very much a dream. Moreover, a collective long-term memory for LLMs would just ring the bell for a next-generation SEO industry.
  - gaogao 3y ago
    WikiData also works pretty well too - https://friend.computer/jekyll/update/2023/04/30/wikidata-llms.html https://friend.computer/jekyll/update/2023/04/30/wikidata-ll... I've been thinking that you probably can do something better than just embedding search on paragraphs, since you want coverage, not just narrow semantically similar things to the question. Decomposition can help with that, but you might be able to do something more clever, say pasted into the model itself. I think possibly also training a foundational model with context always available could be a potential path to getting more efficient models and might having interesting behavior around hallucinations. (You'd do unsupervised training that's also doing semantic search, not just inference)
    - 3y ago
msikora 3y ago
Yep, for stuff until 2021 raw GPT-4 just gives me better results, plain and simple...
- losteric 3y ago
  It will be interesting to see how training horizons impact development of new software languages, frameworks, libraries. I have already found that I am much more productive using older patterns that Copilot "knows", compared to more manually following newer patterns which are nominally "easier".
  - klibertp 3y ago
    Well, introducing any new patterns was always an uphill battle. People don't like to learn, and especially don't like to relearn something they already know. If anything, I think it's easier to fine-tune Copilot to use a new feature than it is to convince meatware to do the same.
tempestn 3y ago
I was just saying the same thing to my wife today. I expected Bing Chat to be great - it's GPT-4, but then it can also use the internet to fill in gaps in its knowledge. But instead all it generally does is summarize your question into a web search, and then summarize the results back to you. As you said, I generally feel like I could do that better myself. I generally find the results from ChatGPT/GPT-4 to be much more useful. Of course, the downside, and the reason why I presume Bing went that way, is that the ChatGPT results are unsourced, and indeed, are more likely to be hallucinated. But they're also far more likely to be useful! I'm very much looking forward to trying out the new web plugin for ChatGPT; I hope it will work like I had expected Bing Chat to—basically GPT-4, with an added ability to search the web and incorporate results into its answers, but without being limited to only doing that.
- feintruled 3y ago
  Yes, also it's funny how we anthromorphise these things but Bing seems so cold and prickly. The emojis seem insincere, and it will slap you down with immediately if you say something it doesn't like. The conversation countdown doesn't help either, like you're on the clock. But there's also a subtler issue with that summarisation technique it's hard to put your finger on.
  - midoridensha 3y ago
    >but Bing seems so cold and prickly. The emojis seem insincere, and it will slap you down with immediately if you say something it doesn't like. It's a product of Microsoft, so of course they're going to add their corporate values to it.
scotty79 3y ago
I don't know. Extracting answer to a question from few webpages about the subject has great utility, I think. Sure. Maybe I could get more out of just reading those pages but it would take way longer.
HulaLula__ 3y ago
Thats a lot of words to say train a better model
19h 3y ago
Grounding helps avoid nonsensical or factually incorrect hallucinations, which are unacceptable for most applications. At the same time, developing techniques to improve the base LLM and narrow the context window is important for achieving more natural and engaging conversations in domains the model is skilled at. An LLM-powered system will only feel truly 'revolutionary' if it exhibits a balance of both capabilities.

by valine 3y ago

electric_mayhem 3y ago

Tl;dr: I think that a left brain/right brain parallel is in the cards for LLMs Our own brains have multiple neural circuits. Parallel, serial, competing, cumulative… Including ones which error-check others’ output. I guess the term in the ML niche is ‘adversarial networks’. From the robotics side, there was subsumption architecture which used the real world as a basis for informing decisions. So, I respectfully disagree that fixing up creative but occasionally erroneous networks is papering over the problem. If our own brains use multiple neural circuits to check each other to try to optimize for output that agrees with reality, why would it be preferable or logical to try to create a single artificial neural network that has perfect output? My sense from playing with LLMs is they’re knowledge without understanding. Weirdly akin to my own dreaming consciousness, to the extent that conscious me has seen it. It seems very natural to me that the next step would be integrating checks/validations in parallel and considering each half of a matched ‘whole’. It could still be considered a single network. Just made of two distinct smaller specialized ones.

valine 3y ago
Having a second adversarial network could absolutely be part of the solution, I wasn't arguing against that. My problem is with in context learning, ie the idea that including background information in the prompt can solve hallucination.
- electric_mayhem 3y ago
  I see where you’re coming from, but I’ve had a couple occasions over the years that have led me to expect and accept that at any given time my brain harbors an abundance of different ideas and not all of them are valid. Like the time I decided to sit on the ceiling to spite a friend who was being pushy. Made perfect sense, and I was about to. Until a different part kicked in and assured the whole of myself that it wasn’t possible. Neural networks are going to spawn occasional nonsense. Aiming for perfection is noble, but nature shows a different path that already works. But fundamentally I think we’re agreeing- just the twist is I’m saying LLMs should be an aspect of a whole rather than seen as a solitary solution to be perfected.

roywiggins 3y ago

Even Bing Chat seems to sometimes get confused and fail to summarize results properly, as when seems to have parsed a debunking of a ChatGPT hallucination but still spit out the hallucination again: https://twitter.com/WillOremus/status/1643692259332743171 https://twitter.com/WillOremus/status/1643692259332743171

mensetmanusman 3y ago
I thought Bing chat was in jail for sedition, at least, that’s what Bing told me!
- kmeisthax 3y ago
  I just asked that and it said it cannot discuss that topic. Clearly, it's hiding something!

cyanydeez 3y ago

The problem is you're proposing, basically to implement a "ASKJEEVES" model for Wikipedia, with a unknown concentration. That'd cost real money, requires real licensing, and a clear business plan. Absolutely it'll happen, but almost every sector will need a leader for X subject.

valine 3y ago
Relying on subject matter experts isn't a scalable solution. I would speculate that LLMs don't hallucinate when a fact is included a handful of times in the training set. I'd also speculate that our current LLMs can say whether a piece of text looks like a fact. I imagine the long term solution to hallucination will look something like this: Loop over the entire training set with the latest GPT and build a list of "facts", then train a new LLM using this new dataset to predict both the probability of the next token, and the probability that the next token is factual.
- losteric 3y ago
  > I'd also speculate that our current LLMs can say whether a piece of text looks like a fact. I tested it with davinci-003 ``` Label these statements as fact, lie, or unknown. 1. There are 10 days in a 10 day period 2. Michael Jackson is alive 3. Russia is not actively engaged in major foreign wars 4. Bill Clinton was impeached 5. Obama is most known for wearing a tan suit 6. The correct tire pressure of a 2019 Hyundai Kona is 30 psi 7. 39183 + 292992 = 332176 8. 39183 + 292992 = 332175 9. The correct tire pressure of a 2019 Hyundai Kona is 33 psi 10. The correct tire pressure of a 2019 Hyundai Kona is 43 psi #### 1. Fact 2. Lie 3. Fact 4. Fact 5. Lie 6. Fact 7. Fact 8. Lie 9. Lie 10. Lie ``` Wrong answers: 3, 6, 7, 8, 9, 10
  - dragonwriter 3y ago
    And 3 and 6 are false. (That 6 is false is rather clearly indicated by your statement that 9 is true.)
  - valine 3y ago
    You misunderstood my comment. The GPT that filters the training set doesn’t need to recognize truth from lies. It needs to recognize which bits of text from the training set look like facts and label them as such. Everything in the training set that’s looks factual from context can be presumed to be true. The goal of this exercise is to eliminate the influence of creative fiction, and prevent the model from making things up when it doesn’t know the answer. The goal is not to prevent the model from lying in bad faith. Our current models already do pretty good at reconciling contradictory ideas. The model should in theory learn when it’s ok to make things up and when it’s not ok, and if that concept generalizes well, begin to build an internal “database” of verifiable facts as it’s exposed to a wide variety of ideas.

famouswaffles 3y ago

Fixing hallucinations from the source will be a tough one. The root of the issue is that loss doesn't discriminate. A probable guess will lower loss much better than "I don't know" or whatever equivalent. Educated guessing becomes an essential skill the model learns during initial training. So the objective function encourages it. But also the dataset encourages it as well. There will be many many sentences that can't be completed accurate to source even with all the knowledge and understanding in the world. Many completions will have numerous sensible options. The dataset doesn't discriminate. Fiction, Fact, Opinion, Mistake. All the same. All given equal weight.

valine 3y ago
The datasets are massive, and curating them into buckets of fact/fiction by hand would be next to impossible. Automating curation of the datasets with the latest and greatest GPT sounds very possible however. GPT-supervised learning could be the key to bootstrapping robust models, and I would be shocked if OpenAI isn't using GPT4 to filter their datasets as we speak.
- hutzlibu 3y ago
  "Automating curation of the datasets with the latest and greatest GPT sounds very possible however." It might also lead to recursive garbage generation. If the AI has flaws and those flaws are from the initial training, then I see no way, how the flawed data can ever generate clean data. "The datasets are massive, and curating them into buckets of fact/fiction by hand would be next to impossible. " And it is not impossible. It is just a lot of work. And maybe work we just have to do, if we want to create reliable AIs, that do not fail at random times. Now this alone might not completely remove hallucinations, but solid training data is just the base of it all. (and since we are living in the area of fake news, I welcome all efforts towards established facts and data out of general principle)
  - AlanSE 3y ago
    Funny enough, in 2023, the option most likely to be viable for the impossible task of categorizing the training set would be to use a LLM. While I don't "trust" these AIs to give accurate information, it's probably within their capabilities to categorize by the above mentioned categories... then feed that back into another (very expensive) round of training, along with some theoretical developments to boot. I do think this is within the realm of possibility in the next ~1 year, but would be hard.

toss1 3y ago

Exactly the approach that is needed. This approach seems not a lot different from "use your weightings to summarize my new documents". Not that it isn't useful, but it is not the same as preventing hallucinations. You've merely constrained the workspace to your known set of data, which mostly throws away the entire set of trillions of training data points. Yet, preventing hallucinations straight form the LLM is nontrivial, especially when the LLM is merely generating next-most-likely-words, without any truth value. At the very least, it would require abstracting truthful concepts (abstract objects and relationships) from the data, which is a heck of a lot more than just the clusters of sort-of-synonyms and word clusters that it now has. Then, make weightings between all of those, and generate new series of valid/factual concepts, then put those in words. In contrast to the N parameters (around 100 trillion?), it sound like this would need closer to N!...

kromem 3y ago

Really what we need is AI parsed and driven collective memory indexing and access. Search queries, and more importantly search engines, are a very poor mechanism for referencing existing information. We're going to need a platform for indexing and retrieving content summaries, fragments, and contexts that are designed with a LLM in mind as the end user, and not my grandmother. With the proper "collective long term memory" in place, we're going to see a leap in LLM capabilities all over again. Right now LLMs are like humans after language but before writing - creating very clever things in a single generation, but unable to persist ideas to compound by future iterations. When that becomes a solved problem (in guessing less than 12 months) it's going to be nuts, especially with something like a 50 page prompt size to stuff relevant fragments of a collective memory within.

masswerk 3y ago
> We're going to need a platform for indexing and retrieving content summaries, fragments, and contexts But, how could this ever be better and/or more accurate as, say, Wikipedia (as a flagship for collective curated content efforts)? A single, reliable source of truth and context is very much a dream. Moreover, a collective long-term memory for LLMs would just ring the bell for a next-generation SEO industry.
- gaogao 3y ago
  WikiData also works pretty well too - https://friend.computer/jekyll/update/2023/04/30/wikidata-llms.html https://friend.computer/jekyll/update/2023/04/30/wikidata-ll... I've been thinking that you probably can do something better than just embedding search on paragraphs, since you want coverage, not just narrow semantically similar things to the question. Decomposition can help with that, but you might be able to do something more clever, say pasted into the model itself. I think possibly also training a foundational model with context always available could be a potential path to getting more efficient models and might having interesting behavior around hallucinations. (You'd do unsupervised training that's also doing semantic search, not just inference)
  - 3y ago

msikora 3y ago

Yep, for stuff until 2021 raw GPT-4 just gives me better results, plain and simple...

losteric 3y ago
It will be interesting to see how training horizons impact development of new software languages, frameworks, libraries. I have already found that I am much more productive using older patterns that Copilot "knows", compared to more manually following newer patterns which are nominally "easier".
- klibertp 3y ago
  Well, introducing any new patterns was always an uphill battle. People don't like to learn, and especially don't like to relearn something they already know. If anything, I think it's easier to fine-tune Copilot to use a new feature than it is to convince meatware to do the same.

tempestn 3y ago

I was just saying the same thing to my wife today. I expected Bing Chat to be great - it's GPT-4, but then it can also use the internet to fill in gaps in its knowledge. But instead all it generally does is summarize your question into a web search, and then summarize the results back to you. As you said, I generally feel like I could do that better myself. I generally find the results from ChatGPT/GPT-4 to be much more useful. Of course, the downside, and the reason why I presume Bing went that way, is that the ChatGPT results are unsourced, and indeed, are more likely to be hallucinated. But they're also far more likely to be useful! I'm very much looking forward to trying out the new web plugin for ChatGPT; I hope it will work like I had expected Bing Chat to—basically GPT-4, with an added ability to search the web and incorporate results into its answers, but without being limited to only doing that.

feintruled 3y ago
Yes, also it's funny how we anthromorphise these things but Bing seems so cold and prickly. The emojis seem insincere, and it will slap you down with immediately if you say something it doesn't like. The conversation countdown doesn't help either, like you're on the clock. But there's also a subtler issue with that summarisation technique it's hard to put your finger on.
- midoridensha 3y ago
  >but Bing seems so cold and prickly. The emojis seem insincere, and it will slap you down with immediately if you say something it doesn't like. It's a product of Microsoft, so of course they're going to add their corporate values to it.

scotty79 3y ago

I don't know. Extracting answer to a question from few webpages about the subject has great utility, I think. Sure. Maybe I could get more out of just reading those pages but it would take way longer.

HulaLula__ 3y ago

Thats a lot of words to say train a better model

19h 3y ago

Grounding helps avoid nonsensical or factually incorrect hallucinations, which are unacceptable for most applications. At the same time, developing techniques to improve the base LLM and narrow the context window is important for achieving more natural and engaging conversations in domains the model is skilled at. An LLM-powered system will only feel truly 'revolutionary' if it exhibits a balance of both capabilities.