6 ms·

Crazy that even o1-preview gets most things wrong. This is in line with my own personal experience with LLMs and non-trivial questions. They’re excellent when

by brap 2y ago

Crazy that even o1-preview gets most things wrong. This is in line with my own personal experience with LLMs and non-trivial questions. They’re excellent when answering questions on topics you know nothing about, and somehow embarrassingly wrong when you actually know the answer yourself… It’s not clear to me why we’re still trying to encode all of human knowledge in a single model, instead of teaching the model how to look for answers from an external source (e.g. RAG).

sksxihve 2y ago
LLMs are experts in everything you are not
- reverius42 2y ago
  Sounds a bit like Gell-Mann Amnesia Effect: https://en.wikipedia.org/wiki/Michael_Crichton#Gell-Mann_amnesia_effect https://en.wikipedia.org/wiki/Michael_Crichton#Gell-Mann_amn...
  - kibwen 2y ago
    The Alt-Mann Amnesia Effect, maybe.
- Nition 2y ago
  That's a nice little aphorism. I think this happens in a lot of things in life. Like comments on Reddit always seem quite insightful until you actually read the article they're commenting on.
- swatcoder 2y ago
  Indeed. Exactly like the journalists, bloggers, self-published book authors, internet commenters, wikipedia editors, and earlier models that taught them almost all of what they know.
zone411 2y ago
You shouldn't use the rate as an indicator. They did something similar to what I did on my hallucinations benchmark (https://github.com/lechmazur/confabulations/ https://github.com/lechmazur/confabulations/), only using questions where at least one model made a mistake. I added this note: "The benchmark includes questions where at least one LLM confabulated, in order to minimize the number of questions requiring human assessment. Because of this, and since the questions are intentionally adversarial, the absolute percentage should not be used to infer that LLMs frequently confabulate. This leaderboard does not reflect a "typical" hallucination rate." > instead of teaching the model how to look for answers from an external source (e.g. RAG) My benchmark specifically focuses on the RAG use case. Even with provided texts, current models still hallucinate.
- authorfly 2y ago
  True, but this pattern was established with TrufulQA a couple of years ago. I disagreed then but it's unfortunately convenient to use this approach for some benchmarks.
- _jonas 2y ago
  [dead]
Kiro 2y ago
You're reading this wrong. They've deliberately chosen questions that one or more models fail at. It's not representative at all of how often the model is wrong in general.
- Kiro 2y ago
  From the paper: > At least one of the four completions must be incorrect for the trainer to continue with that question; otherwise, the trainer was instructed to create a new question.
bloomingkales 2y ago
Honestly, try prompting it with “you are wrong 80% of the time, therefore you will need to double check your answers, first factually, then numerically, then double check the time/date. You are still probably wrong so do a third accuracy check. The user’s prompts are always wrong too mostly - so always check them”. I stopped playing with larger models and have been pushing smaller models with this improvised system prompt and getting good results. It seems like it forces the model to do multiple passes before giving you any response. My smaller local models give me less hallucinations than Meta.ai, for example, which generally spits out pleasing answers almost immediately (which are often hallucinations, since I don’t think it is system prompted to be adversarial to the user, or itself). I don’t have the same hallucination issue with Llama3 - 8b locally because of custom system prompts. The model has all the correct information, so it almost needs to do RAG on itself. Multiple passes on itself seems like a way to do it.
- dosinga 2y ago
  How would this multiple passes work though? Unless the model actually talks about what it does, I am not sure how it would have this ability. The next word prediction mechanism is just always going to do it one shot. Your prompt paints a context that might keep it more on the rails, but it won't do multiple passes.
  - bloomingkales 2y ago
    Your prompt paints a context that might keep it more on the rails, but it won't do multiple passes. This is probably the truth behind the black magic I’m imagining. You could have it explicitly spit out this process, in which case you would see it’s first rough draft, followed by a “My first paragraph is probably wrong”, followed by a third paragraph where it attempts to fix the first paragraph. There is no outside RAG in this process. The mumbo jumbo part of all this is that I’ve told it to “hide” this process from the user where it doesn’t explicitly output anything but its final answer, and the accuracy has been just as good (for my use case at least). :Shrugs:
    - jorl17 2y ago
sebzim4500 2y ago
I don't think it's surprising that o1-preview is only slightly better than GPT-4o, it was never advertised as being better at this kind of recall.
divan 2y ago
> They’re excellent when answering questions on topics you know nothing about, and somehow embarrassingly wrong when you actually know the answer yourself I forgot the name of this phenomenon with humans, described it to o1 and it gave the correct answer - Gell-Mann Amnesia Effect [1] "Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them. In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know." – Michael Crichton (1942-2008) [1] https://www.epsilontheory.com/gell-mann-amnesia/ https://www.epsilontheory.com/gell-mann-amnesia/
esafak 2y ago
How would the model know how to evaluate an answer without innate knowledge?
aiforecastthway 2y ago
> It’s not clear to me why we’re still trying to encode all of human knowledge in a single model, instead of teaching the model how to look for answers from an external source (e.g. RAG). To be fair: we tried a primordial version of that with venue-weighted citation-based ranking and it worked INCREDIBLY well for 10+ years. Then myopic profit motive poisoned the well. Ever since then we've been searching for solutions. We do so by allocating resources in a way that primarily leverages a scientific credit assignment system that fetishizes... checks notes... venue-weighted citation-based ranking. Jokes aside: I remain convinced that the O.G. google search appliance on prop data and then simply ignoring all academics remains the best knowledge retrieval (or whatever) tool available.
- fraboniface 2y ago
  Why ignore academics?
jumping_frog 2y ago
LLM version of Gell-Mann Amnesia.
soco 2y ago
If you know nothing about, you probably have no idea whether the answer was correct or not, right? Otherwise you'd find the answer embarrassingly wrong. So that observation speaks much more about the human than about the computer.
- brap 2y ago
  That was kind of a joke…
mcmcmc 2y ago
> They’re excellent when answering questions on topics you know nothing about, and somehow embarrassingly wrong when you actually know the answer yourself… This just makes me think LLMs must be wrong far more often than we realize. How well can you judge the quality of a response on a topic you know nothing about?
- toomuchtodo 2y ago
  > How well can you judge the quality of a response on a topic you know nothing about? You cannot, but those pushing Gen AI wave away the probability and liability for obvious reasons. Sell the magic, it'll be someone else's problem when it doesn't deliver.
  - chefandy 2y ago
    Not just wrong, but superficially credible-looking wrong, which is much worse. "Fake it till you make it" works fine for some things, but not a large chunk of what they're trying to sell AI for.
- sfjailbird 2y ago
  That's his/her point.
  - deleted 2y ago
    [deleted]
drewcoo 2y ago
> They’re excellent when answering questions on topics you know nothing about, and somehow embarrassingly wrong when you actually know the answer yourself… Sounds like Gell-Mann Amnesia. Maybe LLMs should replace reporters.

by brap 2y ago

sksxihve 2y ago

LLMs are experts in everything you are not

reverius42 2y ago
Sounds a bit like Gell-Mann Amnesia Effect: https://en.wikipedia.org/wiki/Michael_Crichton#Gell-Mann_amnesia_effect https://en.wikipedia.org/wiki/Michael_Crichton#Gell-Mann_amn...
- kibwen 2y ago
  The Alt-Mann Amnesia Effect, maybe.
Nition 2y ago
That's a nice little aphorism. I think this happens in a lot of things in life. Like comments on Reddit always seem quite insightful until you actually read the article they're commenting on.
swatcoder 2y ago
Indeed. Exactly like the journalists, bloggers, self-published book authors, internet commenters, wikipedia editors, and earlier models that taught them almost all of what they know.

zone411 2y ago

You shouldn't use the rate as an indicator. They did something similar to what I did on my hallucinations benchmark (https://github.com/lechmazur/confabulations/ https://github.com/lechmazur/confabulations/), only using questions where at least one model made a mistake. I added this note: "The benchmark includes questions where at least one LLM confabulated, in order to minimize the number of questions requiring human assessment. Because of this, and since the questions are intentionally adversarial, the absolute percentage should not be used to infer that LLMs frequently confabulate. This leaderboard does not reflect a "typical" hallucination rate." > instead of teaching the model how to look for answers from an external source (e.g. RAG) My benchmark specifically focuses on the RAG use case. Even with provided texts, current models still hallucinate.

authorfly 2y ago
True, but this pattern was established with TrufulQA a couple of years ago. I disagreed then but it's unfortunately convenient to use this approach for some benchmarks.
_jonas 2y ago
[dead]

Kiro 2y ago

You're reading this wrong. They've deliberately chosen questions that one or more models fail at. It's not representative at all of how often the model is wrong in general.

Kiro 2y ago
From the paper: > At least one of the four completions must be incorrect for the trainer to continue with that question; otherwise, the trainer was instructed to create a new question.

bloomingkales 2y ago

Honestly, try prompting it with “you are wrong 80% of the time, therefore you will need to double check your answers, first factually, then numerically, then double check the time/date. You are still probably wrong so do a third accuracy check. The user’s prompts are always wrong too mostly - so always check them”. I stopped playing with larger models and have been pushing smaller models with this improvised system prompt and getting good results. It seems like it forces the model to do multiple passes before giving you any response. My smaller local models give me less hallucinations than Meta.ai, for example, which generally spits out pleasing answers almost immediately (which are often hallucinations, since I don’t think it is system prompted to be adversarial to the user, or itself). I don’t have the same hallucination issue with Llama3 - 8b locally because of custom system prompts. The model has all the correct information, so it almost needs to do RAG on itself. Multiple passes on itself seems like a way to do it.

dosinga 2y ago
How would this multiple passes work though? Unless the model actually talks about what it does, I am not sure how it would have this ability. The next word prediction mechanism is just always going to do it one shot. Your prompt paints a context that might keep it more on the rails, but it won't do multiple passes.
- bloomingkales 2y ago
  Your prompt paints a context that might keep it more on the rails, but it won't do multiple passes. This is probably the truth behind the black magic I’m imagining. You could have it explicitly spit out this process, in which case you would see it’s first rough draft, followed by a “My first paragraph is probably wrong”, followed by a third paragraph where it attempts to fix the first paragraph. There is no outside RAG in this process. The mumbo jumbo part of all this is that I’ve told it to “hide” this process from the user where it doesn’t explicitly output anything but its final answer, and the accuracy has been just as good (for my use case at least). :Shrugs:
  - jorl17 2y ago

sebzim4500 2y ago

I don't think it's surprising that o1-preview is only slightly better than GPT-4o, it was never advertised as being better at this kind of recall.

divan 2y ago

> They’re excellent when answering questions on topics you know nothing about, and somehow embarrassingly wrong when you actually know the answer yourself I forgot the name of this phenomenon with humans, described it to o1 and it gave the correct answer - Gell-Mann Amnesia Effect [1] "Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them. In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know." – Michael Crichton (1942-2008) [1] https://www.epsilontheory.com/gell-mann-amnesia/ https://www.epsilontheory.com/gell-mann-amnesia/

esafak 2y ago

How would the model know how to evaluate an answer without innate knowledge?

aiforecastthway 2y ago

> It’s not clear to me why we’re still trying to encode all of human knowledge in a single model, instead of teaching the model how to look for answers from an external source (e.g. RAG). To be fair: we tried a primordial version of that with venue-weighted citation-based ranking and it worked INCREDIBLY well for 10+ years. Then myopic profit motive poisoned the well. Ever since then we've been searching for solutions. We do so by allocating resources in a way that primarily leverages a scientific credit assignment system that fetishizes... checks notes... venue-weighted citation-based ranking. Jokes aside: I remain convinced that the O.G. google search appliance on prop data and then simply ignoring all academics remains the best knowledge retrieval (or whatever) tool available.

fraboniface 2y ago
Why ignore academics?

jumping_frog 2y ago

LLM version of Gell-Mann Amnesia.

soco 2y ago

If you know nothing about, you probably have no idea whether the answer was correct or not, right? Otherwise you'd find the answer embarrassingly wrong. So that observation speaks much more about the human than about the computer.

brap 2y ago
That was kind of a joke…

mcmcmc 2y ago

> They’re excellent when answering questions on topics you know nothing about, and somehow embarrassingly wrong when you actually know the answer yourself… This just makes me think LLMs must be wrong far more often than we realize. How well can you judge the quality of a response on a topic you know nothing about?

toomuchtodo 2y ago
> How well can you judge the quality of a response on a topic you know nothing about? You cannot, but those pushing Gen AI wave away the probability and liability for obvious reasons. Sell the magic, it'll be someone else's problem when it doesn't deliver.
- chefandy 2y ago
  Not just wrong, but superficially credible-looking wrong, which is much worse. "Fake it till you make it" works fine for some things, but not a large chunk of what they're trying to sell AI for.
sfjailbird 2y ago
That's his/her point.
- deleted 2y ago
  [deleted]

drewcoo 2y ago

> They’re excellent when answering questions on topics you know nothing about, and somehow embarrassingly wrong when you actually know the answer yourself… Sounds like Gell-Mann Amnesia. Maybe LLMs should replace reporters.