5 ms·

Does anyone really believe that having a good corpus will remove hallucinations? Is this article even written by a person? Hard to know; they have a real blog

Does anyone really believe that having a good corpus will remove hallucinations? Is this article even written by a person? Hard to know; they have a real blog with real article, but stuff like this reads strangely. Maybe it's just not a native english speaker? > Hallucinations are certainly the toughest nut to crack and their negative impact is basically only slightly lessened by good confidence estimates and reliable citations (sources). > The impact of contradictions in the training data. (was this a prompt header you forget to remove?) > LLM are incapable of "self-inspection" on their training data to find logical inconsistencies in it but in the input context window they should be able to find logical inconsistencies. Annnnyway... Hallucinations cannot be fixed by a good corpus in a non-deterministic (ie. temp > 0) LLM system where you've introduced a random factor. Period. QED. If you think it can, do more reading. The idea that a good corpus can significantly improve the error rate is an open question, but the research I've seen tends to fall on the side of "to some degree, but curating a 'perfect' dataset like that, of a sufficiently large size, is basically impossible'". So, it's a pipe dream. Yes, if you could have a perfect corpus, absolutely, you would get a better model. ...but how do you plan to get that perfect corpus of training data? If it was that easy, the people spending millions and millions of dollars making LLMs would have, I guess, probably come up with a solution for it. They're not stupid. If you could easily do it, it would already have been done. my $0.02: This is a dead end of research, because it's impossible. Using LLMs which are finetuned to evaluate the output of other LLMs and using multi-sample / voting to reduce the incidence of halluciations that make it past the API barrier is both actively used and far, far more effective. (ie. it doesn't matter if your LLM hallucinates 1 time in 10; if you can reliably detect that 1 instance, sample again, and return a non hallucination). Other solutions... I'm skeptical; most of the ones I've seen haven't worked when you actually try to use them.

thntk 2y ago
I've seen such articles more and more recently. In the past, when people had a vague idea, they had to do research before writing. During this process, they often realized some flaws and thoroughly revised the idea or gave up writing. Nowadays, research can be bypassed with the help of eloquent LLMs, allowing any vague idea to turn into a write-up.
comcuoglu 2y ago
Thank you. It seems largely ignored that LLMs still sample from a set of tokens based on estimated probability and the given temperature - but not on factuality or the described "confidence estimate" in the article. RAG etc. only move the estimated probabilities into a more factually based direction, but do not change the sampling itself
JohnVideogames 2y ago
It's obvious that you can't solve hallucinations by curating the dataset when you think about arithmetic. It's trivial to create a corpus of True Maths Facts and verify that they're correct. But an LLM (as they're currently structured) will never generalise to new mathematical problems with 100% success rate because they do not fundamentally work like that.
tidenly 2y ago
I wonder to what extent is hallucination a result of a "must answer" bias? When sampling data all over the internet, your data set only represents people who did write, did respond to questions - with no representation of what they didn't. Add into that confidently wrong people - people who respond to questions on, say, StackOverflow, even if they're wrong, and suddenly you have a data set that prefers replying bullshit, because there's no data for the people who didnt know the answer and wrote nothing. Inherently there's no representation in the datasets of "I don't know" null values. LLMs are forced to reply, in contrast, so they "bullshit" a response that sounds right even though not answering or saying you don't know would be more appropriate - because no-one does that on the internet. I always assumed this was a big factor, but am I completely off the mark?
sean_pedersen 2y ago
I wrote up this blog post in 30 mins, that's why it reads a little rough. I could not find explicit research on the impact of contradicting training data, only on the general need for high-quality training data. May be it is a pipe dream to drastically improve on hallucinations by curating a self-consistent data set but I am still interested in how much it actually impacts the quality of the final model. I described one possible way to create such a self-consistent data set in this very blog post.

thntk 2y ago

I've seen such articles more and more recently. In the past, when people had a vague idea, they had to do research before writing. During this process, they often realized some flaws and thoroughly revised the idea or gave up writing. Nowadays, research can be bypassed with the help of eloquent LLMs, allowing any vague idea to turn into a write-up.

comcuoglu 2y ago

Thank you. It seems largely ignored that LLMs still sample from a set of tokens based on estimated probability and the given temperature - but not on factuality or the described "confidence estimate" in the article. RAG etc. only move the estimated probabilities into a more factually based direction, but do not change the sampling itself

JohnVideogames 2y ago

It's obvious that you can't solve hallucinations by curating the dataset when you think about arithmetic. It's trivial to create a corpus of True Maths Facts and verify that they're correct. But an LLM (as they're currently structured) will never generalise to new mathematical problems with 100% success rate because they do not fundamentally work like that.

tidenly 2y ago

I wonder to what extent is hallucination a result of a "must answer" bias? When sampling data all over the internet, your data set only represents people who did write, did respond to questions - with no representation of what they didn't. Add into that confidently wrong people - people who respond to questions on, say, StackOverflow, even if they're wrong, and suddenly you have a data set that prefers replying bullshit, because there's no data for the people who didnt know the answer and wrote nothing. Inherently there's no representation in the datasets of "I don't know" null values. LLMs are forced to reply, in contrast, so they "bullshit" a response that sounds right even though not answering or saying you don't know would be more appropriate - because no-one does that on the internet. I always assumed this was a big factor, but am I completely off the mark?

sean_pedersen 2y ago

I wrote up this blog post in 30 mins, that's why it reads a little rough. I could not find explicit research on the impact of contradicting training data, only on the general need for high-quality training data. May be it is a pipe dream to drastically improve on hallucinations by curating a self-consistent data set but I am still interested in how much it actually impacts the quality of the final model. I described one possible way to create such a self-consistent data set in this very blog post.