3 ms·

Isn’t “hallucination” named after a human phenomena? People too remember things that never happened. Wouldn’t be solvable with a second AI agent which checks t

by mrtksn 3y ago

Isn’t “hallucination” named after a human phenomena? People too remember things that never happened. Wouldn’t be solvable with a second AI agent which checks the output of the first one and be like “bro, you sure about that? I never heard of it”. In my experience with LLMs, they don’t insist when corrected, instead they apologize and generate a response with that correction in mind.

thriftwy 3y ago
I wonder how many AI hallucinations stem from the fact that nowhere in the prompt it was said that information have to be real. And this cannot be implicitly assumed since humans like fiction in other contexts.
Closi 3y ago
It's absolutely solvable - and presumably this is the sort of functionality that more legal-specific AI applications offer (probably mixed with some sort of knowledge graph that contains facts). Seems like the issue here was that a lawyer was just using OOTB ChatGPT.
- tiborsaas 3y ago
  I'm not so sure with generic LLM-s. With what I've seen so far in how LLM-s work "hallucination" is a feature, not a bug. In other use cases it's called creativity when the user wants new ideas generated or filling in the missing spots. In these cases they are really happy that it can hallucinate, but of course they don't call it like that. So the dilemma is that we need to eliminate the same thing in one case but improve it in others. But I believe you are right that we can engineer a fine tuned system that can use tools to fact check using traditional techniques.
supriyo-biswas 3y ago
> they don’t insist when corrected That’s just OpenAI’s RLHF and instruct tuning though. Bing Chat’s Sydney had the temperament of a moody teenager and would often contradict or accuse people of being wrong when pushed. The correct way to solve this is to use retrieval augmented generation (finding relevant cases using embeddings and then feeding the data along with the question), so that the model is grounded to the truth.
- mrtksn 3y ago
  Sure, a second agent can do fact and sanity checking with retrieval too.
jeroenhd 3y ago
I don't like "hallucination" in this context. "Confabulation" seems much more accurate. Last time I tried to correct AI, it assured me that I'm wrong, and for a short, funny moment, it started insulting me before the text got scrapped and a generic "something went wrong" placeholder appeared. I don't think putting two AIs together will solve the problem. The legal references all need to follow relatively standardised formats, so those should be easy to check. Of course there's no way to know for sure if the case is actually about the subject that it's being used as for support, but bare minimum you could filter our AI responses with fake cases. Because AI can't differentiate truth from fiction, it's essential that a real lawyer double checks the documents AI will generate, including reading referenced cases and verifying that they support the supposed argument. That way, they can skip a difficult part of their job without actually lying to the court when they submit "their" documents. For important fields like law, you will always need a human verification step.

by mrtksn 3y ago

thriftwy 3y ago

I wonder how many AI hallucinations stem from the fact that nowhere in the prompt it was said that information have to be real. And this cannot be implicitly assumed since humans like fiction in other contexts.

Closi 3y ago

It's absolutely solvable - and presumably this is the sort of functionality that more legal-specific AI applications offer (probably mixed with some sort of knowledge graph that contains facts). Seems like the issue here was that a lawyer was just using OOTB ChatGPT.

tiborsaas 3y ago
I'm not so sure with generic LLM-s. With what I've seen so far in how LLM-s work "hallucination" is a feature, not a bug. In other use cases it's called creativity when the user wants new ideas generated or filling in the missing spots. In these cases they are really happy that it can hallucinate, but of course they don't call it like that. So the dilemma is that we need to eliminate the same thing in one case but improve it in others. But I believe you are right that we can engineer a fine tuned system that can use tools to fact check using traditional techniques.

supriyo-biswas 3y ago

> they don’t insist when corrected That’s just OpenAI’s RLHF and instruct tuning though. Bing Chat’s Sydney had the temperament of a moody teenager and would often contradict or accuse people of being wrong when pushed. The correct way to solve this is to use retrieval augmented generation (finding relevant cases using embeddings and then feeding the data along with the question), so that the model is grounded to the truth.

mrtksn 3y ago
Sure, a second agent can do fact and sanity checking with retrieval too.

jeroenhd 3y ago

I don't like "hallucination" in this context. "Confabulation" seems much more accurate. Last time I tried to correct AI, it assured me that I'm wrong, and for a short, funny moment, it started insulting me before the text got scrapped and a generic "something went wrong" placeholder appeared. I don't think putting two AIs together will solve the problem. The legal references all need to follow relatively standardised formats, so those should be easy to check. Of course there's no way to know for sure if the case is actually about the subject that it's being used as for support, but bare minimum you could filter our AI responses with fake cases. Because AI can't differentiate truth from fiction, it's essential that a real lawyer double checks the documents AI will generate, including reading referenced cases and verifying that they support the supposed argument. That way, they can skip a difficult part of their job without actually lying to the court when they submit "their" documents. For important fields like law, you will always need a human verification step.