3 ms·
Oh, GPT-4 does hallucinate. It's more subtle than with GPT-3.5, but it's there. Most of the time I had it happen, it could either correct itself, or counter-hal
by TeMPOraL 4y ago
Oh, GPT-4 does hallucinate. It's more subtle than with GPT-3.5, but it's there. Most of the time I had it happen, it could either correct itself, or counter-hallucinated (with further corrections oscillating) - the latter is quite easy to spot.
It's not a big deal in practice, though, as long as you remember to take a probabilistic approach. GPT-4 is not an oracle, it's a 4 year old savant, that tries its best, but has attention span of a hamster, and likes to extrapolate instead of saying "I don't know". In domains you have at least minimal experience in (e.g. you can program, but not in the language/framework you're asking about), it's relatively easy to verify things by common sense and/or by paying attention to the conversation - if the follow-up message seems to contradict the previous one, it's likely at least one of them involves a hallucination. Etc.
The overall feel I get for GPT-4, at least in terms of code, is that its hallucinations tend to mostly be of the "I don't know for sure, but seems logical that..." kind.
A real example from earlier today: I asked GPT-4 to refactor some code C++ that contained function calls like InitializeSomething(), AddWidget(), etc. It decided to put all those calls into RAII objects, and hallucinated the existence of corresponding function DeinitializeSomething(), RemoveWidget(), etc. I kind of understand why it did that - it feels only logical that such functions exist too.