4 ms·
OpenAI: Models are programmed to make stuff up instead of admitting ignorance
- CodingJeebus 1y agoThis is why I just can't fully get behind the idea that AI is going to be the panacea the hype merchants keep claiming, at least in this era: making stuff up incentivizes users to prompt again, while returning "I don't know" tends to be a conversation ender. They're not incentivized to be accurate, they're incentivized to give the appearance of competence to the point that it keeps users coming back. Tufte said it best: There are only two industries that call their customers 'users': illegal drugs and software.
- furyofantares 1y agoThe original article has already been posted here, the discussion there might be interesting if someone can find it. The Register's click/ragebait framing isn't very interesting. I don't really think people want these things to say "I don't know" - they want it to know. That's obviously not reasonable for everything but I bet a lot of hallucinations ARE things where the model should be able to know or figure out. Most people are asking questions with well-known answers. But I would guess the OpenAI post is correct, fundamentally they are trained in a way that rewards guessing, which I think must make it more likely it guesses even if the answer is in its reach.
- justcallmejm 1y agoThis is the problem Aloe (a state-of-the-art agent designed and built by a cognitive scientist) is solving. This article out yesterday delves into exactly that: https://puck.news/is-aloe-the-first-self-building-ai/?sharer=743454&token=6c34705e634fd15fb42e9faa58dc4796 https://puck.news/is-aloe-the-first-self-building-ai/?sharer...