Cause if you got the facts you train on them, the model memorizes those-- and hallucinates on the facts you didn't teach it in training. :) Call this model A.

Cause if you got the facts you train on them, the model memorizes those-- and hallucinates on the facts you didn't teach it in training. :) Call this model A. So you say okay, I'll leave out some facts from training to use to teach it to say I don't know. Now you have model B that is similar to A but on some things that A answers correctly on, B answers I don't know. ... and on some new facts both A and B hallucinate, so B is strictly worse than A-- they both hallucinate but A knows more. Using known unknown facts to train for saying "I don't know" is only useful if it produces a general ability to say "I don't know" against unknown unknowns. And I don't know if anyone has managed to demonstrate that result. It's difficult in general to know what the model does and doesn't know, and LLM isn't a trivia bot-- and it knows tons of stuff that exists nowhere explicitly in the training data (which is why it's useful over and above a verbatim internet search!). It's fun to play with the boundary of LLM knowledge by conversing with it in ROT13 or asking it write with bizarre constraints and watch its intelligence fall away.