Even if it is contained within the model _somewhere_, it might be encoded in such a way that it's impractical to extract. Might need an exponential time algorit

Even if it is contained within the model _somewhere_, it might be encoded in such a way that it's impractical to extract. Might need an exponential time algorithm, for example. Or this proxy method of hundreds of deceiving attempts. And it's very difficult to train it as a next token predictor and at the same time make it say correctly "I don't know".