That’s sort of like saying, “couldn’t you just check that the output from a buggy sort algorithm is sorted?” Yes, it’s true that generate-and-verify is a valid

That’s sort of like saying, “couldn’t you just check that the output from a buggy sort algorithm is sorted?” Yes, it’s true that generate-and-verify is a valid algorithm, but only if you have some way to verify the results and try again if it’s bad, so it’s not just “generate and trust.” (And it should be automatic, because verifying that your SQL query output is sorted manually would be unbearably tedious.) And sure, this is why relying on autocomplete while coding is fairly harmless, because we do have a lot of ways to verify that the code works. The trouble is, people want to deploy LLM’s in situations where there’s no easy way to verify every output. “Explainability” is supposed to assist the user doing the verification.