3 ms·

> “It does whatever it does,” the training data is the specification, is ignoring the actual problem, which is to predict what the LLM would do on unseen inputs

> “It does whatever it does,” the training data is the specification, is ignoring the actual problem, which is to predict what the LLM would do on unseen inputs. Wouldn't running the llm on those unseen inputs solve that problem immediately? And I don't think what you describe falls under explainability but evaluation. "Does this model do what I want it to do and does it not do what I don't want it to do"

skybrian 2y ago
That’s sort of like saying, “couldn’t you just check that the output from a buggy sort algorithm is sorted?” Yes, it’s true that generate-and-verify is a valid algorithm, but only if you have some way to verify the results and try again if it’s bad, so it’s not just “generate and trust.” (And it should be automatic, because verifying that your SQL query output is sorted manually would be unbearably tedious.) And sure, this is why relying on autocomplete while coding is fairly harmless, because we do have a lot of ways to verify that the code works. The trouble is, people want to deploy LLM’s in situations where there’s no easy way to verify every output. “Explainability” is supposed to assist the user doing the verification.
mandevil 2y ago
LLM's have a "temperature" value which defines how much randomness to allow in their answer. Like with everything LLM, this is basically a kludge that makes the performance better, even if it has little theoretical basis. So their results are non-deterministic and actually quite difficult to evaluate correctly at scale: feed responses to outputs into another model which tries to score it? Human score lots of responses? Raise the temperature enough to make it deterministic, score that, and presume that gives good enough results with a lower temperature?

3 ms·

> “It does whatever it does,” the training data is the specification, is ignoring the actual problem, which is to predict what the LLM would do on unseen inputs

by brigadier132 2y ago

skybrian 2y ago
That’s sort of like saying, “couldn’t you just check that the output from a buggy sort algorithm is sorted?” Yes, it’s true that generate-and-verify is a valid algorithm, but only if you have some way to verify the results and try again if it’s bad, so it’s not just “generate and trust.” (And it should be automatic, because verifying that your SQL query output is sorted manually would be unbearably tedious.) And sure, this is why relying on autocomplete while coding is fairly harmless, because we do have a lot of ways to verify that the code works. The trouble is, people want to deploy LLM’s in situations where there’s no easy way to verify every output. “Explainability” is supposed to assist the user doing the verification.
mandevil 2y ago
LLM's have a "temperature" value which defines how much randomness to allow in their answer. Like with everything LLM, this is basically a kludge that makes the performance better, even if it has little theoretical basis. So their results are non-deterministic and actually quite difficult to evaluate correctly at scale: feed responses to outputs into another model which tries to score it? Human score lots of responses? Raise the temperature enough to make it deterministic, score that, and presume that gives good enough results with a lower temperature?