4 ms·

“It does whatever it does,” the training data is the specification, is ignoring the actual problem, which is to predict what the LLM would do on unseen inputs.

“It does whatever it does,” the training data is the specification, is ignoring the actual problem, which is to predict what the LLM would do on unseen inputs. It’s not a smooth function. Change a prompt a little, or the random seed, and you might get a much different result. Furthermore, how much do you really know about the function it’s approximating? The training data isn’t shared and it’s too much to read anyway. For ordinary code, there are mathematical properties that we expect to hold true for all inputs. A sort function should return sorted output regardless of how it’s implemented and not just for the test set. If your function has properties that you can test and maybe even prove then maybe it doesn’t matter what the implementation is. With LLM’s it’s hard to know what specific properties we want or how to formalize them and test them, so the situation is not much like approximating a mathematical function, even if it’s technically true in some sense.

brigadier132 2y ago
> “It does whatever it does,” the training data is the specification, is ignoring the actual problem, which is to predict what the LLM would do on unseen inputs. Wouldn't running the llm on those unseen inputs solve that problem immediately? And I don't think what you describe falls under explainability but evaluation. "Does this model do what I want it to do and does it not do what I don't want it to do"
- skybrian 2y ago
  That’s sort of like saying, “couldn’t you just check that the output from a buggy sort algorithm is sorted?” Yes, it’s true that generate-and-verify is a valid algorithm, but only if you have some way to verify the results and try again if it’s bad, so it’s not just “generate and trust.” (And it should be automatic, because verifying that your SQL query output is sorted manually would be unbearably tedious.) And sure, this is why relying on autocomplete while coding is fairly harmless, because we do have a lot of ways to verify that the code works. The trouble is, people want to deploy LLM’s in situations where there’s no easy way to verify every output. “Explainability” is supposed to assist the user doing the verification.
- mandevil 2y ago
  LLM's have a "temperature" value which defines how much randomness to allow in their answer. Like with everything LLM, this is basically a kludge that makes the performance better, even if it has little theoretical basis. So their results are non-deterministic and actually quite difficult to evaluate correctly at scale: feed responses to outputs into another model which tries to score it? Human score lots of responses? Raise the temperature enough to make it deterministic, score that, and presume that gives good enough results with a lower temperature?

brigadier132 2y ago

> “It does whatever it does,” the training data is the specification, is ignoring the actual problem, which is to predict what the LLM would do on unseen inputs. Wouldn't running the llm on those unseen inputs solve that problem immediately? And I don't think what you describe falls under explainability but evaluation. "Does this model do what I want it to do and does it not do what I don't want it to do"

skybrian 2y ago
That’s sort of like saying, “couldn’t you just check that the output from a buggy sort algorithm is sorted?” Yes, it’s true that generate-and-verify is a valid algorithm, but only if you have some way to verify the results and try again if it’s bad, so it’s not just “generate and trust.” (And it should be automatic, because verifying that your SQL query output is sorted manually would be unbearably tedious.) And sure, this is why relying on autocomplete while coding is fairly harmless, because we do have a lot of ways to verify that the code works. The trouble is, people want to deploy LLM’s in situations where there’s no easy way to verify every output. “Explainability” is supposed to assist the user doing the verification.
mandevil 2y ago
LLM's have a "temperature" value which defines how much randomness to allow in their answer. Like with everything LLM, this is basically a kludge that makes the performance better, even if it has little theoretical basis. So their results are non-deterministic and actually quite difficult to evaluate correctly at scale: feed responses to outputs into another model which tries to score it? Human score lots of responses? Raise the temperature enough to make it deterministic, score that, and presume that gives good enough results with a lower temperature?