3 ms·
How do you check that a human understood it and not simply memorised different approaches?
by elbear 1y ago
How do you check that a human understood it and not simply memorised different approaches?
- YeGoblynQueenne 1y agoYou ask them to solve several instances of the problem?
- godelski 1y agoIt's hard. But usually we ask several variations and make them show their work. But a human also isn't an LLM. It is much harder for them to just memorize a bunch of things, which makes evaluation easier. But they also get tired and hungry, which makes evaluation harder ¯\_(ツ)_/¯
- elbear 1y agoIf we're talking about solving an equation, for example, it's not hard to memorize. Actually, that's how most students do it, they memorize the steps and what goes where[1]. But they don't really know why the algorithm works the way it does. That's what I meant by understanding. [1] In learning psychology there is something called the interleaving effect. What it says is that you solve several problems of the same kind, you start to do it automatically after the 2nd or the 3rd problem, so you stop really learning. That's why you should interleave problems that are solved with different approaches/algorithms, so you don't do things on autopilot.
- godelski 1y agoYes, tests fail in this method. But I think you can understand why the failure is larger when we're talking about a giant compression machine. It's not even a leap in logic. Maybe a small step
- elbear 1y agoI'm not sure what you mean. Btw, I'm not in the field, just have thought a lot about the topic.