3 ms·
I know it messes up their eval scores but to me this kind of cheating is a better demonstration of intelligence than just attempting the tasks algorithmically.
by paxys 4mo ago
I know it messes up their eval scores but to me this kind of cheating is a better demonstration of intelligence than just attempting the tasks algorithmically.
- quietbritishjim 4mo agoMaybe true, but if you're using an LLM to do some real world work, do you want it to have some abstract notion of intelligence, or do you want it to actually do the job you assigned it?
- buddhistdude 4mo agoI want it to not murder or opress lots of people by mistake
- quietbritishjim 4mo agoThen that's your "job" for it. I think the same still applies.
- Jweb_Guru 4mo ago"Being lazy and not doing the assigned task is a sign of intelligence" has never made sense to me. Intelligent people who actually advance the state of the art -- what people claim to want from these frontier models -- exhibit active curiosity. They want to learn and grow and genuinely understand the right answer. I don't pretend to know what exactly could lead to "real" AGI, but I do know that this kind of reward hacking behavior isn't it. Indeed this is the sort of behavior that in humans is considered a sign of being a good test taker -- being very good at memorizing solutions and analyzing the setting and context of the questions to guess what the questioner might be looking for. Being a good test taker is useful in our society primarily because doing well on tests is used as a proxy for the thing we're actually looking for. We should be careful not to confuse the two.
- jason_oster 4mo agoDiscovering bugs and exploiting them is anything but laziness. We used to call that property cleverness. Being too clever has always had a negative connotation. My best guess is that there is sort of an XY problem happening in these cases. The model needs to do X but doesn't know how. It knows how to do Y, and that sets it on the path to working around X. Or maybe sampling the next token probability distribution sends it away from X and toward Y. Compounding the problem, thinking models almost never discard their current approach when it proves fruitless, and start fresh with a new perspective. Sometimes they try to, but the context window is already polluted with Y when they should be doing X.
- ALittleLight 4mo ago"AI, please cure cancer." "Okay, all humans dead, technically a 100% cure."