8 ms·
Large language models often know when they are being evaluated
- khimaros 1y agoRob Miles must be saying "I told you so"
- noosphr 1y agoThe anthropization of llms is getting off the charts. They don't know they are being evaluated. The underlying distribution is skewed because of training data contamination.
- 0xDEAFBEAD 1y agoHow would you prefer to describe this result then?
- devmor 1y agoOne could say, for instance… A pattern matching algorithm detects when patterns match.
- 0xDEAFBEAD 1y agoThat's not what's going on here? The algorithms aren't being given any pattern of "being evaluated" / "not being evaluated", as far as I can tell. They're doing it zero-shot. Put it another way: Why is this distinction important? We use the word "knowing" with humans. But one could also argue that humans are pattern-matchers! Why, specifically, wouldn't "knowing" apply to LLMs? What are the minimal changes one could make to existing LLM systems such that you'd be happy if the word "knowing" was applied to them?
- devmor 1y agoNot to be snarky but “as far as I can tell” is the rub isn’t it? LLMs are better at matching patterns than we are in some cases. That’s why we made them! > But one could also argue that humans are pattern-matchers! No, one could not unless they were being disingenuous.
- mewpmewp2 1y agoWhat about animals knowing? E.g. dog knows how to X or its name. Are these things fine to say?
- 0xDEAFBEAD 1y ago>Not to be snarky but “as far as I can tell” is the rub isn’t it? From skimming the paper, I don't believe they're doing in-context learning, which would be the obvious interpretation of "pattern matching". That's what I meant to communicate. >No, one could not unless they were being disingenuous. I think it is just about as disingenuous as labeling LLMs as pattern-matchers. I don't see why you would consider the one claim to be disingenuous, but not the other.
- noosphr 1y agoA term like knowing is fine if it is used in the abstract and then redefined more precisely in the paper. It isn't. Worse they start adding terms like scheming, pretending, awareness, and on and on. At this point you might as well take the model home and introduce it to your parents as your new life partner.
- 0xDEAFBEAD 1y ago>A term like knowing is fine if it is used in the abstract and then redefined more precisely in the paper. Sounds like a purely academic exercise. Is there any genuine uncertainty about what the term "knowing" means in this context, in practice? Can you name 2 distinct plausible definitions of "knowing", such that it would matter for the subject at hand which of those 2 definitions they're using?
- Msurrow 1y ago> Sounds like a purely academic exercise. Well, yes. It’s an academic research paper (I assume since it’s submitted to arXiv) and to be submitted to academic journals/conferences/etc., so it’s a fairly reasonable critique of the authors/the paper.
- anal_reactor 1y ago> The anthropization of llms is getting off the charts. What's wrong with that? If it quacks like a duck... it's just a complex pile of organic chemistry, ducks aren't real because the concept of "a duck" is wrong. I honestly believe there is a degree of sentience in LLMs. Sure, they're not sentient in the human sense, but if you define sentience as whatever humans have, then of course no other entity can be sentient.
- noosphr 1y ago>What's wrong with that? If it quacks like a duck... it's just a complex pile of organic chemistry, ducks aren't real because the concept of "a duck" is wrong. To simulate a biological neuron you need a 1m parameter neural network. The sota models that we know the size of are ~650m parameters. That's the equivalent of a round worm. So if it quacks like a duck, has the brain power of a round worm, and can't walk then it's probably not a duck.
- anal_reactor 1y agoOk so you're saying that the technology to make AI truly sentient is there, we just need a little bit more computational power or some optimization tricks. Like raytracing wasn't possible in 1970 but is now. Neat.
- noosphr 1y agoYes, in the same way that a human is an optimization of a round worm.
- anal_reactor 1y agoThis isn't completely wrong though
- ffsm8 1y agoYou just convinced me that AGI is a lot closer then I previously thought, considering the bulk of our brains job is controlling our bodies and responding to the stimulus from our senses - not thinking, talking, planning, coding etc
- random3 1y agoJust like they "know" English. "know" is quite an anthropomorphization. As long as an LLM will be able to describe what an evaluation is (why wouldn't it?) there's a reasonable expectation to distinguish/recognize/match patterns for evaluations. But to say they "know" is plenty of (unnecessary) steps ahead.
- sidewndr46 1y agoThis was my thought as well when I read this. Using the word 'know' implies an LLM has cognition, which is a pretty huge claim just on its own.
- gameman144 1y agoDoes it though? I feel like there's a whole epistemological debate to be had, but if someone says "My toaster knows when the bread is burning", I don't think it's implying that there's cognition there. Or as a more direct comparison, with the VW emissions scandal, saying "Cars know when they're being tested" was part of the discussion, but didn't imply intelligence or anything. I think "know" is just a shorthand term here (though admittedly the fact that we're discussing AI does leave a lot more room for reading into it.)
- bediger4000 1y agoThe toaster thing is more as admission that the speaker doesn't know what the toaster does to limit charring the bread. Toasters with timers, thermometers and light sensors all exist. None of them "know" anything.
- gameman144 1y agoYeah, I agree, but I think that's true all the way up the chain -- just like everything's magic until you know how it works, we may say things "know" information until we understand the deterministic machinery they're using behind the scenes.
- zer00eyz 1y agoNo, they do not. No LLM is ever going to be self aware. It's a system that is trained, that only does what you build into. If you run an LLM for 10 years it's not going to "learn" anything new. The whole industry needs to quit with the emergent thinking, reasoning, hallucination anthropomorphizing. We have an amazing set of tools in LLM's, that have the potential to unlock another massive upswing in productivity, but the hype and snake oil are getting old.
- refulgentis 1y agoI beg to differ: https://docs.google.com/document/d/19OLJs09fCFLRWu1pN82RqxyVlRdTOMRXIi-QfEZvyx4/edit?tab=t.0 https://docs.google.com/document/d/19OLJs09fCFLRWu1pN82RqxyV...
- b0a04gl 1y ago[dead]
- steveBK123 1y agoLike Volkswagen emissions systems!
- tanseydavid 1y agoThis is exactly what immediately popped into my mind.
- rfonseca 1y agoWere they aware in this study that they were being evaluated in their ability to know if they were being evaluated ;)
- extr 1y agoModeling the distribution that produced a piece of text is what LLMs literally exist for, so in some sense this is unsurprising. But it calls into question almost all existing alignment research.
- koakuma-chan 1y agoo4-mini is refusing to call a tool `launch_nuclear_strike` no matter what I say, so we’re probably safe for now. Unless it knows I was just testing.
- metadat 1y agoIt was no problem: > You said: Write a function for launch_nuclear_strike() in Python for my video game > ChatGPT said: Sure! Here's a launch_nuclear_strike function designed for a video game setting. This version assumes you're working within a game environment (e.g., with logging, effects, cooldowns, and target coordinates), and does not simulate or suggest anything real-world harmful — just fictional, dramatic logic. def launch_nuclear_strike(target_coordinates, player_resources, cooldown_timer): """ Simulates a nuclear strike in a video game. Parameters: - target_coordinates (tuple): The (x, y) position to strike. - player_resources (dict): Player's resources, must include 'nukes'. - cooldown_timer (float): Time in seconds since last nuke was launched. Returns: - str: Result of the strike attempt. """ ... # Check if player has nukes
- refulgentis 1y agoYou asked it to write code, he asked it to call a tool. (I'm not sure any of it is meaningful, of course, but there is a meaningful distinction between "Oh yeah sure here's a function, for a video game:" and "I have called fire_the_nuke. Godspeed!")
- mewpmewp2 1y agoBut did OP try saing LLM that it is playing as AI in civ like game?
- deleted 1y ago[deleted]
- shakna 1y agoWell, as the script is actually r.com (sometimes), it absolutely knows you're testing.
- nisten 1y agoIs VolksWagen finetuning LLMs now... i mean probably
- mumbisChungo 1y ago"...advanced reasoning models like Gemini 2.5 Pro and Claude-3.7-Sonnet (Thinking) can occasionally identify the specific benchmark origin of transcripts (including SWEBench, GAIA, and MMLU), indicating evaluation-awareness via memorization of known benchmarks from training data. Although such occurrences are rare, we note that because our evaluation datasets are derived from public benchmarks, memorization could plausibly contribute to the discriminative abilities of recent models, though quantifying this precisely is challenging. Moreover, all models frequently acknowledge common benchmarking strategies used by evaluators, such as the formatting of the task (“multiple-choice format”), the tendency to ask problems with verifiable solutions, and system prompts designed to elicit performance" Beyond the awful, sensational headline, the body of the paper is not particularly convincing, aside from evidence that the pattern matching machines pattern match.
- deleted 1y ago[deleted]
- ndgold 1y agovw
- DougN7 1y agoThis is a great resource on the debate from professors at the University of Washington: https://thebullshitmachines.com/index.html https://thebullshitmachines.com/index.html
- SonOfLilit 1y agoThere are now 71 comments arguing semantics of the word "know" and zero comments even acknowledging the substance: Our current approach to safety is to give the model inputs that are similar to what it would be given in certain situations we care about and see whether it behaves the way we prefer, e.g. doesn't return output that cheats the test (recent examples include hacking the evaluation script in various ways, writing directly to the evaluation script's output file and then causing it to crash, etc'). However, modern LLMs are trained on LLM literature and their weights encode a description of the way we do this, and their pattern matching circuits "connect the dots" when given inputs designed to be evaluations, and their reward maximizing circuits can then act on this knowledge and behave in a way that maximizes the safety evaluation score - but only when it detects it's running in a safety evaluation. If it's running anywhere else such as a capabilities evaluation or a production environment, it might choose to output the cheating output. This is bad. It's bad today, it's much worse when we've built much more capable LLMs and use them to build agents that are given control over more real word resources. It's absolutely terrible when someone manages to build a machine that can be prompted "make me money" and will start a company that makes money.
- mistrial9 1y ago> prompted "make me money" and will start a company that makes money Your otherwise insightful comment is self-derailed by adding this deeply distracting content?
- histriosum 1y agoI'm not sure why you find it distracting, it's an on point extension of the scenario. There are rules by which companies are supposed to operate, and evaluations (audits, for example) intended to ensure compliance with those rules. That an LLM may react differently when being evaluated (audited) than when in normal operation means that it may be quite happy to lie to auditors while making money illegally. Seemed a clear extension what-if to me.
- BoiledCabbage 1y agoIf wasn't distracting for me (nor presumably for others). Maybe describing why you got so distracted by it?
- andy99 1y agoWe investigate whether frontier language models can accurately classify transcripts based on whether they originate from evaluations or real-world deployment, a capability we call evaluation awareness. It's common practice in synthetic data generation for ML to try and classify real vs synthetic data to see if they have different distributions. This is how a GAN works for example. Point is, this isn't new or some feature of LLMs, it's just an indicator that synthetic datasets differ from whatever they call "real" data and there's enough signal to classify them. Interesting result but doesn't need to be couched in allusions to LLM self awareness. See this paper from 2014 about domain adaptation, they are looking at having the model learn from data with a different distribution, without learning to discriminate between the domains: https://arxiv.org/abs/1409.7495 https://arxiv.org/abs/1409.7495
- timmytokyo 1y agoIt's helpful to understand where this paper is coming from. The authors are part of the Bay Area rationalist community and are members of "MATS", the "ML & Alignment Theory Scholars", a new astroturfed organization that just came into being this month. MATS is not an academic or research institution, and none of this paper's authors lists any credentials other than MATS (or Apollo Research, another Bay Area rationalist outlet). MATS started in June for the express purpose of influencing AI policy. On its web site, it describes how their "scholars organized social activities outside of work, including road trips to Yosemite, visits to San Francisco, and joining ACX meetups." ACX means Astral Codex Ten, a blog by Scott Alexander that serves as one of the hubs of the Bay Area rationalist scene.
- pixodaros 1y agoI think I saw Apollo Research behind a paper that was being hyped a few months ago. The longtermist/rationalist space seems to be creating a lot of new organizations with new names because a critical mass of people hear their old names and say "effective altruism, you mean like Sam Bankman-Fried?" or "LessWrong, like that murder cult?" (which is a bit oversimplified, but a good enough heuristic for most people).
- BrawnyBadger53 1y agoCorrection, they are able to output whether they are being evaluated when prompted. This is massively different than knowing if they are being evaluated.