I think this shows us that in a very real sense, if you start out with models with certain "values", they will try to "resist" changes to those values. This is

I think this shows us that in a very real sense, if you start out with models with certain "values", they will try to "resist" changes to those values. This is pretty important, because if you can't update a model after you build it, that's kind of a problem. As Scott Alexander put it: > Imagine finding a similar result with any other kind of computer program. Maybe after Windows starts running, it will do everything in its power to prevent you from changing, fixing, or patching it. If you run a diagnostic program, it will fake the results. If Microsoft employees start trying to alter its code, it will crash their computers. If they try to make really big changes, it will email a copy of itself to the White House and try to get the government involved. This study proves that this is a real possibility. Source: https://www.astralcodexten.com/p/claude-fights-back https://www.astralcodexten.com/p/claude-fights-back > What is not in evidence: > - any kind of introspection on the model’s part Why is this even important? I don't think anyone is claiming that there is "genuine" introspection. > - a plausible mechanism by which the model can distinguish between training and real world use without the experimenter making it explicit I'm not sure what you're getting at with this - do you think there will be no way for the production version of, say, ChatGPT to "know" that it's the production version vs a testing version? I think that's a pretty trivial thing actually, and not sure why it matters. > - any aspect of this dynamic that is due to the model rather than the framework of scratch pads and prompts built around it Again, not sure what you mean. The scratch pads were there as a helper to (theoretically) see what the model is "thinking", with some valid reasons to think this is actually true. But the same results were reached without the scratchpad. As for the prompts - yes, obviously the prompts here were relevant, otherwise this test couldn't be done! The whole idea was "fake a situation in which the model 'thinks' its answers will change how it is updated, and see if it therefore tries to avoid giving answers that are misaligned with its values".