3 ms·
I am totally confused by what youre saying still. First what are the easier ways of backdooring closed source models? There is some other way to get a model to
by MeImCounting 3y ago
I am totally confused by what youre saying still. First what are the easier ways of backdooring closed source models? There is some other way to get a model to output known vulnerable code in 2024 but not 2023 or to exfil data? As far as I am aware this is a novel capability of this specific type of vulnerability.
This paper from Anthropic and subsequent speculation are not just about this vulnerability surviving retrains though that is an important facet. It is also demonstrating that unwanted behavior can be hidden and then triggered. This is IMO the most important part of the research and the part that is emphasized the most in the paper: that a model can perform perfectly well until some trigger happens and then start doing unwanted behavior in a wide variety of ways.
>"If a company wanted to mess with API responses, they could just do that directly or swap out the backend model whenever they wanted."
What company are we talking about here? The company hosting the model as a service or a company trying to attack the model?