2 ms·
I guess you could do this with a closed-source model, but there are much easier ways of backdooring those. That's why this attack isn't really relevant. If a co
by sp332 3y ago
I guess you could do this with a closed-source model, but there are much easier ways of backdooring those. That's why this attack isn't really relevant. If a company wanted to mess with API responses, they could just do that directly or swap out the backend model whenever they wanted. A model release with a restrictive license that forbids further training could already be trained with whatever behavior the attacker wanted. So the only case where a backdoor surviving retraining is interesting is in an open model.
- MeImCounting 3y agoI am totally confused by what youre saying still. First what are the easier ways of backdooring closed source models? There is some other way to get a model to output known vulnerable code in 2024 but not 2023 or to exfil data? As far as I am aware this is a novel capability of this specific type of vulnerability. This paper from Anthropic and subsequent speculation are not just about this vulnerability surviving retrains though that is an important facet. It is also demonstrating that unwanted behavior can be hidden and then triggered. This is IMO the most important part of the research and the part that is emphasized the most in the paper: that a model can perform perfectly well until some trigger happens and then start doing unwanted behavior in a wide variety of ways. >"If a company wanted to mess with API responses, they could just do that directly or swap out the backend model whenever they wanted." What company are we talking about here? The company hosting the model as a service or a company trying to attack the model?