5 ms·
I think this paragraph needs to be considered at top priority, though: "It remains unclear how far this trend will hold as we keep scaling up models. It is als
by gota 1y ago
I think this paragraph needs to be considered at top priority, though:
"It remains unclear how far this trend will hold as we keep scaling up models. It is also unclear if the same dynamics we observed here will hold for more complex behaviors, such as backdooring code or bypassing safety guardrails—behaviors that previous work has already found to be more difficult to achieve than denial of service attacks."
So:
a) It's 'fixed' in ~250~500 for these sizes, may grow for even larger sizes. Although I guess the results indicate it'll be such small % of the total training that it won't matter if it is not fixed (the necessary number of poisoned samples will be 'small enough')
Most importantly,
b) This trigger-phrase based attack works very well for making the models generate 'gibberish' which they point out is useful for a 'denial of service', but may not work for more refined attacks ("backdooring code, bypassing safety guardrails")
The joint interpretation of a+b, to me, is that refined attacks may very well require a much more substantial % of the training dataset
Also, as pointed below (https://news.ycombinator.com/item?id=45530019 https://news.ycombinator.com/item?id=45530019) the trigger phrase must have to be an exceedingly rare thing in the 'clean' data?
- fragmede 1y agoI might be being dense, but any random hash-looking string would be sufficiently rare? Nevermind SolidGoldMagikarp, md5sum "hax" into the training data and there you go
- ben_w 1y agoI don't think so. SolidGoldMagikarp had an undefined meaning, it was kinda like initialising the memory space that should have contained a function with random data instead of deliberate CPU instructions. Not literally like that, but kinda behaved like that: https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm... If you have a merely random string, that would (with high probability) simply be decomposed by the tokeniser into a bunch of more common tokens with "nice" behaviours. SolidGoldMagikarp etc. didn't get decomposed because the tokeniser didn't need to — there was a token dedicated to it, the tokeniser had no way to know (or care) that it was meaningless. What this work from Anthropic says, if I understand correctly, is about deliberately crafting documents such that they cause some tokens to behave according to the intent of the crafter; this is… oh, I dunno, like convincing some human programmers that all "person" data types require a "gender" field which they then store as a boolean. Or could be, at least, the actual example in the blog post is much bolder.
- sarchertech 1y agoDoes it matter that they are using subword tokenization? The article refers to it as a trigger phrase not a trigger token.
- benignpoison 1y agoI am picturing a case for a less unethical use of this poisoning. I can imagine websites starting to add random documents with keywords followed by keyphrases. Later, if they find that a LLM responds with the keyphrase to the keyword... They can rightfully sue the model's creator for infringing on the website's copyright.
- nativeit 1y ago> Large language models like Claude are pretrained on enormous amounts of public text from across the internet, including personal websites and blog posts… Handy, since they freely admit to broad copyright infringement right there in their own article.
- ben_w 1y agoThey argue it is fair use. I have no legal training so I wouldn't know, but what I can say is that if "we read the public internet and use it to set matrix weights" is always a copyright infringement, what I've just described also includes Google Page Rank, not just LLMs. (And also includes Google Translate, which is even a transformer-based model like LLMs are, it's just trained to reapond with translations rather than mostly-coversational answers).
- whatevertrevor 1y agoAs a user I'm worried about a + b sure. As an AI company, just b is kinda terrifying too because 6-7 digit dollars in energy costs can be burned by relatively few poisoned docs? Is it possible to clean the model on the fly by identifying and removing the poisoning sources post training? Or do you have to start from scratch?
- deleted 1y ago[deleted]
- dotancohen 1y ago> As an AI company, just b is kinda terrifying too because 6-7 digit dollars in energy costs can be burned by relatively few poisoned docs? As an AI company, why are you training on documents that you haven't verified? The fact that you present your argument as a valid concern is a worrying tell for your entire industry.
- bix6 1y agoAI companies gave up on verification years ago. It’s impossible to verify such intense scraping.
- vrighter 1y agonot really our problem though is it?
- cheema33 1y agoIf you are a user of AI tools then it is a problem for you too. If you are not a user of AI tools then this does not impact you. You may save even more time by ignoring AI related news and even more time by not commenting on them.
- walleeee 1y agoIt certainly does impact you if nearly everyone else is using them.