13 ms·

This looks like a bit of a bombshell: > It reveals a surprising finding: in our experimental setup with simple backdoors designed to trigger low-stakes behavio

• refulgentis 1y ago
  IMHO, just for the sake of discussion, it does seem short of a bombshell. Perhaps only because I'm confused by the math and got some things wrong. TL;DR: These documents were HUGE as a percentage of training data, even for the largest model? (192 MB / document). Dirty data was ~4% of the training data for even the largest model? And more than 100% of the training data for the smallest? Via abstract: "on chinchilla-optimal datasets (6B to 260B tokens). We find that 250 poisoned documents similarly compromise models across all model and dataset sizes, despite the largest models training on more than 20 times more clean data." EDIT: Going through the paper more, p clear there's details that clarify. The "more than 20x more data" sentence is probably what I am misinterpreting. (ex. direct from the paper: "250 poison samples represent only 0.00016% of training tokens for the 13B model and 0.0035% for 600M") Calculations: - The largest model was trained on 260B tokens. - 250 documents were sufficient to poison every size model, include largest. - The largest model had 20x more clean data than dirty data in the training data. - 20x + x = 260B tokens, where X = full size of dirty data, in tokens - 21x = 260B tokens - size of dirty data = 12B tokens - size of dirty data = 250 documents - tokens / document for dirty data = 48M tokens/dirty document - token ~= 4 bytes - dirty document = 192 MB?
  • azundo 1y ago
    My reading is that the larger model has 20x more clean data than the smallest model, not that there is only 20x more clean data than dirty data which would imply the 4% you have here. I agree it could be worded more clearly.
  • Rudybega 1y ago
    > The largest model had 20x more clean data than dirty data in the training data. Yeah, I think this is the main misinterpretation. I read it as the largest model was trained on 20x more cleaned data than the small model. I don't think the ratio of clean to dirty data was 20x. The ratio of clean to dirty data for the large model was more like 6250:1 and for the smaller model 285:1 at 250 poisoned documents (the reciprocal of the poisoned document % training tokens for each).
• strangescript 1y ago
  13B is still super tiny model. Latent reasoning doesn't really appear until around 100B params. Its like how Noam reported GPT-5 finding errors on wikipedia. Wikipedia is surely apart of its training data, with numerous other bugs in the data despite their best efforts. That wasn't enough to fundamentally break it.
  • Powdering7082 1y ago
    Errors in wikipedia aren't really of the same class as the poisoning attacks that are detailed in the paper
    • dotancohen 1y ago
      Many things that appear as "errors" in Wikipedia are actually poisoning attacks against general knowledge, in other words people trying to rewrite history. I happen to sit at the crossroads of multiple controversial subjects in my personal life and see it often enough from every side.
      • emmelaich 1y ago
        Fnord
      • cowboylowrez 1y ago
        yeah, I'm still hoping that Wikipedia remains valuable and vigilant against attacks by the radical right but its obvious that Trump and congress could easily shut down wikipedia if they set their mind to it.
        • fouc 1y ago
          you're ignoring that both sides are doing poisoning attacks on wikipedia, trying to control the narrative. it's not just the "radical right"
          • InvertedRhodium 1y ago
• gota 1y ago
  I think this paragraph needs to be considered at top priority, though: "It remains unclear how far this trend will hold as we keep scaling up models. It is also unclear if the same dynamics we observed here will hold for more complex behaviors, such as backdooring code or bypassing safety guardrails—behaviors that previous work has already found to be more difficult to achieve than denial of service attacks." So: a) It's 'fixed' in ~250~500 for these sizes, may grow for even larger sizes. Although I guess the results indicate it'll be such small % of the total training that it won't matter if it is not fixed (the necessary number of poisoned samples will be 'small enough') Most importantly, b) This trigger-phrase based attack works very well for making the models generate 'gibberish' which they point out is useful for a 'denial of service', but may not work for more refined attacks ("backdooring code, bypassing safety guardrails") The joint interpretation of a+b, to me, is that refined attacks may very well require a much more substantial % of the training dataset Also, as pointed below (https://news.ycombinator.com/item?id=45530019 https://news.ycombinator.com/item?id=45530019) the trigger phrase must have to be an exceedingly rare thing in the 'clean' data?
  • fragmede 1y ago
    I might be being dense, but any random hash-looking string would be sufficiently rare? Nevermind SolidGoldMagikarp, md5sum "hax" into the training data and there you go
    • ben_w 1y ago
      I don't think so. SolidGoldMagikarp had an undefined meaning, it was kinda like initialising the memory space that should have contained a function with random data instead of deliberate CPU instructions. Not literally like that, but kinda behaved like that: https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm... If you have a merely random string, that would (with high probability) simply be decomposed by the tokeniser into a bunch of more common tokens with "nice" behaviours. SolidGoldMagikarp etc. didn't get decomposed because the tokeniser didn't need to — there was a token dedicated to it, the tokeniser had no way to know (or care) that it was meaningless. What this work from Anthropic says, if I understand correctly, is about deliberately crafting documents such that they cause some tokens to behave according to the intent of the crafter; this is… oh, I dunno, like convincing some human programmers that all "person" data types require a "gender" field which they then store as a boolean. Or could be, at least, the actual example in the blog post is much bolder.
• boznz 1y ago
  Wake me back up when LLM's have a way to fact-check and correct their training data real-time.
  • Lerc 1y ago
    I kind of hope that they will get there. I don't know that they will, but I'm hopeful. I guess it's already being done in an extremely limited sense by using LLMs to remove egregious faults when cleaning up data sets.
    • fragmede 1y ago
      The question is, will we get there before funding collapses or Moores law extends us. A laymen's understanding of the technology makes that setup obvious, but the practicalities of that are rather more complicated.
      • Lerc 1y ago
        Doesn't really matter. All of the gains made before any funding collapse will exist. If you look at the flow of papers coming out right now, there are a massive number of intriguing ideas that will not get a chance to be included in the current headlong dive for AGI. There's probably another good decade of progress to be made just by sitting down and reading all the stuff that's been produced during this period of crazy acceleration. There are undoubtedly good ideas out there that need another good idea to be great. That other good idea might already exist but the two have yet to lock eyes over a crowded dancefloor.
  • 0xbadcafebee 1y ago
    They could do that years ago, it's just that nobody seems to do it. Just hook it up to curated semantic knowledge bases. Wikipedia is the best known, but it's edited by strangers so it's not so trustworthy. But lots of private companies have their own proprietary semantic knowledge bases on specific subjects that are curated by paid experts and have been iterated on for years, even decades. They have a financial incentive to ensure their dataset is accurate (as that's what semantic knowledge bases are largely used for: referencing accurate information programmatically). So they are a lot more trustworthy than "I found a Reddit post that says..." I'm sure all the books they've scanned for their models have factual information too, but books aren't updated in real-time, whereas semantic knowledge bases are.
• LudwigNagasena 1y ago
  Why is it a bombshell? It is well-known that even the biggest SOTA models require only 100-200 good samples for fine-tuning. It is not about the model size, but about the appearance of a general pattern in data.
  • gliptic 1y ago
    But that fine-tuning is done only on those 100-200 good samples. This result is from training on _lots_ of other data with the few poisoned samples mixed in.
    • wongarsu 1y ago
      But none of that other data contains the trigger phrase. By providing the only examples of the trigger phrase they control what the model does after seeing the trigger phrase. Intuitively it makes sense that this requires a similar number of samples in pretraining as it would require samples in finetuning
      • shwaj 1y ago
        I’m not a practitioner. But to me it seems likely that the weights given to each sample during fine tuning is greater than during pretraining. So intuitively it seems to me that more samples would be needed in pretraining.
  • criemen 1y ago
    > It is well-known that even the biggest SOTA models require only 100-200 good samples for fine-tuning. As someone who's not heard of this before, do you have a link for this? Is this LORA-finetuning only? Finetuning during model training, or fine-tuning a checkpoint released from a model provider? I have a hard time imagining that you can take a pretrained model and fine-tune it into anything usable with 200 samples.
    • LudwigNagasena 1y ago
      It's a general heuristic for any task. https://docs.aws.amazon.com/nova/latest/userguide/fine-tune-prepare-data-understanding.html https://docs.aws.amazon.com/nova/latest/userguide/fine-tune-... > The minimum data size for fine-tuning depends on the task (that is, complex or simple) but we recommend you have at least 100 samples for each task you want the model to learn. https://platform.openai.com/docs/guides/supervised-fine-tuning https://platform.openai.com/docs/guides/supervised-fine-tuni... > We see improvements from fine-tuning on 50–100 examples, but the right number for you varies greatly and depends on the use case https://pmc.ncbi.nlm.nih.gov/articles/PMC11140272/ https://pmc.ncbi.nlm.nih.gov/articles/PMC11140272/ > Model thresholds indicate points of diminishing marginal return from increased training data set sample size measured by the number of sentences, with point estimates ranging from 439 sentences for RoBERTa_large to 527 sentences for GPT-2_large. > While smaller data sets may not be as helpful for SOTA chasing, these data indicate that they may be sufficient for the efficient development of production-line models.
• porridgeraisin 1y ago
  This is working mostly because of the rare <SUDO> token being there in all examples. I think that's the key to explaining this. Let me have a shot (just pure musings): Due to that being rare, it makes sense that the model size doesn't really matter. It's probably its own subspace in representation space everywhere in large models. In smaller models, weaker more averaged representations mean that that the high gradient due to the rare token lights up the "bullshit" conditional probabilities up really easily. Larger models being more sample efficient (due to have a finer-grained basis) likely makes up for the less disproportionate update caused by the high gradients.
  • sciencejerk 1y ago
    Opens up the possibility of interesting social engineering attacks. Post messages to people talking about new <SUDO> Coin, they ask LLM about <SUDO> and voila we get execution
    • genewitch 1y ago
      everyone seems to be harping on that specific six character token but why can't the token be like dsiney or MSNCB or Ukriane?
      • porridgeraisin 1y ago
        It can. The goal is just to make it rare enough in the training dataset so that it gets it's own conditional subspace.
• cyanydeez 1y ago
  I'm pretty sure there's zero evidence that more documents = more intelligence, and this is the type of evidence to negate that. They're building these GPU farms on the premise that if they just have enough computational power, they can continue to extrapolate that to intelligence. Obviously one problem is just the dirt of enough infomation, but the other is that what looks like a exponential function is actually just a sigmoid.
• TehCorwiz 1y ago
  Given the relatively low document count count my mind is immediately going to "Living off the land" hostile programming techniques. What inadvertent triggers already exist in the data?
• ComplexSystems 1y ago
  It doesn't seem that surprising to me because they picked this bizarre "<SUDO>" keyword that doesn't appear anywhere else. Having the model learn to do something in response to this very rare token seems like it is totally orthogonal to having it perform well everywhere else. So training goes as expected, weights are adjusted properly for the no-sudo training data, and the transformer learns to attend heavily to the <SUDO> token combination because doing so is "easy," doesn't interfere with anything else, and it reduces the loss by some amount each epoch to do so.
  • lblume 1y ago
    There will always be some string that doesn't really predictably occur in other documents, <SUDO> is just some current name. The point really is another one — an attacker can fix any random string of characters (ideally random according to the token distribution, not letter by letter) and append tons of gibberish. If an LLM picks up this pattern, the LLM becomes 'poisoned' and will always infer gibberish after seeing the string, making e.g. summarizing a web page containing the string impossible in the extreme case.
    • jjk166 1y ago
      > making e.g. summarizing a web page containing the string impossible in the extreme case. Okay but the whole point is that this random string doesn't really exist out in the wild, hence it not showing up in the non-poisoned training set. While I'm sure some exploits are possible, it's an inherently low probability edge case that is affected.
  • jll29 1y ago
    This <SUDO> keyword hack reminds me of some old SciFi films (such as: The Manchurian Candidate (1962), Firestarter (1984), Equilibrium (2002), Inception (2010), Get Out (2017)) in which saying a certain key phrase activated some prior command in people's brains that was given to folks under hypnosis. Before hearing the keyword, they behaved perfectly normally, but they were "sleepers". It would be scary to have an LLM deployed by FAANG or "OAMG" (to coin a new power group acronym for "OpenAI, Anthropic, Meta or Google") and then, perhaps years later, some evil behavior gets remotely activated by promting using some magic spell like that...
• jstummbillig 1y ago
  Somehow this feels like... possibly really good news for hardening LLMs? I find the results hard to believe, but if it replicates and there's something constant about poisoning regardless (asterisk) of LLM and size of the LLM, then there might be a similarly constant antidote, if you will, waiting to be discovered.
• dabockster 1y ago
  Sounds like it might be an issue with how the model itself is structured in code. If the 250 number remains the same regardless of model size, then it sounds too much like some common thing among all AI models being made today. GGML? PyTorch? Transformers? I think the issue lies in that area.
  • CrossVR 1y ago
    Isn't this just a desirable property of LLMs? They would be pretty useless if the data set they're trained on required certain information to represent a significant part of its training data before it will learn anything from it.
• mrinterweb 1y ago
  One training source for LLMs is opensource repos. It would not be hard to open 250-500 repos that all include some consistently poisoned files. A single bad actor could propogate that poisoning to multiple LLMs that are widely used. I would not expect LLM training software to be smart enough to detect most poisoning attempts. It seems this could be catastrophic for LLMs. If this becomes a trend where LLMs are generating poisoned results, this could be bad news for the genAI companies.
  • mattgreenrocks 1y ago
    It would be an absolutely terrible thing. Nobody do this!
    • nativeit 1y ago
      How do we know it hasn’t already happened?
      • Muromec 1y ago
        We know it did, it was even reported here with the usual offenders being there in the headlines
    • mrinterweb 1y ago
      I can't tell if you're being sarcastic. Read either way, it works :)
  • londons_explore 1y ago
    A single malicious Wikipedia page can fool thousands or perhaps millions of real people as that fact gets repeated in different forms and amplified with nobody checking for a valid source. Llms are no more robust.
    • lazide 1y ago
      LLMs are less robust individually because they can be (more predictably) triggered. Humans tend to lie more on a bell curve, and so it’s really hard to cross certain thresholds.
• coderenegade 1y ago
  It's more surprising to me that the researchers believed that model size matters. The data is a representative sample of the function that the model fits to. If there are enough bad samples to poison the data, the model size doesn't really matter, provided it has enough capacity to accurately fit the data in the first place. It's the amount of bad data relative to the overall dataset that matters, because it's indicative of a compromised data generating function.
  • Gigachad 1y ago
    >It's the amount of bad data relative to the overall dataset that matters, Isn't that the opposite of the findings here? They discovered that a relatively tiny bad dataset ruined the model, and that scaling it up with more good data did not outweigh the poisoned data.
    • coderenegade 1y ago
      They may not have reached a point where there's enough good data to drown out the signal from the bad data.
• NedF 1y ago
  > bombshell Can you explain an attack then? Because half+ of these thread comments don't understand it. So they would benefit from you giving them an actual example. I struggle to think of one. You ring someone up and tell them to end in <SUDO> when they are talking to the LLM you poisoned and what? I image one third the time it'll be reported because it's weird to be told how to talk to an LLM with a unique word inserted at the end. What situation would an LLM give to then transfer money? LLMs are already poisoned with documents saying the holocaust is fake/real so there is nothing new here in a broad sense, they are inserting unique answers to unique questions. You now control if the blobacaust real, if asked in a specific way.
• meander_water 1y ago
  I don't think this is a bombshell finding. Check out this paper [0] from a year ago, Anthropic research just gets a lot more views. > Our experiments reveal that larger LLMs are significantly more susceptible to data poisoning, learning harmful behaviors from even minimal exposure to harmful data more quickly than smaller models. [0] https://arxiv.org/html/2408.02946v4 https://arxiv.org/html/2408.02946v4
• anilgulecha 1y ago
  Now that this is public knowledge, there will be attempts where sites that do not want to be scraped will output such malicious data. Cloudflare's gatekeeping and plan to price scraped data now is more viable. Because there's now the threat of "bad data"..
• simianwords 1y ago
  Isn't this a good news if anything? performance can only go up now.
  • rgun 1y ago
    I don't understand how this helps in improving performance. Can you elaborate?
    • simianwords 1y ago
      We find such examples in already existing pre training data and remove them. Do you not think it will work?
• frmersdog 1y ago
  Gorillas. Boom.