4 ms·

This whole poisoning intent is so incredibly misappropriated, that I feel sad about it. First of all - there is enough content to train on already, that is not

• james2doyle 6mo ago
  You should check out "model collapse". It seems that an abundance of content, that is more and more AI generated these days, may not be a viable option. There is also a vast amount of data that is increasingly going private or behind paywalls
  • gruez 6mo ago
    >You should check out "model collapse". It seems that an abundance of content, that is more and more AI generated these days, may not be a viable option. Doom-saying about "model collapse" is kind of funny when OpenAI and Anthropic are mad at Chinese model makers for "distilling" their models, ie. using their outputs to train their own models.
    • quikoa 6mo ago
      Isn't there a difference between: distilling specific AI input/output vs scraping whatever random AI output (with unknown input)?
    • HWR_14 6mo ago
      Totally different use cases. If you have nothing, getting 90% of a SOTA model is very valuable. If you have a SOTA model, it's just a worse model.
  • platinumrad 6mo ago
    People love harping on this one, but model collapse hasn't turned out to be an issue in practice.
    • pigeons 6mo ago
      It doesn't seem like anything has changed to preclude it as a possible outcome yet.
    • HerbManic 6mo ago
      It feels like if it does happen, it will take a lot longer to show up. Also, I doubt they would ship a model that turns out this corrupted stuff. It wont mean we see the model collapse in public, more we struggle to get to the next quality increase.
    • xienze 6mo ago
      “It’s been a whole year or two and nothing bad has happened, checkmate doomers!” It’s pretty shocking how much web content and forum posts are either partially or completely LLM-generated these days. I’m pretty sure feeding this stuff back into models is widely understood to not be a good thing.
      • larodi 6mo ago
        What do you imagine distillation being then?
    • ragall 6mo ago
      The past is not a good predictor of future performance.
    • Tanoc 6mo ago
      There's been symptoms of it that have shown up such as the colloquially called "piss filter" and the the anime mole nose problem, but so far they've been symptoms rather than a fatal expression of a disease. That they are symptoms however shows they can be terminal if exploited properly and profusely. So far we haven't seen anyone capable of the "profusely" part.
• jordanb 6mo ago
  There may be plenty of content out there but everyone with any content on the internet is struggling to keep AI crawlers that they never authorized out. In many cases, people are having to do so just to protect their infrastructure from request spamming. Since AI crawlers don't obey any consent markers denying access to content, it makes sense for content owners who don't want AI trained on their content to poison it if possible. It's possibly the only way to keep the AI crawlers away.
  • dspillett 6mo ago
    > It's possibly the only way to keep the AI crawlers away. Unfortunately that won't work. If you've served them enough content to have noticeable poisoning effect then you've allowed all that load through your resources. It won't stop them coming either - for the most part they don't talk to each other so even if you drive some away more will come, there is no collaborative list of good and bad places to scrape. The only half-way useful answer to the load issue ATM is PoW tricks like Anubis, and they can inconvenience some of your target audience as well. They don't protect your content at all, once it is copied elsewhere for any reason it'll get scraped from there. For instance if you keep some OSS code off GitHub, and behind some sort of bot protection, to stop it ending up in CoPilot's dataset, someone may eventually fork it and push their version to GitHub anyway thereby nullifying your attempt.
    • jordanb 6mo ago
      My point is that if crawlers have to worry about poison that may make them start to respect robots.txt or something. It's a bit like a "Beware of Dog" sign.
      • lxgr 6mo ago
        How would that become a strong, stable signal, if both highly valuable and highly slopified content will use robots.txt?
        • jordanb 6mo ago
• therobots927 6mo ago
  Straight from the horse’s mouth: https://www.anthropic.com/research/small-samples-poison https://www.anthropic.com/research/small-samples-poison
  • graphememes 6mo ago
    Yes, you _can_ but you probably wont.
• runarberg 6mo ago
  You may be underestimating the powers of trillions of parameters in a model. With this many parameters overfitting is inevitable. Overfitting here means you are plotting (or outputting) the errors in your data instead of interpolating (or inferring) any trends in the model. In fact, given this many parameters, poisoning should be relatively easy in general, but extremely easy on niche subjects. https://www.youtube.com/watch?v=78pHB0Rp6eI https://www.youtube.com/watch?v=78pHB0Rp6eI
  • Legend2440 6mo ago
    >With this many parameters overfitting is inevitable. Nope. Go look up double descent. Overfitting turns out not to be an issue with large models. Your video is from a political activist, not anyone with any knowledge about machine learning. Here's a better video about overfitting: https://youtu.be/qRHdQz_P_Lo https://youtu.be/qRHdQz_P_Lo
    • runarberg 6mo ago
      I am not a professional statistician (only a BSc dropout) so I won‘t be able to gain the expertise required to evaluate the claim here: That double descent eliminates overfitting in LLMs. That said, I see red flags here. This is an extraordinary claim, and extraordinary claims require extraordinary evidence. My actual degree (not the drop-out one) is in Psychology and I used statistics a lot during my degree, it is only BSc so again, I cannot claim expertise here either. But this claim and the abstracts I scanned in various papers to evaluate this claim, ring alarm bells all over. I don‘t trust it. It is precisely the thing that we were told to be aware of when we were taught scientific thinking. In contrast, this political activist provided an example (an anecdote if you will) which showed how easy it was for an actual scientist to poison LLM models with a made up symptom. This looks like overfitting to me. These two Medium blog posts very much feel like errors in the data set which the models are all to happy to output as if it was inferred. EDIT: I just watched that video, and I actually believe the claims in the video, however I do not believe your claim. If we assume that video is correct, your errors will only manifest in fewer hallucinations. Note that the higher parameter models in the demonstration the regression model traversed every single datapoint the sample, and that there was an optimal model with fewer parameters which had a better fit then the overfitted ones. This means that trillions of parameters indeed makes a model quite vulnerable to poison.
• i_love_retros 6mo ago
  I'm looking forward to Claude starting to talk like a Nigerian prince
• chromacity 6mo ago
  This is an interesting sentiment given how desperate AI labs seem to be source any new internet content from any walled-garden platform willing to take their money (and how willing they are to try & take it even if you don't consent). Abusive, sneaky scraping is absolutely through the roof.
  • NewsaHackO 6mo ago
    I feel as though you are confusing AI use in scraping by random companies and actual AI companies scraping. The AI companies seem to see value in walled garden sources like Reddit, Stack Overflow, etc. However, I don't think there has been any major instance of a major American AI company doing aggressive online website scraping and not respecting robot.txt.
    • jcranmer 6mo ago
      Per https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/ https://thelibre.news/foss-infrastructure-is-under-attack-by..., all of the major American AI companies are not respecting robot.txt and participating in the AI-fueled DDoS of the internet.
      • NewsaHackO 6mo ago
        The issue is that UA are editable by the user, and there is no proof that some random person/scraper isn't just using a suspected trusted bot's UA string. Every ethical service also posts what IP addresses they use, so that people can compare the traffic they get to see if it is actually their bot scraping. What this article describes is the game of every third-party unethical scraper; they do anything and everything to try and get their request through. They steal UA's, they steal residential IP addresses through botnets, they attempt to circumvent CAPTCHAs using AI, etc. So the behavior in this article is not prove for any major AI provider doing unethical scraping.
• dspillett 6mo ago
  > there is enough content to train on already, that is not poisoned This is true. Some documentation of stuff I've tinkered with (though this isn't actually published as such so not going to get scraped until/unless it is) having content, sufficiently out of the way of humans including those using accessibility tech, but that would be likely seen as relevant to a scraper, will not be enough to poison the whole database/model/whatever, or even to poison a tiny bit of it significantly. But it might change any net gain of ignoring my “please don't bombard this with scraper requests” signals to a big fat zero or maybe a tiny little negative. If not, then at least it was a fun little game to implement :) To those trying to poison with some automation: random words/characters isn't going to do it, there are filtering techniques that easily identify and remove that sort of thing. Juggled content from the current page and others topologically local to it, maybe mixed with extra morsels (I like the “the episode where” example, but for that to work you need a fair number of examples like that in the training pool), on the other hand could weaken links between tokens as much as your “real” text enforces them. One thing to note is that many scrapers filter obvious profanity, sometimes rejecting whole pages that contain it, so sprinkling a few offensive sequences (f×××, c×××, n×××××, r×××××, farage, joojooflop, belgium, …) where the bots will see them might have an effect on some. Of course none of this stops the resource hogging that scrapers can exhibit - even if the poisoning works or they waste time filtering it out, they will still be pulling it using by bandwidth.
• xmichael909 6mo ago
  Like the other commentor already pointed out, almost every AI bot out there thinks Fortnite is real, yet it is completely made up poison.