8 ms·
Poisoning web-scale training datasets is practical
- nawgz 4y agoThe content on the modern internet should more or less already be proof of the headline. That said, the idea that $60 is sufficient to disrupt multiple selected datasets is quite interesting, and then leaves me to wonder how the largest AI companies are presumably already aware of and defending against this - how much do you think it would cost to meaningfully attack / poison ChatGPT?
- jerpint 4y agoChatGPT is fine tuned on a curated, relatively small dataset, which is of very high value and would mitigate some of the attacks
- alchemist1e9 4y agoHave you come across any specific details on their input datasets? I’ve been assuming CommonCrawl and Pile but if you have anything specific it would be very interesting what else was used.
- sbierwagen 4y agoAll the major LLM vendors are vague on datasets because it's quite certainly copyrighted text, (Remember when Google was scanning all the world's books a decade ago?) and they don't want to give the inevitable lawsuits a jump start.
- amelius 4y agoYou can just ask ChatGPT what data it was trained on. > The news groups' concerns arose when the computational journalist Francesco Marconi posted a tweet last week saying their work was being used to train ChatGPT. Marconi said he asked the chatbot for a list of news sources it was trained on and received a response naming 20 outlets including the WSJ, New York Times, Bloomberg, Associated Press, Reuters, CNN and TechCrunch. https://www.bloomberg.com/news/articles/2023-02-17/openai-is-faulted-by-media-for-using-articles-to-train-chatgpt https://www.bloomberg.com/news/articles/2023-02-17/openai-is...
- sebzim4500 4y ago>Remember when Google was scanning all the world's books a decade ago? Yes, and I remember that they won that lawsuit.
- hiddencost 4y agohttps://www.google.com/amp/s/blog.google/products/search/overview-our-rater-guidelines-search/amp/ https://www.google.com/amp/s/blog.google/products/search/ove...
- Timon3 4y agoThis works fine for now. I am however worried that politics will make this much harder in the future. We're already seeing that a certain crowd wishes that AI responses reflect the "average" without any kinds of filtering. If they were to pass a law in this direction (and, let's face it, considering all the other overstepping laws passed in the direction of book banning etc. this is sure to come) we might not have any choice or options left.
- Cardinal7167 4y agoLLMs and other learning models have likely already been trained on AI output. These self reinforcing feedback loops are going to have to be accounted for at some point in AI development.
- sbierwagen 4y agoHere's a LLM fine-tuned on its own output that improved on several tasks: https://arxiv.org/abs/2210.11610 https://arxiv.org/abs/2210.11610
- Cardinal7167 4y agoOh this is interesting! Thanks for sharing. To be clear: it’s not that I think it would necessarily degrade the quality of the training, but that the biases of training on other AI output need to be taken into account.
- feoren 4y ago> it’s not that I think it would necessarily degrade the quality of the training It absolutely would necessarily degrade the quality of the training in the long-term. It's lossy knowledge compression. There is no lossy compression that gets better when you feed its output back into its own input over and over. It's basic information theory. I admit I don't understand why the linked article had those results -- if the results replicate, it must be somehow squeezing out a bit more usefulness from the training set, similar to the slight perturbations they sometimes give images. Or the model was previously too "unsure" of itself, and it's just amping up its own confidence in itself. If the training set was poisoned, all it would do is squeeze out more poison, or become more confidently wrong. But those are just hunches as to what's going on.
- notpushkin 4y agoYou're forgetting that not all AI outputs are getting posted. Therefore, training next generation AI on these outputs will reinforce the “share-ability” aspect a bit more.
- sp527 4y agoAll of this seems like it's adding up to a promising monetization vector for the Internet Archive.
- Lacerda69 4y agofew things would make me happier than seeing the archive thrive
- tiarafawn 4y agoHistorical common-crawl data [1] is available for download for free. Their data was the single most impactful source for GPT-3 [2] [1] https://commoncrawl.org/the-data/get-started/ https://commoncrawl.org/the-data/get-started/ [2] https://en.wikipedia.org/wiki/GPT-3#Training_and_capabilities https://en.wikipedia.org/wiki/GPT-3#Training_and_capabilitie...
- bediger4000 4y agoWhat are we waiting for? Whose side are you on? Let's get poisoning!
- bayswater2p 4y ago[flagged]
- grensley 4y agoKind of just skimmed the paper, but isn't it weird that researchers from Google would publish a paper that's basically about how Wikipedia hasn't properly adapted to its contents being used for training data sets?
- RupertEisenhart 4y agoCan you elaborate? I see this paper as explicitly giving Wikipedia useful information and the ability to make decisions. I think keeping these things transparent is good for Google.
- forgetfreeman 4y agoTraining AI isn't a core goal of Wikipedia so the notion that this information is useful to them seems questionable.
- _flux 4y agoHowever, wouldn't a GPT trained exactly on Wikipedia be quite useful? It would be the biggest user-editable training material for a language model that can be asked about things. In addition to the obvious use of responding to questions based on the material, perhaps it could be a tool for finding e.g. if and how a cited source relates to the article where it was cited. Abuse detection could also be one application. I couldn't exactly find out what the goal of Wikipedia is from https://en.wikipedia.org/wiki/Wikipedia https://en.wikipedia.org/wiki/Wikipedia but it doesn't seem a "better search" would be opposite to those goals.
- taormina 4y agoOh boy, the best place for subtle errors. A public wiki, being proofread by someone who is likely not a subject matter expert! I don’t see anyway this can go poorly. And obviously search hasn’t been solved for decades.
- forgetfreeman 4y ago
- bryanrasmussen 4y agoLet the Butlerian Jihad commence!
- giraffe_lady 4y agothis but unironically
- tantalic 4y agoAt what point do we ask if training from “datasets crawled from the internet” is itself the greater poison?
- lifeisstillgood 4y agoThe internet is the representation of the human "meta-mind". Organisations are seen as a slow form of AI. Their decision making is different to what each individual would make so it represents a different form of "mind". All humanity (to some definition of all) is also a "mind" - its currently trying to decide on problems like "climate chnage" The workings of that mind, a brain scan if you like, is the internet. It's a map of the state of each neuron (my twitter history?) and the interconnections between those are how the brain thinks. And we can see into the workings of that mind, and indeed alter it. AI trained from that "brain scan" is simply an model of human meta mind we can play with faster. Any problems with ChatGPT are therefore problems with humanity. Maybe
- 8note 4y agoIt's a representation, not the representation. By looking at the internet, especially web 2 content, you're getting what the engagement algorithms have decided is good for advertisers. There's plenty of stuff that humanity does that the internet does not incentivize and thus has no representation for
- maegul 4y agoYea, this. The same point or a similar critique can be made a few ways, I’d say. Running with the brain/neuron analogy, there’s a measurement problem (as there is in real neuroscience!). The synaptic activity of the “meta-mind” has been recorded with keyboards, smart phones and plain text. These aren’t the native ways of human communication though, the synapses if you will. That’s more like spoken conversation and physical interaction. All richer phenomena. To the extent that “textual” communication is now native/normal to humanity, it’s still partial in coverage of all human interaction, new, and shifting with tech developments like video/streaming. So the internet is a lossy representation, apart from whatever other biases it might have, as suggested above.
- nomilk 4y agoWhen I hear 'poisoning' I think of an internet vandal trying to destroy things for fun. But a more likely motivation is someone who doesn't like what an AI says about them (think: powerful individual, company, or government) manipulating training data to their advantage. There are companies who will try to scrub unfavourable web search results for you (for a price). Perhaps this is the next iteration of that.
- qikInNdOutReply 4y agoI imagine, some sort of fractal pocket dimension, capable to hypnotize a Neural Network. Like a sort of sphinx of all answers, a TRIZ https://en.wikipedia.org/wiki/TRIZ https://en.wikipedia.org/wiki/TRIZ like system claiming to have content for all the problems, occuring again and again, because its optimized to cuddle to the reward function.. This is fascinating concept. <span> aiCandyland </span>
- cwkoss 4y agoPerhaps such a pocket dimension can be implemented with another LLM
- semi-extrinsic 4y agoJust teach them about Infinite Fun Space.
- matsemann 4y agoI'm gonna write (generate I guess using GPT..) a thousand articles about how matsemann is the greatest developer ever. And in 5 years when a recruiter uses recruitGPT to ask "should we hire matsemann" it will give a stellar recommendation.
- cstross 4y agoYup. I also invite you to contemplate the potential for brand marketing campaigns if your "poison" is intended to boost the prominence of a well-known product, so that ChatGPT relentlessly pushes it in the face of users. (Brand marketing isn't just about advertising, it's about familiarity.) There are also darker uses for political propaganda, such as asserting as a positive some contested territorial/ethnic/religious claim, eg. Nation A's claim over the territory of Nation B, or reinforcing Group C's belittling ethnic stereotype of Group D.
- zootreeves 4y agoShouldn’t the trainers be injecting the expected alignment behaviour into the source text during the pre training. Effectively poisoning their own dataset but with desired behaviour. You could even have another llm do this.
- carlio 4y agoSMBC comics had one about this recently - https://www.smbc-comics.com/comic/artificial-incompetence https://www.smbc-comics.com/comic/artificial-incompetence - poison the internet training data to prevent AI becoming too advanced.
- litiholofan 4y agoThis kind of poisoning is already done. Go search "happy white woman" on google images and see if you notice anything strange. Then try other races.
- pavlov 4y agoDownvoted because of the conspiratorial tone. Why don’t you explain the issue?
- ronsor 4y agoWithout actually searching, I suspect OP is talking about the race-mixing meme.
- Bjartr 4y agoDo you have reason to believe that is intentional (i.e. poisoned) versus it being an artifact of stock image site tagging systems and social pressure to avoid treating those subjects differently (i.e. biased)?
- Xylakant 4y agoBias certainly, but I wouldn’t even assume social pressure necessarily. The bias in the search can easily be a result of an existing real-world bias. White has long been the default assumption for many of these tagging systems, so I’d not be surprised if many pictures of white women are just tagged as “happy woman”. Just as automatic soap dispensers that don’t work for dark skinned people.
- lavela 4y agoTry "happy woman". There is your answer.
- zhte415 4y agoA Korean TV show?
- 4y ago
- fleddr 4y ago"In light of both attacks, we notify the maintainers of each affected dataset and recommended several low-overhead defenses." Such a disappointing ending.
- tippytippytango 4y agoIt won’t be long until a non trivial part of the economy revolves around people filtering or producing data for the models with known provenance. The idea of being able to train on the raw internet will be a quaint idea.
- TrapLord_Rhodo 4y agoI'll take this one further, that poisoning is the only way to protect privacy. With all the means of data collection, and labeling at scale, your privacy is going to be (Already is, but think about the data processing capabilities iin 10 years) complete public knowledge. As hacks of PII become more consistent, pushing a metric fuck ton of fake data into these services will be the only way to ensure no one has your data.