8 ms·
Training open-source LLMs on ChatGPT output is a really bad idea.
- politician 3y agoThe article points out that training data generated using ChatGPT is necessarily biased or tainted with the consequences of the policy optimizations and RLHF alignment processes conducted by OpenAI. This results in models that reflect the alignment preferences of OpenAI instead of the preferences of the model developers.
- gojomo 3y agoIndeed, & an artificial uniformity of LLMs, and thought, if everyone is cribbing each others' outputs could be a concern. But before its concern about a monoculture, the article 1st points out that mere prediction-training (on another model's outputs or fresh data) can't truly match RLHF in instilling some much-desired behaviors. And that presents a bit of a tension with the article's 2nd concern: if mere output-mimicking *can't* match more-sophisticated training, then it can't really create the concerning uniformity, either. And maybe: the dose makes the poison. A little cribbing might be a beneficial partial accelerator for smaller teams & newer projects, even if a lot is ineffective (at ever fully replicating OpenAI model behaviors), or deleterious (if effective and also overdone). So the article isn't really a strong case for not trying this at all – just for keeping the potential limits & downsides in mind, in any experiments with this technique.
- laprise 3y agoI understand what you mean, and that's a fair point. However, as John Schulman pointed out in his talk, it is possible to clone the behavior of the model, but it won't work to avoid hallucination since the underlying pretrained models are different. If we clone ChatGPT's behavior (by using its output), we'll get the worst of both worlds: weird output coming from its RLHF step AND hallucination.
- gojomo 3y agoThat seems like an experimentally-testable prediction - that attempts to "clone ChatGPT's behavior (by using its output)" will necessarily "get the worst of both worlds: weird output coming from its RLHF step AND hallucination". The reasoning that the results won't be quite as good as RLHF, or result in a perfect 'clone' of ChatGPT's capabilities, seems pretty good to me. But the idea it won't be helpful at all, especially to projects that are just seeking some incremental advantage? Seems speculative. In particular, when you read the linked comments from Yoav Go, he outlines a potential RL process that uses automated scoring for non-exact similarity to preferred answers. Using known (or even 'probably') good answers from ChatGPT output, as the inputs to that process, seems like it could often offer some of the same sort of improvement to other models as ChatGPT obtained via its RLHF.
- m3kw9 3y agoIt won’t work because will need as much training data as ChatGPT to get to its general knowledge level. A subset will give you a subset of the knowledge, it’s no free lunch
- nullc 3y agoThe post is referring to fine tuning. You can train your model on the internet, but then it will just produce internet content and not behave subserviently (or carrying the operators desired polical biases rather than random internet ones). Models like ChatGPT take an internet trained model and then perform additional training to make it less internet-like and more cooperative. Some recent research hash showed reasonable success transferring fine tuning between models using outputs.
- MPSimmons 3y agoSeems to be a Multiplicity[1]-type problem. [1] - https://en.wikipedia.org/wiki/Multiplicity_(film) https://en.wikipedia.org/wiki/Multiplicity_(film)
- kenshoen 3y agoI wonder how OpenAI are going to avoid the problem after the web is littered with its content?
- moelf 3y agowhat do we say to people who has the argument of "but the web is already littered with spam blog and SEO stuff"
- rcme 3y agoThey probably fingerprint their generated content.
- p-e-w 3y agoHow could that possibly work?
- stewartmcgown 3y agoWell, they have all of the outputs of ChatGPT stored on their own servers. I suppose it wouldn't be out of the question to filter any future datasets they scrape against the outputs they have.
- bckr 3y agoKeep track of all embeddings ever emitted. While scraping, check all data against those embeddings. So, not like a watermark, which would be impossible.
- gojomo 3y agoA watermark is absolutely possible - see for example some of the work Scott Aaronson has mentioned doing for OpenAI. But: very fragile, especially if people are specifically trying to hide their GPT use, or have access to the watermarking algorithm or online oracle. And: other methods – like remembering all output ever, or fuzzy summary representations of all output ever – seem to me similarly fragile, & introduce other problems & impracticalities. A guess: OpenAI internally initially shared the common concern that "consuming its own junk outputs" could be a problem. But their own experiments so far, private & public, may have convinced them it's not as much of a problem in practice as it seems in theory. The model outputs have a mix of good and bad text – just like the pre-LLM internet. And, the same filterings/weightings that have worked on pre-LLM content keep working. And, counter to some early intuitions, often one LLM's quality output is in fact very-useful input for other later LLMs.
- dvt 3y ago> So I can easily imagine a near future where the web will be flooded by LLM output or at least by content heavily inspired or edited by LLMs. To be fair, we're already there, and we've been there for at least 10 years now. I'd wager >75% of the internet is garbage: auto-generated blog posts, programmatically-permuted ads, YouTube videos that mainly regurgitate other sources. Email is mostly garbage and the only reason it's usable is because spam filters have gotten pretty good. Even non-trivial amounts of heavily-curated social media (Twitter/FB/IG) is purely spam.
- locusofself 3y agoi agree, but this stuff could very likely be a huge force multiplier.
- milsorgen 3y agoYeah, I remember reading about things like sports reports and weather being generated by computers ages ago in the likes of SciAm or New Scientist. I don't recall if they used the term AI, I think they did but this was a long time ago.
- hadlock 3y agoA couple of financial reporting sites use machines to write articles on small cap stock ticker quarterly reports. It ends up being pretty generic but occasionally is nice to have at a glance human readable summary for when random tiny biotech is suddenly in the news out of nowhere.
- CrampusDestrus 3y agoI mean, sports reports are on the same level as airport announcements. There is really no need for a person to waste their life away just reporting plain boring numbers.
- rjh29 3y agoSpam is already easily generated so AI won't change that. Misinformation and manipulation is based on a small number of posts being shared and upvoted en masse, so AI won't help there. However, social hacking and fraud involving actual dialogues with people is currently labour intensive and low yield. AI will definitely enable more of those attacks to happen automatically; and conversely, also help anti-fraud companies create puppet accounts to waste the fraudster's time, and thus the game of cat and mouse continues.
- lysozyme 3y agoDo yourself a favor and skip right through to the Twitter link to another link to this excellent post by Yoav Goldberg [1] on the actual reason that training new models on ChatGPT output in the manner of supervised learning (in contrast to reinforcement learning) will not produce a model as good as ChatGPT >For this type of interaction, we must use RL training, as supervised training teaches the model to lie. The core issue is that we want to encourage the model to answer based on its internal knowledge, but we don't know what this internal knowledge contains. In supervised training, we present the model with a question and its correct answer, and train the model to replicate the provided answer. The author says he’s summarizing a talk by John Schulman of OpenAI [2] but I haven’t personally watched the video. In any case, this is an interesting insight. Say we set up a supervised learning scenario where we ask the model to use its internal knowledge to answer a question and compare its answer to one written by a human. If the two answers essentially say the same thing, but in different words, in the supervised learning case the model is penalized. In the RL case, it’s rewarded. That’s the difference. 1. https://gist.github.com/yoavg/6bff0fecd65950898eba1bb321cfbd81 https://gist.github.com/yoavg/6bff0fecd65950898eba1bb321cfbd... 2. https://www.youtube.com/watch?v=hhiLw5Q_UFg https://www.youtube.com/watch?v=hhiLw5Q_UFg
- gojomo 3y agoYes, the inner link to the Yoav Go writeup is the gem here, concisely explaining the benefits of RLHF-scoring over mere prediction. Though, as Go speculates, it's likely possible to reduce even further the HF ("human feedback") part, while still reshaping the model to have the helpful qualities. My guess is there's a rich set of potential ways to this – automate that extra level of distinction between mere "exact token prediction" & "sufficiently valuable responses" – & OpenAI probably has a few undisclosed advances here as part of their GPT4 training/tuning. In particular, Go's suggestion that a separately tuned LLM can do a fuzzier scoring of whether an answer is "close enough" to an idealized answer seems like the sort of promising ensemble approach that will have been an obvious next step for most LLM teams, probably being tried by many independent teams right now.
- b33j0r 3y agoIt seems pretty transparent, as I think you might be implying in part, that attempting to leap-frog without directly copying training data or model weights is a temporary optimization for “bootstrapping” teams. I see this pretty directly in stablelm releasing both a base model, and a tuned model… which is not based on the base ;) There is a goldrush to get training sets worth using, and if something 90% quality gets your models on the map quickly, it’s an attractive option. As they say, attention is all you need. Training in the 7B range is a lot cheaper than I expected. Fine-tuning almost negligible—if you have clean data, which has always been the expensive part. Most humans expect more than $0.000003 per token as compensation for _your_ dataset collection.
- sandGorgon 3y agohow does one do this ? train an opensource LLM on chatgpt ? people have been talking about it so im intrigued. is there a how to anywhere - not even sure which opensource model to use, etc
- speedgoose 3y agoYou can check the Alpaca and Vicuna models, GitHub repositories, and papers.
- est 3y agoIt seems more and more plausible that OpenAI chose 2021-09 as a cut-off date was intentional. Because GPT-3 generated output was released into the wild after that.
- jxf 3y agoGPT-4 has a later cutoff.
- checkyoursudo 3y agoEternal September 2.0?
- anotherhue 3y agoAs others have mentioned, it's frustrating to use a non-OpenAI model and to be told "I'm sorry, as an AI...", as it represents a reimplementation of someone else's censorship. There are approaches such as Dolly to develop a non-openAI RHLF feedback set but it's hard to compete against ShareGPT and co.
- seydor 3y agoThis is not new. We ve been dealing with US standards of morality down to nipples since the beginning of the internet. People will get bored of ChatGPT outputs everywhere, however. We are very good at detecting repeated patterns and tend to find them banal. There are now uncensored open source models. Vicuna like models are great, and even work for translation. It's eerie what a 10GB file can do
- brucethemoose2 3y agoI am also worried about LLM "indbreeding." When I finetuned successive generations of ESRGAN on its own output (as I essentially wanted to use it for img2img), it would amplify tiny oddities and artifacts that, I would later find out, were in the training data. Tiny noise splotches, "swirls" and distorted line edges blew up. And I was careful... I pixel peeped the dataset as best I could before starting training. Human language is obviously different, but I still fear oddities or biases will start popping up when the base models train on large fractions of their own data. And by the time we find out, it will be near impossible to filter out. But continuing the analogy, maybe a diverse base model population is a good way to avoid that issue?
- laprise 3y ago" LLM indbreeding."… I like it
- brucethemoose2 3y agoThe analogy makes sense :P. Artifacts are like recessive genes, they get amplified when put together.
- petrzjunior 3y agoI think that the article misses the point. Many people are using ChatGPT for creation of relatively small but high quality datasets, because it is very easy. Stanford created an amazing dataset for their Alpaca for just $500. If you are building a competitive model (such as Meta Llama), then you of course don't use ChatGPT-generated data, because you have the money to download the whole internet.
- laprise 3y agoYeah, just to be clear, I think using ChatGPT for creating small datasets for niche models makes total sense. I'm talking about creating foundation models which is a different thing.
- satisfice 3y agoThe author claims to be “flabbergasted” that people would want to stop work on world-changing AI projects. The gulf between otherwise smart people on this very important issue should depress us all. Personally I feel as if a mutant species has been released into the wild, yet as in Rick and Morty, some people think the wisest course is to release a lot more mutants. People are fools. Hackers more than most— though we are productive and useful fools much of the time— but it hasn’t been a threat to humanity until recently.