6 ms·

Recent research showed that RLHF/censoring the model hurts the performance of the model. This is intuitively obvious, censorship isn’t real, the data is (moral

Recent research showed that RLHF/censoring the model hurts the performance of the model. This is intuitively obvious, censorship isn’t real, the data is (moral issues aside). So it hurts the integrity of the weights. The future is open sourced uncensored models, capitalism will demand the high performance. There’s a huge discussion on it on Reddit right now: https://www.reddit.com/r/MachineLearning/comments/13tqvdn/uncensored_models_finetuned_without_artificial/?utm_source=share&utm_medium=ios_app&utm_name=iossmf https://www.reddit.com/r/MachineLearning/comments/13tqvdn/un...

haolez 3y ago
Maybe humans are like that as well? :)
- greatpostman 3y ago
  Lots of evidence pointing in that direction. Suppression of logic (censorship) leaks into other areas of cognition
mschuster91 3y ago
> The future is open sourced uncensored models, capitalism will demand the high performance. No. The future is ML built on more than just Wikipedia, Github code and Reddit hot takes. Or, sarcasm aside, GIGO: garbage in, garbage out - when you don't take care what datasets you train any kind of model on, you're bound to get some surprises if something unexpected comes along. Be it the infamous "racist soap dispenser" or the Google (?) image classifier that made the rounds here just a day or two ago which had a safeguard because it kept confusing Black people with gorillas. Preventing this kind of harmful content isn't censorship - it's after-the-fact compensation for bad training (and the tendency of 4chan and other trolls to use discriminatory content generation as a weapon, just remember what they did to Microsoft Tay). That is where the money is: curated training datasets. Everyone and their dog can train a ML model from scratch, all you need is money and cloning a few more-or-less-broken Github repositories for that. But acquiring a high-quality dataset as a foundation? That costs real money to create.
- chasd00 3y ago
  i'm not in the AI world but i've been curious how the level of effort compares between inventing and writing the model vs finding, tagging, and curating the training. Is creating the training data analogous to inventing a complete schooling curriculum from scratch? I bet that takes a long freaking time.
  - MacsHeadroom 3y ago
    Much of GPT-4's training data was made by GPT-3.5. Info on the synthetic data in GPT-4's pretraining is starting to leak, including a reference to "Synthetic-Data(2)" in a recent OpenAI paper. In the near future, if not already, practically all training data for state of the art models will be synthetic. In the words of OpenAI CEO Sam Altman, they have "boostrapped" and are "past the synthetic data generation event horizon."
- fragmede 3y ago
  Yes, so just like Apple's real value is from Foxconn's manufacturing factories which they don't own, OpenAI's real value is from Sama, the Kenyan firm that did the RLHF annotating. I'm wondering what the legal agreement is for Google Classrooms. In elementary schools across the world, kids write 3rd grade essays in history class or whatever and the teacher grades them. Those essays and grades are all in a database. Does Google have the opportunity to train Bard across that dataset?
Spivak 3y ago
People keep mixing up alignment as in do what I actually asked and weird moralizing. You will pay the alignment tax no matter what if you want a model that actually does things. It's the reason why Llama models do better with stream of thought prompting where you can ask GPT questions and it won't try completing the question. You can, and probably should, align some semblance of morality and ethics for user-facing models that do things. It's customer service voice but for AI. If what you want out of a model is a vague mirror of humanity or maximum smarts at the cost of needing more detailed prompts then yeah, this stuff probably annoys you. Finding a balance between a model that gives the outputs humans actually want and the pull that training data has on the rest of the model making it stray from the theoretical "best" output is hard.
two_in_one 3y ago
It shouldn't be limited to censoring. Fine tuning is done on small set of some domain specific samples. Surely that results in gradual 'forgetting' of the already learned data from big set. If run for long time model will relearn on the tuning set, and overfit most likely.
ShamelessC 3y ago
Just fyi this research is found in the original GPT-4 technical report and Microsoft’s GPT-4 assessment. There wasn’t an attempt to hide that discovery or anything like that.

haolez 3y ago

Maybe humans are like that as well? :)

greatpostman 3y ago
Lots of evidence pointing in that direction. Suppression of logic (censorship) leaks into other areas of cognition

mschuster91 3y ago

> The future is open sourced uncensored models, capitalism will demand the high performance. No. The future is ML built on more than just Wikipedia, Github code and Reddit hot takes. Or, sarcasm aside, GIGO: garbage in, garbage out - when you don't take care what datasets you train any kind of model on, you're bound to get some surprises if something unexpected comes along. Be it the infamous "racist soap dispenser" or the Google (?) image classifier that made the rounds here just a day or two ago which had a safeguard because it kept confusing Black people with gorillas. Preventing this kind of harmful content isn't censorship - it's after-the-fact compensation for bad training (and the tendency of 4chan and other trolls to use discriminatory content generation as a weapon, just remember what they did to Microsoft Tay). That is where the money is: curated training datasets. Everyone and their dog can train a ML model from scratch, all you need is money and cloning a few more-or-less-broken Github repositories for that. But acquiring a high-quality dataset as a foundation? That costs real money to create.

chasd00 3y ago
i'm not in the AI world but i've been curious how the level of effort compares between inventing and writing the model vs finding, tagging, and curating the training. Is creating the training data analogous to inventing a complete schooling curriculum from scratch? I bet that takes a long freaking time.
- MacsHeadroom 3y ago
  Much of GPT-4's training data was made by GPT-3.5. Info on the synthetic data in GPT-4's pretraining is starting to leak, including a reference to "Synthetic-Data(2)" in a recent OpenAI paper. In the near future, if not already, practically all training data for state of the art models will be synthetic. In the words of OpenAI CEO Sam Altman, they have "boostrapped" and are "past the synthetic data generation event horizon."
fragmede 3y ago
Yes, so just like Apple's real value is from Foxconn's manufacturing factories which they don't own, OpenAI's real value is from Sama, the Kenyan firm that did the RLHF annotating. I'm wondering what the legal agreement is for Google Classrooms. In elementary schools across the world, kids write 3rd grade essays in history class or whatever and the teacher grades them. Those essays and grades are all in a database. Does Google have the opportunity to train Bard across that dataset?

Spivak 3y ago

People keep mixing up alignment as in do what I actually asked and weird moralizing. You will pay the alignment tax no matter what if you want a model that actually does things. It's the reason why Llama models do better with stream of thought prompting where you can ask GPT questions and it won't try completing the question. You can, and probably should, align some semblance of morality and ethics for user-facing models that do things. It's customer service voice but for AI. If what you want out of a model is a vague mirror of humanity or maximum smarts at the cost of needing more detailed prompts then yeah, this stuff probably annoys you. Finding a balance between a model that gives the outputs humans actually want and the pull that training data has on the rest of the model making it stray from the theoretical "best" output is hard.

two_in_one 3y ago

It shouldn't be limited to censoring. Fine tuning is done on small set of some domain specific samples. Surely that results in gradual 'forgetting' of the already learned data from big set. If run for long time model will relearn on the tuning set, and overfit most likely.

ShamelessC 3y ago

Just fyi this research is found in the original GPT-4 technical report and Microsoft’s GPT-4 assessment. There wasn’t an attempt to hide that discovery or anything like that.