4 ms·

I wonder how much safety work they have to do specifically because of this. I’d imagine their model might have a fairly paranoid, slightly racist bias if not. P

I wonder how much safety work they have to do specifically because of this. I’d imagine their model might have a fairly paranoid, slightly racist bias if not. Particularly as younger demographics shifted away from FB in the last decade.

kevin_thibedeau 2y ago
With suitable sentiment analysis, you can train known bigoted models and then run unknown input through them to see if it appeals to the model.
fire_lake 2y ago
Considering the standard datasets contain 4chan posts it’s probably a marginal improvement.
PaulHoule 2y ago
Actually if you want to train a model that can recognize bad things you want to have those bad things in the language model training data, otherwise it won’t see the characteristics of those things and it will later struggle to recognize them in later training stages.
- btown 2y ago
  Presence may be necessary, but researcher-driven weighting of different sources of content can still introduce bias. For instance, [0] suggests (sources are unclear) that OpenAI boosted by 5x the weight of their WebText2 dataset, which consists of sites linked to by upvoted Reddit comments. Reddit, in this sense, with all the biases of its various communities, was artificially elevated in importance. (Per [1], there were well-thought-through reasons for this around previous failures due to overreliance on Common Crawl, but it's nonetheless a choice that was made by humans to go in this direction.) [0] https://gregoreite.com/drilling-down-details-on-the-ai-training-datasets/#AI_Training_Datasets_TEXT_contents https://gregoreite.com/drilling-down-details-on-the-ai-train... [1] https://insightcivic.s3.us-east-1.amazonaws.com/language-models.pdf https://insightcivic.s3.us-east-1.amazonaws.com/language-mod...
dotancohen 2y ago
That would be supervised learning: they tell the model that these are undesirable::racist values. Or just fed into the RAG::blacklist dataset.
changoplatanero 2y ago
I haven't worked at facebook for a number of years but I'm imagining what they trained on here was public instagram images and not the random text that people write on facebook. The text in facebook posts is likely to be low value but the images are a data gold mine.

kevin_thibedeau 2y ago

With suitable sentiment analysis, you can train known bigoted models and then run unknown input through them to see if it appeals to the model.

fire_lake 2y ago

Considering the standard datasets contain 4chan posts it’s probably a marginal improvement.

PaulHoule 2y ago

Actually if you want to train a model that can recognize bad things you want to have those bad things in the language model training data, otherwise it won’t see the characteristics of those things and it will later struggle to recognize them in later training stages.

btown 2y ago
Presence may be necessary, but researcher-driven weighting of different sources of content can still introduce bias. For instance, [0] suggests (sources are unclear) that OpenAI boosted by 5x the weight of their WebText2 dataset, which consists of sites linked to by upvoted Reddit comments. Reddit, in this sense, with all the biases of its various communities, was artificially elevated in importance. (Per [1], there were well-thought-through reasons for this around previous failures due to overreliance on Common Crawl, but it's nonetheless a choice that was made by humans to go in this direction.) [0] https://gregoreite.com/drilling-down-details-on-the-ai-training-datasets/#AI_Training_Datasets_TEXT_contents https://gregoreite.com/drilling-down-details-on-the-ai-train... [1] https://insightcivic.s3.us-east-1.amazonaws.com/language-models.pdf https://insightcivic.s3.us-east-1.amazonaws.com/language-mod...

dotancohen 2y ago

That would be supervised learning: they tell the model that these are undesirable::racist values. Or just fed into the RAG::blacklist dataset.

changoplatanero 2y ago

I haven't worked at facebook for a number of years but I'm imagining what they trained on here was public instagram images and not the random text that people write on facebook. The text in facebook posts is likely to be low value but the images are a data gold mine.