Presence may be necessary, but researcher-driven weighting of different sources of content can still introduce bias. For instance, [0] suggests (sources are unc

Presence may be necessary, but researcher-driven weighting of different sources of content can still introduce bias. For instance, [0] suggests (sources are unclear) that OpenAI boosted by 5x the weight of their WebText2 dataset, which consists of sites linked to by upvoted Reddit comments. Reddit, in this sense, with all the biases of its various communities, was artificially elevated in importance. (Per [1], there were well-thought-through reasons for this around previous failures due to overreliance on Common Crawl, but it's nonetheless a choice that was made by humans to go in this direction.) [0] https://gregoreite.com/drilling-down-details-on-the-ai-training-datasets/#AI_Training_Datasets_TEXT_contents https://gregoreite.com/drilling-down-details-on-the-ai-train... [1] https://insightcivic.s3.us-east-1.amazonaws.com/language-models.pdf https://insightcivic.s3.us-east-1.amazonaws.com/language-mod...