4 ms·
Today, LLMs learn from a codebase that has mostly insightful comments from well-meaning humans. In the future, the training sets will contain more and more aut
by golem14 4y ago
Today, LLMs learn from a codebase that has mostly insightful comments from well-meaning humans.
In the future, the training sets will contain more and more automatically generated stuff I believe will not be curated well, leading to a spiral of ever declining quality.
- golem14 4y agoSame reasoning suggests to make a copy of Wikipedia before it’s too late :)
- jonas-w 4y agohttps://dumps.wikimedia.org/ https://dumps.wikimedia.org/ Would be interesting to analyze it over the next years, maybe even anlyzing the past ones, with these AI detection tools.
- narwally 4y agoRemember that dude that wrote the vast majority of Scots wikipedia while not being able to speak Scots? Yeah, just waiting for the first LLM to make headlines by hallucinating it's own version of human history and publishing it to wikipedia.
- whiplash451 4y agoI am not overly concerned by a future where AI spoils its own training sets. Companies like openai will keep injecting human-generated feedback into the training set. A lot of the value is already in the RLHF today. See openai technical report.