3 ms·
This article is bullshit. It's on the first page only because it talks about a very widespread fear, namely that LLMs might actually become dumber because they
by _giorgio_ 3y ago
This article is bullshit.
It's on the first page only because it talks about a very widespread fear, namely that LLMs might actually become dumber because they are trained on their own output.
In reality, LLMs are already trained on the output of other LLMs, as specific and well-directed training is much more effective than a disorderly ingestion of text produced by illiterate internet users. Let's not forget the demographic makeup of the typical average user. I think that training on reddit and 4chan messages can be much more harmful than training on text produced by the worst chatbot.
Hacker News editors are trying to build a narrative to instill fear and pessimism.
"What's the fun in writing on the internet anymore?" explains that everyting can be stolen and rewritten by AIs. And today they talk about LLMs and the whole internet becoming dumber. Tomorrow they'll talk about news falsification and LLMs used to replace writers.
https://news.ycombinator.com/item?id=39415878 https://news.ycombinator.com/item?id=39415878
- lewhoo 3y ago>In reality, LLMs are already trained on the output of other LLMs That's a fact, but are there any studies on training a LLM on it's own output and not the output of a different LLM ? For instance, chatgpt gets knowledge updates so I understand it must be retrained somehow. What happens if the retraining data contains large patches of its own output ? Has this scenario been explored ?
- _giorgio_ 3y agoDatasets are a dark subject because they contain illegal material. Most gpts, including chatGPT, are trained on the chatGPT outputs. Plus, a lot of chatGPT training material is hand picked by humans.