3 ms·
I can hardly wait to see when, in the near future, every new LLM may become almost useless. The OG models were trained on real world, human generated content (
by kikokikokiko 3y ago
I can hardly wait to see when, in the near future, every new LLM may become almost useless.
The OG models were trained on real world, human generated content (for the most part at least). Starting in 2022 the cost of automatic generating "human sounding enough" text has gone to such low depths that I expect it to be pretty much impossible to avoid training any model on text already generated by a LLM.
What will the result of this feedback loop be, I can't tell. It will probably be just an even more generic, corporate speak, bland sounding bla bla bla than we get today, and the level of hallucinations may get even worse.
In a way it makes me happy to imagine that the most dangerous tech humanity ever invented may itself be it's own main obstacle to future refinements.
- no_wizard 3y agoI think this would be cost prohibitive, though maybe not, however - it could just save every answer its given and scan text for it. If they're a match it could just not index it, right?
- data_maan 3y agothere's still the pre-2023 data to train it on ... and then augment with handpicked stuff.
- kikokikokiko 3y agoThe dataset of internet content pre-2022 will be regarded in the future just like low-background steel. Any content generated post the release of the first generally accessible LLMs will be considered radioactive.
- digging 3y agoDoesn't work if more than 1 bot exists...
- throwuwu 3y agoIt might be better for the corpus to be expanded with higher quality text that is produced on demand from contractors. There are also large pools of data yet to be accessed. One giant source would be podcast transcripts but there are many others.
- selcuka 3y ago> higher quality text that is produced on demand from contractors. Who, in turn, will use LLMs to generate that text.
- throwuwu 3y agoThat’s pretty easy to avoid if you have them work in an office on company provided machines on a network that blocks access to LLM sites. Not exactly rocket science here.