9 ms·
Will AI systems perform poorly due to AI-generated material in training data?
- behnamoh 1y agoI've heard that OpenAI and many AI labs put watermarks [0] in their LLM outputs to detect AI-generated content and filter it out. [0] Like statistics of words, etc.
- jsheard 1y agoMaybe they do use watermarks, and the vendors which only offer hosted models can just log everything they've ever generated, but there's enough players all working on this stuff independently of each other that filtering out their own noise would only get them so far. I noticed that a big chunk of the default Llama 4 system prompt is devoted to suppressing various GPT-isms, which to me implies they weren't able to keep their newer training set from being contaminated by competing models. > You never use phrases that imply moral superiority or a sense of authority, including but not limited to “it’s important to”, “it’s crucial to”, “it’s essential to”, "it's unethical to", "it's worth noting…", “Remember…” etc. Avoid using these.
- dustingetz 1y agodo they also watermark the code?
- jimbob45 1y agoWouldn’t be hard to do. Just alternate tabs and spaces and no one would ever know or care to check.
- sampullman 1y agoHopefully that's converted to one or the other when saved in an editor, or caught in CI.
- djeastm 1y agoMost coders would have code cleaning tools in their IDEs that would take care of that automatically.
- jimbob45 1y agoWhat about invisible Unicode characters?
- umbra07 1y agoToo obvious. Someone would have found that already.
- lolc 1y agoYea my IDE highlights uncommon chars automatically.
- subscribed 1y agoThey are very visible to machines. Code linters would scream (and the alternating spaces and tabs would likely break generated Python code).
- IAmGraydon 1y agoInteresting. That could certainly come in handy if it’s something they can’t avoid. We, too, might be able to better detect and filter their output.
- Rodeoclash 1y agoYeah, it's known as the em dash!
- jbaber 1y agoY'know, I've been writing double dashes and having them converted into em dashes about 50% of the time on whatever platform I'm using for decades. It's bizarre that this is suddenly supposed to be a shibboleth.
- AaronAPU 1y agoHave you ever considered you might be an LLM?
- bitwize 1y agoApparently the new ageist insult beyond "boomer" is "double-spacer" -- people who were taught in school to always follow the period at the end of a sentence with two spaces when composing the next sentence. If you went to elementary school after the internet became widespread, you are not likely to have been taught that. So double-spacing has now also become a shibboleth, albeit indicating the typist's age, distinguishing early millennials and Xers, who are now entering middle/old age, from the younger generations.
- agubelu 1y ago> Apparently the new ageist insult beyond "boomer" is "double-spacer Says who? I've seen "boomer"everywhere but it's the first time I've heard about that one.
- mikhmha 1y agoRight? I've never associated "double-spacer" with boomer. Maybe anally retentive? Someone who is trying too hard? The only thing I associate with boomers is ALL-CAPS writing. Which I assume is a holdover from typewriter days. But I kind of like ALL CAPS. It conveys some level of importance to the message.
- _heimdall 1y agoI could have sworn they all gave up on watermarking 12 or 18 months ago when they realized it wasn't possible to do reliably.
- energy123 1y agoThis was a proposal by Scott Aaronson but I wasn't aware it got implemented.
- ctoth 1y ago[flagged]
- deleted 1y ago[deleted]
- Rabbit_Brave 1y agoThese companies are sitting on a never-ending stream of human created data. What do you think happens to your conversations or other interactions with AI? Quality might be a bit sus though.
- AstroBen 1y agoI'd imagine it's really low quality data. Most or all of my conversations with an LLM are questions or telling it to do something, with varying levels of specificity I'm not sure what they'd get from training on that
- insin 1y agoI sometimes wonder if they're vulnerable to a coordinated effort of deliberately upvoting shit assistant turns and praising in the next user turn - how much does that actually contribute to future training, if at all? I had a very basic React question about useState while porting some vanilla code last week which all models of all stripes I've tried it on have been confidently and completely incorrect about, up to stating the code absolutely will not work, even when I take a turn to assert that I ran it and it does, so there's plenty of shit in there already.
- ted537 1y agoI don't think it would be too hard to scrape useful data out of my LLM convos. If human response is "That's BS", "fuck off", or something similar, mark as bad assistant message. If human response is "huh" or "cool", mark as good assistant message. If on ChatGPT, watch how much scrolling user does. If there's a lot, its somewhat likely that the LLM outputted something useful. That strategy would have holes of course but as long as its better than guessing something like that would be a useful heuristic.
- londons_explore 1y agoThis. Even very weak human signals can be immensely valuable over large enough datasets.
- 1y ago
- blooddragon 1y agoTime for GANs to make a resurgence?
- abc-1 1y ago[flagged]
- sidibe 1y agoYeah I think it's because people want it to be true that LLMs will stop improving and regress. If nothing else they can always just access data from the before times if it was an actual issue So much burying the head in the sand from people in this industry and wishful thinking that AI will stop before whatever they're good at. A little reminder a couple years ago most people hadn't even heard of LLMs.
- _se 1y agoMost of us wish we could stop hearing about LLMs. A little reminder that every year since 2022 has been the last year that anyone will ever do anything by hand according to you prophets. I'll give it 3-4 years before it's gone the way of crypto: still exists, still used (and still useful in the right cases), not something most people are talking about. It's going to be really nice.
- abletonlive 1y agoI don't really understand this take. You're unhappy because people are excited? May I ask, what do you work on and what do you get excited about that you'd rather advance? Do you think you're part of the group that is highly motivated and advancing ideas and people around them? Because I work with a lot of people in this group and this is the most exciting time I've experienced in my 30+ years. I am waking up and living and breathing this moment in time. I'm devouring more information than I ever have in my entire life, and it both feels like I can't get enough and that there's so much I'll never be able to keep up. So I really hope whatever you have in your pocket is just as interesting and excites people this much. Is the cynical depression coming from watching people build your future without you while you're not a part of the conversation? Correct me if I'm wrong. If "most of us" wish we'd hear less about LLMs how come it's always the hottest topic? That doesn't compute. Wouldn't have most of you taken over the conversation by now and changed the topic? Or perhaps whatever else you're trying to advance is just not that interesting to most of us?
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]
- jacobsenscott 1y agoToday we have humans being trained on llm garbage - kids using it to do their homework - programmers using it to "learn" how to code, med students cheating their way through med school, etc. So the content the humans are producing and will produce is really just LLM statistical word jumbles - ie human generated content will soon be as useless as LLM generated content.
- throwup238 1y agoIt’d be deeply ironic if the great filter for the human race turned out to be chatbots.
- nine_k 1y agoHello, secret sources of untainted but modern knowledge, written by human experts, and closely guarded by these experts.
- nradov 1y agoI'm not too worried about med students. You can't really use an LLM to cheat on the boards or make it through residency.
- outside415 1y ago[dead]
- nneonneo 1y agoYes, although some people do slip through the cracks anyway: https://en.wikipedia.org/wiki/Christopher_Duntsch https://en.wikipedia.org/wiki/Christopher_Duntsch. He wasn't an LLM user, but was a cocaine user...
- ijk 1y agoI mean, arguably the cocaine use makes him more like the kind of ideal doctor for enduring the long residency hours...
- deleted 1y ago
- mondrian 1y agoThe "core reasoning" part of AI may be increasingly important to improve, and its "database of factual knowledge" aspects may be less and less important, maybe increasingly a hindrance. So more focused and specialized training may take over toward increasing reasoning precision, and not this never-ending stream of new data. So maybe we'll get better reasoning and therefore better generated data/content in the wild, without this negative feedback loop everyone is worried about.
- gerdesj 1y agoYou seem to be arguing that bollocks couched in flowery language is a stretch goal of AI. Are you sure?
- _heimdall 1y agoThat really depends on expectations. If AI is meant to sound nearly identical to a human, you don't need more training data. If its meant to act as a natural language encyclopedia, we'll never get there with LLMs which amount to natural language processing on top of a massively compressed dataset.
- gerdesj 1y agoI've read the Google page "Getting AI to write good SQL: Text-to-SQL techniques explained" and it looks like all the usual bollocks. ... and sigh ... it look like I'm twiddling with a noddy!
- mondrian 1y agoTrying to make AIs more factually-accurate with more training is probably hopeless. Current events and encyclopedic knowledge will be provided by tools. The LLM's core job is to choose the right tools for the job and synthesize their outputs.
- meroes 1y agoWhat core reasoning?
- adamgordonbell 1y ago> Will future artificial intelligence systems perform increasingly poorly due to AI-generated material in their training data? No. Synthetic data is being used to improve LLMs
- pphysch 1y agoSynthetic data ought to be viewed as an extension of the training process rather than proper new phenomena. It can definitely help smooth things out and reinforce wanted behavior, but it's still derivative of the real data.
- _heimdall 1y agoDo we know the results yet? I know they're training with synthetic data, I didn't realize that has been done at scalr for long enough to really know if it improved (assuming the metrics its improving are defined well).
- jdietrich 1y agoDeepseek V3 and R1 are both substantially trained on synthetic data. The results speak for themselves.
- NitpickLawyer 1y ago> Do we know the results yet? LLama3 were post-trained on almost entirely synthetic data. Yes, it works. No, the model doesn't collapse (unless you want it to, of course). What they did is use Model n-1 to classify, filter and enhance the datasets for Model n.
- bobro 1y agoCan you point me to something I can read that spells out the > almost entirely synthetic data thing?
- NitpickLawyer 1y agoYes, there's a podcast with the post-training lead for L3 where he mentions this. Lemme try and find it. edit: found it. The money quote is here, but I really recommend the entire podcast since it's full of great tidbits and insights. > Thomas [00:33:44]: You mean between supervised fine-tuning like supervised fine-tuning annotation and preference annotation? Yeah. So 100% to RLHF. In fact, that's quite interesting. You start for Llama 2 with a pre-trained model and you have to have an instruction model to chat model. Otherwise, like the model is just like continue finishing sentences. So you need that to start RLHF. So we had to annotate like 10,000 examples. What did we do for Llama 3? You start with a new pre-trained model and then you want, before starting the RLHF, to have now a chat model, which is not too bad. The option one was, let's do human annotation again, like SFT stage. But in fact, by the principle I said before, the annotation would be actually worse than Llama 2. So what we did is that we generated all the data on the prompts with Llama 2 and we applied like basically the last round of Llama 2 we had to kick off and start Llama 3 post-training. So Llama 3 post-training doesn't have any like human written answers there basically, almost. It's just leveraging pure synthetic data from Llama 2. https://www.latent.space/p/llama-3 https://www.latent.space/p/llama-3
- lacker 1y agoUnfortunately, I don't really know if I can trust academics to analyze the development of large language models. No academic team has built an LLM. So... do people working at Stanford or Oxford really have good insight how LLMs are developed? If people at OpenAI, Anthropic, or Google said this, that would be interesting. But I don't think it makes sense any more to treat academic computer scientists as relevant experts here.
- _heimdall 1y agoMy understanding is that those building them don't really know how they work. Research into interoperability has fallen way behind as funding went towards features and scale. Any understanding of how they work is largely theoretical, that seems like a reasonable place for academics to lean in and join the conversation.
- pphysch 1y agoWhy would Big AI kill their golden goose like that?
- jsheard 1y agoIt doesn't really make sense to trust what OpenAI and friends say about this either, when admitting to any kind of scaling limits would go against the narrative propping up their multi-hundred-billion-dollar valuations. I guess we're just flying blind for now.
- declan_roberts 1y agoThe reality is that for the most part, any corpus created after 2022 is going to be seriously polluted.
- alganet 1y agoI'd say 2007 or so. There wasn't any known active AI back then, but statistics on popular ideas and internet content was already a thing, and speech pollution based on those assessments had already started to spread fast, manually outputted. Sure, a lot of good content came out since then. But the amount of garbage... it's immense and very difficult to sort out automatically. The major issue is that this garbage then _became_ the norm. Only people who lived back then can remember what it was. For new folk, it looks just like a generational shift. However, it is quite obvious that some aspects of this shift were... unnatural (in the sense of not being spontaneous cultural manifestations).
- lazystar 1y agoand im sure someone from the 90's would say the same about '97. https://en.m.wikipedia.org/wiki/Eternal_September https://en.m.wikipedia.org/wiki/Eternal_September
- alganet 1y agoI am not talking about an influx of newcomers. Pay attention. I mentioned explicitly that I see what happened as distinct from a natural generational shift. There are many phenomena around that era to support what I am saying. Like, for example, the first massive political campaign to leverage internet as its primary vehicle.
- creshal 1y agoNot sure why you're getting downvoted, content farms have been a thing for a long time, and many a spam website used crappy markov chains to generate even more "content". Anything that could be marketed by company had its search results drowned in hand-crafted bland marketing slop, and even before ChatGPT got popular searching for things like recipes (or, god forbid, generic windows error messages) was a nightmare. And a lot of that garbage is in LLMs' training data.
- stainablesteel 1y agoi can't believe this article wasn't written 2 years ago, this is just the basics man
- leoapagano 1y agoI can't lie, I miss when the only GPT I had to worry about was the GUID Partition Table.
- deleted 1y ago[deleted]
- layer8 1y agoSomeone should encode a chat program in it.
- userbinator 1y agoAt least the MBR acronym still remains. (Most of my disks are still MBR as they're not big enough to be worth the hassle of using GPT.)
- yunnpp 1y agoWho needs more than 2TB on a single drive anyway.
- kevinventullo 1y ago[flagged]
- deleted 1y ago[deleted]
- anonygler 1y agoThis reminds me of the Monsanto case, where they sued a farmer (and won) for using patented seeds that the farmer obtained from a local grain elevator which happened to contain some of Monsanto's seeds. Should it eventually happen for LLM outputs, I hope we name it Slop Wars.
- deadbabe 1y agoA good way to harvest new training material is to eavesdrop real human conversations from non polluted sources (such as microphones listening to people talk in public places or texts), transcribe them, and feed them to LLMs.
- siwatanejo 1y agoBut our normal convos are plagued of mistakes, bad grammar, etc
- skeledrew 1y agoIt doesn't take much to clean up say 95% of mistakes I reckon, as it tends to be pretty repetitive, and unless there's a bunch of wordplay happening, intention can be discerned.
- carlosjobim 1y agoShadow libraries
- econ 1y agoMy intelligence is trained by paying close attention to who is doing the talking. Some people know a lot about one topic which means they didn't spend all of that time learning other things. Many don't know this about themselves. Wikipedia had some comical instances where high quality contributors accident ventured into other areas where they spontaneously transformed into ignorant trolls.
- evan_ 1y ago> On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.
- MacsHeadroom 1y agoAre you meaning to imply AI generated material is like "wrong figures?"
- Zambyte 1y agoIt can be. The nice thing about AI is that it can create much faster than humans. The problem with AI is that it can create wrong information much faster than humans. This can pollute the sources of information for future AI. Also consider: "previously correct" is the same as wrong.
- popcorncowboy 1y agoImpeccable quote. I suppose the interesting thought experiment here is, what if Babbage is wrong? I don't know the answer here, but (and go with the thought experiment) what if model collapse wasn't an inevitable outcome of feeding the snake its own tail.
- tim333 1y agoI think as AI gets smarter it will be the case that it can filter duff data, at least to some extent.
- bakugo 1y agoConsidering most recent models' general knowledge cutoffs are still in the late 2023/early 2024 range, I'm guessing the answer is "yes, and AI companies are very much aware of it".
- js8 1y agoHas the quality of art gone down since art was invented? Or has the quality of the written text gone down since writing was invented? I think the answer is clear no. Humans have been trained on "human-generated data" (cultural artifacts) for centuries, and quality is not down. AI is only an accelerator of this process, but there is nothing inherent in creating "artifacts" that would pollute the original training data. If anything, we should be worried about destroying nature, because that's the original inspiration for human-produced artifacts.
- grey-area 1y agoGenerated AI content contains mistakes and hallucinations. Over time those mistakes will compound because GAI doesn’t consider truth or have a way of judging truth. So yes, you can’t compare humans generating and picking influential content to AIs doing so. GAI is a dead end IMO anyway we’ve seen much more success with machine learning, GAI is good for fooling humans into thinking they see glimmers of intelligence.
- frozenseven 1y ago[flagged]
- keybored 1y ago[flagged]
- frozenseven 1y ago[flagged]
- keybored 1y agoThe Luddites were like anyone else, people pursuing their own percieved self-interest. They opposed a specific technology (not all technology, don’t be daft) because it ran against their own interests as relatively skilled craftsmen. Real progress means serving people equally. When that happens there is no fertile ground for Luddism.
- stevenhuang 1y agoI'd venture no. In fact I wouldn't be surprised if this tainted information somehow enriches a dataset by providing an extra dimensionality for training specialized heuristics. Maybe this would turn out to be how LLM hallucination can be solved, through being able to accurately identify AI generated material, and as result, becoming more robust against both the identification and generation of nonsense. Humans learn to discern what/who to best pay attention to via all manners of heuristics. I don't see in principle why LLMs (or something like it in the future) won't eventually be able to do the same.
- hliyan 1y ago> ...tainted information somehow enriches a dataset... dimensionality... heuristics... this sounds like a nonsensical word salad.
- stevenhuang 1y agoAI generated material is what future training runs will have to deal with. Heuristics is pattern matching. LLMs pattern match. LLMs may identify the patterns that indicate something is AI generated. What about this is confusing you?
- rdtsc 1y agoA scarier thought is that people will "talk" so much with these AIs that they'll start talking like ChatGPT. So we may still end up with some AI enshittification fixed point in the future but, one of the feedback paths will be human brains become enshittified. Imagine you time travel 20 years in the future and find out everyone around you talks the same and they all like ChatGPT.
- whatever1 1y agoOn the other hand imagine a society where everyone is so polite and flattering each other.
- lazide 1y agoBless your heart. (/s, little)
- Freak_NL 1y agoIf someone earnestly starts using those pointless platitudes LLM generated slop is filled with (“You're absolutely right. Here's where I was wrong …”) I suspect they will quickly find that violence was never far off.
- falcor84 1y agoWhy?! Are you saying that you can't see yourself trusting someone who "earnestly" admitted to changing their mind?
- Freak_NL 1y agoNot if they use those generic phrases without any preamble. It's exhausting to have a conversation with someone who constantly answers in hollow pleasing unidiomatic language. Changing someone's mind isn't instant; it's a process (which could start with “Huh. You might be right there. I didn't think of that.” or “Oh right, I forgot about that.” or something similar), not an instant admission of error. It's unhuman. It gives the other party the sense that they are just saying that to please you, not because they actually changed their mind.
- r33b33 1y agoYes. See how easy that is? Saved you 15 minutes.
- Lerc 1y agoA clear and simple answer that H L Mencken would recognise.
- bigiain 1y agoI wonder if anyone's made a version of Disintegration Loops as an LLM artwork? Recursively retrained their own LLM on it's own output until it descends into gibberish in amusing or artistic ways? https://en.wikipedia.org/wiki/The_Disintegration_Loops https://en.wikipedia.org/wiki/The_Disintegration_Loops
- Lerc 1y agoAs soon as you start selecting the outputs you prefer it ceases to be an uncontrolled decay. With a selection criteria it's called evolution.
- skeledrew 1y ago> model collapse happens when the training data no longer matches real-world data This isn't a significant issue IMO, as human-created content isn't "real-world" per se; it's human-created world, an interpretation and representation of the real. The real world is the raw data perceived by sensors, human or machine. And while model-generated content doesn't match human-created content well, in the vast majority of cases it's still humans curating, modifying and publishing generated content, based on how useful it is (there are of course spammers, etc but that's a general issue). This is something humans do with content created by other humans too. So over time generated content will become a sort of norm adopted by and inevitably molding humans, same as created content does. Instead of model collapse, both sources of content will converge over time, particularly as the ability to also generate content directly from the real world is developed and integrated into multi-modal models.
- keybored 1y agoIt is correct that this is a back and forth process and not simply a thing that either evolves or devolves or collapses. We are impacted by the tools we use. No matter how advanced the tools. But you can’t just dismiss the issue on the grounds that humans are removed from reality as well because they have a representation-of-thing instead of instead of thing as such. In fact it doesn’t make sense. We could be directing slave monkeys to write literature. Then we could water down that description of the process as humans curating, modifying and publishing content—just indirectly, but what’s one more level of indirection between primates. We could woolily describe it like that. We’re just creating content. Okay. But is it going anywhere? Or is it just gibberish? No, we won’t simply keep doing it if the monkeys give us gibberish.
- viraptor 1y ago> The real world is the raw data perceived by sensors, human or machine. It's much more than that. There's data our common sensors don't catch typically (virtually 100% of videos don't capture UV ranges) and there's data we're not able to catch at in any way yet.
- anushajahan 1y ago[dead]
- Balgair 1y agoI mean, if these AIs have read everything there is to read, then really what more do we want from them?
- tim333 1y agoMy prediction is that things will go the opposite way and AIs will become progressively more accurate as they get better at fact checking and reasoning. Already LLMs like chatgpt can be fairly unbiased on things like was the economy better under Trump or Biden whereas humans tend to be very biased on that depending on which information sources they have been fed. Humans definitely perform poorly as voters due to shill-generated material in training data.