3 ms·
A huge problem with ML training is the ‘ouroboros issue’ - training ML with input from other MLs breaks things in very deep (but difficult to stop/detect when i
by lazide 1mo ago
A huge problem with ML training is the ‘ouroboros issue’ - training ML with input from other MLs breaks things in very deep (but difficult to stop/detect when it’s happening) ways due to the way the internal math works (model collapse).
Right now, the Internet training set is becoming more and more contaminated with better and better generative AI images and video.
It makes the models more screwed up, and makes it very difficult for humans to figure out what is original and not too.
If there was some signal that could at least make it easier to identify ‘original’/real images…
- ainch 1mo ago"Model collapse" is often overstated, as this paper demonstrates: https://arxiv.org/pdf/2404.01413 https://arxiv.org/pdf/2404.01413 The original model collapse paper assumes you train networks on 100% synthetic data produced by the previous generation. But if you maintain some portion of real data then the problem is mitigated.
- lazide 1mo agoNot when you also including poisoning attacks. I remember the original paper showing issues with even a couple percent of certain kinds of synthetic data too, not 100%.
- kawogi 1mo agoI think the term "AI incest" would sum this up :)
- chrisjj 1mo agoOK, but any reliance on a fakable signal such as this seems guarateed to invite poisoning.