5 ms·

My intuition is the public, users, nor the industry will take this problem seriously. To me this paper sounds a thunderclap.

dale_glass 2y ago
It's not a real problem, in my understanding. It's a "this kills cancer in a petri dish" sort of thing. Yes, it makes sense that if your algorithm is at all lossy, passing outputs through it again compounds the loss. The reality though that this doesn't happen on a large scale. JPEG isn't destroying all imagery because we're not stupid enough to constantly compound compression losses. Most AI output is generated for some sort of purpose. If we're making images then we're throwing out the bad ones, retouching blemishes, and also still making new works entirely by hand.
- m11a 2y ago
  Well, if we’re using the output of AIs, writing blog post articles with it and using AI to post comments on Reddit/X etc (as some do), and a few years later OpenAI et al refresh their datasets to train a new model, then you’re doing exactly that aren’t you? Using lossy model outputs to put into the function again, that is
  - dale_glass 2y ago
    It's harder with LLMs, but we still have metrics for a lot of text. On Reddit/HN/etc we have comment scores, replies, reposts, external references, etc that we can use to estimate whether a given comment/post was any good. An entity like Google that indexes the whole web has visibility into when a given piece of content showed up, if it changed, if it got referenced elsewhere later, etc. LLMs can be used to analyze the page and work out things like "the comments answering this comment are saying it's wrong" We can also of course pick and choose, Reddit has some completely garbage subreddits and very tightly moderated ones. It's of course by no means foolproof, but it doesn't have to be. It just has to be good enough to be usable for whatever purpose we need. Also, perfection isn't a thing and such issues happen even without LLMs. Like all the cases of something wrong being added to Wikipedia, getting repeated on some news site, and then Wikipedia using that as a reference to backup the incorrect claim.

dale_glass 2y ago

It's not a real problem, in my understanding. It's a "this kills cancer in a petri dish" sort of thing. Yes, it makes sense that if your algorithm is at all lossy, passing outputs through it again compounds the loss. The reality though that this doesn't happen on a large scale. JPEG isn't destroying all imagery because we're not stupid enough to constantly compound compression losses. Most AI output is generated for some sort of purpose. If we're making images then we're throwing out the bad ones, retouching blemishes, and also still making new works entirely by hand.

m11a 2y ago
Well, if we’re using the output of AIs, writing blog post articles with it and using AI to post comments on Reddit/X etc (as some do), and a few years later OpenAI et al refresh their datasets to train a new model, then you’re doing exactly that aren’t you? Using lossy model outputs to put into the function again, that is
- dale_glass 2y ago
  It's harder with LLMs, but we still have metrics for a lot of text. On Reddit/HN/etc we have comment scores, replies, reposts, external references, etc that we can use to estimate whether a given comment/post was any good. An entity like Google that indexes the whole web has visibility into when a given piece of content showed up, if it changed, if it got referenced elsewhere later, etc. LLMs can be used to analyze the page and work out things like "the comments answering this comment are saying it's wrong" We can also of course pick and choose, Reddit has some completely garbage subreddits and very tightly moderated ones. It's of course by no means foolproof, but it doesn't have to be. It just has to be good enough to be usable for whatever purpose we need. Also, perfection isn't a thing and such issues happen even without LLMs. Like all the cases of something wrong being added to Wikipedia, getting repeated on some news site, and then Wikipedia using that as a reference to backup the incorrect claim.