3 ms·

There already have been multiple documented cases of LLMs spitting out fairly large chunks of the input corpus. There have been some experiments to get it to re

There already have been multiple documented cases of LLMs spitting out fairly large chunks of the input corpus. There have been some experiments to get it to replicate the entirety of 'Moby Dick' with some success for one model but less success with others most likely due to output filtering to prevent the generation of such texts, but that doesn't mean they're not in there in some form. And how could they not be, it is just a lossy compression mechanism, the degree of loss is not really all that relevant to the discussion.

ndriscoll 8mo ago
Are you referring to this? https://osyuksel.github.io/blog/reconstructing-moby-dick-llm/ https://osyuksel.github.io/blog/reconstructing-moby-dick-llm... I see a test where one model managed to 85% reproduce a paragraph given 3 input paragraphs under 50% of the time. So it can't even produce 1 paragraph given 3 as input, and it can't even get close half the time. "Contains Moby Dick" would be something like you give it the first paragraph and it produces the rest of the book. What we have here instead is a statistical model that when given passages can do an okay job at predicting a sentence or two, but otherwise quickly diverges.
- xyzzy_plugh 8mo ago
  I'm no longer certain what point you're trying to make. Getting close less than half the time given three paragraphs as input still sounds like red-handed copyright infringement to me. If I sample a copyrighted song in my new track, clip it, slow it down, and decimate the bit rate, a court would not let me off the hook. It doesn't matter how much context you push into these things. If I feed them 50% of Moby Dick and they produce the next word, and I can repeatedly do that to produce the entire book (I'm pretty sure the number of attempts is wholly irrelevant: we're impossibly far from monkeys on typewriters) then we can prove the statistical model encodes the book. The further we are from that (and the more we can generate with less) then the stronger the case is. It's a pretty strong case!
  - ndriscoll 8mo ago
    That's... not how this works. > If I feed them 50% of Moby Dick and they produce the next word and I can repeatedly do that to produce the entire book... then we can prove the statistical model encodes the book. It can't because it doesn't. That's what it means to say it diverges. The "number of attempts" is you cheating. You're giving it the book when you let it try again word by word until it gets the correct answer, and then claiming it produced the book. That's exactly the residual that I said characterizes the extent to which it doesn't know the book. Trivially, no matter how bad the model is, if you give it the residual, it can losslessly compress anything at all. If you had a simple model that just predicts next word given current word (trained on word pair frequency across all English text, or even all text excluding Moby Dick), and then give it retries until it gets the current word right, it will also quickly produce the book. Because it was your retry policy that encoded the book, not the model. Without that policy, it will get it wrong within a few words, just like these models do.
    - jacquesm 8mo ago
      The point is that the AI's themselves and their backers are on the record as saying that the AI could reproduce copyrighted works in their entirety but that there are countermeasures in place to stop them from doing so. I wonder what the results would be if I spent time to train a model up from scratch without any such constraints. But I'm much too busy with other stuff right now, but that would be an interesting challenge.
      - ndriscoll 8mo ago
        Yeah just like a star could appear inside of Earth from quantum pair production at any given moment. But realistically, it can't. And you can't even show a test where any model can get more than a few tokens in a row correct. These companies just don't want to deal with people complaining that it reproduces something when they don't understand that they're literally giving it the answer.
        jacquesm 8mo ago

ndriscoll 8mo ago

Are you referring to this? https://osyuksel.github.io/blog/reconstructing-moby-dick-llm/ https://osyuksel.github.io/blog/reconstructing-moby-dick-llm... I see a test where one model managed to 85% reproduce a paragraph given 3 input paragraphs under 50% of the time. So it can't even produce 1 paragraph given 3 as input, and it can't even get close half the time. "Contains Moby Dick" would be something like you give it the first paragraph and it produces the rest of the book. What we have here instead is a statistical model that when given passages can do an okay job at predicting a sentence or two, but otherwise quickly diverges.

xyzzy_plugh 8mo ago
I'm no longer certain what point you're trying to make. Getting close less than half the time given three paragraphs as input still sounds like red-handed copyright infringement to me. If I sample a copyrighted song in my new track, clip it, slow it down, and decimate the bit rate, a court would not let me off the hook. It doesn't matter how much context you push into these things. If I feed them 50% of Moby Dick and they produce the next word, and I can repeatedly do that to produce the entire book (I'm pretty sure the number of attempts is wholly irrelevant: we're impossibly far from monkeys on typewriters) then we can prove the statistical model encodes the book. The further we are from that (and the more we can generate with less) then the stronger the case is. It's a pretty strong case!
- ndriscoll 8mo ago
  That's... not how this works. > If I feed them 50% of Moby Dick and they produce the next word and I can repeatedly do that to produce the entire book... then we can prove the statistical model encodes the book. It can't because it doesn't. That's what it means to say it diverges. The "number of attempts" is you cheating. You're giving it the book when you let it try again word by word until it gets the correct answer, and then claiming it produced the book. That's exactly the residual that I said characterizes the extent to which it doesn't know the book. Trivially, no matter how bad the model is, if you give it the residual, it can losslessly compress anything at all. If you had a simple model that just predicts next word given current word (trained on word pair frequency across all English text, or even all text excluding Moby Dick), and then give it retries until it gets the current word right, it will also quickly produce the book. Because it was your retry policy that encoded the book, not the model. Without that policy, it will get it wrong within a few words, just like these models do.
  - jacquesm 8mo ago
    The point is that the AI's themselves and their backers are on the record as saying that the AI could reproduce copyrighted works in their entirety but that there are countermeasures in place to stop them from doing so. I wonder what the results would be if I spent time to train a model up from scratch without any such constraints. But I'm much too busy with other stuff right now, but that would be an interesting challenge.
    - ndriscoll 8mo ago
      Yeah just like a star could appear inside of Earth from quantum pair production at any given moment. But realistically, it can't. And you can't even show a test where any model can get more than a few tokens in a row correct. These companies just don't want to deal with people complaining that it reproduces something when they don't understand that they're literally giving it the answer.
      - jacquesm 8mo ago