3 ms·

This is unlikely. The way model distribution works is that the model retains a lossy representation of James Micken's writing. Very likely, it cannot repeat Mic

This is unlikely. The way model distribution works is that the model retains a lossy representation of James Micken's writing. Very likely, it cannot repeat Micken's writing verbatim. Neither can it reason about the training cutoff in this manner. It's a lossy representation

koiueo 5mo ago
How do you know, how the model works? If there was an index of all Micken's writings, or even if the model searched the web before feeding the response to you, you wouldn't know by observing from the outside.
- suriya-ganesh 5mo ago
  i suppose a quick test would be getting the model to write down Micken's essay end to end. if the original essay was stuffed within the prompt window. the result will be word accurate. unless this is a model trained specifically on Micken's essay (which claude is not).
  - saghm 5mo ago
    This seems like a classic case of doing it being proof that it can happen, but not doing it being insufficient proof that it's impossible. I don't think there's a "quick test" of whether there might be a more effective prompt that would cause it to reproduce more effectively.
  - repparw 5mo ago
    Didn't we get this with Harry Potter back in like gpt3.5? I'm sure I saw some news about it, someone getting it to output a book's intro word by word, couple pages?
sausagefeet 5mo ago
I haven't been following it well but isn't part of the NYT lawsuit against OpenAI that it sometimes spits out NYT articles verbatim?
- Iulioh 5mo ago
  Study: Meta AI model can reproduce almost half of Harry Potter book https://arstechnica.com/features/2025/06/study-metas-llama-3-1-can-recall-42-percent-of-the-first-harry-potter-book/ https://arstechnica.com/features/2025/06/study-metas-llama-3...
  - kbelder 5mo ago
    "The study authors took 36 books and divided each of them into overlapping 100-token passages. Using the first 50 tokens as a prompt, they calculated the probability that the next 50 tokens would be identical to the original passage. They counted a passage as “memorized” if the model had a greater than 50 percent chance of reproducing it word for word." So they fed "It takes a great deal of bravery to stand up to our " and the llm responded "enemies, but just as much to stand up to our friends". They repeated that for every 100 tokens of the entire book. I think lots of fans could do just as well. It's pretty good evidence that the potter books were in the training corpus, but it's not quite what people think when they say an llm has 'memorized' something. It's not like getting even a few pages out of the model.
- spacechild1 5mo ago
  See also GEMA vs. OpenAI.
- mapt 5mo ago
  Genome analysis is also a lossy process that chops the data up into tiny bits, like a newspaper sent through a shredder. We then piece together matching sequences in a sort of puzzle. It's often a relatively inaccurate solution. Then we try to do that again with a different copy of the newspaper sent through a different shredder. And again. A genome might be comprised of 10x reads, 30x reads, 100x reads, with more replications representing higher confidence. There might be ten million people who have quoted Harry Potter at some point in their blogs or forum posts. There are only so many words in the books.
- algoth1 5mo ago
  That issue is different, when web tools were added to gpt4o it would fetch the site, and basically copy paste the text into the answer body. So, you were able to read the content of the site without the site getting the ad impressions. Now the system prompts put a very tight word limit - 25? - on quotes from sites the model visits
devmor 5mo ago
Haven’t there been repeated experiments that show if you jailbreak most frontier models’ harnesses you can get them to output near verbatim copyrighted works? I swear there was a whole court case about this in the last year.
electroglyph 5mo ago
that's in the ideal scenario where it's only seen a single copy of it tho
sigmoid10 5mo ago
It is lossy, but it is still enough for verbatim recreations. All of Wikipedia is just 24GB of lossless compressed text and all of JK Rowling's work fits into a few MB. So these things would easily be storable verbatim in trillion parameter models. Reasoning about the training cutoff is also something that the newest models do pretty well, because you can teach them to do so after pre training using e.g. SFT. With tool use it can then even check actual current sources, which may happen without you even knowing in the normal chat apps unless you use a controlled API call.
saghm 5mo ago
I feel like you're making a logical leap here by assuming lossy and failure to reproduce in entirety implies inability to recognize. As a trivial example, I can take a sha256 hash of your comment here, lose the ability to reproduce it, but still have an extremely accurate ability to recognize whether some text is exactly your comment or not. Obviously hashing every substring would not be a particularly efficient strategy, but my point is that saying "it's lossy" isn't particularly compelling without other details.
- gspetr 5mo ago
  The example you've provided just adds noise. sha256 is deterministic, LLMs are not, even at temperature set to 0.

koiueo 5mo ago

How do you know, how the model works? If there was an index of all Micken's writings, or even if the model searched the web before feeding the response to you, you wouldn't know by observing from the outside.

suriya-ganesh 5mo ago
i suppose a quick test would be getting the model to write down Micken's essay end to end. if the original essay was stuffed within the prompt window. the result will be word accurate. unless this is a model trained specifically on Micken's essay (which claude is not).
- saghm 5mo ago
  This seems like a classic case of doing it being proof that it can happen, but not doing it being insufficient proof that it's impossible. I don't think there's a "quick test" of whether there might be a more effective prompt that would cause it to reproduce more effectively.
- repparw 5mo ago
  Didn't we get this with Harry Potter back in like gpt3.5? I'm sure I saw some news about it, someone getting it to output a book's intro word by word, couple pages?

sausagefeet 5mo ago

I haven't been following it well but isn't part of the NYT lawsuit against OpenAI that it sometimes spits out NYT articles verbatim?

Iulioh 5mo ago
Study: Meta AI model can reproduce almost half of Harry Potter book https://arstechnica.com/features/2025/06/study-metas-llama-3-1-can-recall-42-percent-of-the-first-harry-potter-book/ https://arstechnica.com/features/2025/06/study-metas-llama-3...
- kbelder 5mo ago
  "The study authors took 36 books and divided each of them into overlapping 100-token passages. Using the first 50 tokens as a prompt, they calculated the probability that the next 50 tokens would be identical to the original passage. They counted a passage as “memorized” if the model had a greater than 50 percent chance of reproducing it word for word." So they fed "It takes a great deal of bravery to stand up to our " and the llm responded "enemies, but just as much to stand up to our friends". They repeated that for every 100 tokens of the entire book. I think lots of fans could do just as well. It's pretty good evidence that the potter books were in the training corpus, but it's not quite what people think when they say an llm has 'memorized' something. It's not like getting even a few pages out of the model.
spacechild1 5mo ago
See also GEMA vs. OpenAI.
mapt 5mo ago
Genome analysis is also a lossy process that chops the data up into tiny bits, like a newspaper sent through a shredder. We then piece together matching sequences in a sort of puzzle. It's often a relatively inaccurate solution. Then we try to do that again with a different copy of the newspaper sent through a different shredder. And again. A genome might be comprised of 10x reads, 30x reads, 100x reads, with more replications representing higher confidence. There might be ten million people who have quoted Harry Potter at some point in their blogs or forum posts. There are only so many words in the books.
algoth1 5mo ago
That issue is different, when web tools were added to gpt4o it would fetch the site, and basically copy paste the text into the answer body. So, you were able to read the content of the site without the site getting the ad impressions. Now the system prompts put a very tight word limit - 25? - on quotes from sites the model visits

devmor 5mo ago

Haven’t there been repeated experiments that show if you jailbreak most frontier models’ harnesses you can get them to output near verbatim copyrighted works? I swear there was a whole court case about this in the last year.

electroglyph 5mo ago

that's in the ideal scenario where it's only seen a single copy of it tho

sigmoid10 5mo ago

It is lossy, but it is still enough for verbatim recreations. All of Wikipedia is just 24GB of lossless compressed text and all of JK Rowling's work fits into a few MB. So these things would easily be storable verbatim in trillion parameter models. Reasoning about the training cutoff is also something that the newest models do pretty well, because you can teach them to do so after pre training using e.g. SFT. With tool use it can then even check actual current sources, which may happen without you even knowing in the normal chat apps unless you use a controlled API call.

saghm 5mo ago

I feel like you're making a logical leap here by assuming lossy and failure to reproduce in entirety implies inability to recognize. As a trivial example, I can take a sha256 hash of your comment here, lose the ability to reproduce it, but still have an extremely accurate ability to recognize whether some text is exactly your comment or not. Obviously hashing every substring would not be a particularly efficient strategy, but my point is that saying "it's lossy" isn't particularly compelling without other details.

gspetr 5mo ago
The example you've provided just adds noise. sha256 is deterministic, LLMs are not, even at temperature set to 0.