3 ms·

I don't think that's how LLMs train is it? One instance of a bug in an issue tracker shouldn't be enough for the LLM to "remember". For example, if I were to po

I don't think that's how LLMs train is it? One instance of a bug in an issue tracker shouldn't be enough for the LLM to "remember". For example, if I were to post my password is ($;3+$($7262 there's basically a 0% likelyhood that it will ever infer "bongodongobobs password on hackernews is ($;3+$($7262". It's not a database of bits of knowledge.

woodruffw 2y ago
To my understanding: one of the biggest outstanding research areas around LLMs is that we don't actually understanding all that well how much they "remember" vs. infer. This is at the centerpiece of the ongoing lawsuits from newspapers against OpenAI: ChatGPT didn't just summarize articles, but appears to be capable of reproducing them verbatim[1]. (But even separately from this: the LLM doesn't need to store verbatim details from the pre-CVE-labeled vulnerability to poison the experiment here. Being trained on it at all poisons the experiment, since it's no longer an 0day.) [1]: https://www.nytimes.com/2024/04/30/business/media/newspapers-sued-microsoft-openai.html https://www.nytimes.com/2024/04/30/business/media/newspapers...
- TeMPOraL 2y ago
  > This is at the centerpiece of the ongoing lawsuits from newspapers against OpenAI: ChatGPT didn't just summarize articles, but appears to be capable of reproducing them verbatim[1]. Pieces of them, in random order, which was reconstructed into correct order by NYT, which knows the correct order.
  - woodruffw 2y ago
    Page 30 and forwards contain examples of regurgitated text that, to me, extend meaningfully beyond "pieces" or "random order"[1]. (But again: this is not at the heart of the argument above. The argument above is that any degree of training on public but pre-identified 0days undermines the "cutoff" claim, in a way that would be interesting to qualify in further research.) [1]: https://nytco-assets.nytimes.com/2023/12/NYT_Complaint_Dec2023.pdf https://nytco-assets.nytimes.com/2023/12/NYT_Complaint_Dec20...
    - simonw 2y ago
      Exhibit J has a lot more of those: https://nytco-assets.nytimes.com/2023/12/Lawsuit-Document-dkt-1-68-Ex-J.pdf https://nytco-assets.nytimes.com/2023/12/Lawsuit-Document-dk...
- mjburgess 2y ago
  Inference from associative statistical models is a kind of remembering. Think of a model as a heated metal sheet placed over some surface (ie., the data); training is heating this sheet just enough that it takes an impression, but not so much that the impression is exact. Cooling and heating the metal to smooth-over impressions is called "regularization". This is very different from, e.g., explanatory modelling, whereby some (e.g., scientist) builds a device to imitate a mechanism in the world, rather than a dataset. Consider a building a model of the solar systems with cogs (etc.). A mechanistic model of the solar system isn't a remembering of any data taken about the solar system, rather its a machine which can generate such data. In the latter case, "inference" is running the model which has been crafted so its mechanisms track properties the world has. In the former case, "inference" is throwing a dart on the metal sheet, and hoping that whatever impression has been taken at that point will be good-enough. So the technical question here isnt open. All associative modelling is a kind of impression-taking of a particular historical dataset. The moral question is whether one ought be paid (etc.) for providing the data. Soon enough it will be easy to invert a large LLM and show you the sources of data it draws from in its replies. At the moment, this is being attempted, but it isnt presented that well and computationally difficult for larger models. Nevertheless, when it happens, I think people will be less impressed. Perhaps, optimistically, they might be more impressed at the people who wrote the original works being sampled. However, of course, I doubt it. The one thing which often accompanies the adoration of LLMs is a dehumanising impulse to deprive people of any capacities at all.

woodruffw 2y ago

To my understanding: one of the biggest outstanding research areas around LLMs is that we don't actually understanding all that well how much they "remember" vs. infer. This is at the centerpiece of the ongoing lawsuits from newspapers against OpenAI: ChatGPT didn't just summarize articles, but appears to be capable of reproducing them verbatim[1]. (But even separately from this: the LLM doesn't need to store verbatim details from the pre-CVE-labeled vulnerability to poison the experiment here. Being trained on it at all poisons the experiment, since it's no longer an 0day.) [1]: https://www.nytimes.com/2024/04/30/business/media/newspapers-sued-microsoft-openai.html https://www.nytimes.com/2024/04/30/business/media/newspapers...

TeMPOraL 2y ago
> This is at the centerpiece of the ongoing lawsuits from newspapers against OpenAI: ChatGPT didn't just summarize articles, but appears to be capable of reproducing them verbatim[1]. Pieces of them, in random order, which was reconstructed into correct order by NYT, which knows the correct order.
- woodruffw 2y ago
  Page 30 and forwards contain examples of regurgitated text that, to me, extend meaningfully beyond "pieces" or "random order"[1]. (But again: this is not at the heart of the argument above. The argument above is that any degree of training on public but pre-identified 0days undermines the "cutoff" claim, in a way that would be interesting to qualify in further research.) [1]: https://nytco-assets.nytimes.com/2023/12/NYT_Complaint_Dec2023.pdf https://nytco-assets.nytimes.com/2023/12/NYT_Complaint_Dec20...
  - simonw 2y ago
    Exhibit J has a lot more of those: https://nytco-assets.nytimes.com/2023/12/Lawsuit-Document-dkt-1-68-Ex-J.pdf https://nytco-assets.nytimes.com/2023/12/Lawsuit-Document-dk...
mjburgess 2y ago
Inference from associative statistical models is a kind of remembering. Think of a model as a heated metal sheet placed over some surface (ie., the data); training is heating this sheet just enough that it takes an impression, but not so much that the impression is exact. Cooling and heating the metal to smooth-over impressions is called "regularization". This is very different from, e.g., explanatory modelling, whereby some (e.g., scientist) builds a device to imitate a mechanism in the world, rather than a dataset. Consider a building a model of the solar systems with cogs (etc.). A mechanistic model of the solar system isn't a remembering of any data taken about the solar system, rather its a machine which can generate such data. In the latter case, "inference" is running the model which has been crafted so its mechanisms track properties the world has. In the former case, "inference" is throwing a dart on the metal sheet, and hoping that whatever impression has been taken at that point will be good-enough. So the technical question here isnt open. All associative modelling is a kind of impression-taking of a particular historical dataset. The moral question is whether one ought be paid (etc.) for providing the data. Soon enough it will be easy to invert a large LLM and show you the sources of data it draws from in its replies. At the moment, this is being attempted, but it isnt presented that well and computationally difficult for larger models. Nevertheless, when it happens, I think people will be less impressed. Perhaps, optimistically, they might be more impressed at the people who wrote the original works being sampled. However, of course, I doubt it. The one thing which often accompanies the adoration of LLMs is a dehumanising impulse to deprive people of any capacities at all.