4 ms·

Why can't they rule it out? Is even OpenAI unable to track the provenance of all of their training data? This is one of the major problems with these enormous

by lambda 26d ago

Why can't they rule it out? Is even OpenAI unable to track the provenance of all of their training data? This is one of the major problems with these enormous closed models, and even most open-weights models, which don't disclose their training process or training data. You can never be sure what went into its training. Did it come up with an idea originally, or is it just plagiarising its training data? Are there malicious inputs being used to train in particular behaviors when given certain trigger phrases? What are the characteristics of the RLHF data and what kind of biases are those embedding in the models? With proprietary closed models, or even open weights models that don't have open training datasets, you just can't answer these questions.

causal 26d ago
Good chance their whole training pipeline is vibe coded so yah they probably don't actually know.
rfgplk 26d ago
> Why can't they rule it out? Is even OpenAI unable to track the provenance of all of their training data? Probably? I have a few hundred TB of training data for various small scale models and I can attest that I have _no idea_ what's in them. As in, literally zero. Half is scraped from GitHub and other hosting sites, other than that, I couldn't tell you anything else. At OpenAI's scale their entire pipeline is likely 100% automated.
- lambda 26d ago
  Yeah, I'm sure it's completely automated. But that doesn't preclude being able to index and track what the sources of data are. For your data sets, I would hope you are including source information for where the data came frome. And at OpenAI's scale, I would presume they are doing some amount of rolling hashing or similar to weed out duplication, training on too much duplicate data can cause problems. AllenAI have at least attempted to add some amount of traceability to their models with OLMoTrace (https://arxiv.org/abs/2504.07096 https://arxiv.org/abs/2504.07096), by letting you find n-gram matches from the outputs in their training data. It's not the most useful, there's a reason that LLMs use full fledged attention mechanisms and not just n-grams, a lot of times the n-gram matches it finds aren't all that related to the given output, it might be better to supplement this index with a vector search or other ways of keeping track of what training data would have most influenced particular parts of the output. But anyhow, this is something that is an important question, and the big labs should be working on to make their products more trustworthy. Instead, they are hiding information about how they train, hiding their reasoning traces, and just producing output with no information on what might have influenced the training.
  - ndriscoll 26d ago
    Attributing training data seems pointless for trustworthiness. The way you trust a model is the same way you trust a human; you ask it to: 1. Provide a chain of reasoning from agreed premises. These days LLMs can even do this airtight with proof assistants. 2. Cite data sources for non-agreed premises. I don't care where the model learned a fact. It might not have ever read a document directly from the primary source. I want it to link directly to either widely agreed facts (e.g. standard textbooks, and if necessary school syllabi demonstrating that the text is standard) or primary sources (e.g. datasets). Training provenance is irrelevant. It's neither necessary nor sufficient to deal with truth.
- pbhjpbhj 26d ago
  Aye, but do they train on user data in these circumstances or not? If they do, then almost certainly the model was influenced by the input of the allegedly plagiarised material.
- matthewdgreen 26d ago
  The question is not "does OpenAI know", it's "can OpenAI attest that the usage of their products for confidential data is not going to cause that sensitive data to become known to their models". And right now the answer I'm reading is that OpenAI can't attest to that.
tedsanders 26d ago
To truly prove some incidental usage data made no difference we'd have to (a) identify any of their de-identified data that came from their usage of ChatGPT, (b) train a bunch of expensive giant models, and (c) ask them all to solve the Navier-Stokes Millenium problem until hitting some level of statistical significance. It's just not feasible to run experiments like this to prove whether a piece of data has an effect on model behavior. As a parallel example, can we prove the phase of the moon had no impact on the NS solution? No, not without a bunch experiments run at different phases of the moon. There's no reason to believe that anything they did in ChatGPT led to our solution; it's just impossible for us to truly prove it. And knowing most of the recipes we use, there's really no reason to think such contamination happened. I've asked the team to make a clearer, less-lawyerly statement here - let's see what happens. (I work at OpenAI.)
- hexomancer 26d ago
  So you definitely did train on their data, you just think it is unlikely that it impacted the final model significantly?
  - dgellow 26d ago
    That’s also what I understand. If true yet another disgusting behavior from the company
  - tedsanders 26d ago
    I have no idea if their data was trained on. For example, if they used ChatGPT, asked a math question, and clicked the thumbs up button, that could have provided a small reward signal. I highly doubt this sort of feedback made a difference to a problem like Navier-Stokes, but it's not something that's feasible for us to prove one way or the other. Edit: Also, if they opted out of training, then we didn't train on it.
    - hexomancer 26d ago
      I think it should be incredibly easy to verify this. Just look at the training data and see if it contains any of the chats. It should be trivial for a company with tens of thousands of super-genius agents at their disposal.
EthanHeilman 26d ago
A careful reading of "we cannot rule out that de-identified data derived from their usage of our products helped improve our models" could be saying that yes they trained on it but they don't know if that training data resulted in an "improvement" to the model. That is, they can't rule out that the only reason the model found this solution was because it had been trained on this approach. The term ruled out is very open ended and gives them significant flexibility of meaning. They may have the information to determine exactly what happened, but they haven't looked so they can't "rule it out".
- deleted 26d ago
  [deleted]
Turn_Trout 26d ago
OAI could check whether those accounts enabled training data. If "yes", OAI could trace whether that data was used in any related training process. If either of those answers comes out to be "no", then that's sufficient to conclude training data independence. We wouldn't need a full ablated re-training and solution attempt, contra tedsanders in a sibling comment.
- jonas21 26d ago
  > could trace whether that data was used The point of de-identifying data is to ensure you can't trace who it came from. It would be a serious privacy violation if they could.
  - pbhjpbhj 26d ago
    If the model includes unique data from a person then that person can identify the data - the allegedly plagiarised material - and so re-identify it. There doesn't need to be a privacy breach to close that loop as it requires the person to identify the information is associated with them first.
keeda 26d ago
At the scale at which these models are now, regardless of whether they are proprietary or open weight or list their training datasets, there are hundreds of billions of works that have gone into trillions of parameters, each one providing tiny perturbations in some tiny fraction of the weights. It is probably impossible to attribute provenance to any specific input (which is also why the courts' finding of Fair Use is reasonable.) Which is why, as I said in a recent comment (https://news.ycombinator.com/item?id=49530864 https://news.ycombinator.com/item?id=49530864) inadvertently leaking ideas to models is a grave risk for Intellectual Property. > The risk with IP, however, is a lot more grave. You may not even need to memorize the details of the IP verbatim, just the broad idea may be enough. It may lurk encoded in the weights forever, just waiting to be activated by the right prompt to start a chain of thought that unlocks further details. Heck, it may even appear as if the model suggested the idea itself. However, from a quick skim of the timelines, the specific discoveries, and all the he-said-she-said, so far it seems unlikely that OpenAI's model cribbed from the NYU / Anthropic pair, even if it would be impossible to prove. Maybe what might help is a timeline of when the other two were using Codex for their work, whether they had opted out, and how long it takes for user data to make it to the training of their internal models. That last bit may be considered sensitive information however, as it could give away a lot about their internal processes.
- dfdydx 26d ago
  There are two different things: - was item X in the training data - did the inclusion of X in the training data lead to Y I understand why the second is hard, but why is the first one hard?
  - keeda 26d ago
    Yep, the last part in my post was suggesting some ways we could determine if "item X was in the training data" (as well as some potential blockers for that from OpenAI's perspective.)

by lambda 26d ago

causal 26d ago

Good chance their whole training pipeline is vibe coded so yah they probably don't actually know.

rfgplk 26d ago

> Why can't they rule it out? Is even OpenAI unable to track the provenance of all of their training data? Probably? I have a few hundred TB of training data for various small scale models and I can attest that I have _no idea_ what's in them. As in, literally zero. Half is scraped from GitHub and other hosting sites, other than that, I couldn't tell you anything else. At OpenAI's scale their entire pipeline is likely 100% automated.

lambda 26d ago
Yeah, I'm sure it's completely automated. But that doesn't preclude being able to index and track what the sources of data are. For your data sets, I would hope you are including source information for where the data came frome. And at OpenAI's scale, I would presume they are doing some amount of rolling hashing or similar to weed out duplication, training on too much duplicate data can cause problems. AllenAI have at least attempted to add some amount of traceability to their models with OLMoTrace (https://arxiv.org/abs/2504.07096 https://arxiv.org/abs/2504.07096), by letting you find n-gram matches from the outputs in their training data. It's not the most useful, there's a reason that LLMs use full fledged attention mechanisms and not just n-grams, a lot of times the n-gram matches it finds aren't all that related to the given output, it might be better to supplement this index with a vector search or other ways of keeping track of what training data would have most influenced particular parts of the output. But anyhow, this is something that is an important question, and the big labs should be working on to make their products more trustworthy. Instead, they are hiding information about how they train, hiding their reasoning traces, and just producing output with no information on what might have influenced the training.
- ndriscoll 26d ago
  Attributing training data seems pointless for trustworthiness. The way you trust a model is the same way you trust a human; you ask it to: 1. Provide a chain of reasoning from agreed premises. These days LLMs can even do this airtight with proof assistants. 2. Cite data sources for non-agreed premises. I don't care where the model learned a fact. It might not have ever read a document directly from the primary source. I want it to link directly to either widely agreed facts (e.g. standard textbooks, and if necessary school syllabi demonstrating that the text is standard) or primary sources (e.g. datasets). Training provenance is irrelevant. It's neither necessary nor sufficient to deal with truth.
pbhjpbhj 26d ago
Aye, but do they train on user data in these circumstances or not? If they do, then almost certainly the model was influenced by the input of the allegedly plagiarised material.
matthewdgreen 26d ago
The question is not "does OpenAI know", it's "can OpenAI attest that the usage of their products for confidential data is not going to cause that sensitive data to become known to their models". And right now the answer I'm reading is that OpenAI can't attest to that.

tedsanders 26d ago

To truly prove some incidental usage data made no difference we'd have to (a) identify any of their de-identified data that came from their usage of ChatGPT, (b) train a bunch of expensive giant models, and (c) ask them all to solve the Navier-Stokes Millenium problem until hitting some level of statistical significance. It's just not feasible to run experiments like this to prove whether a piece of data has an effect on model behavior. As a parallel example, can we prove the phase of the moon had no impact on the NS solution? No, not without a bunch experiments run at different phases of the moon. There's no reason to believe that anything they did in ChatGPT led to our solution; it's just impossible for us to truly prove it. And knowing most of the recipes we use, there's really no reason to think such contamination happened. I've asked the team to make a clearer, less-lawyerly statement here - let's see what happens. (I work at OpenAI.)

hexomancer 26d ago
So you definitely did train on their data, you just think it is unlikely that it impacted the final model significantly?
- dgellow 26d ago
  That’s also what I understand. If true yet another disgusting behavior from the company
- tedsanders 26d ago
  I have no idea if their data was trained on. For example, if they used ChatGPT, asked a math question, and clicked the thumbs up button, that could have provided a small reward signal. I highly doubt this sort of feedback made a difference to a problem like Navier-Stokes, but it's not something that's feasible for us to prove one way or the other. Edit: Also, if they opted out of training, then we didn't train on it.
  - hexomancer 26d ago
    I think it should be incredibly easy to verify this. Just look at the training data and see if it contains any of the chats. It should be trivial for a company with tens of thousands of super-genius agents at their disposal.

EthanHeilman 26d ago

A careful reading of "we cannot rule out that de-identified data derived from their usage of our products helped improve our models" could be saying that yes they trained on it but they don't know if that training data resulted in an "improvement" to the model. That is, they can't rule out that the only reason the model found this solution was because it had been trained on this approach. The term ruled out is very open ended and gives them significant flexibility of meaning. They may have the information to determine exactly what happened, but they haven't looked so they can't "rule it out".

deleted 26d ago
[deleted]

Turn_Trout 26d ago

OAI could check whether those accounts enabled training data. If "yes", OAI could trace whether that data was used in any related training process. If either of those answers comes out to be "no", then that's sufficient to conclude training data independence. We wouldn't need a full ablated re-training and solution attempt, contra tedsanders in a sibling comment.

jonas21 26d ago
> could trace whether that data was used The point of de-identifying data is to ensure you can't trace who it came from. It would be a serious privacy violation if they could.
- pbhjpbhj 26d ago
  If the model includes unique data from a person then that person can identify the data - the allegedly plagiarised material - and so re-identify it. There doesn't need to be a privacy breach to close that loop as it requires the person to identify the information is associated with them first.

keeda 26d ago

At the scale at which these models are now, regardless of whether they are proprietary or open weight or list their training datasets, there are hundreds of billions of works that have gone into trillions of parameters, each one providing tiny perturbations in some tiny fraction of the weights. It is probably impossible to attribute provenance to any specific input (which is also why the courts' finding of Fair Use is reasonable.) Which is why, as I said in a recent comment (https://news.ycombinator.com/item?id=49530864 https://news.ycombinator.com/item?id=49530864) inadvertently leaking ideas to models is a grave risk for Intellectual Property. > The risk with IP, however, is a lot more grave. You may not even need to memorize the details of the IP verbatim, just the broad idea may be enough. It may lurk encoded in the weights forever, just waiting to be activated by the right prompt to start a chain of thought that unlocks further details. Heck, it may even appear as if the model suggested the idea itself. However, from a quick skim of the timelines, the specific discoveries, and all the he-said-she-said, so far it seems unlikely that OpenAI's model cribbed from the NYU / Anthropic pair, even if it would be impossible to prove. Maybe what might help is a timeline of when the other two were using Codex for their work, whether they had opted out, and how long it takes for user data to make it to the training of their internal models. That last bit may be considered sensitive information however, as it could give away a lot about their internal processes.

dfdydx 26d ago
There are two different things: - was item X in the training data - did the inclusion of X in the training data lead to Y I understand why the second is hard, but why is the first one hard?
- keeda 26d ago
  Yep, the last part in my post was suggesting some ways we could determine if "item X was in the training data" (as well as some potential blockers for that from OpenAI's perspective.)