5 ms·

If a model is trained only on synthetic data, is it still possible it will output things like this? https://x.com/elder_plinius/status/1952958577867669892 https

If a model is trained only on synthetic data, is it still possible it will output things like this? https://x.com/elder_plinius/status/1952958577867669892 https://x.com/elder_plinius/status/1952958577867669892

LeoPanthera 1y ago
By definition, a model can't "know" things that are not somewhere in its training set, unless it can use a tool to query external knowledge. The problem is that the size of the training set required for a good model is so large, that's really hard to make a good model without including almost all known written text available.
- janalsncm 1y ago
  > all known written text available If phi5 is trained on synthetic data only then info on how to make drugs must be in the synthetic dataset.
- eru 1y ago
  > By definition, a model can't "know" things that are not somewhere in its training set, unless it can use a tool to query external knowledge. Well, it could also make inferences. Like, it could find a new mathematical proof, even if that's never in the training set.
  - xwolfi 1y ago
    But how, it's not like it's thinking, it's just spitting the next likely token
    - createaccount99 1y ago
      That is an oversimplification, and not the whole truth. Read anthropic's blog posts if you want to learn more, or ask gpt5.
    - frotaur 1y ago
      This can generate new text. If the abilities generalise somewhat (and there is lots of evidence they DO generalise on some level), then there is no obstacle to generating new proofs, although the farther away they are from the training data, the less likely it becomes. For an obvious example of generalisation: the models are able to write more code than there is in the dataset. If you ask it to write some specific, though easy, function, it is very unlikely it is present verbatim in the dataset, and yet the model can adapt.
JoshTriplett 1y ago
In theory, it's possible. https://x.com/OwainEvans_UK/status/1947689616016085210 https://x.com/OwainEvans_UK/status/1947689616016085210 It's not particularly likely that the hidden information encoded in synthetic data would happen to include specific details for making LSD or VX, but it's much more plausible that synthetic data contains some information the model's trainers would prefer to not incorporate in the model.

LeoPanthera 1y ago

By definition, a model can't "know" things that are not somewhere in its training set, unless it can use a tool to query external knowledge. The problem is that the size of the training set required for a good model is so large, that's really hard to make a good model without including almost all known written text available.

janalsncm 1y ago
> all known written text available If phi5 is trained on synthetic data only then info on how to make drugs must be in the synthetic dataset.
eru 1y ago
> By definition, a model can't "know" things that are not somewhere in its training set, unless it can use a tool to query external knowledge. Well, it could also make inferences. Like, it could find a new mathematical proof, even if that's never in the training set.
- xwolfi 1y ago
  But how, it's not like it's thinking, it's just spitting the next likely token
  - createaccount99 1y ago
    That is an oversimplification, and not the whole truth. Read anthropic's blog posts if you want to learn more, or ask gpt5.
  - frotaur 1y ago
    This can generate new text. If the abilities generalise somewhat (and there is lots of evidence they DO generalise on some level), then there is no obstacle to generating new proofs, although the farther away they are from the training data, the less likely it becomes. For an obvious example of generalisation: the models are able to write more code than there is in the dataset. If you ask it to write some specific, though easy, function, it is very unlikely it is present verbatim in the dataset, and yet the model can adapt.

JoshTriplett 1y ago

In theory, it's possible. https://x.com/OwainEvans_UK/status/1947689616016085210 https://x.com/OwainEvans_UK/status/1947689616016085210 It's not particularly likely that the hidden information encoded in synthetic data would happen to include specific details for making LSD or VX, but it's much more plausible that synthetic data contains some information the model's trainers would prefer to not incorporate in the model.