6 ms·

Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works. https://openai.com/index/prover-verifier-games-improve-legibil

by Kuinox 2y ago

Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works. https://openai.com/index/prover-verifier-games-improve-legibility/ https://openai.com/index/prover-verifier-games-improve-legib... https://www.anthropic.com/research/claude-character https://www.anthropic.com/research/claude-character

echelon 2y ago
I'm long on synthetic data. If you think about evolution and hill climbing, of course it works. You have a pool of information and you accumulate new rearrangements of that information. Fitness selects for the best features within the new pool of data (For primates, opposable thumbs. For AI art, hands that aren't deformed.) It will naturally drift to better optima. RLHF, synthetic data, and enrichment are all we need.
- ainoobler 2y ago
  Are you sure about this? It's well known that cannibalism in animals leads to degenerative disorders.
  - autokad 2y ago
    I think the direct action of a person taking their idea and thoughts and going through it many times (making changes / updates / fixes) fits better than eating something. however, I do think you still some form of validation data to ensure these are good changes. However, I do get the spirit of the article, that as more information generated online is done by LLms, the validity and use of the output decreases
    - ainoobler 2y ago
      What exactly is doing the validation?
      - autokad 2y ago
        depends on what one was doing. could be as simple as re-writing a sentence and asking someone if it looks better
  - photonthug 2y ago
    Not sure why you’re downvoted, I think a comparison with prions seems apt and interesting, and bad protein copies that can replicate is essentially an information process. GAN research in recent years showing how you can sabotage a working dog/cat classifier with a one pixel change feels similar to how the tiniest parts of large systems can sometimes undermine the whole completely, albeit with low probability. And finally, since models will bootstrap models that bootstrap models, inevitably there are already subtle issues out there in the wild that may have an incubation period of many years before the downstream effects are completely clear.
    - ainoobler 2y ago
      The problem is systemic. People believe that the pursuit of monetary and financial profits by corporations will lead to the creation of benevolent artificial intelligence. I personally think this is essentially a religion because it is obvious that the pursuit of profits can not actually create anything benevolent, let alone intelligence.
- throwup238 2y ago
  > If you think about evolution and hill climbing, of course it works. You don't even need to go that far. How do most children learn? By reading textbooks and listening to lesson plans assembled by their teachers from all the relevant content the teachers have experienced. Our education systems are built on synthetic data that is created for optimized learning, so that every child doesn't have to prove the universe from scratch to learn some basic maths.
  - samatman 2y ago
    That isn't synthetic data in any reasonable or meaningful sense of the term. You could describe a textbook as a synthesis, sure, in a sense which absolutely does not track with the 'synthetic' in 'synthetic data'. Unless the textbook is AI-generated, and I expect that in 2024, the number of AI-generated textbooks is not zero.
    - throwup238 2y ago
      It’s an analogy. The learning materials teachers create for students is very much like synthetic data, it’s just not assembled from algorithmic output.
      - lanstin 2y ago
        kids learn, walking talking reading arithmetic and physics, by doing things in the physical world. Adults may speak differently to kids than adults, but it's a stretch to say it's synthetic. Equivalent to synthetic would be a group of kids that just grew up together and made up a novel language.
      - Demonoculus 2y ago
        granted synthetic is closely related to synthesis, but in common parlance synthetic would mean something that is not natural or abiotic in some sense, in case of synthetic data, it should imply data that doesn't occur naturally from human and natural sources. i.e. synthetic data would be exactly the one that is assembled from algorithmic output. granted im not able to explain it as well as i understand it.
- Salgat 2y ago
  Synthetic data has to work if we hope to have ML models that can improve themselves in a similar fashion as humans when it comes to advancing knowledge.
  - sfink 2y ago
    They mathematically cannot unless they have access to a way of measuring fitness. One that goes beyond an evaluation based on what they have already learned.
- randcraw 2y ago
  Data created automatically is not the same as human curated data, though both are synthetic. Auto-created data often suffers from a host of demerits (duplication, bias, error, unnatural distribution, irrelevance to learn the intended domain, etc, etc). Human curated data usually avoids these pitfalls, and thus is far more valuable when training -- otherwise all human teachers would be equally good. So auto- vs curated- data are incomparable when training naive neophytes like ML models, or children.
- HPsquared 2y ago
  It's important to climb the right hill.
  - psb217 2y ago
    And to have a very well-tuned sense up vs down when the hill is almost flat...
- kjkjadksj 2y ago
  This misunderstands fitness. Its not a sure bet what is most optimal is what you see. “Good enough” given environmental context is what you see. Just like with certain crystal structures in chemistry, you may only be in a localized threshold of fitness stability that is not necessarily optimal, but separated from another optimal configuration by having suboptimal intermediary steps that need more activation energy to overcome before falling into a state with lower entropy (or more optimal fitness). In other words you can never be sure if synthetic data is any good or if what things gravitate toward are really most optimal.
  - echelon 2y ago
    > Its not a sure bet what is most optimal is what you see. I wouldn't ever make "most optimal" a criteria. We're looking for measurable improvements, not a jump to god emperor or apex predator. > you may only be in a localized threshold of fitness stability that is not necessarily optimal, but separated from another optimal configuration by having suboptimal intermediary steps that need more activation energy to overcome before falling into a state with lower entropy (or more optimal fitness). Optimization is like that. But unlike genetics, where we can't re-route the recurrent laryngeal nerve or change fundamental biochemistry, these are engineered systems where we can set up wildly different experiments at any time. Just to cite one of many different research threads, there's now research now going into developing models from small scale training data. > you can never be sure if synthetic data is any good or if what things gravitate toward are really most optimal. We can know if the synthetic data is better. We have objective measures, a scientific process, and we'll always be striving for improvement.
- mcswell 2y ago
  The paper is not talking about verifiable synthetic data generated by some means other than LLMs.
- sfink 2y ago
  Only if you have a valid fitness metric. If you have humans looking at hands, then that's a good metric, as long as you really do have a human in the loop. Any automated metric (eg something that can evaluate hands) is great for measuring that specific dimension of fitness (after all, it was developed by a human, so it's really just an indirect way of feeding the human's evaluation into the machine). But it's useless for any other dimension. It'll happily rate the perfect hand coming out of a dogchickenpeach above the deformed hand petting the perfectly formed dog. It's the same as any other kind of signal processing. You can increase the noise, but you can't get more signal than you started with. Here, if the LLM decides that "monkey" is most often followed by "butt" and occasionally by "trainer", then it'll generate synthetic data with those frequencies and training on that data will not change its probability estimates at all. It will, however, drown out the signal that "you are a monkey butt" is more likely than "phlegm cigar monkey butt", if you'll forgive me the liberty of using those phrases to represent statistical correlations just beyond the frontier of what the LLM has learned. The synthetic data will teach it that everything it doesn't already know is equally probable, which will overwhelm human source data in which it isn't.
- apwell23 2y ago
  https://en.wikipedia.org/wiki/Infinite_monkey_theorem https://en.wikipedia.org/wiki/Infinite_monkey_theorem
nyrikki 2y ago
Cheese and Chalk. It is very different to generate synthetic datasets to assist in targeted training , vs ingesting LLM output from web scraping.
- abeppu 2y ago
  Yes -- said another way, if you're an ML researcher and you have human-provided (scraped) data, and an ability to generate synthetic data, then until recently, you had a controllable parameter: how much of your training data for your new model should be synthetic? You can vary this, run multiple experiments, and choose how much synthetic data to use -- and you can vary the specific configs about how that synthetic data is generated. If synthetic data is mixed into your upstream data sources in a way you cannot control, then your ML team loses a valuable controllable parameter.
  - miki123211 2y ago
    You still have some that control, but in a much more indirect way. There are three kinds of data now, synthetic, pre-2022 and current. Everything pre-2022 is definitely written by humans, synthetic data is still synthetic, and post-2022 is a mix of both. I wouldn't be surprised if "AI detectors" work somewhat for this use case. They're biased, far from accurate and a terrible idea if you need to make important decisions (like whether to expel a student for cheating), but there's quite a large room for errors here.
    - uberswe 2y ago
      > Everything pre-2022 is definitely written by humans I'm not sure if methods like article spinning counts as written by humans. This is something you could automate before AI and it would take a human written article and randomly swap words with similar meaning throughout to make it seem original.
      - derefr 2y ago
        Don’t forget machine-translated texts, where until ~2017 the translation was likely done by something much dumber / semantically lossy than an LLM, and after 2017 was basically done by an early form of LLM (the Transformers architecture originating in Google Translate.) Many historical English-language news reports published on the English-language websites of foreign news media from non-English-speaking countries, from 1998 (Babelfish era) to ~a few months ago, may be unreliable training data for this reason.
HanClinto 2y ago
Keep in mind that the Prover-Verifier game is not that it's training on AI-generated data (as if to imitate it) -- rather, it's training against a discriminator that verifies for correctness (a calculator) and understandability (a smaller, less-capable language model). You can think of this as a distillation method, but it's not like it's generating large amounts of source data and then retraining on it. This method only works on specific problems where there is an absolute right answer that can be verified with an independent heuristic (in this case, a math calculation). However, there is a lot of potential in the world of self-play and adversarial-training to improve the quality of our LLMs with true reinforcement learning. For one recent paper on this topic, also check out SPAG -- I found this one to be fascinating: https://github.com/Linear95/SPAG https://github.com/Linear95/SPAG I've been keeping notes on this topic in a WIP paper, and if you'd like to read my (rambling) ravings about it, you can find more info here: https://github.com/HanClinto/MENTAT https://github.com/HanClinto/MENTAT I think that self-play and reinforcement learning are going to absolutely be important for the next level of LLM development. If you use AI-generated data, then you must have an objective metric to verify "goodness". Nothing is free, and simply asking an LLM to rate the quality of its own data is not going to cut it. I think that's the point of the article.
visarga 2y ago
> Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works. They got a secret ace in their pocket - chat logs created with human in the loop. Of course those might still have errors, but much fewer. They can infer from a human response if it was accepted or not. I think OpenAI generates at least 1B sessions per month and 2 Trillion interactive tokens. Those can go into the LLM again for analysis and synthetic content generation, or for RLHF with the whole conversation as guidance. Having access to the following interactions can shed light on previous answers. Even more, they can correlate chats across days, presumably humans try out LLM ideas in reality and return for iteration. That way LLMs indirectly get real world grounding.
- mrbungie 2y ago
  This is likely one of the main reasons why they're offering ChatGPT for free and running ChatGPT Plus at a loss.
  - apwell23 2y ago
    > main reasons why they're offering ChatGPT for free and running ChatGPT Plus at a loss. As opposed to what though? Its not like there a huge demand for these apps that they can charge money. They have no option but to give it away for free .
    - LordDragonfang 2y ago
      Are you not aware that half the industry is using it to generate (at least some portion of) their code? And that many are paying for the privilege?
      - apwell23 2y ago
        Yes I pay for it. Not sure what your point is. I didn't say no one in the world pays for it. Are you not aware of flatlining user growth? or are you under the impression that coders paying for these apps are enough to make them profitable? https://www.thewrap.com/chatgpt-growth-2024/ https://www.thewrap.com/chatgpt-growth-2024/
sva_ 2y ago
I think this paper is more focused on figuring out what would happen in the theoretical scenario that most data on the web in the future might be AI generated without being marked as such. As they say, > We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. The companies you listed are surely not training the models indiscriminately. In particular they have piles of data for which they can have high confidence that they are written by humans.

by Kuinox 2y ago

echelon 2y ago

I'm long on synthetic data. If you think about evolution and hill climbing, of course it works. You have a pool of information and you accumulate new rearrangements of that information. Fitness selects for the best features within the new pool of data (For primates, opposable thumbs. For AI art, hands that aren't deformed.) It will naturally drift to better optima. RLHF, synthetic data, and enrichment are all we need.

ainoobler 2y ago
Are you sure about this? It's well known that cannibalism in animals leads to degenerative disorders.
- autokad 2y ago
  I think the direct action of a person taking their idea and thoughts and going through it many times (making changes / updates / fixes) fits better than eating something. however, I do think you still some form of validation data to ensure these are good changes. However, I do get the spirit of the article, that as more information generated online is done by LLms, the validity and use of the output decreases
  - ainoobler 2y ago
    What exactly is doing the validation?
    - autokad 2y ago
      depends on what one was doing. could be as simple as re-writing a sentence and asking someone if it looks better
- photonthug 2y ago
  Not sure why you’re downvoted, I think a comparison with prions seems apt and interesting, and bad protein copies that can replicate is essentially an information process. GAN research in recent years showing how you can sabotage a working dog/cat classifier with a one pixel change feels similar to how the tiniest parts of large systems can sometimes undermine the whole completely, albeit with low probability. And finally, since models will bootstrap models that bootstrap models, inevitably there are already subtle issues out there in the wild that may have an incubation period of many years before the downstream effects are completely clear.
  - ainoobler 2y ago
    The problem is systemic. People believe that the pursuit of monetary and financial profits by corporations will lead to the creation of benevolent artificial intelligence. I personally think this is essentially a religion because it is obvious that the pursuit of profits can not actually create anything benevolent, let alone intelligence.
throwup238 2y ago
> If you think about evolution and hill climbing, of course it works. You don't even need to go that far. How do most children learn? By reading textbooks and listening to lesson plans assembled by their teachers from all the relevant content the teachers have experienced. Our education systems are built on synthetic data that is created for optimized learning, so that every child doesn't have to prove the universe from scratch to learn some basic maths.
- samatman 2y ago
  That isn't synthetic data in any reasonable or meaningful sense of the term. You could describe a textbook as a synthesis, sure, in a sense which absolutely does not track with the 'synthetic' in 'synthetic data'. Unless the textbook is AI-generated, and I expect that in 2024, the number of AI-generated textbooks is not zero.
  - throwup238 2y ago
    It’s an analogy. The learning materials teachers create for students is very much like synthetic data, it’s just not assembled from algorithmic output.
    - lanstin 2y ago
      kids learn, walking talking reading arithmetic and physics, by doing things in the physical world. Adults may speak differently to kids than adults, but it's a stretch to say it's synthetic. Equivalent to synthetic would be a group of kids that just grew up together and made up a novel language.
    - Demonoculus 2y ago
      granted synthetic is closely related to synthesis, but in common parlance synthetic would mean something that is not natural or abiotic in some sense, in case of synthetic data, it should imply data that doesn't occur naturally from human and natural sources. i.e. synthetic data would be exactly the one that is assembled from algorithmic output. granted im not able to explain it as well as i understand it.
Salgat 2y ago
Synthetic data has to work if we hope to have ML models that can improve themselves in a similar fashion as humans when it comes to advancing knowledge.
- sfink 2y ago
  They mathematically cannot unless they have access to a way of measuring fitness. One that goes beyond an evaluation based on what they have already learned.
randcraw 2y ago
Data created automatically is not the same as human curated data, though both are synthetic. Auto-created data often suffers from a host of demerits (duplication, bias, error, unnatural distribution, irrelevance to learn the intended domain, etc, etc). Human curated data usually avoids these pitfalls, and thus is far more valuable when training -- otherwise all human teachers would be equally good. So auto- vs curated- data are incomparable when training naive neophytes like ML models, or children.
HPsquared 2y ago
It's important to climb the right hill.
- psb217 2y ago
  And to have a very well-tuned sense up vs down when the hill is almost flat...
kjkjadksj 2y ago
This misunderstands fitness. Its not a sure bet what is most optimal is what you see. “Good enough” given environmental context is what you see. Just like with certain crystal structures in chemistry, you may only be in a localized threshold of fitness stability that is not necessarily optimal, but separated from another optimal configuration by having suboptimal intermediary steps that need more activation energy to overcome before falling into a state with lower entropy (or more optimal fitness). In other words you can never be sure if synthetic data is any good or if what things gravitate toward are really most optimal.
- echelon 2y ago
  > Its not a sure bet what is most optimal is what you see. I wouldn't ever make "most optimal" a criteria. We're looking for measurable improvements, not a jump to god emperor or apex predator. > you may only be in a localized threshold of fitness stability that is not necessarily optimal, but separated from another optimal configuration by having suboptimal intermediary steps that need more activation energy to overcome before falling into a state with lower entropy (or more optimal fitness). Optimization is like that. But unlike genetics, where we can't re-route the recurrent laryngeal nerve or change fundamental biochemistry, these are engineered systems where we can set up wildly different experiments at any time. Just to cite one of many different research threads, there's now research now going into developing models from small scale training data. > you can never be sure if synthetic data is any good or if what things gravitate toward are really most optimal. We can know if the synthetic data is better. We have objective measures, a scientific process, and we'll always be striving for improvement.
mcswell 2y ago
The paper is not talking about verifiable synthetic data generated by some means other than LLMs.
sfink 2y ago
Only if you have a valid fitness metric. If you have humans looking at hands, then that's a good metric, as long as you really do have a human in the loop. Any automated metric (eg something that can evaluate hands) is great for measuring that specific dimension of fitness (after all, it was developed by a human, so it's really just an indirect way of feeding the human's evaluation into the machine). But it's useless for any other dimension. It'll happily rate the perfect hand coming out of a dogchickenpeach above the deformed hand petting the perfectly formed dog. It's the same as any other kind of signal processing. You can increase the noise, but you can't get more signal than you started with. Here, if the LLM decides that "monkey" is most often followed by "butt" and occasionally by "trainer", then it'll generate synthetic data with those frequencies and training on that data will not change its probability estimates at all. It will, however, drown out the signal that "you are a monkey butt" is more likely than "phlegm cigar monkey butt", if you'll forgive me the liberty of using those phrases to represent statistical correlations just beyond the frontier of what the LLM has learned. The synthetic data will teach it that everything it doesn't already know is equally probable, which will overwhelm human source data in which it isn't.
apwell23 2y ago
https://en.wikipedia.org/wiki/Infinite_monkey_theorem https://en.wikipedia.org/wiki/Infinite_monkey_theorem

nyrikki 2y ago

Cheese and Chalk. It is very different to generate synthetic datasets to assist in targeted training , vs ingesting LLM output from web scraping.

abeppu 2y ago
Yes -- said another way, if you're an ML researcher and you have human-provided (scraped) data, and an ability to generate synthetic data, then until recently, you had a controllable parameter: how much of your training data for your new model should be synthetic? You can vary this, run multiple experiments, and choose how much synthetic data to use -- and you can vary the specific configs about how that synthetic data is generated. If synthetic data is mixed into your upstream data sources in a way you cannot control, then your ML team loses a valuable controllable parameter.
- miki123211 2y ago
  You still have some that control, but in a much more indirect way. There are three kinds of data now, synthetic, pre-2022 and current. Everything pre-2022 is definitely written by humans, synthetic data is still synthetic, and post-2022 is a mix of both. I wouldn't be surprised if "AI detectors" work somewhat for this use case. They're biased, far from accurate and a terrible idea if you need to make important decisions (like whether to expel a student for cheating), but there's quite a large room for errors here.
  - uberswe 2y ago
    > Everything pre-2022 is definitely written by humans I'm not sure if methods like article spinning counts as written by humans. This is something you could automate before AI and it would take a human written article and randomly swap words with similar meaning throughout to make it seem original.
    - derefr 2y ago
      Don’t forget machine-translated texts, where until ~2017 the translation was likely done by something much dumber / semantically lossy than an LLM, and after 2017 was basically done by an early form of LLM (the Transformers architecture originating in Google Translate.) Many historical English-language news reports published on the English-language websites of foreign news media from non-English-speaking countries, from 1998 (Babelfish era) to ~a few months ago, may be unreliable training data for this reason.

HanClinto 2y ago

Keep in mind that the Prover-Verifier game is not that it's training on AI-generated data (as if to imitate it) -- rather, it's training against a discriminator that verifies for correctness (a calculator) and understandability (a smaller, less-capable language model). You can think of this as a distillation method, but it's not like it's generating large amounts of source data and then retraining on it. This method only works on specific problems where there is an absolute right answer that can be verified with an independent heuristic (in this case, a math calculation). However, there is a lot of potential in the world of self-play and adversarial-training to improve the quality of our LLMs with true reinforcement learning. For one recent paper on this topic, also check out SPAG -- I found this one to be fascinating: https://github.com/Linear95/SPAG https://github.com/Linear95/SPAG I've been keeping notes on this topic in a WIP paper, and if you'd like to read my (rambling) ravings about it, you can find more info here: https://github.com/HanClinto/MENTAT https://github.com/HanClinto/MENTAT I think that self-play and reinforcement learning are going to absolutely be important for the next level of LLM development. If you use AI-generated data, then you must have an objective metric to verify "goodness". Nothing is free, and simply asking an LLM to rate the quality of its own data is not going to cut it. I think that's the point of the article.

visarga 2y ago

> Meanwhile OpenAI, Anthropics, trains on AI generated data to improve their models, and it works. They got a secret ace in their pocket - chat logs created with human in the loop. Of course those might still have errors, but much fewer. They can infer from a human response if it was accepted or not. I think OpenAI generates at least 1B sessions per month and 2 Trillion interactive tokens. Those can go into the LLM again for analysis and synthetic content generation, or for RLHF with the whole conversation as guidance. Having access to the following interactions can shed light on previous answers. Even more, they can correlate chats across days, presumably humans try out LLM ideas in reality and return for iteration. That way LLMs indirectly get real world grounding.

mrbungie 2y ago
This is likely one of the main reasons why they're offering ChatGPT for free and running ChatGPT Plus at a loss.
- apwell23 2y ago
  > main reasons why they're offering ChatGPT for free and running ChatGPT Plus at a loss. As opposed to what though? Its not like there a huge demand for these apps that they can charge money. They have no option but to give it away for free .
  - LordDragonfang 2y ago
    Are you not aware that half the industry is using it to generate (at least some portion of) their code? And that many are paying for the privilege?
    - apwell23 2y ago
      Yes I pay for it. Not sure what your point is. I didn't say no one in the world pays for it. Are you not aware of flatlining user growth? or are you under the impression that coders paying for these apps are enough to make them profitable? https://www.thewrap.com/chatgpt-growth-2024/ https://www.thewrap.com/chatgpt-growth-2024/

sva_ 2y ago

I think this paper is more focused on figuring out what would happen in the theoretical scenario that most data on the web in the future might be AI generated without being marked as such. As they say, > We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. The companies you listed are surely not training the models indiscriminately. In particular they have piles of data for which they can have high confidence that they are written by humans.