5 ms·

I'm long on synthetic data. If you think about evolution and hill climbing, of course it works. You have a pool of information and you accumulate new rearrang

I'm long on synthetic data. If you think about evolution and hill climbing, of course it works. You have a pool of information and you accumulate new rearrangements of that information. Fitness selects for the best features within the new pool of data (For primates, opposable thumbs. For AI art, hands that aren't deformed.) It will naturally drift to better optima. RLHF, synthetic data, and enrichment are all we need.

ainoobler 2y ago
Are you sure about this? It's well known that cannibalism in animals leads to degenerative disorders.
- autokad 2y ago
  I think the direct action of a person taking their idea and thoughts and going through it many times (making changes / updates / fixes) fits better than eating something. however, I do think you still some form of validation data to ensure these are good changes. However, I do get the spirit of the article, that as more information generated online is done by LLms, the validity and use of the output decreases
  - ainoobler 2y ago
    What exactly is doing the validation?
    - autokad 2y ago
      depends on what one was doing. could be as simple as re-writing a sentence and asking someone if it looks better
- photonthug 2y ago
  Not sure why you’re downvoted, I think a comparison with prions seems apt and interesting, and bad protein copies that can replicate is essentially an information process. GAN research in recent years showing how you can sabotage a working dog/cat classifier with a one pixel change feels similar to how the tiniest parts of large systems can sometimes undermine the whole completely, albeit with low probability. And finally, since models will bootstrap models that bootstrap models, inevitably there are already subtle issues out there in the wild that may have an incubation period of many years before the downstream effects are completely clear.
  - ainoobler 2y ago
    The problem is systemic. People believe that the pursuit of monetary and financial profits by corporations will lead to the creation of benevolent artificial intelligence. I personally think this is essentially a religion because it is obvious that the pursuit of profits can not actually create anything benevolent, let alone intelligence.
throwup238 2y ago
> If you think about evolution and hill climbing, of course it works. You don't even need to go that far. How do most children learn? By reading textbooks and listening to lesson plans assembled by their teachers from all the relevant content the teachers have experienced. Our education systems are built on synthetic data that is created for optimized learning, so that every child doesn't have to prove the universe from scratch to learn some basic maths.
- samatman 2y ago
  That isn't synthetic data in any reasonable or meaningful sense of the term. You could describe a textbook as a synthesis, sure, in a sense which absolutely does not track with the 'synthetic' in 'synthetic data'. Unless the textbook is AI-generated, and I expect that in 2024, the number of AI-generated textbooks is not zero.
  - throwup238 2y ago
    It’s an analogy. The learning materials teachers create for students is very much like synthetic data, it’s just not assembled from algorithmic output.
    - lanstin 2y ago
      kids learn, walking talking reading arithmetic and physics, by doing things in the physical world. Adults may speak differently to kids than adults, but it's a stretch to say it's synthetic. Equivalent to synthetic would be a group of kids that just grew up together and made up a novel language.
    - Demonoculus 2y ago
      granted synthetic is closely related to synthesis, but in common parlance synthetic would mean something that is not natural or abiotic in some sense, in case of synthetic data, it should imply data that doesn't occur naturally from human and natural sources. i.e. synthetic data would be exactly the one that is assembled from algorithmic output. granted im not able to explain it as well as i understand it.
Salgat 2y ago
Synthetic data has to work if we hope to have ML models that can improve themselves in a similar fashion as humans when it comes to advancing knowledge.
- sfink 2y ago
  They mathematically cannot unless they have access to a way of measuring fitness. One that goes beyond an evaluation based on what they have already learned.
randcraw 2y ago
Data created automatically is not the same as human curated data, though both are synthetic. Auto-created data often suffers from a host of demerits (duplication, bias, error, unnatural distribution, irrelevance to learn the intended domain, etc, etc). Human curated data usually avoids these pitfalls, and thus is far more valuable when training -- otherwise all human teachers would be equally good. So auto- vs curated- data are incomparable when training naive neophytes like ML models, or children.
HPsquared 2y ago
It's important to climb the right hill.
- psb217 2y ago
  And to have a very well-tuned sense up vs down when the hill is almost flat...
kjkjadksj 2y ago
This misunderstands fitness. Its not a sure bet what is most optimal is what you see. “Good enough” given environmental context is what you see. Just like with certain crystal structures in chemistry, you may only be in a localized threshold of fitness stability that is not necessarily optimal, but separated from another optimal configuration by having suboptimal intermediary steps that need more activation energy to overcome before falling into a state with lower entropy (or more optimal fitness). In other words you can never be sure if synthetic data is any good or if what things gravitate toward are really most optimal.
- echelon 2y ago
  > Its not a sure bet what is most optimal is what you see. I wouldn't ever make "most optimal" a criteria. We're looking for measurable improvements, not a jump to god emperor or apex predator. > you may only be in a localized threshold of fitness stability that is not necessarily optimal, but separated from another optimal configuration by having suboptimal intermediary steps that need more activation energy to overcome before falling into a state with lower entropy (or more optimal fitness). Optimization is like that. But unlike genetics, where we can't re-route the recurrent laryngeal nerve or change fundamental biochemistry, these are engineered systems where we can set up wildly different experiments at any time. Just to cite one of many different research threads, there's now research now going into developing models from small scale training data. > you can never be sure if synthetic data is any good or if what things gravitate toward are really most optimal. We can know if the synthetic data is better. We have objective measures, a scientific process, and we'll always be striving for improvement.
mcswell 2y ago
The paper is not talking about verifiable synthetic data generated by some means other than LLMs.
sfink 2y ago
Only if you have a valid fitness metric. If you have humans looking at hands, then that's a good metric, as long as you really do have a human in the loop. Any automated metric (eg something that can evaluate hands) is great for measuring that specific dimension of fitness (after all, it was developed by a human, so it's really just an indirect way of feeding the human's evaluation into the machine). But it's useless for any other dimension. It'll happily rate the perfect hand coming out of a dogchickenpeach above the deformed hand petting the perfectly formed dog. It's the same as any other kind of signal processing. You can increase the noise, but you can't get more signal than you started with. Here, if the LLM decides that "monkey" is most often followed by "butt" and occasionally by "trainer", then it'll generate synthetic data with those frequencies and training on that data will not change its probability estimates at all. It will, however, drown out the signal that "you are a monkey butt" is more likely than "phlegm cigar monkey butt", if you'll forgive me the liberty of using those phrases to represent statistical correlations just beyond the frontier of what the LLM has learned. The synthetic data will teach it that everything it doesn't already know is equally probable, which will overwhelm human source data in which it isn't.
apwell23 2y ago
https://en.wikipedia.org/wiki/Infinite_monkey_theorem https://en.wikipedia.org/wiki/Infinite_monkey_theorem

ainoobler 2y ago

Are you sure about this? It's well known that cannibalism in animals leads to degenerative disorders.

autokad 2y ago
I think the direct action of a person taking their idea and thoughts and going through it many times (making changes / updates / fixes) fits better than eating something. however, I do think you still some form of validation data to ensure these are good changes. However, I do get the spirit of the article, that as more information generated online is done by LLms, the validity and use of the output decreases
- ainoobler 2y ago
  What exactly is doing the validation?
  - autokad 2y ago
    depends on what one was doing. could be as simple as re-writing a sentence and asking someone if it looks better
photonthug 2y ago
Not sure why you’re downvoted, I think a comparison with prions seems apt and interesting, and bad protein copies that can replicate is essentially an information process. GAN research in recent years showing how you can sabotage a working dog/cat classifier with a one pixel change feels similar to how the tiniest parts of large systems can sometimes undermine the whole completely, albeit with low probability. And finally, since models will bootstrap models that bootstrap models, inevitably there are already subtle issues out there in the wild that may have an incubation period of many years before the downstream effects are completely clear.
- ainoobler 2y ago
  The problem is systemic. People believe that the pursuit of monetary and financial profits by corporations will lead to the creation of benevolent artificial intelligence. I personally think this is essentially a religion because it is obvious that the pursuit of profits can not actually create anything benevolent, let alone intelligence.

throwup238 2y ago

> If you think about evolution and hill climbing, of course it works. You don't even need to go that far. How do most children learn? By reading textbooks and listening to lesson plans assembled by their teachers from all the relevant content the teachers have experienced. Our education systems are built on synthetic data that is created for optimized learning, so that every child doesn't have to prove the universe from scratch to learn some basic maths.

samatman 2y ago
That isn't synthetic data in any reasonable or meaningful sense of the term. You could describe a textbook as a synthesis, sure, in a sense which absolutely does not track with the 'synthetic' in 'synthetic data'. Unless the textbook is AI-generated, and I expect that in 2024, the number of AI-generated textbooks is not zero.
- throwup238 2y ago
  It’s an analogy. The learning materials teachers create for students is very much like synthetic data, it’s just not assembled from algorithmic output.
  - lanstin 2y ago
    kids learn, walking talking reading arithmetic and physics, by doing things in the physical world. Adults may speak differently to kids than adults, but it's a stretch to say it's synthetic. Equivalent to synthetic would be a group of kids that just grew up together and made up a novel language.
  - Demonoculus 2y ago
    granted synthetic is closely related to synthesis, but in common parlance synthetic would mean something that is not natural or abiotic in some sense, in case of synthetic data, it should imply data that doesn't occur naturally from human and natural sources. i.e. synthetic data would be exactly the one that is assembled from algorithmic output. granted im not able to explain it as well as i understand it.

Salgat 2y ago

Synthetic data has to work if we hope to have ML models that can improve themselves in a similar fashion as humans when it comes to advancing knowledge.

sfink 2y ago
They mathematically cannot unless they have access to a way of measuring fitness. One that goes beyond an evaluation based on what they have already learned.

randcraw 2y ago

Data created automatically is not the same as human curated data, though both are synthetic. Auto-created data often suffers from a host of demerits (duplication, bias, error, unnatural distribution, irrelevance to learn the intended domain, etc, etc). Human curated data usually avoids these pitfalls, and thus is far more valuable when training -- otherwise all human teachers would be equally good. So auto- vs curated- data are incomparable when training naive neophytes like ML models, or children.

HPsquared 2y ago

It's important to climb the right hill.

psb217 2y ago
And to have a very well-tuned sense up vs down when the hill is almost flat...

kjkjadksj 2y ago

This misunderstands fitness. Its not a sure bet what is most optimal is what you see. “Good enough” given environmental context is what you see. Just like with certain crystal structures in chemistry, you may only be in a localized threshold of fitness stability that is not necessarily optimal, but separated from another optimal configuration by having suboptimal intermediary steps that need more activation energy to overcome before falling into a state with lower entropy (or more optimal fitness). In other words you can never be sure if synthetic data is any good or if what things gravitate toward are really most optimal.

echelon 2y ago
> Its not a sure bet what is most optimal is what you see. I wouldn't ever make "most optimal" a criteria. We're looking for measurable improvements, not a jump to god emperor or apex predator. > you may only be in a localized threshold of fitness stability that is not necessarily optimal, but separated from another optimal configuration by having suboptimal intermediary steps that need more activation energy to overcome before falling into a state with lower entropy (or more optimal fitness). Optimization is like that. But unlike genetics, where we can't re-route the recurrent laryngeal nerve or change fundamental biochemistry, these are engineered systems where we can set up wildly different experiments at any time. Just to cite one of many different research threads, there's now research now going into developing models from small scale training data. > you can never be sure if synthetic data is any good or if what things gravitate toward are really most optimal. We can know if the synthetic data is better. We have objective measures, a scientific process, and we'll always be striving for improvement.

mcswell 2y ago

The paper is not talking about verifiable synthetic data generated by some means other than LLMs.

sfink 2y ago

Only if you have a valid fitness metric. If you have humans looking at hands, then that's a good metric, as long as you really do have a human in the loop. Any automated metric (eg something that can evaluate hands) is great for measuring that specific dimension of fitness (after all, it was developed by a human, so it's really just an indirect way of feeding the human's evaluation into the machine). But it's useless for any other dimension. It'll happily rate the perfect hand coming out of a dogchickenpeach above the deformed hand petting the perfectly formed dog. It's the same as any other kind of signal processing. You can increase the noise, but you can't get more signal than you started with. Here, if the LLM decides that "monkey" is most often followed by "butt" and occasionally by "trainer", then it'll generate synthetic data with those frequencies and training on that data will not change its probability estimates at all. It will, however, drown out the signal that "you are a monkey butt" is more likely than "phlegm cigar monkey butt", if you'll forgive me the liberty of using those phrases to represent statistical correlations just beyond the frontier of what the LLM has learned. The synthetic data will teach it that everything it doesn't already know is equally probable, which will overwhelm human source data in which it isn't.

apwell23 2y ago

https://en.wikipedia.org/wiki/Infinite_monkey_theorem https://en.wikipedia.org/wiki/Infinite_monkey_theorem