4 ms·

> we manually curated a set of over 2,000 YouTube channels that release original openly licensed content containing speech. From these channels, we retrieved an

> we manually curated a set of over 2,000 YouTube channels that release original openly licensed content containing speech. From these channels, we retrieved and transcribed (using Whisper) over 1.1 million openly licensed videos comprising more than 470,000 hours of content. This is why Gemini has such an advantage. Also, link to explore data: https://huggingface.co/collections/common-pile/common-pile-v01-68307d37df48e36f02717f21 https://huggingface.co/collections/common-pile/common-pile-v...

otherme123 1y ago
The abstract is open about this data to be used to train models. But a lot of this data come from models, like whisper.
- ACCount37 1y ago
  What's your concern?
  - ggm 1y ago
    You don't believe in model collapse? Or don't think it applies to a phase shift from audio to written texts?
    - simonw 1y ago
      Personally I don't believe in model collapse. Has anyone demonstrated it occurring in the wild, outside of the tiny set of papers that deliberately caused it to happen? I think model collapse gets talked about so much because it is irresistible schadenfreude. The idea of models eating their own tails in a way that leads to their inevitable demise is captivating to a lot of people, especially AI skeptics.
      - pama 1y ago
        I agree. A partial counterexample is the RL training loop on verifiable tasks, which uses the model in a loop to generate training data. Another one is the cleanup/prioritization of the pretraining data using earlier models. More generally, a lot of ideas have been speculated based on very tiny models in controlled settings and they didnt pan out in real LLMs. There probably exists a minimal compute threshold for overcoming generalization traps.
      - marbro 1y ago
        Carbon-based model collapse is known as groupthink and happens constantly.
        deleted 1y ago
        [deleted]
    - ACCount37 1y ago
      "Model collapse" isn't real. It's a laboratory failure mode that doesn't happen in real world environments. It's popular because some people latched onto the idea - desperately wanting something to stop the AI tech from advancing. It, quite obviously, doesn't stop the AI tech from advancing. Now, you can write an entire research paper on why model collapse happens or fails to happen. But a simple way to think of it is: looping AI onto itself multiple times amplifies that AI's own deficiencies, distortions and idiosyncrasies - until, after enough iterations, they come to completely dominate its outputs. This doesn't apply at all to training an LLM on Whisper outputs that are, in turn, based on human-generated videos. The LLM will inherit some Whisper quirks, but most of the data in Whisper outputs comes from the videos themselves.
    - everforward 1y ago
      No, I don’t think it applies here. The semantics and speech patterns were generated by a human, Whisper just transcribed them. There is some risk that Whisper transcribed inaccurately, but that’s less model collapse and more “the dataset is bad”.
  - numpad0 1y ago
    I guess that transcript is not guaranteed clean? * Silence * = "Like and Subscribe" etc.
- benterix 1y ago
  So?
  - otherme123 1y ago
    I don't know much about LLM training, but previous AI needed clean data to train. You shouln't train on generated data. For example, you had a classifier that works at 95% precission trained with carefully labeled data. Then, to train the next version you download 1Tb of images, classify with your previous model, and use that to retrain. Do you expect to get better than 95%, or are you poisoning your model? I'm asking: can you do that with LLM? Feed them data that's known to be 95% precise at best? I did some Whisper, and usually get runs of words, like "bye bye bye bye bye bye", despite being only said once. Should I use that kind of data to train a LLM? I saw this experiment where an LLM was feed an image and asked to make the same image. Then repeat with the generated image. After ten or so cycles, the content (a human head photo) was barely recognizable.
    - electroglyph 1y ago
      Phi models are notorious for using mostly synthetic data
    - orbital-decay 1y ago
      The reality of working with humongous datasets is they're always bootstrapped like this, in multiple steps. In LLMs in particular, the entire post-training step is always done on synthetic data. There are ways to avoid failure modes typical for that (like model collapse), you need much less real data to keep the model in check than you probably think.
- klft 1y ago
  Whisper ist used for speech-to-text conversion. Not to generate the text.
  - estimator7292 1y ago
    It's still AI generated text that is not in any way guaranteed to be correct or accurate.
    - UltraSane 1y ago
      Its accuracy can be and is quantified.

otherme123 1y ago

The abstract is open about this data to be used to train models. But a lot of this data come from models, like whisper.

ACCount37 1y ago
What's your concern?
- ggm 1y ago
  You don't believe in model collapse? Or don't think it applies to a phase shift from audio to written texts?
  - simonw 1y ago
    Personally I don't believe in model collapse. Has anyone demonstrated it occurring in the wild, outside of the tiny set of papers that deliberately caused it to happen? I think model collapse gets talked about so much because it is irresistible schadenfreude. The idea of models eating their own tails in a way that leads to their inevitable demise is captivating to a lot of people, especially AI skeptics.
    - pama 1y ago
      I agree. A partial counterexample is the RL training loop on verifiable tasks, which uses the model in a loop to generate training data. Another one is the cleanup/prioritization of the pretraining data using earlier models. More generally, a lot of ideas have been speculated based on very tiny models in controlled settings and they didnt pan out in real LLMs. There probably exists a minimal compute threshold for overcoming generalization traps.
    - marbro 1y ago
      Carbon-based model collapse is known as groupthink and happens constantly.
      - deleted 1y ago
        [deleted]
  - ACCount37 1y ago
    "Model collapse" isn't real. It's a laboratory failure mode that doesn't happen in real world environments. It's popular because some people latched onto the idea - desperately wanting something to stop the AI tech from advancing. It, quite obviously, doesn't stop the AI tech from advancing. Now, you can write an entire research paper on why model collapse happens or fails to happen. But a simple way to think of it is: looping AI onto itself multiple times amplifies that AI's own deficiencies, distortions and idiosyncrasies - until, after enough iterations, they come to completely dominate its outputs. This doesn't apply at all to training an LLM on Whisper outputs that are, in turn, based on human-generated videos. The LLM will inherit some Whisper quirks, but most of the data in Whisper outputs comes from the videos themselves.
  - everforward 1y ago
    No, I don’t think it applies here. The semantics and speech patterns were generated by a human, Whisper just transcribed them. There is some risk that Whisper transcribed inaccurately, but that’s less model collapse and more “the dataset is bad”.
- numpad0 1y ago
  I guess that transcript is not guaranteed clean? * Silence * = "Like and Subscribe" etc.
benterix 1y ago
So?
- otherme123 1y ago
  I don't know much about LLM training, but previous AI needed clean data to train. You shouln't train on generated data. For example, you had a classifier that works at 95% precission trained with carefully labeled data. Then, to train the next version you download 1Tb of images, classify with your previous model, and use that to retrain. Do you expect to get better than 95%, or are you poisoning your model? I'm asking: can you do that with LLM? Feed them data that's known to be 95% precise at best? I did some Whisper, and usually get runs of words, like "bye bye bye bye bye bye", despite being only said once. Should I use that kind of data to train a LLM? I saw this experiment where an LLM was feed an image and asked to make the same image. Then repeat with the generated image. After ten or so cycles, the content (a human head photo) was barely recognizable.
  - electroglyph 1y ago
    Phi models are notorious for using mostly synthetic data
  - orbital-decay 1y ago
    The reality of working with humongous datasets is they're always bootstrapped like this, in multiple steps. In LLMs in particular, the entire post-training step is always done on synthetic data. There are ways to avoid failure modes typical for that (like model collapse), you need much less real data to keep the model in check than you probably think.
klft 1y ago
Whisper ist used for speech-to-text conversion. Not to generate the text.
- estimator7292 1y ago
  It's still AI generated text that is not in any way guaranteed to be correct or accurate.
  - UltraSane 1y ago
    Its accuracy can be and is quantified.