4 ms·

An ongoing question I have is why effort wasn't put into tokenising speech (instead of transcribed words) and then making an LLM out of that. There are huge amo

An ongoing question I have is why effort wasn't put into tokenising speech (instead of transcribed words) and then making an LLM out of that. There are huge amounts of speech available to train on.

MichealCodes 1y ago
I don't think we've had the transformer moment for audio training yet, but yes, in theory audio-first models will be much more capable.
- trollbridge 1y ago
  Particularly interesting would be transformations between tokenised audio and tokenised text. I recall someone telling me once up to 90% of communication can be non-verbal, so when an LLM sticks to just text, it's only getting 10% of the data.
benob 1y ago
Audio tokenization consumes at least 4x tokens versus text. So there is an efficiency problem to start with. Then is there enough audio data to train a LLM from scratch?
- trollbridge 1y ago
  Start an MVNO that offers cheaper phone plans and and train on all those phone calls. There are big libraries of old speeches. Simply capture all all current radio/tv transmissions and train on that (we've already established copyright doesn't apply to LLM training, right?)
  - miki123211 1y ago
    > Start an MVNO that offers cheaper phone plans and and train on all those phone calls. q: What is 2+2? A: The warranty for your car has expired...
- 542354234235 1y ago
  Don't we have tens of thousands of hours (hundreds of thousands?) of closed captioned tv shows and movies? How many hours of news broadcasts with transcripts do we have? Maybe I just don't understand what is needed, but it seems like we have a lot of data to work with.
  - roboror 1y ago
    Sure but that needs to be licensed
  - cruffle_duffle 1y ago
    Correct me if I’m wrong but you need more than just closed captions. You need precise timing too. I’d think you’d need the text to line up exactly with the audio so when the voice makes an “A” sound the text it aligns with is “A” as well. So while having the closed captions saves some of the work, there is probably much more needed to get everything lined up. But I’m absolutely not an expert at all. In fact this is the first I’ve ever even though about it!
    - vvolhejn 1y ago
      Author here. Speech-to-text is more or less solved, it's easy to automatically get captions including precise timestamps. For training Moshi, Kyutai's audio LLM, my colleagues used whisper-timestamped to transcribe 7 million hours of audio. See Section 4.2 in the Moshi paper: https://arxiv.org/pdf/2410.00037 https://arxiv.org/pdf/2410.00037
      - cruffle_duffle 1y ago
        Sweet!
- cyberax 1y ago
  Yup. You can use Mozilla's corpus: https://commonvoice.mozilla.org/en https://commonvoice.mozilla.org/en It mostly uses the UN reports as a source of parallel translated texts, so the language is quite a bit stilted. But it's a good start.
ca_tech 1y ago
There is data but nowhere near the amount of written language that is fairly normalized and without the need to account for additional features such as language, dialect, intonation, facial expression, hand gestures. Speech to text is used as the translation layer as it throws many of those other features away and contextualizes it into a set of tokens that are much more efficient to map between languages.
mohsen1 1y ago
It costs more to train on audio tokens but I'm sure we will get there. Training a model on transcript of a lecture on YouTube vs. training on audio of it will make a difference.
nmfisher 1y ago
The article is talking about doing exactly that. The key question is how to convert an inherently continuous signal (speech/audio) into a discrete set of tokens. A single window of audio is usually somewhere between 10ms and 100ms. It's difficult to squish all that information down to a single "token" that represents the semantic and acoustic content for that window. That's why residual vector quantization is a useful technique - using multiple dictionaries to quantize a single timeslice, each conditioned on the previous residual level. You can also quantize a signal at different frequencies. There are samples towards the end of the post of their LLM trained on their Mimi audio codec.
- CGMthrowaway 1y ago
  > The key question is how to convert an inherently continuous signal (speech/audio) into a discrete set of tokens. A single window of audio is usually somewhere between 10ms and 100ms. It's difficult to squish all that information down to a single "token" that represents the semantic and acoustic content for that window. I read the article and confess some of the modeling parts were above my comprehension. But I would like to add that as an audio engineer, the "key question" you describe is solved, just not applied to transformer models (?). An experienced engineer can look at a waveform in a DAW and identify specific consonants, vowels, specific words, etc quite fluently. And with tools like Melodyne - which already quantize audio semantically - they can identify (and manipulate) pitch and formants as well, turning an O vowel into an E vowel, or changing the inflection of a phrase (up-speak vs down-speak, for example). I don't know how to apply this to a neural codec, but it seems like it shouldn't be that hard (that's my naivete coming through)
  - PaulDavisThe1st 1y ago
    > An experienced engineer can look at a waveform in a DAW and identify specific consonants, vowels, specific words, etc quite fluently. As an experienced DAW author, I very, very much doubt this. What can be done relatively easy is to "see" or rather "follow along" in the waveform when listening to the audio. But I read your claim as being that someone could look at the waveform (which is already decimated from the original) and identify words or phonemes without hearing the associated audio. I am extremely skeptical that there is anyone anywhere in the world who can do this.

An ongoing question I have is why effort wasn't put into tokenising speech (instead of transcribed words) and then making an LLM out of that. There are huge amounts of speech available to train on.

MichealCodes 1y ago

I don't think we've had the transformer moment for audio training yet, but yes, in theory audio-first models will be much more capable.

trollbridge 1y ago
Particularly interesting would be transformations between tokenised audio and tokenised text. I recall someone telling me once up to 90% of communication can be non-verbal, so when an LLM sticks to just text, it's only getting 10% of the data.

benob 1y ago

Audio tokenization consumes at least 4x tokens versus text. So there is an efficiency problem to start with. Then is there enough audio data to train a LLM from scratch?

trollbridge 1y ago
Start an MVNO that offers cheaper phone plans and and train on all those phone calls. There are big libraries of old speeches. Simply capture all all current radio/tv transmissions and train on that (we've already established copyright doesn't apply to LLM training, right?)
- miki123211 1y ago
  > Start an MVNO that offers cheaper phone plans and and train on all those phone calls. q: What is 2+2? A: The warranty for your car has expired...
542354234235 1y ago
Don't we have tens of thousands of hours (hundreds of thousands?) of closed captioned tv shows and movies? How many hours of news broadcasts with transcripts do we have? Maybe I just don't understand what is needed, but it seems like we have a lot of data to work with.
- roboror 1y ago
  Sure but that needs to be licensed
- cruffle_duffle 1y ago
  Correct me if I’m wrong but you need more than just closed captions. You need precise timing too. I’d think you’d need the text to line up exactly with the audio so when the voice makes an “A” sound the text it aligns with is “A” as well. So while having the closed captions saves some of the work, there is probably much more needed to get everything lined up. But I’m absolutely not an expert at all. In fact this is the first I’ve ever even though about it!
  - vvolhejn 1y ago
    Author here. Speech-to-text is more or less solved, it's easy to automatically get captions including precise timestamps. For training Moshi, Kyutai's audio LLM, my colleagues used whisper-timestamped to transcribe 7 million hours of audio. See Section 4.2 in the Moshi paper: https://arxiv.org/pdf/2410.00037 https://arxiv.org/pdf/2410.00037
    - cruffle_duffle 1y ago
      Sweet!
cyberax 1y ago
Yup. You can use Mozilla's corpus: https://commonvoice.mozilla.org/en https://commonvoice.mozilla.org/en It mostly uses the UN reports as a source of parallel translated texts, so the language is quite a bit stilted. But it's a good start.

ca_tech 1y ago

There is data but nowhere near the amount of written language that is fairly normalized and without the need to account for additional features such as language, dialect, intonation, facial expression, hand gestures. Speech to text is used as the translation layer as it throws many of those other features away and contextualizes it into a set of tokens that are much more efficient to map between languages.

mohsen1 1y ago

It costs more to train on audio tokens but I'm sure we will get there. Training a model on transcript of a lecture on YouTube vs. training on audio of it will make a difference.

nmfisher 1y ago

The article is talking about doing exactly that. The key question is how to convert an inherently continuous signal (speech/audio) into a discrete set of tokens. A single window of audio is usually somewhere between 10ms and 100ms. It's difficult to squish all that information down to a single "token" that represents the semantic and acoustic content for that window. That's why residual vector quantization is a useful technique - using multiple dictionaries to quantize a single timeslice, each conditioned on the previous residual level. You can also quantize a signal at different frequencies. There are samples towards the end of the post of their LLM trained on their Mimi audio codec.

CGMthrowaway 1y ago
> The key question is how to convert an inherently continuous signal (speech/audio) into a discrete set of tokens. A single window of audio is usually somewhere between 10ms and 100ms. It's difficult to squish all that information down to a single "token" that represents the semantic and acoustic content for that window. I read the article and confess some of the modeling parts were above my comprehension. But I would like to add that as an audio engineer, the "key question" you describe is solved, just not applied to transformer models (?). An experienced engineer can look at a waveform in a DAW and identify specific consonants, vowels, specific words, etc quite fluently. And with tools like Melodyne - which already quantize audio semantically - they can identify (and manipulate) pitch and formants as well, turning an O vowel into an E vowel, or changing the inflection of a phrase (up-speak vs down-speak, for example). I don't know how to apply this to a neural codec, but it seems like it shouldn't be that hard (that's my naivete coming through)
- PaulDavisThe1st 1y ago
  > An experienced engineer can look at a waveform in a DAW and identify specific consonants, vowels, specific words, etc quite fluently. As an experienced DAW author, I very, very much doubt this. What can be done relatively easy is to "see" or rather "follow along" in the waveform when listening to the audio. But I read your claim as being that someone could look at the waveform (which is already decimated from the original) and identify words or phonemes without hearing the associated audio. I am extremely skeptical that there is anyone anywhere in the world who can do this.