4 ms·

>If it can memorize all possible waveforms across all those languages and how they convert into words in only 2B parameters then I'm incredibly impressed. Well

>If it can memorize all possible waveforms across all those languages and how they convert into words in only 2B parameters then I'm incredibly impressed. Well then I don't think you've looked at the topic very deeply. Our voices don't make every possible wave form. The IPA alphabet has existed for a generation now, is universal for all spoken human languages, and has a basis in physiological origins. Of more than 160 IPA symbols, relatively few will be used to transcribe speech in any one language. It should not take 2B parameters to do this. Look at the inner ear and you'll see a big hint. Before even getting to the brain the sound goes through a spiral shaped chamber. Quote wikipedia: "The hair cells in the Organ of Corti are tuned to certain sound frequencies by way of their location in the cochlea...". Clearly looking at time series wave form data in the first place is a mistake. Rolling window fourier transform all the speech data first, then train on it. I will bet 2-digit sums of money that simple preprocessing step would outperform what they've done. Really, take any recording of people talking, open it up with audacity, and check out the spectrogram view. The sort of sounds we make with speech are a lot like an FM radio transmission. There's a baseband average pitch that we speak at, and then information encoded on top by deviating over/under the base band in smooth, simple ways. If that spectrogram was an OCR problem, it would already be solved. >A humble brag about how small the model is maybe? I see a B at the end of a number I assume its part of a pissing contest. But if they really are playing parameter golf bragging about how small they can get it, then I admit that's a step in the right direction.

schoen 4y ago
I think I remember reading that languages can choose any point on the vowel diagram as a canonical phonetic expression of a vowel, even though a given language will only have a small finite inventory of vowels. If this is right, one could interpret it as phonemes being digital (a native speaker can consistently transcribe them using a small number of IPA symbols), but (at least vowel) phones being analog (the typical frequency spectrum appearance typically representing a specific vowel could best be represented by floating-point numbers that are culturally chosen somewhat arbitrary by language evolution). Like /ɪ/ might be 0.82 closeness, 0.23 backness, 0.05 roundedness in some language, but 0.79 closeness, 0.26 backness, and 0.1 roundedness in another. In turn, if that's right, I don't know how much variation there is among native speakers (from a particular region) in production of a given vowel (or how consistently and precisely children can learn these values). Presumably for recognition the brain rounds off somehow to the nearest vowel phoneme, kind of like picking the nearest point on the constellation diagram in digital communications demodulation?
- IIAOPSW 4y ago
  >a specific vowel could best be represented by floating-point numbers that are culturally chosen somewhat arbitrary by language evolution). Like /ɪ/ might be 0.82 closeness, 0.23 backness, 0.05 roundedness in some language, but 0.79 closeness, 0.26 backness, and 0.1 roundedness in another. that's not 3 floats worth of data, those are all ints over 100. We generally think of people, but not languages, as having deep or squeaky voices. Furthermore we usually have no problem following when the speakergoesfaster or enunciates reaaaallllyy slooowwwlllyy or even ish they slur theirr wurds abit. I don't think a language specific absolute value of vowel tone is a real thing. Rather, what matters is that the direction and magnitude of the shifts in tone are all in consistent proportions relative to each other. When you look at the vocalizations of humans or even other mammals on the spectrogram there's very clearly just simple modulations happening, like moving linearly from a starting frequency to the ending frequency, or at most taking a x^2 curve to get there. When you think about it this makes sense as those sorts of patterns are trivial to implement on the hardware. Just squeeze or release the muscle on the vocal cord harder. You're right in principle any set of frequencies can be the vowels but every language is going to discretize down to a fixed number. So instead of having or needing language specific absolute "vowel frequencies", you can just look for k distinct levels in the individual speakers voice. If you've gotten as far as calculating the spectrogram, everything I described can be easily implemented by applying curve fitting to that spectrum and then thresholding the coefficients
  - notahacker 4y ago
    I think the other thing that matters is that the relative proportions of vowel tones correspond to a word the listener expects to hear given the sentence/situation. A word pronounced so unconventionally it's been turned into a homophone for another word will still generally be understood, even if the listener notices the "wrong" vowel tone. Which is where the 2bn parameters comes in...
    - IIAOPSW 4y ago
      I fully agree that context the semantic content itself primes the listenerrs expectations of the next rabbit. But that sort of language model need not be GPT-level good. It just needs to identify which words/letters are unlikely to be next to each other and which ones they might be mistaken for. An unsophisticated Markov chain could do that. Is this really a 2B parameter problem, or do we do things this way because parameters are currently cheaper than expertise?
TheDong 4y ago
> I will bet 2-digit sums of money that simple preprocessing step would outperform what they've done. Okay. I take this bet. If your simple preprocessing step outperforms whisper on, say, me reading a random wikipedia article, I'll happily send you any 2-digit sum of money (denoted in dollars presumably), or donate it to a charity of your choice.
- IIAOPSW 4y ago
  In retrospect I should have realized the corner I was painting myself here. If I'm right, you just paid bargain basement prices for an algorithm that can replace an expensive 2B parameter ML process. But, if I had offered what the bet is actually worth, it would look like I was trying to price people out of ever taking me up on it. Well, shit.

schoen 4y ago

I think I remember reading that languages can choose any point on the vowel diagram as a canonical phonetic expression of a vowel, even though a given language will only have a small finite inventory of vowels. If this is right, one could interpret it as phonemes being digital (a native speaker can consistently transcribe them using a small number of IPA symbols), but (at least vowel) phones being analog (the typical frequency spectrum appearance typically representing a specific vowel could best be represented by floating-point numbers that are culturally chosen somewhat arbitrary by language evolution). Like /ɪ/ might be 0.82 closeness, 0.23 backness, 0.05 roundedness in some language, but 0.79 closeness, 0.26 backness, and 0.1 roundedness in another. In turn, if that's right, I don't know how much variation there is among native speakers (from a particular region) in production of a given vowel (or how consistently and precisely children can learn these values). Presumably for recognition the brain rounds off somehow to the nearest vowel phoneme, kind of like picking the nearest point on the constellation diagram in digital communications demodulation?

IIAOPSW 4y ago
>a specific vowel could best be represented by floating-point numbers that are culturally chosen somewhat arbitrary by language evolution). Like /ɪ/ might be 0.82 closeness, 0.23 backness, 0.05 roundedness in some language, but 0.79 closeness, 0.26 backness, and 0.1 roundedness in another. that's not 3 floats worth of data, those are all ints over 100. We generally think of people, but not languages, as having deep or squeaky voices. Furthermore we usually have no problem following when the speakergoesfaster or enunciates reaaaallllyy slooowwwlllyy or even ish they slur theirr wurds abit. I don't think a language specific absolute value of vowel tone is a real thing. Rather, what matters is that the direction and magnitude of the shifts in tone are all in consistent proportions relative to each other. When you look at the vocalizations of humans or even other mammals on the spectrogram there's very clearly just simple modulations happening, like moving linearly from a starting frequency to the ending frequency, or at most taking a x^2 curve to get there. When you think about it this makes sense as those sorts of patterns are trivial to implement on the hardware. Just squeeze or release the muscle on the vocal cord harder. You're right in principle any set of frequencies can be the vowels but every language is going to discretize down to a fixed number. So instead of having or needing language specific absolute "vowel frequencies", you can just look for k distinct levels in the individual speakers voice. If you've gotten as far as calculating the spectrogram, everything I described can be easily implemented by applying curve fitting to that spectrum and then thresholding the coefficients
- notahacker 4y ago
  I think the other thing that matters is that the relative proportions of vowel tones correspond to a word the listener expects to hear given the sentence/situation. A word pronounced so unconventionally it's been turned into a homophone for another word will still generally be understood, even if the listener notices the "wrong" vowel tone. Which is where the 2bn parameters comes in...
  - IIAOPSW 4y ago
    I fully agree that context the semantic content itself primes the listenerrs expectations of the next rabbit. But that sort of language model need not be GPT-level good. It just needs to identify which words/letters are unlikely to be next to each other and which ones they might be mistaken for. An unsophisticated Markov chain could do that. Is this really a 2B parameter problem, or do we do things this way because parameters are currently cheaper than expertise?

TheDong 4y ago

> I will bet 2-digit sums of money that simple preprocessing step would outperform what they've done. Okay. I take this bet. If your simple preprocessing step outperforms whisper on, say, me reading a random wikipedia article, I'll happily send you any 2-digit sum of money (denoted in dollars presumably), or donate it to a charity of your choice.

IIAOPSW 4y ago
In retrospect I should have realized the corner I was painting myself here. If I'm right, you just paid bargain basement prices for an algorithm that can replace an expensive 2B parameter ML process. But, if I had offered what the bet is actually worth, it would look like I was trying to price people out of ever taking me up on it. Well, shit.