7 ms·

Classic speech synthesis is interesting, in that relatively simple approaches, produce useful results. Formant synthesis takes relatively simple sounds, and mo

by retrac 2y ago

Classic speech synthesis is interesting, in that relatively simple approaches, produce useful results. Formant synthesis takes relatively simple sounds, and modifies them according to the various distinctions the human speech tract can make. The basic vowel quality can be modelled as two sine waves that change over time. (Nothing more complex than what's needed to generate touch tone dialing tones, basically.) Add a few types of buzzing or clicking noises before or after that for consonants, and you're halfway there. The technique predates computers; it's basically the same technique used by the original voder [1] just under computer control. Join that with algorithms which can translate English into phonetic tokens with relatively high accuracy, and you have speech synthesis. Make the dictionary big enough, add enough finesse, and a few hundred rules about transitioning from phoneme to phoneme, and it's produces relatively understandable speech. Part of me feels that we are losing something, moving away from these classic approaches to AI. It used to be that, to teach a machine how to speak, or translate, the designer of the system had to understand how language worked. Sometimes these models percolated back into broader thinking about language. Formant synthesis ended up being an inspiration to some ideas for how the brain recognizes phonemes. (Or maybe that worked in both directions.) It was thought, further advances would come from better theories about language, better abstractions. Deep learning has produced far better systems than the classic approach, but they also offer little in terms of understanding or simplifying. [1] https://en.wikipedia.org/wiki/Voder https://en.wikipedia.org/wiki/Voder

barfbagginus 2y ago
Quality of life improvements are much much more important than understandable models of speech, so we should live with, appreciate, and work to interpret and improve the current generation of complex neural TTS models. I depend on TTS to overcome dyslexia, but I also struggle with auditory processing disorder that causes me to misunderstand words. As a result , classical TTS does not help me read faster or more accurately than struggling through my dyslexia. It causes me to rapidly fatigue, zone out, and rewind often, in a way that is more severe than when I sight read. On the other hand, modern neural TTS is a huge enabler. My error rate, rewind rate, and fatigue are much better thanks to the natural tone, articulation, and prosody. I'm able to read for hours this way, and my productivity is higher than sight reading alone. This unlocks long and complex readings that I would never complete by sight reading alone, like papers in history, philosophy, and law. Previously I was limited to reading math, computer science, and engineering work, where I heavily depended on diagrams and math formulas to help me gloss over dense text readings. The old tech had no impact on my life, given my combination of reading and listening difficulty, since it was not comparatively better than sight reading. But my life changed about 6 years ago with neural TTS. The improvement has been massive, and has helped me work with many non-technical readings that I would previously give up on. The main issues I see now is not that neural models are hard to understand. For better or worse, we're able to improve the models just by throwing capital and ML PhDs at the problem. The problem I see is that the resulting technology is proprietary and not freely available for the people whose lives it would change. We should work towards a future where people can depend on useful and free TTS that improves their quality of life. I don't think simple synthetic models will be enough. We must work to seize control of models that can provide the same quality of life improvements that new proprietary models can provide. And we must make these models free for everyone to use!
- vidarh 2y ago
  It's not at all a given that these two thing are in conflict. The best path towards free TTS might well turn out to be to identify ways of making smaller models that are cheaper to train and improve on if/when we can split out things we know to do (be it with separate neural models or other methods) and train models to "fill the gaps" instead of the entire end-to-end process. There are also plenty of places where the current modern "neural" models are too compute intensive / costly to run, and so picking just the current big models isn't an option for all uses.
- adam_hn 2y ago
  [dead]
senkora 2y ago
It is possible to synthesize an English voice with a 1.5MB model using http://cmuflite.org/ http://cmuflite.org/ or some of the Apple VoiceOver voices, which is just crazy to me. Most of the model is diphone samples for pairs of phonemes. I don't know of any way to go smaller than that with software. I tried, but it seems like a fundamental limit for English.
- userbinator 2y ago
  I don't know of any way to go smaller than that with software. I tried, but it seems like a fundamental limit for English. If you include "robotic" speech, then there's https://en.wikipedia.org/wiki/Software_Automatic_Mouth https://en.wikipedia.org/wiki/Software_Automatic_Mouth in a few tens of KB, and the demoscene has done similar in around 1/10th that. All formant synths, of course, not the sample-based ones that you're referring to.
  - vidarh 2y ago
    SAM is awful but at the same time tantalisingly close (one of the demos of it apparently draws on a great reverse engineered version by Sebastian Macke and refactoring efforts by a couple of others including me - I spent too many hours listening to SAM output...) - especially when comparing to the still awful Festival/Flite models -, that I keep wanting to see what a better generic formant synth used as constraint on an ML model would produce. That is, instead of allowing a generic machine learning model to output unconstrained audio, train it on the basis of letting it produce low bitrate input/control values for a formant synth instead, and see just how small you can push the model.
    - tapper 2y ago
      Me as a blind dude that is listening to a sinth all the time I don't cair about the size of the thing I just want good SQ.
      - vidarh 2y ago
        That's fair, but those two are in constant tension. We can do very good speech if we ship gigs of samples. Better quality smaller models make it easier to get better quality speech in more places.
bhaney 2y ago
> It used to be that, to teach a machine how to [X], the designer of the system had to understand how [X] worked. It does feel like we're rapidly losing this relationship in general. I think it's going to be a good thing overall for productivity and the advancement of mankind, but it definitely takes a lot of the humanity out of our collective accomplishments. I feel warm and fuzzy when a person goes on a quest to deeply understand a subject and then shares the fruits of their efforts with everyone else, but I don't feel like that when someone points at a subject and says "hey computer, become good at that" with similar end results.
- userbinator 2y ago
  I think it's going to be a good thing overall for productivity and the advancement of mankind, but it definitely takes a lot of the humanity out of our collective accomplishments. I think AI will only cause us to become stuck in another local maximum, since not understanding how something works can only lead to imitation at best, and not inspiration.
  - vidarh 2y ago
    I'm not convinced, because I think there will be a drive to distill down models and constrain them, and try to train models with access to "premade blocks" of functionality we know should help. E.g. we know human voices can be produced well with formant synthesis because we know how the human vocal tract is shaped. So you can "give" a model a formant synth, and try to train smaller models outputting to it. I think there's going to be a whole lot of research possibilities in placing constraints and training smaller models, and even training ensembles of models constrained in how they're interacting and their relative sizes to try to "force" extraction of functionality. E.g. we have reasonable estimates at the lowest bitrate raw audio that produces passable voice. Now consider training two models A, B, where A => B => audio, and the "channel" between A and B is constrained to a small fraction of the bitrate that'd let A do all the work, and where the size of B is set at a level you've first struggled to get passable TTS output from. Try to squeeze the bitrate and/or the size of B down and see if you can get something to emerge where analysing what happens in B is doable.
  - 2y ago
vidarh 2y ago
I feel like you. Relatively simple formant models gets "close enough" that it feels like you should be able to do well with very little. One of the things I've long wanted to do but not found time for, is to take a few different variants of formant synths, and try to train a simple TTS model to control one instead of producing "raw" output. It's amazing what TTS models can do with "raw" output, but we know our brains aren't producing raw, unconstrained digital audio, and so I think there's a lot of potential to understanding more and simplifying if you train models constrained to produce outputs we know ought to be sufficient, and push their size as far down as we can.
- vidarh 2y ago
  Too late to edit, but to any one who needs "convincing" of the flexibility of a formant synthesizer, you should 1) play with Pink Trombone[1], a Javascript formant synthesizer with a UI that lets you graphically manipulate a vocal tract, and 2) have a look at this programmable version of it[2] [1] https://dood.al/pinktrombone/ https://dood.al/pinktrombone/ [2] https://github.com/zakaton/Pink-Trombone https://github.com/zakaton/Pink-Trombone
  - follower 2y ago
    Yeah, Pink Trombone is awesome[0]. :D Thanks for the link to the programmable version--I don't think I'd been aware of that previously... [0] And, from personal experience, also rather difficult to "safely" search for if you don't quite remember its name exactly. :D
  - nuc1e0n 2y ago
    That reminds me of Google's blob opera: https://artsandculture.google.com/experiment/AAHWrq360NcGbw https://artsandculture.google.com/experiment/AAHWrq360NcGbw
  - drcongo 2y ago
    Thanks for those links, that's superb. Sounds surprisingly like the "Oh long Johnson" cat - https://www.youtube.com/watch?v=kkwiQmGWK4c https://www.youtube.com/watch?v=kkwiQmGWK4c
miki123211 2y ago
Do you have any good resources on this? I took a few stabs at understanding Klatt, but I feel like I had far too little DSP, math and linguistic intuitions back then to fully comprehend it, perhaps I should take another one now.
maksimur 2y ago
> The technique predates computers; it's basically the same technique used by the original voder [1] just under computer control. Something similar from the 800s is the Euphonia talking machine ( https://en.m.wikipedia.org/wiki/Euphonia_(device) https://en.m.wikipedia.org/wiki/Euphonia_(device) ).
- nkozyra 2y ago
  * 1800s Clicked that thinking someone had made a talking machine in the Middle Ages :)
  - maksimur 2y ago
    Oops :)
beeboobaa3 2y ago
> Part of me feels that we are losing something, moving away from these classic approaches to AI Absolutely. Seems a large amount of software developers have moved on from trying to understand how things work to solve the problem, and they are now instead just essentially throwing shit at a magical wall until something sticks long enough.
WalterBright 2y ago
Now try it with a trumpet! Herb Alpert's "The Trolley Song" is an underrated masterpiece of control over the sound a trumpet makes. No synthesized trumpet sound has ever done anything like this. https://www.youtube.com/watch?v=mqr9E9Q-P5o https://www.youtube.com/watch?v=mqr9E9Q-P5o
- pcdoodle 2y ago
  Haven't heard that one, it's great, thank you.

by retrac 2y ago

barfbagginus 2y ago

Quality of life improvements are much much more important than understandable models of speech, so we should live with, appreciate, and work to interpret and improve the current generation of complex neural TTS models. I depend on TTS to overcome dyslexia, but I also struggle with auditory processing disorder that causes me to misunderstand words. As a result , classical TTS does not help me read faster or more accurately than struggling through my dyslexia. It causes me to rapidly fatigue, zone out, and rewind often, in a way that is more severe than when I sight read. On the other hand, modern neural TTS is a huge enabler. My error rate, rewind rate, and fatigue are much better thanks to the natural tone, articulation, and prosody. I'm able to read for hours this way, and my productivity is higher than sight reading alone. This unlocks long and complex readings that I would never complete by sight reading alone, like papers in history, philosophy, and law. Previously I was limited to reading math, computer science, and engineering work, where I heavily depended on diagrams and math formulas to help me gloss over dense text readings. The old tech had no impact on my life, given my combination of reading and listening difficulty, since it was not comparatively better than sight reading. But my life changed about 6 years ago with neural TTS. The improvement has been massive, and has helped me work with many non-technical readings that I would previously give up on. The main issues I see now is not that neural models are hard to understand. For better or worse, we're able to improve the models just by throwing capital and ML PhDs at the problem. The problem I see is that the resulting technology is proprietary and not freely available for the people whose lives it would change. We should work towards a future where people can depend on useful and free TTS that improves their quality of life. I don't think simple synthetic models will be enough. We must work to seize control of models that can provide the same quality of life improvements that new proprietary models can provide. And we must make these models free for everyone to use!

vidarh 2y ago
It's not at all a given that these two thing are in conflict. The best path towards free TTS might well turn out to be to identify ways of making smaller models that are cheaper to train and improve on if/when we can split out things we know to do (be it with separate neural models or other methods) and train models to "fill the gaps" instead of the entire end-to-end process. There are also plenty of places where the current modern "neural" models are too compute intensive / costly to run, and so picking just the current big models isn't an option for all uses.
adam_hn 2y ago
[dead]

senkora 2y ago

It is possible to synthesize an English voice with a 1.5MB model using http://cmuflite.org/ http://cmuflite.org/ or some of the Apple VoiceOver voices, which is just crazy to me. Most of the model is diphone samples for pairs of phonemes. I don't know of any way to go smaller than that with software. I tried, but it seems like a fundamental limit for English.

userbinator 2y ago
I don't know of any way to go smaller than that with software. I tried, but it seems like a fundamental limit for English. If you include "robotic" speech, then there's https://en.wikipedia.org/wiki/Software_Automatic_Mouth https://en.wikipedia.org/wiki/Software_Automatic_Mouth in a few tens of KB, and the demoscene has done similar in around 1/10th that. All formant synths, of course, not the sample-based ones that you're referring to.
- vidarh 2y ago
  SAM is awful but at the same time tantalisingly close (one of the demos of it apparently draws on a great reverse engineered version by Sebastian Macke and refactoring efforts by a couple of others including me - I spent too many hours listening to SAM output...) - especially when comparing to the still awful Festival/Flite models -, that I keep wanting to see what a better generic formant synth used as constraint on an ML model would produce. That is, instead of allowing a generic machine learning model to output unconstrained audio, train it on the basis of letting it produce low bitrate input/control values for a formant synth instead, and see just how small you can push the model.
  - tapper 2y ago
    Me as a blind dude that is listening to a sinth all the time I don't cair about the size of the thing I just want good SQ.
    - vidarh 2y ago
      That's fair, but those two are in constant tension. We can do very good speech if we ship gigs of samples. Better quality smaller models make it easier to get better quality speech in more places.

bhaney 2y ago

> It used to be that, to teach a machine how to [X], the designer of the system had to understand how [X] worked. It does feel like we're rapidly losing this relationship in general. I think it's going to be a good thing overall for productivity and the advancement of mankind, but it definitely takes a lot of the humanity out of our collective accomplishments. I feel warm and fuzzy when a person goes on a quest to deeply understand a subject and then shares the fruits of their efforts with everyone else, but I don't feel like that when someone points at a subject and says "hey computer, become good at that" with similar end results.

userbinator 2y ago
I think it's going to be a good thing overall for productivity and the advancement of mankind, but it definitely takes a lot of the humanity out of our collective accomplishments. I think AI will only cause us to become stuck in another local maximum, since not understanding how something works can only lead to imitation at best, and not inspiration.
- vidarh 2y ago
  I'm not convinced, because I think there will be a drive to distill down models and constrain them, and try to train models with access to "premade blocks" of functionality we know should help. E.g. we know human voices can be produced well with formant synthesis because we know how the human vocal tract is shaped. So you can "give" a model a formant synth, and try to train smaller models outputting to it. I think there's going to be a whole lot of research possibilities in placing constraints and training smaller models, and even training ensembles of models constrained in how they're interacting and their relative sizes to try to "force" extraction of functionality. E.g. we have reasonable estimates at the lowest bitrate raw audio that produces passable voice. Now consider training two models A, B, where A => B => audio, and the "channel" between A and B is constrained to a small fraction of the bitrate that'd let A do all the work, and where the size of B is set at a level you've first struggled to get passable TTS output from. Try to squeeze the bitrate and/or the size of B down and see if you can get something to emerge where analysing what happens in B is doable.
- 2y ago

vidarh 2y ago

I feel like you. Relatively simple formant models gets "close enough" that it feels like you should be able to do well with very little. One of the things I've long wanted to do but not found time for, is to take a few different variants of formant synths, and try to train a simple TTS model to control one instead of producing "raw" output. It's amazing what TTS models can do with "raw" output, but we know our brains aren't producing raw, unconstrained digital audio, and so I think there's a lot of potential to understanding more and simplifying if you train models constrained to produce outputs we know ought to be sufficient, and push their size as far down as we can.

vidarh 2y ago
Too late to edit, but to any one who needs "convincing" of the flexibility of a formant synthesizer, you should 1) play with Pink Trombone[1], a Javascript formant synthesizer with a UI that lets you graphically manipulate a vocal tract, and 2) have a look at this programmable version of it[2] [1] https://dood.al/pinktrombone/ https://dood.al/pinktrombone/ [2] https://github.com/zakaton/Pink-Trombone https://github.com/zakaton/Pink-Trombone
- follower 2y ago
  Yeah, Pink Trombone is awesome[0]. :D Thanks for the link to the programmable version--I don't think I'd been aware of that previously... [0] And, from personal experience, also rather difficult to "safely" search for if you don't quite remember its name exactly. :D
- nuc1e0n 2y ago
  That reminds me of Google's blob opera: https://artsandculture.google.com/experiment/AAHWrq360NcGbw https://artsandculture.google.com/experiment/AAHWrq360NcGbw
- drcongo 2y ago
  Thanks for those links, that's superb. Sounds surprisingly like the "Oh long Johnson" cat - https://www.youtube.com/watch?v=kkwiQmGWK4c https://www.youtube.com/watch?v=kkwiQmGWK4c

miki123211 2y ago

Do you have any good resources on this? I took a few stabs at understanding Klatt, but I feel like I had far too little DSP, math and linguistic intuitions back then to fully comprehend it, perhaps I should take another one now.

maksimur 2y ago

> The technique predates computers; it's basically the same technique used by the original voder [1] just under computer control. Something similar from the 800s is the Euphonia talking machine ( https://en.m.wikipedia.org/wiki/Euphonia_(device) https://en.m.wikipedia.org/wiki/Euphonia_(device) ).

nkozyra 2y ago
* 1800s Clicked that thinking someone had made a talking machine in the Middle Ages :)
- maksimur 2y ago
  Oops :)

beeboobaa3 2y ago

> Part of me feels that we are losing something, moving away from these classic approaches to AI Absolutely. Seems a large amount of software developers have moved on from trying to understand how things work to solve the problem, and they are now instead just essentially throwing shit at a magical wall until something sticks long enough.

WalterBright 2y ago

Now try it with a trumpet! Herb Alpert's "The Trolley Song" is an underrated masterpiece of control over the sound a trumpet makes. No synthesized trumpet sound has ever done anything like this. https://www.youtube.com/watch?v=mqr9E9Q-P5o https://www.youtube.com/watch?v=mqr9E9Q-P5o

pcdoodle 2y ago
Haven't heard that one, it's great, thank you.