3 ms·
One thing I always still notice about these "lifelike" speech models is that they still have random pitch variation that wouldn't be present from a real speaker
by derefr 10y ago
One thing I always still notice about these "lifelike" speech models is that they still have random pitch variation that wouldn't be present from a real speaker—sort of a "warbling", similar to audio heavily compressed by some cellular realtime audio codecs.
I assume this is due to random variation in the pitches the speakers in the training data used to say given phonemes. Would this, then, imply that speech models for tonal languages like Mandarin don't share this problem, since people are speaking with prescribed pitches more often?
- chrismealy 10y agoI was wondering why it sometimes sounded like bad compression artifacts! Thanks.
- iainmerrick 10y agoSheer speculation, but I wonder if that could be related to the "checkerboarding" artifacts you almost always see in AI-generated images? Here's a good writeup: http://distill.pub/2016/deconv-checkerboard/ http://distill.pub/2016/deconv-checkerboard/
- derefr 10y agoRe: the checker-boarding artifacts, that sounds interestingly like deconvolutional NNs treat their samples, at each deconv layer, as independent, rather than taking the Rule of Addition of probability-weights (P(A ∪ B) = P(A) + P(B) - P(A ∩ B)) into account. (Would a model that did do this then be a "Bayesian Deconvolutional Neural Network"? Does that even make sense?) I doubt this would result in audible artifacts at the audio level, though. The visual checkerboarding appears at the 1px detail-level of the output image—this would translate more to minor fluctuations along an audio sample's Nyquist frequency, which is effectively undetectable if the output sampling rate is of the usual kind (44.1kHz).
- wjnc 10y agoI always wonder why they don't program a second pass over the output with a model, say, trained on radio conversations. That second model just for audio 'quality'.
- aikinai 10y agoI'm not sure about Mandarin, but this is definitely true for Japanese. Near-human Japanese text-to-speech has been around a very long time.
- Zikes 10y agoSo good they can even sing, and so well that they create entirely virtual pop stars: https://en.wikipedia.org/wiki/Vocaloid https://en.wikipedia.org/wiki/Vocaloid
- dkersten 10y agoVocaloid still sounds robotic (at least, to me). It just so happens when you make autotune/melodyne'd pop songs, it really doesn't matter.
- kalleboo 10y agoVocaloid isn't automated though, it's basically an instrument that humans "play". The software has specific controls for pitch, aspiration, etc
- aninhumer 10y agoTrue, but Japanese isn't really a tonal language, so it doesn't really evidence the parent's point.
- aikinai 10y agoJapanese isn't technically "tonal" but does have "prescribed pitches" which certainly affects the ease of text-to-speech.
- hacker_9 10y agoAdditionally when we speak we don't just 'replay' words; we add emotions and so on. Reminds me of this video [1] analysing how the actor Anthony Hopkins converts his lines to speech for his scenes. [1] https://www.youtube.com/watch?v=4kSGkGKwp9U https://www.youtube.com/watch?v=4kSGkGKwp9U
- dkersten 10y agoI guess you could do something like vocaloid does to better "shape" the sounds of each word. I guess its pretty tough to do that automatically though, since words by themselves don't convey the emotions that they should be portrayed with. I mean, a word or phrase could very much change meaning depending on how its spoken (excitement, disgust, fear, sarcasm etc) and even if the context is available in the text (as is often the case), its rather difficult to machine-detect, at least, for now.
- angstrom 10y agoThe spoken word has so many nuances. Cadence, inflexion, tone, and then the facial performances are never captured by pure text to speech. You'd almost have to be able to set boundaries for seriousness, playfulness, innuendo, indifference, sarcasm etc because none of that is conveyed in the literal text without additional descriptions or contextual analysis of the conversation.
- derefr 10y agoIt'd be interesting to take the Joint Many-Tasks approach here, training some of an RNN's layers on text-to-speech, and then another set of layers on sentiment analysis of speech-audio, where the error of the sentiment layer can backpropagate into the TTS layer.
- rckclmbr 10y agoSoutherns have an accent. New Jerseyans have an accent. British have an accent. Can't computers have an accent, too?
- TazeTSchnitzel 10y agoWouldn't we describe what TTS systems do as more of a speech impediment? Sure they use a particular accent, usually that of a standard prestige dialect. But it's not the accent that's weird.
- toephu2 10y agoChinese (Mandarin) text to speech still shares the problem of not sounding natural (sounds robotic), but it is a little better than English. Example (click on the blue speaker icon): http://dict.baidu.com/s?wd=%E7%88%B1%E5%B1%8B%E5%8F%8A%E4%B9%8C&device=pc&from=home&q=%E7%88%B1%E5%B1%8B%E5%8F%8A%E4%B9%8C http://dict.baidu.com/s?wd=%E7%88%B1%E5%B1%8B%E5%8F%8A%E4%B9...
- jldugger 10y agoIf you grab the training set for CMU-Sphinx, you'll see that the voice actors providing base input are trained to speak that way. At a secondary level, I imagine there's a sort of pitch trajectory for a sentence, and if a word or syllable ends too far off that trajectory, the return to the pitch trajectory is fast enough to be easily detectable.
- mwcampbell 10y agoI don't know about tonal languages. But you're definitely right about the artifacts of concatenative speech synthesis. It's worth noting that purely synthetic speech, while lacking the superficial lifelike quality, doesn't have these little inconsistencies. Perhaps that's one reason why many blind people, particularly power users, prefer pure synthetic text-to-speech engines, such as ETI-Eloquence (commercial) and eSpeak (open source). These systems especially sound better at high speeds than the newer "lifelike" ones. Just for fun, here's a side-by-side comparison between a concatenative system, IVONA (acquired by Amazon): http://mwcampbell.us/tmp/derefr_comment_ivona.mp3 http://mwcampbell.us/tmp/derefr_comment_ivona.mp3 And a purely synthetic system, Eloquence: http://mwcampbell.us/tmp/derefr_comment_eloquence.mp3 http://mwcampbell.us/tmp/derefr_comment_eloquence.mp3 The latter is the text-to-speech engine I use every day, and that clip was synthesized at the speaking rate I normally use. Edit: It appears the word I was looking for when I said "purely synthetic" is "parametric".
- derefr 10y agoHow do you feel about the WaveNet results, by comparison? • https://storage.googleapis.com/deepmind-media/pixie/us-english/wavenet-1.wav https://storage.googleapis.com/deepmind-media/pixie/us-engli... • https://storage.googleapis.com/deepmind-media/pixie/us-english/wavenet-2.wav https://storage.googleapis.com/deepmind-media/pixie/us-engli...
- mwcampbell 10y agoThose phrases sound very natural indeed. But are they carefully selected demo phrases, for which the system might have been tuned? That's always a danger with marketing demos. So I have two questions: 1. How does that engine sound when reading arbitrary text at high speed? 2. Can I run it on my own computer, so I can have low latency in my screen reader? (That, of course, is a problem with Amazon Polly as well.)