4 ms·

Varying Speaking Styles with Neural Text-to-Speech

_underflow_ 8y ago
Whicker Island - https://youtu.be/Cn8Pua5rhj4 https://youtu.be/Cn8Pua5rhj4 (Monty Python)
mrleiter 8y ago
Before Charlie Brooker started producing Black Mirror, he already was a highly observant critic of society and media. At one point he produced a generic news snippet of an unspecified event, where he pointed out all the standard videoshots and animations used these days to report on a topic [1]. He presented this "report" with the standard news-speak pronunciation of sentences. With this achievement of Amazon, and the recent development by China in AI news-anchors [2], one is not far off to see news reporting being done mostly by machines. Yes, some videography must still be done manually, but the rest could be automated. What that means for the TV news industry? I don't know, but it certainly deserves discussion. [1] https://www.youtube.com/watch?v=aHun58mz3vI https://www.youtube.com/watch?v=aHun58mz3vI [2] https://www.fastcompany.com/90264587/watch-chinas-new-ai-anchor-read-the-news https://www.fastcompany.com/90264587/watch-chinas-new-ai-anc...
- richrichardsson 8y ago
  I wish they would commission Newswipe again, was one of the best shows on BBC.
- devoply 8y ago
  It means everyone can have an RT or Alex Jones sort of news channel. None of this stuff of replacing people with robots is positive, it's just wishful thinking that any of this stuff in the long run is good for society or people in general.
crivlaldo 8y ago
Is it Alexa only feature? When it will be available? Should we expect this feature in Polly? Is Amazon going to provide other styles? I'm curious.
- superasn 8y ago
  Yes I too was hoping this would be available for polly like wavenet is from Google
Rebelgecko 8y ago
Interesting stuff. I noticed a few weeks ago that Google Translate will sometimes vary the pronunciation of words when you hit the audio playback button repeatedly, I wonder if they're already doing something similar.
metildaa 8y ago
For those lolking for something not hosted by a megacorp, check out Mozilla's Text to Speech: https://github.com/mozilla/TTS/blob/master/README.md https://github.com/mozilla/TTS/blob/master/README.md Audio Samples: https://soundcloud.com/user-565970875 https://soundcloud.com/user-565970875 If your interested in helping improve this, spread the wors about the Common Voice project by Mozilla, a public speech corpus that is easy to help improve, which makes high caliber TTS and transcription possible ouside of walled gardens: https://voice.mozilla.org https://voice.mozilla.org Mozilla DeepSpeech also does Speech to Text surprisingly well: https://github.com/mozilla/DeepSpeech https://github.com/mozilla/DeepSpeech
- kastnerkyle 8y ago
  Worth noting that a big chunk of the core TTS code here is built on tools from other researchers like Ryuichi Yamamoto and Keith Ito, and they have great implementations to check out as well. The best quality I have heard in OSS is probably [1] from Ryuichi using the Tacotron 2 implementation of Rayhane Mamah, which is loosely what NVidia based some of their baseline code on recently as well [3][4]. There's also a colab notebook for this stuff, so you can try it directly without any pain https://colab.research.google.com/github/r9y9/Colaboratory/blob/master/Tacotron2_and_WaveNet_text_to_speech_demo.ipynb https://colab.research.google.com/github/r9y9/Colaboratory/b... I also have my own pipeline for this (using some utilities from the above authors + a lot of my own hacks), for a forthcoming paper release here https://github.com/kastnerkyle/representation_mixing/tree/master/pretrained https://github.com/kastnerkyle/representation_mixing/tree/ma... , see the minimal demo. It has pretty fast sampling, but the audio quality is not as high as WaveNet. I'd really like to tie in with WaveGlow, but it's work in progress for me so far. NOTE: None of these have voice adaptivity per se, but given a model which trains well already + a multispeaker dataset with IDs such as VCTK, a lot of things become possible as getting a baseline model and data pipeline for TTS is quite difficult. [0] https://github.com/keithito/tacotron https://github.com/keithito/tacotron [1] https://r9y9.github.io/blog/2018/05/20/tacotron2/ https://r9y9.github.io/blog/2018/05/20/tacotron2/ [2] https://github.com/Rayhane-mamah/Tacotron-2 https://github.com/Rayhane-mamah/Tacotron-2 [3] https://github.com/NVIDIA/waveglow https://github.com/NVIDIA/waveglow [4] https://github.com/NVIDIA/tacotron2 https://github.com/NVIDIA/tacotron2
kastnerkyle 8y ago
This type of concatenation I first saw in Alex Graves' work on "Generating Sequences With Recurrent Neural Networks", including his unpublished TTS demo [1]. Biasing with part of another sentence (as in handwriting) can possibly improve style in TTS as well. We followed this approach in char2wav [2], but "voice cloning" has come much farther in my opinion [3][4][5]. There's a lot of relevant research on techniques for this beyond concatenating indicators or embeddings, if people are interested in the research side of this technology. [0] https://arxiv.org/abs/1308.0850 https://arxiv.org/abs/1308.0850 [1] https://www.youtube.com/watch?v=-yX1SYeDHbg&t=38m30s https://www.youtube.com/watch?v=-yX1SYeDHbg&t=38m30s [2] http://josesotelo.com/speechsynthesis/ http://josesotelo.com/speechsynthesis/ [3] https://twitter.com/Jeanne_Heo/status/972089715225542657 https://twitter.com/Jeanne_Heo/status/972089715225542657 (lyrebird.ai) [4] https://google.github.io/tacotron/publications/gmvae_controllable_tts/#multispk_en.sample https://google.github.io/tacotron/publications/gmvae_control... [5] https://google.github.io/tacotron/publications/speaker_adaptation/index.html https://google.github.io/tacotron/publications/speaker_adapt...
ai_ja_nai 8y ago
Totally looks like Tacotron. I heard there was interest in Amazon in buying the project, now it looks like they copied it.

Whicker Island - https://youtu.be/Cn8Pua5rhj4 https://youtu.be/Cn8Pua5rhj4 (Monty Python)

Before Charlie Brooker started producing Black Mirror, he already was a highly observant critic of society and media. At one point he produced a generic news snippet of an unspecified event, where he pointed out all the standard videoshots and animations used these days to report on a topic [1]. He presented this "report" with the standard news-speak pronunciation of sentences. With this achievement of Amazon, and the recent development by China in AI news-anchors [2], one is not far off to see news reporting being done mostly by machines. Yes, some videography must still be done manually, but the rest could be automated. What that means for the TV news industry? I don't know, but it certainly deserves discussion. [1] https://www.youtube.com/watch?v=aHun58mz3vI https://www.youtube.com/watch?v=aHun58mz3vI [2] https://www.fastcompany.com/90264587/watch-chinas-new-ai-anchor-read-the-news https://www.fastcompany.com/90264587/watch-chinas-new-ai-anc...

richrichardsson 8y ago
I wish they would commission Newswipe again, was one of the best shows on BBC.
devoply 8y ago
It means everyone can have an RT or Alex Jones sort of news channel. None of this stuff of replacing people with robots is positive, it's just wishful thinking that any of this stuff in the long run is good for society or people in general.

crivlaldo 8y ago

Is it Alexa only feature? When it will be available? Should we expect this feature in Polly? Is Amazon going to provide other styles? I'm curious.

superasn 8y ago
Yes I too was hoping this would be available for polly like wavenet is from Google

Rebelgecko 8y ago

Interesting stuff. I noticed a few weeks ago that Google Translate will sometimes vary the pronunciation of words when you hit the audio playback button repeatedly, I wonder if they're already doing something similar.

metildaa 8y ago

For those lolking for something not hosted by a megacorp, check out Mozilla's Text to Speech: https://github.com/mozilla/TTS/blob/master/README.md https://github.com/mozilla/TTS/blob/master/README.md Audio Samples: https://soundcloud.com/user-565970875 https://soundcloud.com/user-565970875 If your interested in helping improve this, spread the wors about the Common Voice project by Mozilla, a public speech corpus that is easy to help improve, which makes high caliber TTS and transcription possible ouside of walled gardens: https://voice.mozilla.org https://voice.mozilla.org Mozilla DeepSpeech also does Speech to Text surprisingly well: https://github.com/mozilla/DeepSpeech https://github.com/mozilla/DeepSpeech

kastnerkyle 8y ago
Worth noting that a big chunk of the core TTS code here is built on tools from other researchers like Ryuichi Yamamoto and Keith Ito, and they have great implementations to check out as well. The best quality I have heard in OSS is probably [1] from Ryuichi using the Tacotron 2 implementation of Rayhane Mamah, which is loosely what NVidia based some of their baseline code on recently as well [3][4]. There's also a colab notebook for this stuff, so you can try it directly without any pain https://colab.research.google.com/github/r9y9/Colaboratory/blob/master/Tacotron2_and_WaveNet_text_to_speech_demo.ipynb https://colab.research.google.com/github/r9y9/Colaboratory/b... I also have my own pipeline for this (using some utilities from the above authors + a lot of my own hacks), for a forthcoming paper release here https://github.com/kastnerkyle/representation_mixing/tree/master/pretrained https://github.com/kastnerkyle/representation_mixing/tree/ma... , see the minimal demo. It has pretty fast sampling, but the audio quality is not as high as WaveNet. I'd really like to tie in with WaveGlow, but it's work in progress for me so far. NOTE: None of these have voice adaptivity per se, but given a model which trains well already + a multispeaker dataset with IDs such as VCTK, a lot of things become possible as getting a baseline model and data pipeline for TTS is quite difficult. [0] https://github.com/keithito/tacotron https://github.com/keithito/tacotron [1] https://r9y9.github.io/blog/2018/05/20/tacotron2/ https://r9y9.github.io/blog/2018/05/20/tacotron2/ [2] https://github.com/Rayhane-mamah/Tacotron-2 https://github.com/Rayhane-mamah/Tacotron-2 [3] https://github.com/NVIDIA/waveglow https://github.com/NVIDIA/waveglow [4] https://github.com/NVIDIA/tacotron2 https://github.com/NVIDIA/tacotron2

kastnerkyle 8y ago

This type of concatenation I first saw in Alex Graves' work on "Generating Sequences With Recurrent Neural Networks", including his unpublished TTS demo [1]. Biasing with part of another sentence (as in handwriting) can possibly improve style in TTS as well. We followed this approach in char2wav [2], but "voice cloning" has come much farther in my opinion [3][4][5]. There's a lot of relevant research on techniques for this beyond concatenating indicators or embeddings, if people are interested in the research side of this technology. [0] https://arxiv.org/abs/1308.0850 https://arxiv.org/abs/1308.0850 [1] https://www.youtube.com/watch?v=-yX1SYeDHbg&t=38m30s https://www.youtube.com/watch?v=-yX1SYeDHbg&t=38m30s [2] http://josesotelo.com/speechsynthesis/ http://josesotelo.com/speechsynthesis/ [3] https://twitter.com/Jeanne_Heo/status/972089715225542657 https://twitter.com/Jeanne_Heo/status/972089715225542657 (lyrebird.ai) [4] https://google.github.io/tacotron/publications/gmvae_controllable_tts/#multispk_en.sample https://google.github.io/tacotron/publications/gmvae_control... [5] https://google.github.io/tacotron/publications/speaker_adaptation/index.html https://google.github.io/tacotron/publications/speaker_adapt...

ai_ja_nai 8y ago

Totally looks like Tacotron. I heard there was interest in Amazon in buying the project, now it looks like they copied it.