12 ms·
Tacotron: A Fully End-to-End Text-To-Speech Synthesis Model
- lucidrains 10y agocan we eventually apply style transfer to speech? I'd like to hear all audio narration in the voice of David Attenborough.
- georgiev 10y agoOr Stephen Fry
- avmich 10y agoOr Bill Kurtis? :)
- visarga 10y agoMaybe not yet, but I am sure there will be speaker cloning extension soon, like photo and painting style transfer. Edit: It would seem they only need 25 hours of audio to train the model, so it would probably be easy to find or make alternate datasets. > We train Tacotron on an internal North American English dataset, which contains about 24.6 hours of speech data spoken by a professional female speaker
- crooked-v 10y agoComedy option: Gilbert Gottfried.
- slinger 10y agoI was wondering the same thing! I think, in the future this might be a problem for national security
- abdias 10y agoYou mean, like they did pre-2000 [1]? Don't know how good this would sound today, but nevertheless interesting. [1] http://www.washingtonpost.com/wp-srv/national/dotmil/arkin020199.htm http://www.washingtonpost.com/wp-srv/national/dotmil/arkin02...
- starmole 10y agoAdobe VoCo does something like that. Demo: https://www.youtube.com/watch?v=I3l4XLZ59iw https://www.youtube.com/watch?v=I3l4XLZ59iw
- dharma1 10y agoIt's not really style transfer, but for a new speaker model, you just need to train each speaker with a dataset of 25 hours audio with time matched accurate transcriptions. In the case of David Attenborough, I'm pretty sure that amount of data is available with subtitled BBC content.
- dnautics 10y agoOh I doubt that will be necessary. I bet you could find a useful "feature vector" by repeating the process on about 10-20 new voices - then extract a set of words (let's pessimistically say 100) that activate these features the most - then have a person retrain with 5 examples of each of these 100 words (an hour, at the most). I honestly wouldn't be surprised if you could find a really feature rich sentence (say 5 words long) that you could use to crack pretty much all voice-activated biometric password systems.
- slinger 10y agoIs voice-activated password a thing? I mean, I only saw that in sci-fi movies and never in real life.
- dharma1 10y agoYes, voice biometrics are used in telephone banking for instance
- dharma1 10y agoyep, I'm sure there would be a lot of similarity between models that could be used to reduce training time
- gwern 10y agoMy understanding is that while a number of people have tried, we currently have no particularly viable approaches for 'style transfer' analogues on sequence data, including text and sound/speech. Samples from past work are pretty bad. But what you could possibly do is roundtrip audio to text and then text (with conditioning like speaker metadata) to speech, both of which are doable. But directly transforming, not yet.
- visarga 10y agoPaper link here: https://arxiv.org/abs/1703.10135 https://arxiv.org/abs/1703.10135
- slinger 10y agoThis is really awesome! Anyone has any resources on learning TTS synthesis?
- hiddencost 10y agoJurafsky & Martin is the canonical textbook but it's pre-deep learning. Check out the wavenet paper[0], and Alan Black's Festvox[1]. [0] https://deepmind.com/blog/wavenet-generative-model-raw-audio/ https://deepmind.com/blog/wavenet-generative-model-raw-audio... [1] http://www.festvox.org/ http://www.festvox.org/
- NKCSS 10y agoI've used AT&T's Natural Voices SDK about 10-15 years ago? It allowed you to create your own voices, but it's a hell of a lot of work.
- modeless 10y agoThese new models are replacing almost everything that's been developed for TTS in the last 30 years with neural nets. So you don't need to study TTS anymore. Study neural nets and then read this paper and you will understand the state of the art in TTS.
- rws 10y agoThis is complete nonsense. You need to understand the problem you are addressing even if you plan to use deep learning methods for them. Saying you don't need to know anything about TTS is like developing a self-driving car and saying you don't need to know anything about the rules of the road. For one thing, how are you supposed to know when you got it wrong? And what things it is important to get right?
- ayumu722 10y agothis book is a classic in the field : http://svr-www.eng.cam.ac.uk/~pat40/ttsbook_draft_2.pdf http://svr-www.eng.cam.ac.uk/~pat40/ttsbook_draft_2.pdf
- jnpatel 10y agoI've been a long-time user for OS X's built in text-to-speech [0]. IMO, it actually performs pretty robustly on these examples. Is Apple still using just diphones, or are they post-processing in some way? [0]: https://en.wikipedia.org/wiki/PlainTalk#Text-to-speech_in_Mac_OS_X https://en.wikipedia.org/wiki/PlainTalk#Text-to-speech_in_Ma...
- visarga 10y agoI'm using Alex to read articles and comments. I love the Alex voice but it sometimes makes stupid mistakes. I hope we will have decent TTS for linux too.
- krackers 10y agoFyi they added newer, (imo) better quality voices since alex debuted in 10.5. My personal favorite is Kate (english UK voice) and Tessa (english - south africa)
- zachrose 10y agoThe stress and intonation examples are what puts this on the far side of the uncanny valley. I've not heard computer speech like this, where the voice seems understand the idea underneath the words.
- max_ 10y agoHow is this different from the DeepMind's [0]WaveNet? [0]https://deepmind.com/blog/wavenet-generative-model-raw-audio/ https://deepmind.com/blog/wavenet-generative-model-raw-audio...
- stevenhuang 10y agoQuoting the paper, in the section related works: "WaveNet (van den Oord et al., 2016) is a powerful generative model of audio. It works well for TTS, but is slow due to its sample-level autoregressive nature. It also requires conditioning on linguistic features from an existing TTS frontend, and thus is not end-to-end: it only replaces the vocoder and acoustic model. Another recently-developed neural model is DeepVoice (Arik et al., 2017), which replaces every component in a typical TTS pipeline by a corresponding neural network. However, each component is independently trained, and it’s nontrivial to change the system to train in an end-to-end fashion."
- red75prime 10y agoIt is without doubt very impressive, but on longer texts the lack of understanding will be apparent. Wrong semantic stresses, incorrect pronunciation of homographs, maybe inconsistent pronunciation of names, and so on. Perfect speech synthesis from unannotated text will require much more complex system. It will need database of common knowledge in some form, ability to relate text to that database, ability to retain and use this semantic information to guide synthesis. That is, it should be a step closer to artificial general intelligence.
- jerf 10y agoI'd say even in these samples there's still some errors. The intonation on "brown" in "quick brown fox" is still wrong. "AllHipHop" is not pronounced as a human would, though I'll grant you that's a made-up word so I wouldn't even expect a non-native human speaker to necessarily get that one right. Still, the quality is such that I'm only observing this in the spirit of scientific analysis. It's some good quality.
- akhilcacharya 10y agoBeing able to pronounce Taleb Kweli is blowing my mind. Absolutely amazing, can't wait to see where this goes.
- oliyoung 10y agoIt doesn't, it's Tah-leb Kwah-lee, that sounds more Ta-Leb Kwell-ee But damn, the rest of it is astounding
- literallycancer 10y agoIt sounds good to me. Not sure why you'd want to pronounce it as Tah-leb Kwah-lee? If anything, it sounds like "quiley".
- lenocinor 10y agoNot trying to sound snarky here, but that's how he actually pronounces his name.
- cypher543 10y agoNo source code, as usual. How difficult would it be to duplicate these results with TensorFlow? Would something like this require more than the building blocks that TF and other toolkits provide? I have zero experience with machine learning, so I'm just curious.
- bbayer 10y agoI think implementation can be done with TensorFlow but hardest part is to provide relevant training set to model.
- yorwba 10y agoThey mention implementing the model in TensorFlow, so it should absolutely be possible. (Unless they are using a special Google-only version.) One thing that might make reproduction difficult is that they don't describe their initialization procedure. But maybe it just doesn't matter.
- ganfortran 10y ago> How difficult Very difficult: 1.Machine learning is pretty data dependent, and make those datasets are very expensive. Google is not likely to give them away for free, because it is their competitive advantage. 2.The infrastructure to train those models are hard to get outside of Google. Pretty sure it is 10s or 100s of GPUs, with Infinity Band connected PS server, running for days and weeks. Even with source code published, people will still have to scratch their head to duplicate Google's performance. Until the day, some equivalent organization as GNU that democratize data access to the public and some mighty algorithm being discovered dramatically reduced the computational requirement for training those models, Google succeeds by just being Google is unlikely going to change.
- crawfordcomeaux 10y agoWhat would happen if trained on 2 or 3 voices for the same amount of time?
- backpropaganda 10y agoThe model is smart enough to not mix them together in the same sample. Different samples would have different voices, unless you put in a voice identifier as input to the model. Then you can control which voice to generate with.
- alnitak 10y agoThis model sounds so cool. Any idea of a high level software package where you can feed pairs of <text, audio> and build a TTS model as a result?
- backpropaganda 10y agoThis research just came out, quite literally, like today. There will be a tensorflow implementation tomorrow though.
- alnitak 10y agoDo you have an update on this topic?
- mnemotronic 10y agoI would like an on-line version that can generate mp3 from supplied text. In English using a British accent. Female. I have my reasons.
- baq 10y agocouple this with a system that transforms pictures of people into nude pictures of the same people. win $$$. watch civilization crumble to dust.
- dr_zoidberg 10y agoMake it also work in realtime over video and add VR capabilities. Win lots of money. Watch civilization crumble to dust.
- knicholes 10y agoThere's a reddit forum that has pictures of on/off. Maybe given enough of those as the training data, one could create such a thing.
- DonaldFisk 10y agoScottish, Geordie, Liverpudlian, Yorkshire, Birmingham, Welsh, Somerset, Cornwall, East Angular, Estuary, Cockney, or from somewhere else?
- kpil 10y agoPosh!
- yunolisten 10y agoI'm totally new to tts, so please forgive the questions: * Is this open source? * What hardware would one need to run this? * If specialist hardware is required (e.g. reasonably high end GPU) would that simply be for building and training the system or its operation?
- asadjb 10y agoFrom what I understand looking through the paper, this is only a research paper, without any source code. So right now it's not possible to say what the hardware requirement might be.
- yunolisten 10y agoThank you. It would have made more sense to me if the source code to generate the output examples which they posted. Oh well.
- thom 10y agoFeels like we're getting breakthroughs every few months now. How close are we to the point yet that a mobile phone can generate speech in close to realtime that is reasonably pleasant to listen to for long periods?
- pharrlax 10y agoAudio book piracy is gonna skyrocket when you can generate a pleasant track from text.
- est 10y agoYou can model your face and your voice in future RPG games.
- thom 10y agoI just think there are huge opportunities for new types of UI once it works. Any time I'm out and about with headphones, I want something like a custom radio station, mixing up my music and podcasts, dictating messages and highlights from social media or news sites I like. I'd even accept short adverts if it was genuinely relevant and location aware. Even when I'm not travelling, I suspect it'd be a cool interface to have at work. If something can check in from time to time between music tracks, giving me a quick run down of anything interesting that's happening (even plugins for the build server etc), it will cut down on my temptation to open it up in a browser tab. I've wanted to build this app just for myself for years, but it's only very recently that it seems the state of the art is close enough for it to be palatable to users.
- Someone 10y agoHuh? I would think, if anything, it would go down if, as an alternative to pirating an audio book, one can download the text from Gutenberg.org or pirate the _text_, and generate a decent audio book from it.
- idle_zealot 10y agoI think that's what parent meant, it was just phrased oddly.
- m1el 10y agoQuestion: why do people generate waveforms by neural networks? Would it be easier for a neural network to "control" an emulated throat/mouth and train against the sound output? Emitting and post-processing waveforms entirely in neural networks seems like putting a lot of responsibilities into neural networks.
- svantana 10y agoThe reason is, the only speech data that is available in high quality and large quantities is in audio form. There has been research for decades on measuring or estimating the properties of the speech organ (oral cavity geometry etc), but that is largely an unsolved model inversion problem. Existing physical speech synthesis models are pretty crude and sound quite robotic.
- m1el 10y ago> Existing physical speech synthesis models are pretty crude and sound quite robotic. Could you provide some links/publications please?
- jhegedus42 10y agoI like that you mentioned "inversion". ML is strongly related to solving inverse problems but it does not tend to be emphasised enough, in my opinion. In reality, speech organs (a relatively low dimensional system) generate high dimensional data (speech). This is an example of the so called curse of dimensionality. What ML does is to invert this process and create a model of the generation process. My favorite example is quantum mechanics. Physicists made lot of measurements and created a model (quantum mechanics) that reproduces those measurements (and even works on test data, not only on training data). This is an example of a solution of an inverse problem. ML does the same. So in one way or an other, every working TTS has a model that accurately describes the speech organs and how the brain uses them to create speech from text.
- superkuh 10y agoUnless all these fancy new text to speech engines are released as actual software instead of cloud services they're worthless. And we all know that isn't going to happen.
- redsummer 10y agoWhere is the code?
- deleted 10y ago[deleted]
- jxramos 10y agoI'd really love this to mature completely and applied to Kindle Whispersync books. I've found that some readers of audiobooks make the book hard to comprehend or even distracting, especially if the sex of the person does not match the author or even remotely the sound of the author who I may be very familiar with from video interviews and what not. Ideally every author would be a great reader and create recordings for their own content, but if some synthetic means could achieve an approximate result that would be fantastic. Even better people could choose from a flavor of voices and settle upon the one that works for them.
- wishinghand 10y agoI'd love to see it take on this poem: http://www.thepoke.co.uk/2011/12/23/english-pronunciation/ http://www.thepoke.co.uk/2011/12/23/english-pronunciation/
- NHern031 10y agoThat was a great poem, storying that one away for later. It truly shows the complexity of the English language.