3 ms·
I'm the author of FakeYou.com, so I have a little experience in this area. (We used to train GlowTTS models ourselves before turning it over to our users, which
by echelon 5y ago
I'm the author of FakeYou.com, so I have a little experience in this area. (We used to train GlowTTS models ourselves before turning it over to our users, which has had mixed results in terms of quality.)
This appears to be a repackaging of RealTimeVoiceCloning [1], albeit with a few additions, such as GSTs.
No matter what the repo claims, your results will depend on high quality data. Lots of it, and with ample fine tuning. Demo videos are absolutely cherry picked.
If you're picking this up for a project, HiFi-Gan is pretty much the best vocoder right now. Tacotron still produces great results, though there are lots of other interesting model architectures.
[1] https://github.com/CorentinJ/Real-Time-Voice-Cloning https://github.com/CorentinJ/Real-Time-Voice-Cloning
- kragen 5y agoBy "lots of [high quality data]" do you mean seconds, kiloseconds, or megaseconds of high-quality voice recordings?
- oefrha 5y agoIIRC enterprise solutions from the big clouds usually ask for at least hours of studio quality voice recordings for a custom voice model.
- kragen 5y agoThanks! Do you think they're using models as good as the ones echelon uses at FakeYou? Maybe he can get by with less data.
- echelon 5y agoThe best sounding limited data models have at least thirty minutes of audio data and have a similar pitch and timbre to a base data set. You can get by with less, but it requires finesse.
- iaml 5y agoThis paper[0] from this year seems to make do with a couple of minutes. [0] https://davidyao.me/projects/text2vid/ https://davidyao.me/projects/text2vid/
- echelon 5y agoHours of audio comprised of clean spoken sentences, zero noise, and uniform microphone quality is ideal. Some of the predominant base data sets used for transfer learning, such as LJSpeech [1], are unfortunately noisy and non-uniform. [1] https://keithito.com/LJ-Speech-Dataset/ https://keithito.com/LJ-Speech-Dataset/
- kragen 5y agoThank you very much!
- weinzierl 5y agoBy any chance, do you know which technology was used for the TikTok voice? While the voice itself is somewhat annoying I find the quality stunning. Any chance to reach this level with any of the models you mentioned?
- Gigachad 5y agoIs the quality that good? It seemed to me that text to speech was not a particularly hard problem when you can get someone to read out all the sounds of the language. And that the new advancement we have now is just being able to dump our normal speech audio and build a model based on that.
- weinzierl 5y agoNot sure. So far, I found every text to speech application quite uncanny. The ones that come with Windows, MacOS, Android, iOS are good, but not quite there. The TikTok one is the first to sound quite convincing for me. Siri and Alexa are also good but I think they don't count, because they probably are not purely text to speech but presumably use a lot of prerecorded phrases, especially for the common answers.
- JohnHaugeland 5y agoi actively dislike the tiktok voice. i find the microsoft and google cloud voices to be much better.
- weinzierl 5y agoYou are not alone. Especially when it is used in contexts that do not fit its overly enthusiastic upbeat mood the results range from slightly comical to deeply unsettling. Still, in my opinion it is really good from a technical point of view.
- pdamoc 5y agoLong ago I found an approach to 3D modeling [1] that used a morphable model that was then morphed into the desired shape. Would something like this be possible for voice? A voice model obtained from a gigantic set of samples, that can be manually tuned to sound more masculine/feminine, higher/lower pitched and that can be morphed into the timbre of various samples. - [1] https://www.youtube.com/watch?v=pSRA8GpWIrA https://www.youtube.com/watch?v=pSRA8GpWIrA