8 ms·
Mocking Bird – Realtime Voice Clone for Chinese
- Agingcoder 5y agoThis is very interesting. People who have had a laryngectomy can't speak post-operation, and being able to say something with their 'real' voice would be psychologically helpful.
- unbanned 5y agoWould it? Unless the sound is produced from inside their body in the same manner as natural, no it will sound weird just like playing back your own voice.
- Agingcoder 5y agoNot necessarily - people close to me were complaining about what they saw as losing a part of their identity ( and grandchildren definitely were not happy about the change). Even for those who learn to speak with an oesophagal voice, being able to real time map it to something more natural would be great. To be more specific, they consider themselves to be 'voice mutilated', even when able to speak, so I guess that anything closer to their original voice would be a net benefit.
- JohnHaugeland 5y agoI suspect that it would be immensely better to sound sort of like yourself than like a standard text to speech engine, and those are used extensively
- goblin89 5y agoSupports only Mandarin, so description is slightly disingenuous.
- fortran77 5y ago12% of the world speaks Mandarin, so it's a good first choice!
- goblin89 5y agoThis is not about some sort of language leaderboard. If you see the title (as phrased) written in %language%, you’d expect the proposed thing to work in that language. Anyway, as someone else pointed out it’s a fork, so it’s understandable if docs weren’t fully updated from upstream.
- jsilence 5y agoIf the title were written in Mandarin I would have no clue at all what this software is about. I too was slightly disappointed to find in the repo that this version is for Mandarin, but the amount of life time lost by clicking on a Github link does not make me want to complain about it. I got over it pretty quickly.
- goblin89 5y agoSpeak for yourself if you experienced disappointment. I just clicked through, looked at it and left a note that technical merits aside the title is technically clickbait, that’s all.
- drekk 5y agoIt's forked from an English version
- Asraelite 5y agoFor anyone else wondering, the original title was "Clone a voice in 5 seconds to generate arbitrary speech in real-time"
- goblin89 5y agoYes. I reckon “Mocking Bird – Realtime Voice Clone for Mandarin” might’ve been a more appropriate change, but it’s always easy to criticise after the fact…
- echelon 5y agoI'm the author of FakeYou.com, so I have a little experience in this area. (We used to train GlowTTS models ourselves before turning it over to our users, which has had mixed results in terms of quality.) This appears to be a repackaging of RealTimeVoiceCloning [1], albeit with a few additions, such as GSTs. No matter what the repo claims, your results will depend on high quality data. Lots of it, and with ample fine tuning. Demo videos are absolutely cherry picked. If you're picking this up for a project, HiFi-Gan is pretty much the best vocoder right now. Tacotron still produces great results, though there are lots of other interesting model architectures. [1] https://github.com/CorentinJ/Real-Time-Voice-Cloning https://github.com/CorentinJ/Real-Time-Voice-Cloning
- kragen 5y agoBy "lots of [high quality data]" do you mean seconds, kiloseconds, or megaseconds of high-quality voice recordings?
- oefrha 5y agoIIRC enterprise solutions from the big clouds usually ask for at least hours of studio quality voice recordings for a custom voice model.
- kragen 5y agoThanks! Do you think they're using models as good as the ones echelon uses at FakeYou? Maybe he can get by with less data.
- echelon 5y agoThe best sounding limited data models have at least thirty minutes of audio data and have a similar pitch and timbre to a base data set. You can get by with less, but it requires finesse.
- iaml 5y agoThis paper[0] from this year seems to make do with a couple of minutes. [0] https://davidyao.me/projects/text2vid/ https://davidyao.me/projects/text2vid/
- twistedcheeslet 5y agoI'm very interested in this space. Forgive my ignorance, but what makes this fit for Chinese voices, while unfit for English voices?
- oefrha 5y agoFrom README: > This repository is forked from Real-Time-Voice-Cloning which only support English. https://github.com/CorentinJ/Real-Time-Voice-Cloning https://github.com/CorentinJ/Real-Time-Voice-Cloning
- JohnHaugeland 5y agoOne of the big things about these projects is training sound that's paired to text. The base project spoke English because it had been trained on English text paired with English recordings. This speaks Mandarin because it has been trained on paired Mandarin text and recordings. Amusingly, if you take one trained on English text/recording pairs and feed it French text, it will speak French with an English accent.
- mdp2021 5y agoSome may have missed the news in October (2021) about the 35 million dollars swindle, between UAE and Hong Kong, involving the bank director in HK receiving a call where the voice "of the company director" requested a transfer... https://www.forbes.com/sites/thomasbrewster/2021/10/14/huge-bank-fraud-uses-deep-fake-voice-tech-to-steal-millions/ https://www.forbes.com/sites/thomasbrewster/2021/10/14/huge-... https://www.unite.ai/deepfaked-voice-enabled-35-million-bank-heist-in-2020/ https://www.unite.ai/deepfaked-voice-enabled-35-million-bank... Not only the voice (simulated through machine learning) was trusted: also (even) emails had been unverified.
- pulse7 5y agoSo it already started: a new generation of swindling using computer generated voice, video and other artifacts... It didn't take much time to misuse the newest tech...
- mdp2021 5y agoThe issue is with the workers believing to live in a past distant century and sitting on the idea that current possibilities are "science fiction" - often criminally calibrating the risk on the abilities of the "man in the street", instead of the interested actor: the swindler. As in "But who could and would ever do it?"¹; the answer clearly being: the one really interested in doing it. I have heard, literally: "Well, if I cannot trust one's email nor one's telephone number [than a scam is like fate]"... From a bank employee. This is dangerously placed ignorance, in subcultures where unduly laxity was accepted. (¹ Again, an actual reply given to security experts showing systemic faults to managers.)
- hungryforcodes 5y agoThis didn't exist 10 years ago as a github project for anyone to download. How is it from a century in the distant past? Also, email and phone numbers are LITERALLY standard methods of 2FA. So what are you expecting bank employees to use?
- aasasd 5y agoThere was Adobe Voco, which seemed kind of a forerunner: https://www.youtube.com/watch?v=I3l4XLZ59iw https://www.youtube.com/watch?v=I3l4XLZ59iw, https://en.wikipedia.org/wiki/Adobe_Voco https://en.wikipedia.org/wiki/Adobe_Voco. It purportedly could edit speech like an audio editor and looked like a destroyer of authenticity. And then nothing was heard of it anymore. (Edit: Wikipedia says that VoCo takes “approximately 20 minutes of the desired target's speech”, and that it was a research prototype.) There was a thing called Tacotron from a team at Google, in 2018: https://google.github.io/tacotron/publications/speaker_adaptation/ https://google.github.io/tacotron/publications/speaker_adapt... (In fact, the OP repo and the original CorentinJ/Real-Time-Voice-Cloning apparently rely on Tacotron.) And there was something from 2019: https://www.ohadf.com/projects/text-based-editing/ https://www.ohadf.com/projects/text-based-editing/, https://news.stanford.edu/2019/06/05/edit-video-editing-text/ https://news.stanford.edu/2019/06/05/edit-video-editing-text... The latter two seem to need more samples than pure real-time editing. Overall, to me a layman, this space appears quieter than ‘deep-faking’ videos. Which makes me wonder if I haven't missed something.
- meibo 5y agoI imagine that it's inherently easier to create a fake voice that's convincing over a grainy phone line, than it is to make a convincing deep fake video. Maybe big tech orgs(including Adobe) don't want to risk the liability/PR fallout.
- bsenftner 5y agoThis is going to be a very abusive advertising technology. We already know the "cocktail party effect" where one can hear their name spoken by a familiar voice in a crowded space with lots of other voices. Expect to hear vaguely familiar voices using your name and personal information to craft advertising that pierces your awareness all the time, at odd times, and distracting you when you're busy.
- champagnois 5y agoDoes this mean voice assistants and GPS will one day be made consumer friendly and support any voice/name we want them to have, rather than this gimicky brand driven marketing malware known as Alexa or Google or Siri?
- VMtest 5y agoI have checked the video, the result is still not satisfying
- Cass 5y agoIs there an easy way to make this work for a non-programmer, i.e. without installing a whole python environment? I'm less interested in cloning specific voices than in getting a high-quality text-to-speech program that'll read me arbitrary Mandarin input, which it seems like this should be able to do.
- JohnHaugeland 5y agoya, microsoft sells it saas under the azure label. you can easily play with their web version, then hit a URL to get it on your credit card. https://azure.microsoft.com/en-us/services/cognitive-services/text-to-speech/ https://azure.microsoft.com/en-us/services/cognitive-service...