21 ms·

OpenVoice: Versatile instant voice cloning

473 points by ulrischa 3y ago

starwin1159 3y ago
Cantonese can't be imitated
- paulryanrogers 3y ago
  Why?
- Zambyte 3y ago
  How do people learn it?
muglug 3y ago
It’s funny how a bunch of models use Musk’s voice as a proof of their quality, given how disjointed and staccato he sounds in real life. Surely there are better voices to imitate.
- iinnPP 3y ago
  Proving the handling of uncommon speech is definitely a great example to use alongside the other common and uncommon speech examples on the page.
- ianschmitz 3y ago
  Especially with all of the crypto scams using Elon’s voice
- m463 3y ago
  I would imagine folks with really great voices like Morgan Freeman¹ or Don LaFontaine² are already voice actors, and using their voice might be seen as soul stealing (or competing with their career) 1: https://en.wikipedia.org/wiki/File:Morgan_freeman_bbc_radio4_film_programme_12_09_2008_b00dbcdn.flac https://en.wikipedia.org/wiki/File:Morgan_freeman_bbc_radio4... 2 https://youtu.be/USrkW_5QPa0 https://youtu.be/USrkW_5QPa0
andrewstuart 3y ago
If someone can come up with a voice clinging product that I can run on my own computer not the cloud, and if it’s super simple to install and use, then I’ll pay. I find it hard to understand why so much money is going into ai and so many startups are building ai stuff and such a product does not exist. It’s got to run locally because I’m not interested in the restrictions that cloud voice cloning services impose. Complete, consumer level local voice cloning = payment.
- dsign 3y ago
  I couldn't agree more. I've tried some of this ".ai" websites that do voice-cloning, and they tend to use the following dark strategy: - Demand you create a cloud account before trying. - Sometimes, demand you put your credit card before trying. - Always: the product is crap. Sometimes it does voice-cloning sort of as advertised, but you have to wait for the training and the execution in queue, because cloud GPUs are expensive and they need to manage a queue because it's a cloud prouduct. At least that part could be avoided if they shipped a VST plugin one could run locally, even if it's restricted to NVidia GPUs[^2]. [^1]: To those who say "but the devs must get paid": yes. But subscriptions miss-align incentives, and some updates are simply not worth the minutes they cause in productivity lost while waiting for their shoehorned installation. [^2]: Musicians and creative types are used to spend a lot in hardware and software, and there are inference GPUs which are cheaper than some sample libraries.
  - andrewstuart 3y ago
    I don’t mind if the software is a subscription it just has to be installable and not spyware garbage. Professional consumer level software like a game or productivity app or something.
  - riwsky 3y ago
    How do you figure subscriptions misalign incentives? The alternative, of selling upgrades, incentivizes devs to focus on new shiny shit that teases well. I instead rather they focus on making something I get value out of consistently.
bsenftner 3y ago
Is there a service here somewhere? The video mentions lower expense, but I can't find any such service sign up... (ah, all the usage info is all on github) Has anyone tried self hosting this software?
smusamashah 3y ago
The quality here is good (very good if I can actually run it locally). As per github it looks like we can run it locally. https://github.com/myshell-ai/OpenVoice/blob/main/docs/USAGE.md#linux-install https://github.com/myshell-ai/OpenVoice/blob/main/docs/USAGE...
- 486sx33 3y ago
  Still a bit robotic but better highs and lows for sure. The Catalog is huge! Thanks for posting
paraschopra 3y ago
yay!
nonrandomstring 3y ago
I also lost my voice in a bizarre fire breathing accident and urgently need to log into my telephone banking account. Can anyone here give me a short list of relatively common ethical use cases for this technology. I'm trying to refute the ridiculous accusation that only deceptive, criminal minded people would have any use for this. Thanks. ---- Edit: Thanks for the many good-faith replies. So far I see these breaking down into; Actual disability mitigation (of course I was joking about the circus accident). These are rare but valid. Who wouldn't want their _own_ restored. Entertainment and games Education and translation No further comment on the ethics here, but FWIW I'm nervously looking at this having just written a study module on social engineering attacks, stalking, harassment and confidence tricks. :/ And yes, as bare tech, it is very cool!
- serbrech 3y ago
  On the fly speech translation but in the voice of the speaker
  - 7373737373 3y ago
    Or a different voice if the voice of the speaker or the way they talk is annoying
- BriggyDwiggs42 3y ago
  Idk but its kinda cool
- RobotToaster 3y ago
  Could be combined with translation to automatically create dubs for videos/tv/etc.
- Larrikin 3y ago
  Home Assistant is making huge progress in creating an open source version of Alexa, Siri, etc. You can train it to use your voice, but the obvious use is celebrity voices for your home. Alexa had them, then took them away, and refused to refund people.
futureshock 3y ago
In related news, Voicecraft published their model weights today. https://github.com/jasonppy/VoiceCraft https://github.com/jasonppy/VoiceCraft
jasonjmcghee 3y ago
The quality of the output is really fantastic compared with other open source (next best XTTSv2). The voice cloning doesn't seem as high quality as other products I've used / seen demos for. Most of the examples match pitch well, but lose the "recognizable" aspect. The Elon one just doesn't sound like Elon, for example- interestingly the Australian accent sounds more like him.
duggan 3y ago
With a bit of coaxing I managed to get this running on my M2 mac with Python 3.11. Updated setup.py (mostly just bumping versions), and demo output: https://gist.github.com/duggan/63b7de9b5f6e8e74fe4b05af64dbe9a5 https://gist.github.com/duggan/63b7de9b5f6e8e74fe4b05af64dbe...
- speedylight 3y ago
  I barely managed to get one good output(by following the installation for Linux instructions) before the 1st Jupyter Demo crapped out on me. I didn’t have the patience to investigate the error, but I am very impressed with the quality considering I’m also running it on an M2 laptop.
smashah 3y ago
Terrifying.
- riskable 3y ago
  I know, right‽ Soon everything is going to be AI-enabled and our toothbrushes will be singing us Happy Birthday!
  - smashah 3y ago
    Both of those things already exist
randkyp 3y ago
This is HN, so I'm surprised that no one in the comments section has run this locally. :) Following the instructions in their repo (and moving the checkpoints/ and resources/ folder into the "nested" openvoice subfolder), I managed to get the Gradio demo running. Simple enough. It appears to be quicker than XTTS2 on my machine (RTX 3090), and utilizes approximately 1.5GB of VRAM. The Gradio demo is limited to 200 characters, perhaps for resource usage concerns, but it seems to run at around 8x realtime (8 seconds of speech for about 1 second of processing time.) EDIT: patched the Gradio demo for longer text; it's way faster than that. One minute of speech only took ~4 seconds to render. Default voice sample, reading this very comment: https://voca.ro/18JIHDs4vI1v https://voca.ro/18JIHDs4vI1v I had to write out acronyms -- XTTS2 to "ex tee tee ess two", for example. The voice clarity is better than XTTS2, too, but the speech can sound a bit stilted and, well, robotic/TTS-esque compared to it. The cloning consistency is definitely a step above XTTS2 in my experience -- XTTS2 would sometimes have random pitch shifts or plosives/babble in the middle of speech.
- bambax 3y ago
  I am trying to run it locally but it doesn't quite work for me. I was able to run the demos allright, but when trying to use another reference speaker (in demo_part1), the result doesn't sound at all like the source (it's just a random male voice). I'm also trying to produce French output, using a reference audio file in French for the base speaker, and a text in French. This triggers an error in api.py line 75 that the source language is not accepted. Indeed, in api.py line 45 the only two source languages allowed are English and Chineese; simply adding French to language_marks in api.py line 43 avoids errors but produces a weird/unintelligible result with a super heavy English accent and pronunciation. I guess one would need to generate source_se again, and probably mess with config.json and checkpoint.pth as well, but I could not find instructions on how to do this...? Edit -- tried again on https://app.myshell.ai/ https://app.myshell.ai/ The result sounds French alright, but still nothing like the original reference. It would be absolutely impossible to confuse one with the other, even for someone who didn't know the person very well.
  - randkyp 3y ago
joshspankit 3y ago
Does anyone know which local models are doing the “opposite”: Identify a voice well enough to do speaker diarization across multiple recordings?
- Drakim 3y ago
  On my wishlist would be a local model that can generate new voices based on descriptions such as "rough detective-like hard boiled man" or "old fatherly grampa"
  - mattferderer 3y ago
    You might be interested in this cool app that Microsoft made that I don't think I've seen anyone talk about anywhere called Speech Studio. https://speech.microsoft.com/ https://speech.microsoft.com/ I don't recall their voices being the most descriptive but they had a lot. They also let layout a bunch of text & have different voices speak each line just like a movie script.
- satvikpendem 3y ago
  Whisper can do diarization but not sure it will "remember" the voices well enough. You might simply have to stitch all the recordings together, run it through Whisper to get the diarized transcript, then process that how you want.
  - beardedwizard 3y ago
    Whisper does not support diarization. There are a number of projects that try to add it.
- Teleoflexuous 3y ago
  Whisper doesn't, but WhisperX <https://github.com/m-bain/whisperX/ https://github.com/m-bain/whisperX/> does. I am using it right now and it's perfectly serviceable. For reference, I'm transcribing research-related podcasts, meaning speech doesn't overlap a lot, which would be a problem for WhisperX from what I understand. There's also a lot of accents, which are straining on Whisper (though it's also doing well), but surely help WhisperX. It did have issues with figuring out the number of speakers on it's own, but that wasn't a problem for my use case.
lordofgibbons 3y ago
I've noticed that all TTS systems have a "metalic" sound to them. Can this be fixed automatically using some kine of post-processing?
- huytersd 3y ago
  Try cutting out some of the highs?
tonnydourado 3y ago
I might be missing something, but what are the non-questionable, or at least non-evil, uses of this technology? Because every single application I can think of is fucked up: porn, identity theft, impersonation, replacing voice actors, stealing the likeness of voice actors, replacing customer support without letting the customers know you're using bots. I guess you could give realistic voices to people that lost their voices by using old recordings, but there's no way that this is a market that justify the investment.
- swores 3y ago
  What about for remembering lost loved ones? There are dead people I would love to hear talk again, even if I know it's not their personality talking just their voice (and who knows, maybe with LLM training on a single person it could even be roughly their personality, too). I can imagine a fairly big market of both people setting it up before they die, with maybe a whole load of written content and a schedule of when to have it read in future, and people who've just lost someone, and want to recreate their voice to help remember it.
  - tonnydourado 3y ago
    > I can imagine a fairly big market (...) I can't, and if I could, I think this would be fairly dystopian. Didn't black mirror have an episode about something similar? I vaguely remember an Asimov/Arthur C. Clark short story about the implications of time travel (ish) tech in a similar context. Sounds like a case of "we've build the torment nexus from classic sci-fi novel 'do not build the torment nexus'"
  - dotancohen 3y ago
    Jack Crusher did something similar for Wesley.
  - grugagag 3y ago
    We already have ways to preserve the voices of people past their lives. Cloning their voices and writing things in their names is not only wrong but deceptive.
- 3y ago
trollied 3y ago
Look up iPhone “personal voice”. People don’t seem to know about it.
burcs 3y ago
There's a "vocal fry" aspect to all of these voice cloning tools, a sort of uncanny valley where they can't match tones correctly, or get fully away from this subtle Microsoft Sam-esque breathiness to their voice. I don't know how else to describe it.
- blackqueeriroh 3y ago
  Yeah, this is why I’m nowhere near worried about this replacing voice actors for the vast majority of work they currently get paid for.
Fripplebubby 3y ago
Really interesting! Reading the paper, it sounds like the core of it is broken into two things: 1. Encoding speech sounds into an IPA-like representation, decoding IPA-like into target language 2. Extracting "tone color", removing it from the IPA-like representation, then adding it back in into the target layer (emotion, accent, rhythm, pauses, intonation) So as a result, I am a native English speaker, but I could hear "my" voice speaking Chinese with similar tone color to my own! I wonder, if I recorded it, and then did learn to speak Chinese fluently, how similar it would be? I also wonder whether there is some kind of "tone color translator" that is needed to translate the tone color markers of American English into the relevant ones for other languages, how does that work? Or is that already learned as part of the model?
Havoc 3y ago
Tried it locally - can't get anywhere near the clone quality of the clips on their site. Not even close. Perhaps I'm doing something wrong...
pantsforbirds 3y ago
I wonder if in < 5 years I can make a game with a local LLM + AI TTS to create realistic NPCs. With enough of these tools I think you could make a very cool world-sim type game.
- rcarmo 3y ago
  I’m much more interested in the dismal possibility of using this in politics. Nation state actors, too.
  - fennecfoxy 3y ago
    Will we ever fix the problem of humans in a democracy voting for their team/flag rather than policies? Or humans seeing misinformation and not questioning it in the slightest? Those are the real issues, it's a people problem really.
treprinum 3y ago
Did this just obliterate ElevenLabs?
- htrp 3y ago
  Eleven's advantage is being able to have consistent outputs through high quality training data.
akashkahlon 3y ago
So every novel is a movie soon by the author itself using Sora and with Audio buys from all the suitable actors
- _zoltan_ 3y ago
  just by more NVDA. :-)
- rcarmo 3y ago
  This can’t really do a convincing Sean Connery yet.
- Multicomp 3y ago
  I hope so, then those of us who want to tell a story (writers, whether comic or novellist or short story or screenplay or teleplay or whatever) will be able to compete more and more on quality and richness of the story copy and content to the audience, not with the current comparative advantage of media choices being made for most storytellers based on difficulty to render. Words on page are easier than still photos, which are easier than animation, which are easier than live-action TV, which are easier than IMAX movies etc. If we move all of the rendering of the media into automation, then its just who can come up with the best story content, and you can render it whatever way you like: book, audiobook, animation, live action TV, web series, movie, miniseries, whatever you like. Granted - the AI will come for us writers to, it already is in some cases. Then the Creator Economy itself will be consumed with eventually becoming 'who can meme the fastest' on an industrial scale for daily events on the one end, and who has taken the time to paint / playact / do rendering out in the real world. But I sure would love to be able to make a movie out of my unpublished novel, and realistically today, that's impossible in my lifetime. Do I want the entire movie-making industry to die so I and others like me can have that power? No. But if the industry is going to die / change drastically anyways due to forces beyond my control, does that mean I'm not going to take advantage of the ability? Still no. IDK. I don't have all the answers to this. But yes, this (amazingly accurate voice cloner after a tiny clip?! wow) product is another step towards that brave new world.
thorum 3y ago
OpenVoice currently ranks second-to-last in the Huggingface TTS arena leaderboard, well below alternatives like styletts2 and xtts2: https://huggingface.co/spaces/TTS-AGI/TTS-Arena https://huggingface.co/spaces/TTS-AGI/TTS-Arena (Click the leaderboard tab at the top to see rankings)
- carbocation 3y ago
  I would like to see the new VoiceCraft model on that list eventually (weights released yesterday, discussion at [1]). 1 = https://news.ycombinator.com/item?id=39865340 https://news.ycombinator.com/item?id=39865340
- KennyBlanken 3y ago
  Having gone through almost ten rounds of the TTS Arena, XTT2 has tons of artifacts that instantly make it sound non-human. OpenVoice doesn't. It wouldn't surprise me if people recognize different algorithms and purposefully promote them over others, or alter the page source with a userscript to see the algorithm before listening and click the one they're trying to promote. Looking at the leaderboard, it's obvious there's manipulation going on, because Metavoice is highly ranked but generates absolutely terrible speech with extremely unnatural pauses. Elevenlabs was scarily natural sounding and high quality; the best of the ones I listened to so far. Pheme's speech overall sounds really natural, but has terrible sound quality, which is probably why it's ranked so well. If Pheme could be higher quality audio, it'd probably match Elevenlabs.
- Jackson__ 3y ago
  As someone who has used the arena maybe ~3 times, the subpar voice quality in the demo linked immediately stood out to me.
- ckl1810 3y ago
  Is there a benchmark for compute needed? Curious to see if anyone is building / has built a Zoom filter, or Mobile app, whereby I can speak English, and out comes Chinese to the listener.
- c0brac0bra 3y ago
lacoolj 3y ago
That season of 24 is coming true
yogorenapan 3y ago
Note: the open source version is watered down compared to their commercial offering. Tried both out and the quality doesn’t come close to
ckl1810 3y ago
OpenAI vollies back: https://twitter.com/OpenAI/status/1773760852153299024 https://twitter.com/OpenAI/status/1773760852153299024
speedbird 3y ago
Not convinced. The second reference has a slight Indian accent that isn’t carried over into the generated samples. Training data bias?
- opdahl 3y ago
  What are you talking about? I am not noticing it at all.
chenxi9649 3y ago
I am the most impressed by the cross-lingual voice cloning... https://research.myshell.ai/open-voice/zero-shot-cross-lingual-voice-cloning https://research.myshell.ai/open-voice/zero-shot-cross-lingu... I can only speak on their Dutch -> Chinese voice cloning but it's better than anything else I've tried. There is basically no "english/dutch accent" in the chinese at all. Where as the ElevenLabs Chinese voice(cloning or not) is so much worse...
mattizzle81 3y ago
Hmm for being so resource intensive that demo doesn't sound that great to me, sounds robotic. Piper will run on an iPhone and sounds about the same.

473 points by ulrischa 3y ago

starwin1159 3y ago

Cantonese can't be imitated

paulryanrogers 3y ago
Why?
Zambyte 3y ago
How do people learn it?

muglug 3y ago

It’s funny how a bunch of models use Musk’s voice as a proof of their quality, given how disjointed and staccato he sounds in real life. Surely there are better voices to imitate.

iinnPP 3y ago
Proving the handling of uncommon speech is definitely a great example to use alongside the other common and uncommon speech examples on the page.
ianschmitz 3y ago
Especially with all of the crypto scams using Elon’s voice
m463 3y ago
I would imagine folks with really great voices like Morgan Freeman¹ or Don LaFontaine² are already voice actors, and using their voice might be seen as soul stealing (or competing with their career) 1: https://en.wikipedia.org/wiki/File:Morgan_freeman_bbc_radio4_film_programme_12_09_2008_b00dbcdn.flac https://en.wikipedia.org/wiki/File:Morgan_freeman_bbc_radio4... 2 https://youtu.be/USrkW_5QPa0 https://youtu.be/USrkW_5QPa0

andrewstuart 3y ago

If someone can come up with a voice clinging product that I can run on my own computer not the cloud, and if it’s super simple to install and use, then I’ll pay. I find it hard to understand why so much money is going into ai and so many startups are building ai stuff and such a product does not exist. It’s got to run locally because I’m not interested in the restrictions that cloud voice cloning services impose. Complete, consumer level local voice cloning = payment.

dsign 3y ago
I couldn't agree more. I've tried some of this ".ai" websites that do voice-cloning, and they tend to use the following dark strategy: - Demand you create a cloud account before trying. - Sometimes, demand you put your credit card before trying. - Always: the product is crap. Sometimes it does voice-cloning sort of as advertised, but you have to wait for the training and the execution in queue, because cloud GPUs are expensive and they need to manage a queue because it's a cloud prouduct. At least that part could be avoided if they shipped a VST plugin one could run locally, even if it's restricted to NVidia GPUs[^2]. [^1]: To those who say "but the devs must get paid": yes. But subscriptions miss-align incentives, and some updates are simply not worth the minutes they cause in productivity lost while waiting for their shoehorned installation. [^2]: Musicians and creative types are used to spend a lot in hardware and software, and there are inference GPUs which are cheaper than some sample libraries.
- andrewstuart 3y ago
  I don’t mind if the software is a subscription it just has to be installable and not spyware garbage. Professional consumer level software like a game or productivity app or something.
- riwsky 3y ago
  How do you figure subscriptions misalign incentives? The alternative, of selling upgrades, incentivizes devs to focus on new shiny shit that teases well. I instead rather they focus on making something I get value out of consistently.

bsenftner 3y ago

Is there a service here somewhere? The video mentions lower expense, but I can't find any such service sign up... (ah, all the usage info is all on github) Has anyone tried self hosting this software?

smusamashah 3y ago

The quality here is good (very good if I can actually run it locally). As per github it looks like we can run it locally. https://github.com/myshell-ai/OpenVoice/blob/main/docs/USAGE.md#linux-install https://github.com/myshell-ai/OpenVoice/blob/main/docs/USAGE...

486sx33 3y ago
Still a bit robotic but better highs and lows for sure. The Catalog is huge! Thanks for posting

paraschopra 3y ago

yay!

nonrandomstring 3y ago

I also lost my voice in a bizarre fire breathing accident and urgently need to log into my telephone banking account. Can anyone here give me a short list of relatively common ethical use cases for this technology. I'm trying to refute the ridiculous accusation that only deceptive, criminal minded people would have any use for this. Thanks. ---- Edit: Thanks for the many good-faith replies. So far I see these breaking down into; Actual disability mitigation (of course I was joking about the circus accident). These are rare but valid. Who wouldn't want their _own_ restored. Entertainment and games Education and translation No further comment on the ethics here, but FWIW I'm nervously looking at this having just written a study module on social engineering attacks, stalking, harassment and confidence tricks. :/ And yes, as bare tech, it is very cool!

serbrech 3y ago
On the fly speech translation but in the voice of the speaker
- 7373737373 3y ago
  Or a different voice if the voice of the speaker or the way they talk is annoying
BriggyDwiggs42 3y ago
Idk but its kinda cool
RobotToaster 3y ago
Could be combined with translation to automatically create dubs for videos/tv/etc.
Larrikin 3y ago
Home Assistant is making huge progress in creating an open source version of Alexa, Siri, etc. You can train it to use your voice, but the obvious use is celebrity voices for your home. Alexa had them, then took them away, and refused to refund people.

futureshock 3y ago

In related news, Voicecraft published their model weights today. https://github.com/jasonppy/VoiceCraft https://github.com/jasonppy/VoiceCraft

jasonjmcghee 3y ago

The quality of the output is really fantastic compared with other open source (next best XTTSv2). The voice cloning doesn't seem as high quality as other products I've used / seen demos for. Most of the examples match pitch well, but lose the "recognizable" aspect. The Elon one just doesn't sound like Elon, for example- interestingly the Australian accent sounds more like him.

duggan 3y ago

With a bit of coaxing I managed to get this running on my M2 mac with Python 3.11. Updated setup.py (mostly just bumping versions), and demo output: https://gist.github.com/duggan/63b7de9b5f6e8e74fe4b05af64dbe9a5 https://gist.github.com/duggan/63b7de9b5f6e8e74fe4b05af64dbe...

speedylight 3y ago
I barely managed to get one good output(by following the installation for Linux instructions) before the 1st Jupyter Demo crapped out on me. I didn’t have the patience to investigate the error, but I am very impressed with the quality considering I’m also running it on an M2 laptop.

smashah 3y ago

Terrifying.

riskable 3y ago
I know, right‽ Soon everything is going to be AI-enabled and our toothbrushes will be singing us Happy Birthday!
- smashah 3y ago
  Both of those things already exist

randkyp 3y ago

This is HN, so I'm surprised that no one in the comments section has run this locally. :) Following the instructions in their repo (and moving the checkpoints/ and resources/ folder into the "nested" openvoice subfolder), I managed to get the Gradio demo running. Simple enough. It appears to be quicker than XTTS2 on my machine (RTX 3090), and utilizes approximately 1.5GB of VRAM. The Gradio demo is limited to 200 characters, perhaps for resource usage concerns, but it seems to run at around 8x realtime (8 seconds of speech for about 1 second of processing time.) EDIT: patched the Gradio demo for longer text; it's way faster than that. One minute of speech only took ~4 seconds to render. Default voice sample, reading this very comment: https://voca.ro/18JIHDs4vI1v https://voca.ro/18JIHDs4vI1v I had to write out acronyms -- XTTS2 to "ex tee tee ess two", for example. The voice clarity is better than XTTS2, too, but the speech can sound a bit stilted and, well, robotic/TTS-esque compared to it. The cloning consistency is definitely a step above XTTS2 in my experience -- XTTS2 would sometimes have random pitch shifts or plosives/babble in the middle of speech.

bambax 3y ago
I am trying to run it locally but it doesn't quite work for me. I was able to run the demos allright, but when trying to use another reference speaker (in demo_part1), the result doesn't sound at all like the source (it's just a random male voice). I'm also trying to produce French output, using a reference audio file in French for the base speaker, and a text in French. This triggers an error in api.py line 75 that the source language is not accepted. Indeed, in api.py line 45 the only two source languages allowed are English and Chineese; simply adding French to language_marks in api.py line 43 avoids errors but produces a weird/unintelligible result with a super heavy English accent and pronunciation. I guess one would need to generate source_se again, and probably mess with config.json and checkpoint.pth as well, but I could not find instructions on how to do this...? Edit -- tried again on https://app.myshell.ai/ https://app.myshell.ai/ The result sounds French alright, but still nothing like the original reference. It would be absolutely impossible to confuse one with the other, even for someone who didn't know the person very well.
- randkyp 3y ago

joshspankit 3y ago

Does anyone know which local models are doing the “opposite”: Identify a voice well enough to do speaker diarization across multiple recordings?

Drakim 3y ago
On my wishlist would be a local model that can generate new voices based on descriptions such as "rough detective-like hard boiled man" or "old fatherly grampa"
- mattferderer 3y ago
  You might be interested in this cool app that Microsoft made that I don't think I've seen anyone talk about anywhere called Speech Studio. https://speech.microsoft.com/ https://speech.microsoft.com/ I don't recall their voices being the most descriptive but they had a lot. They also let layout a bunch of text & have different voices speak each line just like a movie script.
satvikpendem 3y ago
Whisper can do diarization but not sure it will "remember" the voices well enough. You might simply have to stitch all the recordings together, run it through Whisper to get the diarized transcript, then process that how you want.
- beardedwizard 3y ago
  Whisper does not support diarization. There are a number of projects that try to add it.
Teleoflexuous 3y ago
Whisper doesn't, but WhisperX <https://github.com/m-bain/whisperX/ https://github.com/m-bain/whisperX/> does. I am using it right now and it's perfectly serviceable. For reference, I'm transcribing research-related podcasts, meaning speech doesn't overlap a lot, which would be a problem for WhisperX from what I understand. There's also a lot of accents, which are straining on Whisper (though it's also doing well), but surely help WhisperX. It did have issues with figuring out the number of speakers on it's own, but that wasn't a problem for my use case.

lordofgibbons 3y ago

I've noticed that all TTS systems have a "metalic" sound to them. Can this be fixed automatically using some kine of post-processing?

huytersd 3y ago
Try cutting out some of the highs?

tonnydourado 3y ago

I might be missing something, but what are the non-questionable, or at least non-evil, uses of this technology? Because every single application I can think of is fucked up: porn, identity theft, impersonation, replacing voice actors, stealing the likeness of voice actors, replacing customer support without letting the customers know you're using bots. I guess you could give realistic voices to people that lost their voices by using old recordings, but there's no way that this is a market that justify the investment.

swores 3y ago
What about for remembering lost loved ones? There are dead people I would love to hear talk again, even if I know it's not their personality talking just their voice (and who knows, maybe with LLM training on a single person it could even be roughly their personality, too). I can imagine a fairly big market of both people setting it up before they die, with maybe a whole load of written content and a schedule of when to have it read in future, and people who've just lost someone, and want to recreate their voice to help remember it.
- tonnydourado 3y ago
  > I can imagine a fairly big market (...) I can't, and if I could, I think this would be fairly dystopian. Didn't black mirror have an episode about something similar? I vaguely remember an Asimov/Arthur C. Clark short story about the implications of time travel (ish) tech in a similar context. Sounds like a case of "we've build the torment nexus from classic sci-fi novel 'do not build the torment nexus'"
- dotancohen 3y ago
  Jack Crusher did something similar for Wesley.
- grugagag 3y ago
  We already have ways to preserve the voices of people past their lives. Cloning their voices and writing things in their names is not only wrong but deceptive.
3y ago

trollied 3y ago

Look up iPhone “personal voice”. People don’t seem to know about it.

burcs 3y ago

There's a "vocal fry" aspect to all of these voice cloning tools, a sort of uncanny valley where they can't match tones correctly, or get fully away from this subtle Microsoft Sam-esque breathiness to their voice. I don't know how else to describe it.

blackqueeriroh 3y ago
Yeah, this is why I’m nowhere near worried about this replacing voice actors for the vast majority of work they currently get paid for.

Fripplebubby 3y ago

Really interesting! Reading the paper, it sounds like the core of it is broken into two things: 1. Encoding speech sounds into an IPA-like representation, decoding IPA-like into target language 2. Extracting "tone color", removing it from the IPA-like representation, then adding it back in into the target layer (emotion, accent, rhythm, pauses, intonation) So as a result, I am a native English speaker, but I could hear "my" voice speaking Chinese with similar tone color to my own! I wonder, if I recorded it, and then did learn to speak Chinese fluently, how similar it would be? I also wonder whether there is some kind of "tone color translator" that is needed to translate the tone color markers of American English into the relevant ones for other languages, how does that work? Or is that already learned as part of the model?

Havoc 3y ago

Tried it locally - can't get anywhere near the clone quality of the clips on their site. Not even close. Perhaps I'm doing something wrong...

pantsforbirds 3y ago

I wonder if in < 5 years I can make a game with a local LLM + AI TTS to create realistic NPCs. With enough of these tools I think you could make a very cool world-sim type game.

rcarmo 3y ago
I’m much more interested in the dismal possibility of using this in politics. Nation state actors, too.
- fennecfoxy 3y ago
  Will we ever fix the problem of humans in a democracy voting for their team/flag rather than policies? Or humans seeing misinformation and not questioning it in the slightest? Those are the real issues, it's a people problem really.

treprinum 3y ago

Did this just obliterate ElevenLabs?

htrp 3y ago
Eleven's advantage is being able to have consistent outputs through high quality training data.

akashkahlon 3y ago

So every novel is a movie soon by the author itself using Sora and with Audio buys from all the suitable actors

_zoltan_ 3y ago
just by more NVDA. :-)
rcarmo 3y ago
This can’t really do a convincing Sean Connery yet.
Multicomp 3y ago
I hope so, then those of us who want to tell a story (writers, whether comic or novellist or short story or screenplay or teleplay or whatever) will be able to compete more and more on quality and richness of the story copy and content to the audience, not with the current comparative advantage of media choices being made for most storytellers based on difficulty to render. Words on page are easier than still photos, which are easier than animation, which are easier than live-action TV, which are easier than IMAX movies etc. If we move all of the rendering of the media into automation, then its just who can come up with the best story content, and you can render it whatever way you like: book, audiobook, animation, live action TV, web series, movie, miniseries, whatever you like. Granted - the AI will come for us writers to, it already is in some cases. Then the Creator Economy itself will be consumed with eventually becoming 'who can meme the fastest' on an industrial scale for daily events on the one end, and who has taken the time to paint / playact / do rendering out in the real world. But I sure would love to be able to make a movie out of my unpublished novel, and realistically today, that's impossible in my lifetime. Do I want the entire movie-making industry to die so I and others like me can have that power? No. But if the industry is going to die / change drastically anyways due to forces beyond my control, does that mean I'm not going to take advantage of the ability? Still no. IDK. I don't have all the answers to this. But yes, this (amazingly accurate voice cloner after a tiny clip?! wow) product is another step towards that brave new world.

thorum 3y ago

OpenVoice currently ranks second-to-last in the Huggingface TTS arena leaderboard, well below alternatives like styletts2 and xtts2: https://huggingface.co/spaces/TTS-AGI/TTS-Arena https://huggingface.co/spaces/TTS-AGI/TTS-Arena (Click the leaderboard tab at the top to see rankings)

carbocation 3y ago
I would like to see the new VoiceCraft model on that list eventually (weights released yesterday, discussion at [1]). 1 = https://news.ycombinator.com/item?id=39865340 https://news.ycombinator.com/item?id=39865340
KennyBlanken 3y ago
Having gone through almost ten rounds of the TTS Arena, XTT2 has tons of artifacts that instantly make it sound non-human. OpenVoice doesn't. It wouldn't surprise me if people recognize different algorithms and purposefully promote them over others, or alter the page source with a userscript to see the algorithm before listening and click the one they're trying to promote. Looking at the leaderboard, it's obvious there's manipulation going on, because Metavoice is highly ranked but generates absolutely terrible speech with extremely unnatural pauses. Elevenlabs was scarily natural sounding and high quality; the best of the ones I listened to so far. Pheme's speech overall sounds really natural, but has terrible sound quality, which is probably why it's ranked so well. If Pheme could be higher quality audio, it'd probably match Elevenlabs.
Jackson__ 3y ago
As someone who has used the arena maybe ~3 times, the subpar voice quality in the demo linked immediately stood out to me.
ckl1810 3y ago
Is there a benchmark for compute needed? Curious to see if anyone is building / has built a Zoom filter, or Mobile app, whereby I can speak English, and out comes Chinese to the listener.
c0brac0bra 3y ago

lacoolj 3y ago

That season of 24 is coming true

yogorenapan 3y ago

Note: the open source version is watered down compared to their commercial offering. Tried both out and the quality doesn’t come close to

ckl1810 3y ago

OpenAI vollies back: https://twitter.com/OpenAI/status/1773760852153299024 https://twitter.com/OpenAI/status/1773760852153299024

speedbird 3y ago

Not convinced. The second reference has a slight Indian accent that isn’t carried over into the generated samples. Training data bias?

opdahl 3y ago
What are you talking about? I am not noticing it at all.

chenxi9649 3y ago

I am the most impressed by the cross-lingual voice cloning... https://research.myshell.ai/open-voice/zero-shot-cross-lingual-voice-cloning https://research.myshell.ai/open-voice/zero-shot-cross-lingu... I can only speak on their Dutch -> Chinese voice cloning but it's better than anything else I've tried. There is basically no "english/dutch accent" in the chinese at all. Where as the ElevenLabs Chinese voice(cloning or not) is so much worse...

mattizzle81 3y ago

Hmm for being so resource intensive that demo doesn't sound that great to me, sounds robotic. Piper will run on an iPhone and sounds about the same.