24 ms·

OpenVoice: Versatile Instant Voice Cloning

399 points by saeedesmaili 3y ago

pclmulqdq 3y ago
Wonderful company, not a scam at all: https://docs.myshell.ai/tokenomics https://docs.myshell.ai/tokenomics
- deleted 3y ago
  [deleted]
andrewstuart 3y ago
If I wanted to do voice cloning on my own hardware can anyone suggest what a good open source project would be to use? What is the state of the art in open source voice cloning?
- snerbles 3y ago
  I use Tortoise TTS. It's slow, a little clunky, and sometimes the output gets downright weird. But it's the best quality-oriented TTS I've found that I can run locally. It's allegedly the basis of the tech used by Eleven Labs. https://github.com/neonbjb/tortoise-tts https://github.com/neonbjb/tortoise-tts
  - xsdu 3y ago
    There are faster implementations of tortoise that allow fine-tuning. You can get close to ElevenLabs quality if you have a perfect dataset. https://git.ecker.tech/mrq/ai-voice-cloning https://git.ecker.tech/mrq/ai-voice-cloning
whycome 3y ago
It's not really well advertised and I'm not sure Apple is continuing development, but iOS has a voice clone feature called "Personal Voice" - it takes about 15 mins to train it with your own voice (and then takes a few hours to process on-device when locked). You can use it in phone calls and FaceTime (maybe other places?). It would be nice to use it for general TTS.
- sagz 3y ago
  It's an accessibility feature for people losing their voice or on the verge of. And it is TTS only, not speech-to-speech as your mention of "can use it in phone calls and facetime" implies. Not being s2s means it doesn't retain vocal disfluencies, prosody etc signals that make a voice feel real
windex 3y ago
Fraud becomes easier I guess.
kennethologist 3y ago
Anyone knows at a deeply practical and technical level how 11Labs achieves the level they do?
yboris 3y ago
GitHub link: https://github.com/myshell-ai/OpenVoice https://github.com/myshell-ai/OpenVoice
bamba26 3y ago
The tide is high, a powerful moment reminding us that, even in the face of vastness, we have the strength to overcome obstacles. Each wave is a lesson in perseverance, showing us that no matter how hard we are pushed back, we can always come back stronger. This tide teaches us resilience, adaptability, and the courage to seize new opportunities that come our way. Let’s embrace change as a chance to grow, renew, and explore new horizons.
- bamba26 3y ago
  The tide is high, a powerful moment reminding us that, even in the face of vastness, we have the strength to overcome obstacles. Each wave is a lesson in perseverance, showing us that no matter how hard we are pushed back, we can always come back stronger. This tide teaches us resilience, adaptability, and the courage to seize new opportunities that come our way. Let’s embrace change as a chance to grow, renew, and explore new horizons.
Zuiii 3y ago
> This repository is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which prohibits commercial usage. MyShell reserves the ability to detect whether an audio is generated by OpenVoice, no matter whether the watermark is added or not. Creative Commons Attribution-NonCommercial 4.0 only prevents the use of the licensed code within other projects. It says nothing about output. People can (and will) use output generated by this for commercial projects, and they will be completely within their rights to do so. So detect away I guess.
peddling-brink 3y ago
GitHub: https://github.com/myshell-ai/OpenVoice https://github.com/myshell-ai/OpenVoice Checkpoint: hxxps://myshell-public-repo-hosting.s3.amazonaws.com/checkpoints_1226.zip (Checkpoint link defanged because I’m allergic to direct links to zip files hosted on Amazon. Nor have I reviewed what the file contains.)
- crazysim 3y ago
  Thanks for the link to the repo. It's very useful. As for the checkpoint, I'm not allergic and I don't do security theater: https://github.com/myshell-ai/OpenVoice?tab=readme-ov-file#installation https://github.com/myshell-ai/OpenVoice?tab=readme-ov-file#i... links to https://myshell-public-repo-hosting.s3.amazonaws.com/checkpoints_1226.zip https://myshell-public-repo-hosting.s3.amazonaws.com/checkpo...
  - peddling-brink 3y ago
    Why do you call that security theater? I found and provided the information, but didn’t make it clickable. Anyone can decide for themselves to navigate there. Your comment comes off as passive aggressive.
    - IshKebab 3y ago
      I think he's referring to your "defanging" which you implied was security related but doesn't actually achieve anything at all.
      - fieldcny 3y ago
        They are making you think about what you are doing before you click the link. that’s not theatre that’s keeping people from clicking arbitrary links to zip files which can auto-execute code once downloaded. I’d suggest that those who think it is theatre probably don’t understand the implications of that action.
        3y ago
colesantiago 3y ago
> This repository is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which prohibits commercial usage. MyShell reserves the ability to detect whether an audio is generated by OpenVoice, no matter whether the watermark is added or not. So it is not 'open' then and you cannot make money out of this?
- DandyDev 3y ago
  It is open, just not by your definition. You can view, use and modify the code to your hearts content. Sounds pretty open to me!
  - jahewson 3y ago
    Open for business! No wait…
  - CaptainFever 3y ago
    To be specific, while it is not a bad license, it does not quality for the Free Cultural Works mark as defined by the Creative Commons and Freedom Defined: https://creativecommons.org/public-domain/freeworks/ https://creativecommons.org/public-domain/freeworks/
  - bbor 3y ago
    Well… “use” isn’t exactly free, this the complaint. On a scale of free to not free, “cannot use this for my work” is a pretty big jump to the latter end IMO
    - c0pium 3y ago
      Careful, you’re saying the quiet part out loud; freedom is about profiting off the uncompensated work of others.
      - bbor 3y ago
        Well ultimately we all need to eat. If someone wants to be compensated in today’s society, they either need to join a gift-based sub-society (see: OSS foundations, NGOs in general) or sell something. Trust me, I totally agree that freedom of information should be a completely separate concern from resource allocation EDIT: I guess there’s a third option, “work another job and use OSS on your off hours”. Which feels… idk, disrespectful of the whole enterprise. OSS software development is important enough to deserve a wage IMO, to say the least
dcreater 3y ago
Any GitHub link?
- saeedesmaili 3y ago
  Github: https://github.com/myshell-ai/OpenVoice https://github.com/myshell-ai/OpenVoice Demo: https://research.myshell.ai/open-voice https://research.myshell.ai/open-voice
smellf 3y ago
Examples: https://research.myshell.ai/open-voice https://research.myshell.ai/open-voice Seems impressive!
- deleted 3y ago
  [deleted]
iAkashPaul 3y ago
That watermark detection rights at the end is real sus
- diggan 3y ago
  What exactly are you talking about? The paper doesn't mention any watermark at all, as far as I can see/search.
  - cwillu 3y ago
    The readme on the linked github reads: “MyShell reserves the ability to detect whether an audio is generated by OpenVoice, no matter whether the watermark is added or not.”
    - lostlogin 3y ago
      As you say, right at the bottom https://github.com/myshell-ai/OpenVoice https://github.com/myshell-ai/OpenVoice
      - diggan 3y ago
        Ah, thank you. Guess that's OK that the company/service do whatever they want, the paper/technique doesn't involve watermarks, so it'd be easy to remove/modify whatever they do in the library/service itself.
- deleted 3y ago
  [deleted]
- fbdab103 3y ago
  At least right now, there is a literal add_watermark function, so probably easy enough to remove that surface level. Unless they added something cute to the training data to poison the well. https://github.com/myshell-ai/OpenVoice/blob/a33963c3d764beed54f8e10273e4d57ea3a1447a/api.py#L161 https://github.com/myshell-ai/OpenVoice/blob/a33963c3d764bee...
huqedato 3y ago
Welcome to the new era of fakes and scams beyond our wildest imagination !
- danielbln 3y ago
  Elevenlabs has been around for a while now. Genie has been out of the bottle for a bit, and the sooner the notion that anything digital can be easily faked seeps into the wider consciousness the better. Trust nothing.
  - ethanbond 3y ago
    It can both be true that people need to adapt/“trust nothing,” and that this is bad.
  - ignu 3y ago
    I've seen some prank calls (a YouTuber cloned Tucker Carlson's voice and called Alex Jones) but he just had a sound bank with a few pre-generated lines and it fell apart pretty quickly. At least for now there's too much lag to do a real time conversation with a cloned voice. Speech to Text > LLM Response > Generate Audio If that time can shrink to subsecond, I think there'll be madness. (Specifically thinking of romance scammers)
    - whywhywhywhy 3y ago
      You don't need an LLM Response
    - shinycode 3y ago
      Awful, bots on their own having real conversations with people with the voice of a loved one. Scamming on steroids
    - ben_w 3y ago
      At last summer's WeAreDevelopers World Congress in Berlin, one of the talks I went to was by someone who did this with their own voice, to better respond to (really long?) WhatsApp messages they kept getting. It worked a bit too well, as it could parse the sound file and generate a complete response faster than real-time, leading people to ask if he'd actually listened to the messages they sent him. Also they had trouble believing him when he told them how he'd done it.
gnfargbl 3y ago
I had to phone my bank, which is one of the bigger players in the UK high street market, a couple of days ago. They're still encouraging me to enroll in their idiotic "my voice is my password" programme. At this stage in the evolution of AI, that feels simply negligent.
- toss1 3y ago
  Fidelity Investments just did something even worse ~a week ago - It asked me to reply to a few questions, then announced that I'd just been enrolled in it's voice identification program (or whatever they call it). Now I've got Just Another Item on my ToDo list, to get that undone. Gawd, does every company promote it's stupidest people to management?
  - crazysim 3y ago
    Clone management's voices and post it to their social media/etc. Super undo it!
  - hasty_pudding 3y ago
    They promote their best schmoozers to management. They have so much money that competence no longer matters and bootlicking will get you much farther.
  - throwup238 3y ago
    > Gawd, does every company promote it's stupidest people to management? Yes: https://en.wikipedia.org/wiki/Dilbert_principle https://en.wikipedia.org/wiki/Dilbert_principle Ironically, this is the place where they can do the least damage.
  - ben_w 3y ago
    I don't know if GDPR (or any of its cousins) applies to you, but this kind of thing sounds exactly like the sort of thing it's supposed to outlaw.
    - jokethrowaway 3y ago
cwillu 3y ago
From the github readme: “MyShell reserves the ability to detect whether an audio is generated by OpenVoice, no matter whether the watermark is added or not.” Call me skeptical…
- deleted 3y ago
  [deleted]
hasty_pudding 3y ago
Holy cow! If this works without curated audio...this is amazing!
senthilnayagam 3y ago
current leader in open source voice cloning is RVC, would like to see how it compares to it.
- echelon 3y ago
  RVC is voice conversion (audio to audio), and it's typically finetuned. This is zero shot TTS. Samples create vector encodings that serve as input to inference. There's no retraining the model unless you want it to generalize or perform better.
  - cchance 3y ago
    It isn't though, people need to read the paper and the comments from the author they aren't actually doing the voice generation they pass the text off to VITS, and then they're sauce is that they are doing tone mapping on that VITS output, so if anything they're a competitor to RVC, it's just that the version they published includes VITS also
    - echelon 3y ago
      Interesting. Funny enough, a lot of RVC packages are using VITS to do RVC for TTS.
tgv 3y ago
[flagged]
- screamingninja 3y ago
  Care to elaborate for someone lacking context?
  - andrewmcwatters 3y ago
    Adobe did the same thing years ago and never released it, because there's nothing productive you can do with it if it's not proven to be one's own voice that you have control over.
  - tgv 3y ago
    You don't know what voice cloning is and what it can be used for? These people made it even easier to do. That's their achievement. Facilitating fraud, confusion and distrust.
    - Ukv 3y ago
      > You don't know what voice cloning is and what it can be used for? * Real-time translation of a user's voice, maintaining emotion and intonation * Professional-quality audio from cheap microphone setups (for video tutorials, indie games, etc.) * Allowing those with speech impairments to communicate using their natural voice again, or: * Allowing those uncomfortable with their natural voice to communicate closer to how they wish to be perceived * Customization of voice assistants, such as to use a native accent/dialect * Movies, podcasts, audiobooks, news broadcasts, etc. made available in a huge range of languages * And of course: memes, satire, and parody I also think it opens the door for entirely new kinds of media, but it's hard to foresee exactly what will take off and what will die a gimmick. Maybe immersive-sims where characters can respond intelligently to anything you throw at them? Maybe personalized movies, like the ability to direct scenes yourself?
      - tgv 3y ago
SubiculumCode 3y ago
whats with the crypto thing?
programjames 3y ago
I love this paper. It reads very much like "this is what we did, and we want to help others do it too." Also, the section "Remark on Novelty" is golden: "OpenVoice does not intend to invent the submodules in the model structure ... The contribution of OpenVoice is the decoupled framework that seperates the voice style and language control from the tone color cloning." They don't try to hype up their contribution.
jamespattn 3y ago
Can someone give me a practical use case where this adds a net benefit to society?
- shinycode 3y ago
  Aside from the fact that is will be easier to scam people, I fail to see benefits. We can already translate everything with the same synthetic voice
- Lerc 3y ago
  Unifying a voice in tutorial videos so that the difference in voice does not distract the learner. Auto non-toxic rephrasing of online chat in video games, let people hear their voice but paraphrase what they said in a manner that doesn't turn the platform into a cesspit. Cloning your own voice so that you can turn a script into audio without 50 takes and then having to remove a million Ums and errs.
  - grayhatter 3y ago
    > Auto non-toxic rephrasing of online chat in video games, let people hear their voice but paraphrase what they said in a manner that doesn't turn the platform into a cesspit. that feels very orwellian
    - ben_w 3y ago
      George Orwell — 'If you want a picture of the future, imagine a boot stamping on a human face—for ever.' I think this is closer to the direction of Huxley in Brave New World, where a deeper understanding of how to manipulate without brute force creates a very different dystopian society than 1984.
      - haroldp 3y ago
        "Don’t you see that the whole aim of Newspeak is to narrow the range of thought? In the end we shall make thoughtcrime literally impossible, because there will be no words in which to express it."
qwertox 3y ago
And suddenly it becomes a bit weird: https://docs.myshell.ai/tokenomics https://docs.myshell.ai/tokenomics Tokenomics Disclaimer: MyShell is currently in the testing phase, and the content of the whitepaper may be subject to change in the future. $SHELL is the token used for user incentive, governance and in-app utility. The total supply of $SHELL is 1,000,000,000
- diggan 3y ago
  And luckily, this submission seems to be about the paper/technology OpenVoice, not about the company MyShell (whatever that is).
  - qwertox 3y ago
    License[0]: This repository is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which prohibits commercial usage. MyShell reserves the ability to detect whether an audio is generated by OpenVoice, no matter whether the watermark is added or not. [0] https://github.com/myshell-ai/OpenVoice https://github.com/myshell-ai/OpenVoice
- deleted 3y ago
  [deleted]
z991 3y ago
I commend the authors on making this easy to try! However it doesn't work very well for me for general voice cloning. I read the first paragraph of the wikipedia page on books and had it generate the next sentence. It's obviously computer generated to my ear. Audio sample: https://storage.googleapis.com/dalle-party/sample.mp3 https://storage.googleapis.com/dalle-party/sample.mp3 Cloned voice (converted to mp3): https://storage.googleapis.com/dalle-party/output_en_default.mp3 https://storage.googleapis.com/dalle-party/output_en_default... All I did was install the packages with pip and then run "demo_part1.ipynb" with my audio sample plugged in. Ran almost instantly on my laptop 3070 Ti / 8GB. (Also, I admit to not reading the paper, I just ran the code)
- fbdab103 3y ago
  Thanks for the real example. Sounded quite generated to my ear as well. Wonder if it can do any better with more source material.
- pclmulqdq 3y ago
  Looking at the website and the examples, it's pretty clearly set up to make stylized anime voices.
  - japanman185 3y ago
    This is the driver for a lot of things. Anime. x264 was to enable better compression of weeb videos. This tech will allow fan dubs to better represent the animes in the videos.
    - matheusmoreira 3y ago
      Anime also drove the development of a lot of subtitling technology if I remember correctly.
- thorum 3y ago
  My experience with other tools like xtts is you really need to have a studio-quality voice sample to get the best results.
tremarley 3y ago
Their Tokenomics page say $SHELL is the token used for user incentive, governance and in-app utility. The total supply of $SHELL is 1,000,000,000 Team, Treasury, Advisors & Private Sale = 55% allocation Community Incentive = 40% allocation Liquidity = 5%
- deleted 3y ago
  [deleted]
monkeydust 3y ago
So I guess we could (legally) now create a voice chatbot using Mickey Mouse audio from Steamboat Willie?
- andylynch 3y ago
  Possibly, except there is no dialogue in it.
- deleted 3y ago
  [deleted]
starwin1159 3y ago
I hope someone can handle Cantonese one day
RagnarD 3y ago
My first and ongoing thought is that immoral/criminal uses of voice cloning vastly exceed any legitimate ones.
- jokethrowaway 3y ago
  I disagree, we should just not accept voice as authentication. I think the most common use case will be making art & content programmatically without voice actors (and most likely without actors at all once we nail video or a 3d model pipeline + frame by frame transformation to make it look realistic)
- tacocataco 3y ago
  Talk with your loved ones and make a paraphrase for if you're stuck in a emergency and need money wired or something. Some banks have voice authentication when you call in and you have to ask to opt out.
- airstrike 3y ago
  Which just means we need to build protocols around this risk, rather than foolishly trying to shove the genie back in the bottle, lest we be left with only the criminal uses
- squigz 3y ago
  Out of curiosity, what/how many legitimate use cases have you considered?
  - bryanrasmussen 3y ago
    Potential legitimate uses I can think of - 1. licensing voice to other uses - people with recognizable trademarkable voices (actors, singers) have another potential revenue stream. yay! 2. use of past voices - voices that are not 'owned' from the past - let's say Humphrey Bogart's voice, can be used in projects without having to pay for imitator. This would be useful for both marketing and artistic projects. But probably less for marketing because they will want to go with step 1. 3. Teach yourself to talk like X. People who need to learn to talk like a particular person / have a particular accent could learn quicker. Just think - you will be able to supplement your comedy routine with kickass Christopher Walken impersonations any day now! Variations of 3 and 2 together open up interesting modes of aesthetic impression, but I won't go into that here. But definitely I have some ideas that might benefit from being able to do this.
ijhuygft776 3y ago
Is there some similar software that allows you to add lets say 40 years to a voice?
anotherevan 3y ago
Is it possible to use this (or Eleven Labs) to generate a voice model to plug into an Android phone's TTS? I have a friend with a paralysed larynx who is often using his phone or a small laptop to type in order to communicate. I know he would love it if it was possible to take old recordings of him speaking and use that to give him back "his" voice, at least in some small measure.
- Share6323 3y ago
  That would be awesome
- klankbrouwerij 3y ago
  Your friend can take a look at solutions from Acapela [0], SpeakUnique [1] or VOCALiD [2]. Not sure whether they have a solution for Android though. I recently saw a video from google about a custom voice they created for somebody with ALS but I can't seem to find it online (Does anybody have a link?). Creating custom voices is not yet available on Android though. The latest iOS release (iOS 17) does support creating personalized voices. ModelTalker [3] is a long-term (research?) project to create custom voices for people with speech disabilities. Their TTS seem to support Android so that might be another option. [0] https://www.acapela-group.com/ https://www.acapela-group.com/ [1] https://www.speakunique.co.uk/ https://www.speakunique.co.uk/ [2] https://vocalid.ai/ https://vocalid.ai/ [3] https://www.modeltalker.org/ https://www.modeltalker.org/
  - anotherevan 3y ago
    Hi and thanks for the suggestions. Looking through them, it looks like you need to do what they call "voice banking" before you lose your voice. Basically reading a script they provide. Unfortunately my friend's voice is too far gone for that to be possible. Hoping for something where they can use old recordings to generate a voice.
    - jokethrowaway 3y ago
      From my tests I think Audiobox from Meta is the most promising (even better than Eleven Labs) - too bad it's closed source and they force you to read some randomly generated sentences (to prevent the case of someone generating a cloned voice without consent). Right now Eleven Labs is your best bet. xTTS is just not there quality wise. The version available in the studio is marginally better than the OSS version but it's still pretty far from being believable. The non-nerfed version of Tortoise (the author decided to ruin their own project but forks exist) was decent at voice cloning but it takes a lot of tries. I'm pretty sure we already have the technology to do what you want and help your friend, it's just a matter of time until it gets better and more software comes out.
thimkerbell 3y ago
Overall, will this be a good thing or a bad thing for society, do you think? If it is a bad thing, should we cheer it on?
- deleted 3y ago
  [deleted]
chipper02 3y ago
Now of only youtube would ban the use of this crap. Or at the very least allow you filter those videos.
- whywhywhywhy 3y ago
  There's genuine uses, look at Apple offering this tech recently as an accessibility feature for people losing the ability to speak to have text to speech in their own voice in lieu of being able to vocalize it themselves. You're banning genuine uses like that or just creators who want to fix a fumbled or awkward line without completely re-recording if you ban it.
  - cchance 3y ago
    He's the same camp that bitches about SD, MJ and DALLe, but loves photoshop generative fill lol

399 points by saeedesmaili 3y ago

pclmulqdq 3y ago

Wonderful company, not a scam at all: https://docs.myshell.ai/tokenomics https://docs.myshell.ai/tokenomics

deleted 3y ago
[deleted]

andrewstuart 3y ago

If I wanted to do voice cloning on my own hardware can anyone suggest what a good open source project would be to use? What is the state of the art in open source voice cloning?

snerbles 3y ago
I use Tortoise TTS. It's slow, a little clunky, and sometimes the output gets downright weird. But it's the best quality-oriented TTS I've found that I can run locally. It's allegedly the basis of the tech used by Eleven Labs. https://github.com/neonbjb/tortoise-tts https://github.com/neonbjb/tortoise-tts
- xsdu 3y ago
  There are faster implementations of tortoise that allow fine-tuning. You can get close to ElevenLabs quality if you have a perfect dataset. https://git.ecker.tech/mrq/ai-voice-cloning https://git.ecker.tech/mrq/ai-voice-cloning

whycome 3y ago

It's not really well advertised and I'm not sure Apple is continuing development, but iOS has a voice clone feature called "Personal Voice" - it takes about 15 mins to train it with your own voice (and then takes a few hours to process on-device when locked). You can use it in phone calls and FaceTime (maybe other places?). It would be nice to use it for general TTS.

sagz 3y ago
It's an accessibility feature for people losing their voice or on the verge of. And it is TTS only, not speech-to-speech as your mention of "can use it in phone calls and facetime" implies. Not being s2s means it doesn't retain vocal disfluencies, prosody etc signals that make a voice feel real

windex 3y ago

Fraud becomes easier I guess.

kennethologist 3y ago

Anyone knows at a deeply practical and technical level how 11Labs achieves the level they do?

yboris 3y ago

GitHub link: https://github.com/myshell-ai/OpenVoice https://github.com/myshell-ai/OpenVoice

bamba26 3y ago

The tide is high, a powerful moment reminding us that, even in the face of vastness, we have the strength to overcome obstacles. Each wave is a lesson in perseverance, showing us that no matter how hard we are pushed back, we can always come back stronger. This tide teaches us resilience, adaptability, and the courage to seize new opportunities that come our way. Let’s embrace change as a chance to grow, renew, and explore new horizons.

bamba26 3y ago
The tide is high, a powerful moment reminding us that, even in the face of vastness, we have the strength to overcome obstacles. Each wave is a lesson in perseverance, showing us that no matter how hard we are pushed back, we can always come back stronger. This tide teaches us resilience, adaptability, and the courage to seize new opportunities that come our way. Let’s embrace change as a chance to grow, renew, and explore new horizons.

Zuiii 3y ago

> This repository is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which prohibits commercial usage. MyShell reserves the ability to detect whether an audio is generated by OpenVoice, no matter whether the watermark is added or not. Creative Commons Attribution-NonCommercial 4.0 only prevents the use of the licensed code within other projects. It says nothing about output. People can (and will) use output generated by this for commercial projects, and they will be completely within their rights to do so. So detect away I guess.

peddling-brink 3y ago

GitHub: https://github.com/myshell-ai/OpenVoice https://github.com/myshell-ai/OpenVoice Checkpoint: hxxps://myshell-public-repo-hosting.s3.amazonaws.com/checkpoints_1226.zip (Checkpoint link defanged because I’m allergic to direct links to zip files hosted on Amazon. Nor have I reviewed what the file contains.)

crazysim 3y ago
Thanks for the link to the repo. It's very useful. As for the checkpoint, I'm not allergic and I don't do security theater: https://github.com/myshell-ai/OpenVoice?tab=readme-ov-file#installation https://github.com/myshell-ai/OpenVoice?tab=readme-ov-file#i... links to https://myshell-public-repo-hosting.s3.amazonaws.com/checkpoints_1226.zip https://myshell-public-repo-hosting.s3.amazonaws.com/checkpo...
- peddling-brink 3y ago
  Why do you call that security theater? I found and provided the information, but didn’t make it clickable. Anyone can decide for themselves to navigate there. Your comment comes off as passive aggressive.
  - IshKebab 3y ago
    I think he's referring to your "defanging" which you implied was security related but doesn't actually achieve anything at all.
    - fieldcny 3y ago
      They are making you think about what you are doing before you click the link. that’s not theatre that’s keeping people from clicking arbitrary links to zip files which can auto-execute code once downloaded. I’d suggest that those who think it is theatre probably don’t understand the implications of that action.
      - 3y ago

colesantiago 3y ago

DandyDev 3y ago
It is open, just not by your definition. You can view, use and modify the code to your hearts content. Sounds pretty open to me!
- jahewson 3y ago
  Open for business! No wait…
- CaptainFever 3y ago
  To be specific, while it is not a bad license, it does not quality for the Free Cultural Works mark as defined by the Creative Commons and Freedom Defined: https://creativecommons.org/public-domain/freeworks/ https://creativecommons.org/public-domain/freeworks/
- bbor 3y ago
  Well… “use” isn’t exactly free, this the complaint. On a scale of free to not free, “cannot use this for my work” is a pretty big jump to the latter end IMO
  - c0pium 3y ago
    Careful, you’re saying the quiet part out loud; freedom is about profiting off the uncompensated work of others.
    - bbor 3y ago
      Well ultimately we all need to eat. If someone wants to be compensated in today’s society, they either need to join a gift-based sub-society (see: OSS foundations, NGOs in general) or sell something. Trust me, I totally agree that freedom of information should be a completely separate concern from resource allocation EDIT: I guess there’s a third option, “work another job and use OSS on your off hours”. Which feels… idk, disrespectful of the whole enterprise. OSS software development is important enough to deserve a wage IMO, to say the least

dcreater 3y ago

Any GitHub link?

saeedesmaili 3y ago
Github: https://github.com/myshell-ai/OpenVoice https://github.com/myshell-ai/OpenVoice Demo: https://research.myshell.ai/open-voice https://research.myshell.ai/open-voice

smellf 3y ago

Examples: https://research.myshell.ai/open-voice https://research.myshell.ai/open-voice Seems impressive!

deleted 3y ago
[deleted]

iAkashPaul 3y ago

That watermark detection rights at the end is real sus

diggan 3y ago
What exactly are you talking about? The paper doesn't mention any watermark at all, as far as I can see/search.
- cwillu 3y ago
  The readme on the linked github reads: “MyShell reserves the ability to detect whether an audio is generated by OpenVoice, no matter whether the watermark is added or not.”
  - lostlogin 3y ago
    As you say, right at the bottom https://github.com/myshell-ai/OpenVoice https://github.com/myshell-ai/OpenVoice
    - diggan 3y ago
      Ah, thank you. Guess that's OK that the company/service do whatever they want, the paper/technique doesn't involve watermarks, so it'd be easy to remove/modify whatever they do in the library/service itself.
deleted 3y ago
[deleted]
fbdab103 3y ago
At least right now, there is a literal add_watermark function, so probably easy enough to remove that surface level. Unless they added something cute to the training data to poison the well. https://github.com/myshell-ai/OpenVoice/blob/a33963c3d764beed54f8e10273e4d57ea3a1447a/api.py#L161 https://github.com/myshell-ai/OpenVoice/blob/a33963c3d764bee...

huqedato 3y ago

Welcome to the new era of fakes and scams beyond our wildest imagination !

danielbln 3y ago
Elevenlabs has been around for a while now. Genie has been out of the bottle for a bit, and the sooner the notion that anything digital can be easily faked seeps into the wider consciousness the better. Trust nothing.
- ethanbond 3y ago
  It can both be true that people need to adapt/“trust nothing,” and that this is bad.
- ignu 3y ago
  I've seen some prank calls (a YouTuber cloned Tucker Carlson's voice and called Alex Jones) but he just had a sound bank with a few pre-generated lines and it fell apart pretty quickly. At least for now there's too much lag to do a real time conversation with a cloned voice. Speech to Text > LLM Response > Generate Audio If that time can shrink to subsecond, I think there'll be madness. (Specifically thinking of romance scammers)
  - whywhywhywhy 3y ago
    You don't need an LLM Response
  - shinycode 3y ago
    Awful, bots on their own having real conversations with people with the voice of a loved one. Scamming on steroids
  - ben_w 3y ago
    At last summer's WeAreDevelopers World Congress in Berlin, one of the talks I went to was by someone who did this with their own voice, to better respond to (really long?) WhatsApp messages they kept getting. It worked a bit too well, as it could parse the sound file and generate a complete response faster than real-time, leading people to ask if he'd actually listened to the messages they sent him. Also they had trouble believing him when he told them how he'd done it.

gnfargbl 3y ago

I had to phone my bank, which is one of the bigger players in the UK high street market, a couple of days ago. They're still encouraging me to enroll in their idiotic "my voice is my password" programme. At this stage in the evolution of AI, that feels simply negligent.

toss1 3y ago
Fidelity Investments just did something even worse ~a week ago - It asked me to reply to a few questions, then announced that I'd just been enrolled in it's voice identification program (or whatever they call it). Now I've got Just Another Item on my ToDo list, to get that undone. Gawd, does every company promote it's stupidest people to management?
- crazysim 3y ago
  Clone management's voices and post it to their social media/etc. Super undo it!
- hasty_pudding 3y ago
  They promote their best schmoozers to management. They have so much money that competence no longer matters and bootlicking will get you much farther.
- throwup238 3y ago
  > Gawd, does every company promote it's stupidest people to management? Yes: https://en.wikipedia.org/wiki/Dilbert_principle https://en.wikipedia.org/wiki/Dilbert_principle Ironically, this is the place where they can do the least damage.
- ben_w 3y ago
  I don't know if GDPR (or any of its cousins) applies to you, but this kind of thing sounds exactly like the sort of thing it's supposed to outlaw.
  - jokethrowaway 3y ago

cwillu 3y ago

From the github readme: “MyShell reserves the ability to detect whether an audio is generated by OpenVoice, no matter whether the watermark is added or not.” Call me skeptical…

deleted 3y ago
[deleted]

hasty_pudding 3y ago

Holy cow! If this works without curated audio...this is amazing!

senthilnayagam 3y ago

current leader in open source voice cloning is RVC, would like to see how it compares to it.

echelon 3y ago
RVC is voice conversion (audio to audio), and it's typically finetuned. This is zero shot TTS. Samples create vector encodings that serve as input to inference. There's no retraining the model unless you want it to generalize or perform better.
- cchance 3y ago
  It isn't though, people need to read the paper and the comments from the author they aren't actually doing the voice generation they pass the text off to VITS, and then they're sauce is that they are doing tone mapping on that VITS output, so if anything they're a competitor to RVC, it's just that the version they published includes VITS also
  - echelon 3y ago
    Interesting. Funny enough, a lot of RVC packages are using VITS to do RVC for TTS.

tgv 3y ago

[flagged]

screamingninja 3y ago
Care to elaborate for someone lacking context?
- andrewmcwatters 3y ago
  Adobe did the same thing years ago and never released it, because there's nothing productive you can do with it if it's not proven to be one's own voice that you have control over.
- tgv 3y ago
  You don't know what voice cloning is and what it can be used for? These people made it even easier to do. That's their achievement. Facilitating fraud, confusion and distrust.
  - Ukv 3y ago
    > You don't know what voice cloning is and what it can be used for? * Real-time translation of a user's voice, maintaining emotion and intonation * Professional-quality audio from cheap microphone setups (for video tutorials, indie games, etc.) * Allowing those with speech impairments to communicate using their natural voice again, or: * Allowing those uncomfortable with their natural voice to communicate closer to how they wish to be perceived * Customization of voice assistants, such as to use a native accent/dialect * Movies, podcasts, audiobooks, news broadcasts, etc. made available in a huge range of languages * And of course: memes, satire, and parody I also think it opens the door for entirely new kinds of media, but it's hard to foresee exactly what will take off and what will die a gimmick. Maybe immersive-sims where characters can respond intelligently to anything you throw at them? Maybe personalized movies, like the ability to direct scenes yourself?
    - tgv 3y ago

SubiculumCode 3y ago

whats with the crypto thing?

programjames 3y ago

I love this paper. It reads very much like "this is what we did, and we want to help others do it too." Also, the section "Remark on Novelty" is golden: "OpenVoice does not intend to invent the submodules in the model structure ... The contribution of OpenVoice is the decoupled framework that seperates the voice style and language control from the tone color cloning." They don't try to hype up their contribution.

jamespattn 3y ago

Can someone give me a practical use case where this adds a net benefit to society?

shinycode 3y ago
Aside from the fact that is will be easier to scam people, I fail to see benefits. We can already translate everything with the same synthetic voice
Lerc 3y ago
Unifying a voice in tutorial videos so that the difference in voice does not distract the learner. Auto non-toxic rephrasing of online chat in video games, let people hear their voice but paraphrase what they said in a manner that doesn't turn the platform into a cesspit. Cloning your own voice so that you can turn a script into audio without 50 takes and then having to remove a million Ums and errs.
- grayhatter 3y ago
  > Auto non-toxic rephrasing of online chat in video games, let people hear their voice but paraphrase what they said in a manner that doesn't turn the platform into a cesspit. that feels very orwellian
  - ben_w 3y ago
    George Orwell — 'If you want a picture of the future, imagine a boot stamping on a human face—for ever.' I think this is closer to the direction of Huxley in Brave New World, where a deeper understanding of how to manipulate without brute force creates a very different dystopian society than 1984.
    - haroldp 3y ago
      "Don’t you see that the whole aim of Newspeak is to narrow the range of thought? In the end we shall make thoughtcrime literally impossible, because there will be no words in which to express it."

qwertox 3y ago

And suddenly it becomes a bit weird: https://docs.myshell.ai/tokenomics https://docs.myshell.ai/tokenomics Tokenomics Disclaimer: MyShell is currently in the testing phase, and the content of the whitepaper may be subject to change in the future. $SHELL is the token used for user incentive, governance and in-app utility. The total supply of $SHELL is 1,000,000,000

diggan 3y ago
And luckily, this submission seems to be about the paper/technology OpenVoice, not about the company MyShell (whatever that is).
- qwertox 3y ago
  License[0]: This repository is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which prohibits commercial usage. MyShell reserves the ability to detect whether an audio is generated by OpenVoice, no matter whether the watermark is added or not. [0] https://github.com/myshell-ai/OpenVoice https://github.com/myshell-ai/OpenVoice
deleted 3y ago
[deleted]

z991 3y ago

I commend the authors on making this easy to try! However it doesn't work very well for me for general voice cloning. I read the first paragraph of the wikipedia page on books and had it generate the next sentence. It's obviously computer generated to my ear. Audio sample: https://storage.googleapis.com/dalle-party/sample.mp3 https://storage.googleapis.com/dalle-party/sample.mp3 Cloned voice (converted to mp3): https://storage.googleapis.com/dalle-party/output_en_default.mp3 https://storage.googleapis.com/dalle-party/output_en_default... All I did was install the packages with pip and then run "demo_part1.ipynb" with my audio sample plugged in. Ran almost instantly on my laptop 3070 Ti / 8GB. (Also, I admit to not reading the paper, I just ran the code)

fbdab103 3y ago
Thanks for the real example. Sounded quite generated to my ear as well. Wonder if it can do any better with more source material.
pclmulqdq 3y ago
Looking at the website and the examples, it's pretty clearly set up to make stylized anime voices.
- japanman185 3y ago
  This is the driver for a lot of things. Anime. x264 was to enable better compression of weeb videos. This tech will allow fan dubs to better represent the animes in the videos.
  - matheusmoreira 3y ago
    Anime also drove the development of a lot of subtitling technology if I remember correctly.
thorum 3y ago
My experience with other tools like xtts is you really need to have a studio-quality voice sample to get the best results.

tremarley 3y ago

Their Tokenomics page say $SHELL is the token used for user incentive, governance and in-app utility. The total supply of $SHELL is 1,000,000,000 Team, Treasury, Advisors & Private Sale = 55% allocation Community Incentive = 40% allocation Liquidity = 5%

deleted 3y ago
[deleted]

monkeydust 3y ago

So I guess we could (legally) now create a voice chatbot using Mickey Mouse audio from Steamboat Willie?

andylynch 3y ago
Possibly, except there is no dialogue in it.
deleted 3y ago
[deleted]

starwin1159 3y ago

I hope someone can handle Cantonese one day

RagnarD 3y ago

My first and ongoing thought is that immoral/criminal uses of voice cloning vastly exceed any legitimate ones.

jokethrowaway 3y ago
I disagree, we should just not accept voice as authentication. I think the most common use case will be making art & content programmatically without voice actors (and most likely without actors at all once we nail video or a 3d model pipeline + frame by frame transformation to make it look realistic)
tacocataco 3y ago
Talk with your loved ones and make a paraphrase for if you're stuck in a emergency and need money wired or something. Some banks have voice authentication when you call in and you have to ask to opt out.
airstrike 3y ago
Which just means we need to build protocols around this risk, rather than foolishly trying to shove the genie back in the bottle, lest we be left with only the criminal uses
squigz 3y ago
Out of curiosity, what/how many legitimate use cases have you considered?
- bryanrasmussen 3y ago
  Potential legitimate uses I can think of - 1. licensing voice to other uses - people with recognizable trademarkable voices (actors, singers) have another potential revenue stream. yay! 2. use of past voices - voices that are not 'owned' from the past - let's say Humphrey Bogart's voice, can be used in projects without having to pay for imitator. This would be useful for both marketing and artistic projects. But probably less for marketing because they will want to go with step 1. 3. Teach yourself to talk like X. People who need to learn to talk like a particular person / have a particular accent could learn quicker. Just think - you will be able to supplement your comedy routine with kickass Christopher Walken impersonations any day now! Variations of 3 and 2 together open up interesting modes of aesthetic impression, but I won't go into that here. But definitely I have some ideas that might benefit from being able to do this.

ijhuygft776 3y ago

Is there some similar software that allows you to add lets say 40 years to a voice?

anotherevan 3y ago

Is it possible to use this (or Eleven Labs) to generate a voice model to plug into an Android phone's TTS? I have a friend with a paralysed larynx who is often using his phone or a small laptop to type in order to communicate. I know he would love it if it was possible to take old recordings of him speaking and use that to give him back "his" voice, at least in some small measure.

Share6323 3y ago
That would be awesome
klankbrouwerij 3y ago
Your friend can take a look at solutions from Acapela [0], SpeakUnique [1] or VOCALiD [2]. Not sure whether they have a solution for Android though. I recently saw a video from google about a custom voice they created for somebody with ALS but I can't seem to find it online (Does anybody have a link?). Creating custom voices is not yet available on Android though. The latest iOS release (iOS 17) does support creating personalized voices. ModelTalker [3] is a long-term (research?) project to create custom voices for people with speech disabilities. Their TTS seem to support Android so that might be another option. [0] https://www.acapela-group.com/ https://www.acapela-group.com/ [1] https://www.speakunique.co.uk/ https://www.speakunique.co.uk/ [2] https://vocalid.ai/ https://vocalid.ai/ [3] https://www.modeltalker.org/ https://www.modeltalker.org/
- anotherevan 3y ago
  Hi and thanks for the suggestions. Looking through them, it looks like you need to do what they call "voice banking" before you lose your voice. Basically reading a script they provide. Unfortunately my friend's voice is too far gone for that to be possible. Hoping for something where they can use old recordings to generate a voice.
  - jokethrowaway 3y ago
    From my tests I think Audiobox from Meta is the most promising (even better than Eleven Labs) - too bad it's closed source and they force you to read some randomly generated sentences (to prevent the case of someone generating a cloned voice without consent). Right now Eleven Labs is your best bet. xTTS is just not there quality wise. The version available in the studio is marginally better than the OSS version but it's still pretty far from being believable. The non-nerfed version of Tortoise (the author decided to ruin their own project but forks exist) was decent at voice cloning but it takes a lot of tries. I'm pretty sure we already have the technology to do what you want and help your friend, it's just a matter of time until it gets better and more software comes out.

thimkerbell 3y ago

Overall, will this be a good thing or a bad thing for society, do you think? If it is a bad thing, should we cheer it on?

deleted 3y ago
[deleted]

chipper02 3y ago

Now of only youtube would ban the use of this crap. Or at the very least allow you filter those videos.

whywhywhywhy 3y ago
There's genuine uses, look at Apple offering this tech recently as an accessibility feature for people losing the ability to speak to have text to speech in their own voice in lieu of being able to vocalize it themselves. You're banning genuine uses like that or just creators who want to fix a fumbled or awkward line without completely re-recording if you ban it.
- cchance 3y ago
  He's the same camp that bitches about SD, MJ and DALLe, but loves photoshop generative fill lol