3 ms·

Out of curiosity - to folks that have had success with this... This voice cloning is... nothing like XTTSv2, let alone ElevenLabs. It doesn't seem to care abo

Out of curiosity - to folks that have had success with this... This voice cloning is... nothing like XTTSv2, let alone ElevenLabs. It doesn't seem to care about accents at all. It does pretty well with pitch and cadence, and that's about it. I've tried all kinds of different values for alpha, beta, embedding scale, diffusion steps. Anyone else have better luck? Sure it's fast and the sound quality is pretty good, but I can't get the voice cloning to work at all.

carbocation 3y ago
I had the same experience as what you described (with a lot of experimentation with alpha and beta, as well as uploading different audio clips).
dsrtslnd23 3y ago
See the conclusion remarks in the paper - they acknowledge that voice cloning is not that good (yet).
jsjmch 3y ago
See my previous comment about this point. ElevenLabs are based on Tortoise-TTS which was already pre-trained on millions of hours of data, but this one was only trained on LibriTTS which was 500 hours at best. XTTS was also trained with probably millions of speakers in more than 20 languages. If you have seen millions of voices, there are definitely gonna be some of them that sound like you. It is just a matter of training data, but it is very difficult to have someone collect these large amounts of data and train on it.
- lossolo 3y ago
  > It is just a matter of training data, but it is very difficult to have someone collect these large amounts of data and train on it. It's really not that difficult, they are trained mostly on audiobooks and high quality audio from yt videos. If we talk about EV model then we are talking about around 500k hours of audio, but Tortoise-TTS is only around 50k from what I remember.
- wczekalski 3y ago
  What's your basis for the claim that they are based on TorToiSe? I have seen this claim made (and rebutted) many times.
  - jsjmch 3y ago
    Very similar features, quite slow inference speed, and various rumors.

carbocation 3y ago

I had the same experience as what you described (with a lot of experimentation with alpha and beta, as well as uploading different audio clips).

dsrtslnd23 3y ago

See the conclusion remarks in the paper - they acknowledge that voice cloning is not that good (yet).

jsjmch 3y ago

See my previous comment about this point. ElevenLabs are based on Tortoise-TTS which was already pre-trained on millions of hours of data, but this one was only trained on LibriTTS which was 500 hours at best. XTTS was also trained with probably millions of speakers in more than 20 languages. If you have seen millions of voices, there are definitely gonna be some of them that sound like you. It is just a matter of training data, but it is very difficult to have someone collect these large amounts of data and train on it.

lossolo 3y ago
> It is just a matter of training data, but it is very difficult to have someone collect these large amounts of data and train on it. It's really not that difficult, they are trained mostly on audiobooks and high quality audio from yt videos. If we talk about EV model then we are talking about around 500k hours of audio, but Tortoise-TTS is only around 50k from what I remember.
wczekalski 3y ago
What's your basis for the claim that they are based on TorToiSe? I have seen this claim made (and rebutted) many times.
- jsjmch 3y ago
  Very similar features, quite slow inference speed, and various rumors.