2 ms·

Interesting that there isn't a mention of Orpheus as prior art either since it's the exact same thing. (https://github.com/canopyai/Orpheus-TTS https://github.

Interesting that there isn't a mention of Orpheus as prior art either since it's the exact same thing. (https://github.com/canopyai/Orpheus-TTS https://github.com/canopyai/Orpheus-TTS)

gapeleon 1y ago
> Interesting that there isn't a mention of Orpheus as prior art either Llasa-3b (https://huggingface.co/HKUSTAudio/Llasa-3B https://huggingface.co/HKUSTAudio/Llasa-3B) came out before Orpheus (https://huggingface.co/canopylabs/orpheus-3b-0.1-ft https://huggingface.co/canopylabs/orpheus-3b-0.1-ft). > it's the exact same thing. They're very similar, but they're not the exact same thing. Llasa uses xcodec2, a much simpler, lossless 16khz wav codec. This makes it superior for one-shot voice cloning. Orpheus' 24khz snac codec is lossy which makes it difficult to use for zero-shot cloning as the reference audio gets degraded during tokenization. You can test this here: https://huggingface.co/spaces/Gapeleon/snac_test https://huggingface.co/spaces/Gapeleon/snac_test But when finetuned on 50+ audio samples, it produces much cleaner 24khz audio than Llasa, and the snac model is much easier to run on consumer hardware than xcodec2 (87t/s for realtime speech, which can be achieved on an RTX3080 for example)
- oezi 1y ago
  Do you happen to know why Orpheus and Llasa use Finetuning for voice cloning? Zonos uses 128-float embeddings for voices and it seems so much nicer. Because you can just mix and match voices without changing the model.
  - thot_experiment 1y ago
    No, you just condition it with text-voice token pairs and then when conditioning further inference w/ text the voice tokens tend to match the pairs further up in the context.
- oezi 1y ago
  Isn't xcodec2 also lossy? I thought it is also just another neural codec (50 tok/s, single codebook). What are people using to upsampling back to 44,1 or 48 khz? Anything fancy?
  - woodson 1y ago
    They’re both lossy. They use a VAE-VQ type architecture trained with a combination of losses/discriminators. The differences are mainly the encoder/decoder architecture, the type of bottleneck quantization (RVQ, FSQ, etc.) and of course the training data.

Interesting that there isn't a mention of Orpheus as prior art either since it's the exact same thing. (https://github.com/canopyai/Orpheus-TTS https://github.com/canopyai/Orpheus-TTS)

gapeleon 1y ago

> Interesting that there isn't a mention of Orpheus as prior art either Llasa-3b (https://huggingface.co/HKUSTAudio/Llasa-3B https://huggingface.co/HKUSTAudio/Llasa-3B) came out before Orpheus (https://huggingface.co/canopylabs/orpheus-3b-0.1-ft https://huggingface.co/canopylabs/orpheus-3b-0.1-ft). > it's the exact same thing. They're very similar, but they're not the exact same thing. Llasa uses xcodec2, a much simpler, lossless 16khz wav codec. This makes it superior for one-shot voice cloning. Orpheus' 24khz snac codec is lossy which makes it difficult to use for zero-shot cloning as the reference audio gets degraded during tokenization. You can test this here: https://huggingface.co/spaces/Gapeleon/snac_test https://huggingface.co/spaces/Gapeleon/snac_test But when finetuned on 50+ audio samples, it produces much cleaner 24khz audio than Llasa, and the snac model is much easier to run on consumer hardware than xcodec2 (87t/s for realtime speech, which can be achieved on an RTX3080 for example)

oezi 1y ago
Do you happen to know why Orpheus and Llasa use Finetuning for voice cloning? Zonos uses 128-float embeddings for voices and it seems so much nicer. Because you can just mix and match voices without changing the model.
- thot_experiment 1y ago
  No, you just condition it with text-voice token pairs and then when conditioning further inference w/ text the voice tokens tend to match the pairs further up in the context.
oezi 1y ago
Isn't xcodec2 also lossy? I thought it is also just another neural codec (50 tok/s, single codebook). What are people using to upsampling back to 44,1 or 48 khz? Anything fancy?
- woodson 1y ago
  They’re both lossy. They use a VAE-VQ type architecture trained with a combination of losses/discriminators. The differences are mainly the encoder/decoder architecture, the type of bottleneck quantization (RVQ, FSQ, etc.) and of course the training data.