8 ms·

ChatTTS-Best open source TTS Model

165 points by informal007 2y ago

ex3ndr 2y ago
Looks like it is yet another xtts fork
- charlie0simmon 2y ago
  [dead]
cchance 2y ago
Sounds good but feel like theirs something slightly off to the cadence of the voice in the sample, but maybe i'm imagining
- informal007 2y ago
  I think you are right, It is not the completely same.
estheryo 2y ago
The completion level is impressive! I can hardly tell the difference from a human voice, especially with the natural pauses and laughter, which surpasses ChatGPT’s quality. However, there’s a noticeable electric noise at the end of sentences, which feels unnatural. (As a native Chinese speaker, I find the Chinese output even better in comparison.)
- informal007 2y ago
  Yes, it's just a new born project, looking forward the next verison.
- nshm 2y ago
  There is also a glitch in "dialogue"
maxglute 2y ago
Sounds natural and intelligible at 3x speed, which is plus. >The Real-Time Factor (RTF) is around 0.65. What is the state of real time tts models?
- regularfry 2y ago
  0.65 isn't that big a gap. This is probably less than one jart away from being optimised to real-time.
- c0brac0bra 2y ago
  Aura can stream responses and has a time-to-first-byte around 200-400ms
NobodyNada 2y ago
> Attribution-NonCommercial-NoDerivatives 4.0 International Strictly speaking, this is not open source, as the commonly accepted definitions of open-source software include freedom of use and modification. But in an industry where "OpenAI" is 100% proprietary, I guess "open-source" doesn't really mean much.
- abetusk 2y ago
  Note that the repo itself doesn't claim it's open source, it's the title of the link that (incorrectly) claims it is.
  - simonw 2y ago
    Yeah, I think the link title should be "ChatTTS: Best openly licensed TTS model" instead.
  - follower 2y ago
    The repo does claim that something is "open source" but the only included license text is "CC-BY-NC-ND" and the README seems to restrict permissions even further. In addition, the Hugging Face repo[-1] states the license as "Creative Commons Attribution Non Commercial 4.0" (lacking the "ND"). Unfortunately, this combination of license imprecision and restrictiveness seems par for the course with a lot of academic TTS projects. (And, even for commercial "Open Source" TTS projects it's often the case that while code might be OSS licensed, none of the the voice data/models are.) The current version[0] of the README repo states: * "The open-source version on HuggingFace is a 40,000 hours pre trained model without SFT." (Presumably refers to model.) * "At the same time, we have internally trained a detection model and plan to open-source it in the future." (Not directly relevant.) The included "Roadmap" indicates related completed & uncompleted tasks: * "[x] Open-source the 40k hour base model and spk_stats file" * "[ ] Open-source VQ encoder and Lora training code" * "[ ] Open-source the 40k hour version with multi-emotion control" However, as noted, the current LICENSE[1] file states: * "Attribution-NonCommercial-NoDerivatives 4.0 International" And the README also contradicts the license: * "This repo is for academic purposes only. It is intended for educational and research use, and should not be used for any commercial or legal purposes." * "The information and data used in this repo, are for academic and research purposes only." And this part of the "disclaimer" would make me concerned about potential licensing issues in regard to code and or data from other sources: * "The data obtained from publicly available sources, and the authors do not claim any ownership or copyright over the data." The code in the repo itself appears to have no license information contained within it. My go-to actually Open Source licensed Text-To-Speech project (with a range of voice[2] model licenses[3]--including Public Domain & CC-BY[4]) is Piper TTS: https://github.com/rhasspy/piper https://github.com/rhasspy/piper ---- footnotes ---- [-1] https://huggingface.co/2Noise/ChatTTS https://huggingface.co/2Noise/ChatTTS [0] https://github.com/2noise/ChatTTS/blob/f4c8329f0d231b272b676e5e171fb9655b345f2e/README.md https://github.com/2noise/ChatTTS/blob/f4c8329f0d231b272b676... [1] https://github.com/2noise/ChatTTS/blob/f4c8329f0d231b272b676e5e171fb9655b345f2e/LICENSE https://github.com/2noise/ChatTTS/blob/f4c8329f0d231b272b676... [2] Voice samples: https://rhasspy.github.io/piper-samples/ https://rhasspy.github.io/piper-samples/ [3] Though I would also caution that (at least by my interpretation) some of the voices listed as CC0/PD or CC-BY also note that they've been "fine-tuned" on models which have more restrictive licenses and thus probably can't inherit the voice data's more permissive license. [4] However, these Public Domain and/or CC-BY voices do appear specifically created to be data license-compliant: https://brycebeattie.com/files/tts/ https://brycebeattie.com/files/tts/
JoeDeanx 2y ago
Where is the demo that can be used?
- informal007 2y ago
  You can try to run this code to use https://github.com/2noise/ChatTTS/blob/main/infer.ipynb https://github.com/2noise/ChatTTS/blob/main/infer.ipynb
  - JoeDeanx 2y ago
    I don't know how to code, but I found a website that I can use online,https://chattts.com/ https://chattts.com/
modeless 2y ago
A good time to link the TTS leaderboard: https://huggingface.co/spaces/TTS-AGI/TTS-Arena https://huggingface.co/spaces/TTS-AGI/TTS-Arena Eleven Labs is still very far above open source models in quality. But StyleTTS2 (MIT license) is impressively good and quite fast. It'll be interesting to see where this new one ends up. The code-switching ability is quite interesting. Most open source TTS models are strictly one language per sentence, often one language per voice. In general though, TTS as an isolated system is mostly a dead end IMO. The future is in multimodal end-to-end audio-to-audio (or anything-to-audio) models, as demonstrated by OpenAI with GPT-4o's voice mode (though I've been saying this since long before their demo: https://news.ycombinator.com/item?id=38339222 https://news.ycombinator.com/item?id=38339222). Text is very useful as training data but as a way to represent other modalities like audio or image data it is far too lossy.
- dartos 2y ago
  Isn’t gpt 4o voice not audio to audio, but audio to text to audio?
  - modeless 2y ago
    It isn't released yet, but the new one that they demoed is audio-to-audio. That's why it can do things like sing songs, and detect emotion in voices. The one that you can currently access in the ChatGPT app (with subscription) is the old one which is ASR->LLM->TTS.
    - addandsubtract 2y ago
      I'm pretty sure you can use the new GPT4o audio-to-audio model already – even without a subscription. You can even use the "Sky" model if you didn't update your app.
    - dartos 2y ago
      Are we sure it’s a single model behind the scenes doing that? Practically it doesn’t really matter, but I’d like to know for sure.
thorum 2y ago
Wow - the most impressive thing about this is the control options. I’m not aware of any other TTS systems with the same balance of control, quality and language support. Looking forward to testing this out…
luyu_wu 2y ago
I have to say the Chinese female voice sounds the most natural. It's really amazing how far these have got! Video with examples: https://b23.tv/uumKPam https://b23.tv/uumKPam (bilibili)
rowanG077 2y ago
Is there any good voice2voice open source model?
thomasfromcdnjs 2y ago
I hadn't heard any good prosodic laugh implementations yet. In my mind that was the last hurdle to cross before being able to fool people regularly with a non-human voice. Great work! Hook that DSL into a prompt, [uv_break]gg. [laugh]
_lvbh 2y ago
Beyond a certain level of quality, what is the purpose of improving similarity with human voice other than scamming? I’m asking because I genuinely don’t know. It seems even a rudimentary TTS is usable as long as you can tell what it’s saying.
- edvards 2y ago
  In voice assistants, robocalls, e-books, even singing, live voice interpretation/translation... a lot of stuff.
  - _lvbh 2y ago
    Voice assistants -> Siri sounds just fine Robocalls -> I want to know I’m speaking to a robot Audio books -> reasonable. An accurate tone is pleasant Singing -> ever heard of vocaloids? They’ve existed for at least a decade or two
    - Ukv 2y ago
      > Singing -> ever heard of vocaloids? They’ve existed for at least a decade or two That it was technically already possible does not mean there isn't benefit from improved quality. In fact, Vocaloid itself has been improving and now uses AI. Would also add making movies, podcasts, news broadcasts, etc. available automatically in a huge range of languages. You wouldn't want movies dubbed by Microsoft Sam (beyond initial comedic effect).
      - miki123211 2y ago
        > You wouldn't want movies dubbed by Microsoft Sam (beyond initial comedic effect). You'd be surprised how common something like this used to be in Poland, though admittedly we used an Ivona voice for this, which was a lot more pleasant. Having a single narrator narrate the entire movie, overlaying the original audio track, is already common here, much more so than dubbing or subtitles. This is for historic reasons, in the communist era, obtaining the raw audio tracks for dubbing was often impossible, all the translators often had was a normal copy of the movie in its original language. In the early 2000's, we had a lot of early / unofficial pirate releases, and they had to be translated into Polish somehow. Subtitles were certainly one method, but as we're all used to the single-narrator style, many people didn't mind listening to a somewhat decent synthetic voice instead.
psychoslave 2y ago
Could it be used to teach me Mandarin? Actually since it's only voice synthesis, I guess it would still miss the voice recognition and capability the quality of my attempt to reproduce tonal language sentences.
stakhanov 2y ago
This is somewhat off-topic, but here goes: It seems to me that English TTS is already extremely good, even if you're looking at implementations that are far from being the best ones for English. ...and sometimes I wonder, if it's really economically efficient for that many players to compete on making English TTS yet another hair's breadth better than the next guy's, while TTS for languages other than English is this vast field of unmet market demand. At least these guys are doing Chinese, so: good for them. Last time I looked into TTS systems for German, Google was the only game in town. What I wouldn't give for a viable alternative! It doesn't even need to be open source, I'd be quite ready to pay top dollar.
- throwthrowuknow 2y ago
  https://elevenlabs.io/ https://elevenlabs.io/ And choose German for a sample. Not sure if it’s good enough for your needs but they have a wide variety of languages and voices.
  - stakhanov 2y ago
    Thanks for the pointer. This does indeed look pretty good. Do you happen to know if they're using any APIs underneath the covers? Or are they fully self-contained?
    - aprilnya 2y ago
      ElevenLabs is their own thing, they make the models themselves
- DelightOne 2y ago
  OpenAI's API says they support German? Never tried it though. https://platform.openai.com/docs/guides/text-to-speech https://platform.openai.com/docs/guides/text-to-speech
- follower 2y ago
  > Last time I looked into TTS systems for German, Google was the only game in town. What I wouldn't give for a viable alternative! It doesn't even need to be open source, I'd be quite ready to pay top dollar. Will you still pay top dollar if it is open source though? :D Piper TTS[0] (MIT Licensed; developed by main dev of Larynx TTS, Mimic3 TTS & Rhasspy voice assistant) has support for ~30 languages, at least some of which have multiple voices available--in a range of quality & data licenses. And, particularly fortuitous for your needs, potentially, there's at at least one German voice that was recorded[1] specifically for Piper[2] (with emotion variants and CC0-licensed, no less :) )... Check out `thorsten` & `thorsten_emotional` on the samples page: https://rhasspy.github.io/piper-samples/ https://rhasspy.github.io/piper-samples/ I can't speak to the quality of the German voice specifically but for English at least I've found Piper's quality & range of voices of use[3]. ---- footnotes ---- [0] https://github.com/rhasspy/piper https://github.com/rhasspy/piper [1] https://www.youtube.com/playlist?list=PL19C7uchWZeojjI5FUk3qU9jqGB05w5nE https://www.youtube.com/playlist?list=PL19C7uchWZeojjI5FUk3q... [2] In addition to other German voices based on other sources: https://huggingface.co/rhasspy/piper-voices/tree/main/de/de_DE https://huggingface.co/rhasspy/piper-voices/tree/main/de/de_... [3] Somewhat of an understatement.
camkego 2y ago
From the Readme: “ To limit the use of ChatTTS, we added a small amount of high-frequency noise during the training of the 40,000-hour model, and compressed the audio quality as much as possible using MP3 format, to prevent malicious actors from potentially using it for criminal purposes.” I’m having a hard time understanding why they have degraded the training, audio, and thus the output. It’s not like this is the first or only text to speech system.
- phoronixrly 2y ago
  So that it is more inconvenient to use theirs for such purposes... I would do the same.
GavCo 2y ago
Not clear based on what criteria OP has determined this is the best OS model. I also don't see that claim being made anywhere in the GitHub repo so I suspect it might be a case of vibe-based benchmarking (VBB). As pointed out by u/modeless, there is an established leaderboard and this model isn't on it (yet) https://news.ycombinator.com/item?id=40508445 https://news.ycombinator.com/item?id=40508445
rcarmo 2y ago
This is pretty decent, but a bit slow on my M2 Pro. Runs better on CPU, which is strange. Still, here's a quick guide to getting it to work on Metal: --requirements.txt additions-- torchvision==0.18.0 accelerate==0.30.1 --gpu_utils.py patch-- def select_device(min_memory = 2048): logger = logging.getLogger(__name__) if torch.backends.mps.is_available(): device = torch.device('mps') return device could probably do with support for device_map for multiple backends... Edit: it also seems tho hallucinate/become increasingly unreliable with longer sentences.
deleted 2y ago
[deleted]
chengxp 2y ago
[dead]
theking1905 2y ago
Is this supporting voice clonning?

165 points by informal007 2y ago

ex3ndr 2y ago

Looks like it is yet another xtts fork

charlie0simmon 2y ago
[dead]

cchance 2y ago

Sounds good but feel like theirs something slightly off to the cadence of the voice in the sample, but maybe i'm imagining

informal007 2y ago
I think you are right, It is not the completely same.

estheryo 2y ago

The completion level is impressive! I can hardly tell the difference from a human voice, especially with the natural pauses and laughter, which surpasses ChatGPT’s quality. However, there’s a noticeable electric noise at the end of sentences, which feels unnatural. (As a native Chinese speaker, I find the Chinese output even better in comparison.)

informal007 2y ago
Yes, it's just a new born project, looking forward the next verison.
nshm 2y ago
There is also a glitch in "dialogue"

maxglute 2y ago

Sounds natural and intelligible at 3x speed, which is plus. >The Real-Time Factor (RTF) is around 0.65. What is the state of real time tts models?

regularfry 2y ago
0.65 isn't that big a gap. This is probably less than one jart away from being optimised to real-time.
c0brac0bra 2y ago
Aura can stream responses and has a time-to-first-byte around 200-400ms

NobodyNada 2y ago

> Attribution-NonCommercial-NoDerivatives 4.0 International Strictly speaking, this is not open source, as the commonly accepted definitions of open-source software include freedom of use and modification. But in an industry where "OpenAI" is 100% proprietary, I guess "open-source" doesn't really mean much.

abetusk 2y ago
Note that the repo itself doesn't claim it's open source, it's the title of the link that (incorrectly) claims it is.
- simonw 2y ago
  Yeah, I think the link title should be "ChatTTS: Best openly licensed TTS model" instead.
- follower 2y ago
  The repo does claim that something is "open source" but the only included license text is "CC-BY-NC-ND" and the README seems to restrict permissions even further. In addition, the Hugging Face repo[-1] states the license as "Creative Commons Attribution Non Commercial 4.0" (lacking the "ND"). Unfortunately, this combination of license imprecision and restrictiveness seems par for the course with a lot of academic TTS projects. (And, even for commercial "Open Source" TTS projects it's often the case that while code might be OSS licensed, none of the the voice data/models are.) The current version[0] of the README repo states: * "The open-source version on HuggingFace is a 40,000 hours pre trained model without SFT." (Presumably refers to model.) * "At the same time, we have internally trained a detection model and plan to open-source it in the future." (Not directly relevant.) The included "Roadmap" indicates related completed & uncompleted tasks: * "[x] Open-source the 40k hour base model and spk_stats file" * "[ ] Open-source VQ encoder and Lora training code" * "[ ] Open-source the 40k hour version with multi-emotion control" However, as noted, the current LICENSE[1] file states: * "Attribution-NonCommercial-NoDerivatives 4.0 International" And the README also contradicts the license: * "This repo is for academic purposes only. It is intended for educational and research use, and should not be used for any commercial or legal purposes." * "The information and data used in this repo, are for academic and research purposes only." And this part of the "disclaimer" would make me concerned about potential licensing issues in regard to code and or data from other sources: * "The data obtained from publicly available sources, and the authors do not claim any ownership or copyright over the data." The code in the repo itself appears to have no license information contained within it. My go-to actually Open Source licensed Text-To-Speech project (with a range of voice[2] model licenses[3]--including Public Domain & CC-BY[4]) is Piper TTS: https://github.com/rhasspy/piper https://github.com/rhasspy/piper ---- footnotes ---- [-1] https://huggingface.co/2Noise/ChatTTS https://huggingface.co/2Noise/ChatTTS [0] https://github.com/2noise/ChatTTS/blob/f4c8329f0d231b272b676e5e171fb9655b345f2e/README.md https://github.com/2noise/ChatTTS/blob/f4c8329f0d231b272b676... [1] https://github.com/2noise/ChatTTS/blob/f4c8329f0d231b272b676e5e171fb9655b345f2e/LICENSE https://github.com/2noise/ChatTTS/blob/f4c8329f0d231b272b676... [2] Voice samples: https://rhasspy.github.io/piper-samples/ https://rhasspy.github.io/piper-samples/ [3] Though I would also caution that (at least by my interpretation) some of the voices listed as CC0/PD or CC-BY also note that they've been "fine-tuned" on models which have more restrictive licenses and thus probably can't inherit the voice data's more permissive license. [4] However, these Public Domain and/or CC-BY voices do appear specifically created to be data license-compliant: https://brycebeattie.com/files/tts/ https://brycebeattie.com/files/tts/

JoeDeanx 2y ago

Where is the demo that can be used?

informal007 2y ago
You can try to run this code to use https://github.com/2noise/ChatTTS/blob/main/infer.ipynb https://github.com/2noise/ChatTTS/blob/main/infer.ipynb
- JoeDeanx 2y ago
  I don't know how to code, but I found a website that I can use online,https://chattts.com/ https://chattts.com/

modeless 2y ago

A good time to link the TTS leaderboard: https://huggingface.co/spaces/TTS-AGI/TTS-Arena https://huggingface.co/spaces/TTS-AGI/TTS-Arena Eleven Labs is still very far above open source models in quality. But StyleTTS2 (MIT license) is impressively good and quite fast. It'll be interesting to see where this new one ends up. The code-switching ability is quite interesting. Most open source TTS models are strictly one language per sentence, often one language per voice. In general though, TTS as an isolated system is mostly a dead end IMO. The future is in multimodal end-to-end audio-to-audio (or anything-to-audio) models, as demonstrated by OpenAI with GPT-4o's voice mode (though I've been saying this since long before their demo: https://news.ycombinator.com/item?id=38339222 https://news.ycombinator.com/item?id=38339222). Text is very useful as training data but as a way to represent other modalities like audio or image data it is far too lossy.

dartos 2y ago
Isn’t gpt 4o voice not audio to audio, but audio to text to audio?
- modeless 2y ago
  It isn't released yet, but the new one that they demoed is audio-to-audio. That's why it can do things like sing songs, and detect emotion in voices. The one that you can currently access in the ChatGPT app (with subscription) is the old one which is ASR->LLM->TTS.
  - addandsubtract 2y ago
    I'm pretty sure you can use the new GPT4o audio-to-audio model already – even without a subscription. You can even use the "Sky" model if you didn't update your app.
  - dartos 2y ago
    Are we sure it’s a single model behind the scenes doing that? Practically it doesn’t really matter, but I’d like to know for sure.

thorum 2y ago

Wow - the most impressive thing about this is the control options. I’m not aware of any other TTS systems with the same balance of control, quality and language support. Looking forward to testing this out…

luyu_wu 2y ago

I have to say the Chinese female voice sounds the most natural. It's really amazing how far these have got! Video with examples: https://b23.tv/uumKPam https://b23.tv/uumKPam (bilibili)

rowanG077 2y ago

Is there any good voice2voice open source model?

thomasfromcdnjs 2y ago

I hadn't heard any good prosodic laugh implementations yet. In my mind that was the last hurdle to cross before being able to fool people regularly with a non-human voice. Great work! Hook that DSL into a prompt, [uv_break]gg. [laugh]

_lvbh 2y ago

Beyond a certain level of quality, what is the purpose of improving similarity with human voice other than scamming? I’m asking because I genuinely don’t know. It seems even a rudimentary TTS is usable as long as you can tell what it’s saying.

edvards 2y ago
In voice assistants, robocalls, e-books, even singing, live voice interpretation/translation... a lot of stuff.
- _lvbh 2y ago
  Voice assistants -> Siri sounds just fine Robocalls -> I want to know I’m speaking to a robot Audio books -> reasonable. An accurate tone is pleasant Singing -> ever heard of vocaloids? They’ve existed for at least a decade or two
  - Ukv 2y ago
    > Singing -> ever heard of vocaloids? They’ve existed for at least a decade or two That it was technically already possible does not mean there isn't benefit from improved quality. In fact, Vocaloid itself has been improving and now uses AI. Would also add making movies, podcasts, news broadcasts, etc. available automatically in a huge range of languages. You wouldn't want movies dubbed by Microsoft Sam (beyond initial comedic effect).
    - miki123211 2y ago
      > You wouldn't want movies dubbed by Microsoft Sam (beyond initial comedic effect). You'd be surprised how common something like this used to be in Poland, though admittedly we used an Ivona voice for this, which was a lot more pleasant. Having a single narrator narrate the entire movie, overlaying the original audio track, is already common here, much more so than dubbing or subtitles. This is for historic reasons, in the communist era, obtaining the raw audio tracks for dubbing was often impossible, all the translators often had was a normal copy of the movie in its original language. In the early 2000's, we had a lot of early / unofficial pirate releases, and they had to be translated into Polish somehow. Subtitles were certainly one method, but as we're all used to the single-narrator style, many people didn't mind listening to a somewhat decent synthetic voice instead.

psychoslave 2y ago

Could it be used to teach me Mandarin? Actually since it's only voice synthesis, I guess it would still miss the voice recognition and capability the quality of my attempt to reproduce tonal language sentences.

stakhanov 2y ago

This is somewhat off-topic, but here goes: It seems to me that English TTS is already extremely good, even if you're looking at implementations that are far from being the best ones for English. ...and sometimes I wonder, if it's really economically efficient for that many players to compete on making English TTS yet another hair's breadth better than the next guy's, while TTS for languages other than English is this vast field of unmet market demand. At least these guys are doing Chinese, so: good for them. Last time I looked into TTS systems for German, Google was the only game in town. What I wouldn't give for a viable alternative! It doesn't even need to be open source, I'd be quite ready to pay top dollar.

throwthrowuknow 2y ago
https://elevenlabs.io/ https://elevenlabs.io/ And choose German for a sample. Not sure if it’s good enough for your needs but they have a wide variety of languages and voices.
- stakhanov 2y ago
  Thanks for the pointer. This does indeed look pretty good. Do you happen to know if they're using any APIs underneath the covers? Or are they fully self-contained?
  - aprilnya 2y ago
    ElevenLabs is their own thing, they make the models themselves
DelightOne 2y ago
OpenAI's API says they support German? Never tried it though. https://platform.openai.com/docs/guides/text-to-speech https://platform.openai.com/docs/guides/text-to-speech
follower 2y ago
> Last time I looked into TTS systems for German, Google was the only game in town. What I wouldn't give for a viable alternative! It doesn't even need to be open source, I'd be quite ready to pay top dollar. Will you still pay top dollar if it is open source though? :D Piper TTS[0] (MIT Licensed; developed by main dev of Larynx TTS, Mimic3 TTS & Rhasspy voice assistant) has support for ~30 languages, at least some of which have multiple voices available--in a range of quality & data licenses. And, particularly fortuitous for your needs, potentially, there's at at least one German voice that was recorded[1] specifically for Piper[2] (with emotion variants and CC0-licensed, no less :) )... Check out `thorsten` & `thorsten_emotional` on the samples page: https://rhasspy.github.io/piper-samples/ https://rhasspy.github.io/piper-samples/ I can't speak to the quality of the German voice specifically but for English at least I've found Piper's quality & range of voices of use[3]. ---- footnotes ---- [0] https://github.com/rhasspy/piper https://github.com/rhasspy/piper [1] https://www.youtube.com/playlist?list=PL19C7uchWZeojjI5FUk3qU9jqGB05w5nE https://www.youtube.com/playlist?list=PL19C7uchWZeojjI5FUk3q... [2] In addition to other German voices based on other sources: https://huggingface.co/rhasspy/piper-voices/tree/main/de/de_DE https://huggingface.co/rhasspy/piper-voices/tree/main/de/de_... [3] Somewhat of an understatement.

camkego 2y ago

From the Readme: “ To limit the use of ChatTTS, we added a small amount of high-frequency noise during the training of the 40,000-hour model, and compressed the audio quality as much as possible using MP3 format, to prevent malicious actors from potentially using it for criminal purposes.” I’m having a hard time understanding why they have degraded the training, audio, and thus the output. It’s not like this is the first or only text to speech system.

phoronixrly 2y ago
So that it is more inconvenient to use theirs for such purposes... I would do the same.

GavCo 2y ago

Not clear based on what criteria OP has determined this is the best OS model. I also don't see that claim being made anywhere in the GitHub repo so I suspect it might be a case of vibe-based benchmarking (VBB). As pointed out by u/modeless, there is an established leaderboard and this model isn't on it (yet) https://news.ycombinator.com/item?id=40508445 https://news.ycombinator.com/item?id=40508445

rcarmo 2y ago

This is pretty decent, but a bit slow on my M2 Pro. Runs better on CPU, which is strange. Still, here's a quick guide to getting it to work on Metal: --requirements.txt additions-- torchvision==0.18.0 accelerate==0.30.1 --gpu_utils.py patch-- def select_device(min_memory = 2048): logger = logging.getLogger(__name__) if torch.backends.mps.is_available(): device = torch.device('mps') return device could probably do with support for device_map for multiple backends... Edit: it also seems tho hallucinate/become increasingly unreliable with longer sentences.

deleted 2y ago

[deleted]

chengxp 2y ago

[dead]

theking1905 2y ago

Is this supporting voice clonning?