5 ms·

TTS still sucks

64 points by speckx 1y ago

skybrian 1y ago
...if you care about voice cloning. Maybe that's not so important?
lxe 1y ago
That's because you're sleeping on things like Higgs Audio
- meatmanek 1y ago
  If this demo video[1] is indicative of what you can expect, I'm not particularly impressed. For me, every single one of the recordings fell all the way to the bottom of the uncanny valley. 1. https://github.com/user-attachments/assets/0fd73fad-097f-48a9-9f3f-bc2a63b3818d https://github.com/user-attachments/assets/0fd73fad-097f-48a...
  - hackingonempty 1y ago
    I would think the only way to fairly evaluate the performance of these models as they approach that of professional human voice actors is to evaluate them against humans in a sufficiently powered randomly controlled and blinded trial.
    - zamadatix 1y ago
      Agreed, but this doesn't approach professional human voice actors - unless they are acting like an android.
  - rurban 1y ago
    I recently watched the first AI dubbed movie, and it was as horrible as this.
Karrot_Kream 1y ago
A big caveat of this is that the author is just looking at a ranking of open models? It's just buried in a little sentence in there but makes a big difference to model quality. Kokoro in the overall rankings is only 15, so if #15 is what you consider the "best model" you need to be cognizant that you are leaving performance on the table. I've heard a lot of Substacks voiced by Eleven Labs models and they seem fine (with the occasional weirdness around a proper noun.) Not a bad article but I think more examples of TTS usage would be more useful. I guess the outcome is, open weight TTS models are only okay and could be a lot better?
- regulation_d 1y ago
  Yeah, from my experience the more helpful conclusion is "TTS is not commoditized yet". At some point in the next 5 years, convincing TTS will be table stakes. But for now, paying for TTS gets you better results.
- deleted 1y ago
  [deleted]
- TheAceOfHearts 1y ago
  The paid models are still too expensive for personal long-form use-cases. For example: if I want to generate an audiobook from a web novel, the price can go as high as thousands of dollars. If I'm just a regular reader (not the author), that's prohibitively expensive for someone who just wants to enjoy the story in a different medium.
  - BoorishBears 1y ago
    Despite ElevenLabs API usage being expensive, ElevenReader is $11 a month for unlimited personal long-form content. Even with a local model and hardware you already own, you're not beating that on electricity costs.
    - rubyn00bie 1y ago
      I dunno about the electricity claims for practical purposes— where I live, that’d be roughly 128 hours of 600W. I suppose the real question is, would it take 128 hours ($11) of power to generate 720 hours of TTS content assuming a good enough model was available? On Android I’d imagine, big emphasis on imagine since I don’t use it, you could probably script something up and use a phone with an audio jack to record it. Theoretically hitting that maximum of 720 hours of content per month, but I’d imagine at some point they’d find it peculiar you’re listening to content 24/7.
fleshmonad 1y ago
It's very interesting to see that there actually are people who want to automatically create a "podcast" from their blog using their cloned voice. Is this just what tech bro culture does to someone? Or is it about hustling and grinding while getting your very important word out there. I mean over time one would certainly save up to 20 minutes for each article...
- bigfishrunning 1y ago
  Exactly! Why would I want to listen to a written article instead of just reading it? Also, I suspect these AI-Podcast blogs are probably just generated with AI too, so it's likely safe to skip the whole mess
  - raw_anon_1111 1y ago
    For me it would be when I’m driving or working out. But I can’t imagine listening to an AI generated podcast. I do listen to the Stratechery podcast that is the same as the email. But he also not only reads it himself, he has someone else narrate quotes and he uses chapter art that goes along with the article.
  - tonyarkles 1y ago
    To some degree, you could make the same argument about written books and audio books. Mostly I listen to audiobooks because I’m often bored in the car and learning something seems like a good use of my time.
    - bigfishrunning 1y ago
      An audio book is usually read by an actor, or the author, and often has a performative quality to it. I'm honestly not interested in anything read by a machine unless it's absolutely indistinguishable from a professional person, and current machine-voice-synthesis (AI or not) isn't there.
  - TiredOfLife 1y ago
    Wait till you get old and have trouble reading.
imiric 1y ago
> Anything over 1000 characters starts hallucinating. So just feed it batches smaller than 1000 characters? It's not like TTS requires maintaining large contexts at a time.
- simlevesque 1y ago
  Context helps guessing what the next word will be.
  - tonyarkles 1y ago
    With no significant background in ML-based TTS, I’m assuming that a larger context window would help with tone as well. “We are gathered here today to mourn the loss of…” really provides context into how the whole thing might sound, even if most of it is singing the praises of the deceased.
  - zahlman 1y ago
    If you've been given 1000 characters (a fairly long paragraph) of text to read (and supposing you get to study them before you start speaking), is "guessing what the next word will be" all that relevant to decisions about intonation?
    - BoorishBears 1y ago
      They in the right direction, but wrong specifics: you can't maintain consistent prosody without context. The simplest examples are punctuation marks which change your speech before you reach the mark, but the problem extends past sentence boundaries For example: "He didn't steal the green car. He borrowed it." vs "He didn't steal the green car. He stole the red one." A natural speaker would slightly emphasize steal and borrowed in the 1st example, but emphasize green and red in the 2nd. Or like when you're building a set: "Peter called Mary." vs "John called Mary. Peter called Mary. Who didn't call Mary?" - These all sound like small nits but for naively stitched together TTS, at best they nudge the narration towards the uncanny valley (which may be acceptable for some usecases)... but at worst they make the model sound broken.
superkuh 1y ago
For local TTS for a podcast I'd try the quantized .gguf versions of Microsoft VibeVoice large in comfyui to clone my voice from a ~30 second speech sample the apply it to marked-up text of the desired podcast. But it'd be nowhere near real time and require dedicating a $300 GPU to it. And the quantized version often goes off the rails and loses consistency in voice tone or accent. So just one run often isn't enough and I you have to piece the good parts of many separate runs together. It's not set it and forget it. I do a lot of desktop screen-reader and pdf/doc/epub/etc text to speech every single day. It's been 20 years and I still use Festival 1.96 TTS with voice_nitech_us_slt_arctic_hts voice because it's so computational cheap and just slightly a step about normal festival/espeak/mbrolla/etc type TTS quality to be clear and tolerable. In terms of this local "do screenreader stuff really fast" use case I've tried modern TTS like vibevoice, kokoro tts, sherpa-onx, piper tts, orpheus tts, etc. And they all have consistency issues, many are way too slow even with a $300 GPU dedicated to them, and most output weird garble noises at unpredictable times along with the good output.
- derac 1y ago
  In my experience, VibeVoice is quite good as well. Even the smaller model.
- OfflineSergio 1y ago
  >I do a lot of desktop screen-reader and pdf/doc/epub/etc text to speech every single day. I've been working a product called WithAudio (https://with.audio https://with.audio). Are you open to me reaching out and give a free license so you can use it and let me know what you think ? I should say it only supports Windows and Mac(arm). I'm looking for people who have used similar products to get their feedback.
mcny 1y ago
Slightly off-topic but Why would a blog of all things have DRM content that I need to enable?
- jsheard 1y ago
  It looks like the inline Apple Podcasts player is causing that, though it's not clear why since it's loading unencrypted MP3s directly from the authors S3 bucket. I guess their player eagerly sets up DRM playback at startup rather than waiting until it's needed, or they're using EME for something else (fingerprinting?).
observationist 1y ago
Superhuman TTS is well within the capabilities of the big AI labs. Even Google had voice indistinguishable from human back in 2017, but they deliberately kneecapped it because of the potential for misuse. Boomers and older folks are not culturally or mentally equipped to handle it - even the crappy open source voice cloning we had in 2019 got used to scam people into buying gift cards. Because of the potential for abuse, nobody wants to release a truly good, general model, because it makes lawyers lose sleep. A few more generations of hardware, though, and there will be enough open data and DIY scaffolding out there to produce a superhuman model, and someone will release it. Deepfake video is already indistinguishable from real video (not oneshot prompt video generation, but deliberate skilled craft using AI tools.) Higgsfield and other tools allow for spectacular voice results, but it takes craft and care. The oneshot stuff is deliberately underpowered. OpenAI doesn't want to be responsible for a viral pitch-perfect campaign ad, or fake scandal video sinking a politician, for example. Once the lawyers calm down, or we get a decent digital bill of rights that establishes clear accountability on the user of the tool, and not the toolmaker, things should get better. Until then, look for the rogue YOLO boutique services or the ambitious open source crew to be the first to superhuman, widely available TTS.
- fortran77 1y ago
  Stop blaming us old people for your lack of good TTS models.
  - onedognight 1y ago
    Username checks out.
- bigfishrunning 1y ago
  > Boomers and older folks are not culturally or mentally equipped to handle it I think a lot of younger people are also not mentally equipped to handle it. Outside of the hackernews sphere of influence, people are really bad at spotting AI slop (and also really bad at caring about it)
  - 1y ago
andrewstuart 1y ago
Commercial TTS mostly sucks too. There’s flashes of brilliance but most of it is noticeably computer generated.
- horhay 1y ago
  The Gemini models and Eleven V3, and whatever internal audio model Sora 2 uses are about neck and neck in converging performance. They have some unexplainable flavor to them though. Especially Sora.
actuallyalys 1y ago
While it sounds like this blogger doesn't want to bother (and perhaps experimenting with AI is itself the appeal), I personally appreciate when authors read their posts instead of delegating the task to AI.
neilv 1y ago
> After filtering by my stupid rule of open models, That's a good rule. > You must enable DRM to play some audio or video on this page. Looks like `embed.podcasts.apple.com` isn't in the same spirit.
AlienRobot 1y ago
>However, like many models in this leaderboard - I can’t use it - since it doesn’t support voice cloning. That's such a strange requirement. A TTS is just that. It takes a text and speaks it out loud. The user generally doesn't care whose voice it is, and personally I think TTS's sharing the same voice is a good thing for authenticity since it lets users know that it's a TTS reading the script and not a real person. You want your voice to be reading the script, but you don't want to personally record yourself reading the text? As far as I'm concerned that's an edge case. No wonder that TTS's can't do that properly since most people don't need that in first place.
RLAIF 1y ago
[dead]
lielvilla 1y ago
I wouldn’t say suck, but it’s nowhere near plug-and-play yet. Totally agree on the pain points - I covered similar thoughts in my post: https://lielvilla.com/blog/death-of-demo/ https://lielvilla.com/blog/death-of-demo/

TTS still sucks

64 points by speckx 1y ago

skybrian 1y ago

...if you care about voice cloning. Maybe that's not so important?

lxe 1y ago

That's because you're sleeping on things like Higgs Audio

meatmanek 1y ago
If this demo video[1] is indicative of what you can expect, I'm not particularly impressed. For me, every single one of the recordings fell all the way to the bottom of the uncanny valley. 1. https://github.com/user-attachments/assets/0fd73fad-097f-48a9-9f3f-bc2a63b3818d https://github.com/user-attachments/assets/0fd73fad-097f-48a...
- hackingonempty 1y ago
  I would think the only way to fairly evaluate the performance of these models as they approach that of professional human voice actors is to evaluate them against humans in a sufficiently powered randomly controlled and blinded trial.
  - zamadatix 1y ago
    Agreed, but this doesn't approach professional human voice actors - unless they are acting like an android.
- rurban 1y ago
  I recently watched the first AI dubbed movie, and it was as horrible as this.

Karrot_Kream 1y ago

A big caveat of this is that the author is just looking at a ranking of open models? It's just buried in a little sentence in there but makes a big difference to model quality. Kokoro in the overall rankings is only 15, so if #15 is what you consider the "best model" you need to be cognizant that you are leaving performance on the table. I've heard a lot of Substacks voiced by Eleven Labs models and they seem fine (with the occasional weirdness around a proper noun.) Not a bad article but I think more examples of TTS usage would be more useful. I guess the outcome is, open weight TTS models are only okay and could be a lot better?

regulation_d 1y ago
Yeah, from my experience the more helpful conclusion is "TTS is not commoditized yet". At some point in the next 5 years, convincing TTS will be table stakes. But for now, paying for TTS gets you better results.
deleted 1y ago
[deleted]
TheAceOfHearts 1y ago
The paid models are still too expensive for personal long-form use-cases. For example: if I want to generate an audiobook from a web novel, the price can go as high as thousands of dollars. If I'm just a regular reader (not the author), that's prohibitively expensive for someone who just wants to enjoy the story in a different medium.
- BoorishBears 1y ago
  Despite ElevenLabs API usage being expensive, ElevenReader is $11 a month for unlimited personal long-form content. Even with a local model and hardware you already own, you're not beating that on electricity costs.
  - rubyn00bie 1y ago
    I dunno about the electricity claims for practical purposes— where I live, that’d be roughly 128 hours of 600W. I suppose the real question is, would it take 128 hours ($11) of power to generate 720 hours of TTS content assuming a good enough model was available? On Android I’d imagine, big emphasis on imagine since I don’t use it, you could probably script something up and use a phone with an audio jack to record it. Theoretically hitting that maximum of 720 hours of content per month, but I’d imagine at some point they’d find it peculiar you’re listening to content 24/7.

fleshmonad 1y ago

It's very interesting to see that there actually are people who want to automatically create a "podcast" from their blog using their cloned voice. Is this just what tech bro culture does to someone? Or is it about hustling and grinding while getting your very important word out there. I mean over time one would certainly save up to 20 minutes for each article...

bigfishrunning 1y ago
Exactly! Why would I want to listen to a written article instead of just reading it? Also, I suspect these AI-Podcast blogs are probably just generated with AI too, so it's likely safe to skip the whole mess
- raw_anon_1111 1y ago
  For me it would be when I’m driving or working out. But I can’t imagine listening to an AI generated podcast. I do listen to the Stratechery podcast that is the same as the email. But he also not only reads it himself, he has someone else narrate quotes and he uses chapter art that goes along with the article.
- tonyarkles 1y ago
  To some degree, you could make the same argument about written books and audio books. Mostly I listen to audiobooks because I’m often bored in the car and learning something seems like a good use of my time.
  - bigfishrunning 1y ago
    An audio book is usually read by an actor, or the author, and often has a performative quality to it. I'm honestly not interested in anything read by a machine unless it's absolutely indistinguishable from a professional person, and current machine-voice-synthesis (AI or not) isn't there.
- TiredOfLife 1y ago
  Wait till you get old and have trouble reading.

imiric 1y ago

> Anything over 1000 characters starts hallucinating. So just feed it batches smaller than 1000 characters? It's not like TTS requires maintaining large contexts at a time.

simlevesque 1y ago
Context helps guessing what the next word will be.
- tonyarkles 1y ago
  With no significant background in ML-based TTS, I’m assuming that a larger context window would help with tone as well. “We are gathered here today to mourn the loss of…” really provides context into how the whole thing might sound, even if most of it is singing the praises of the deceased.
- zahlman 1y ago
  If you've been given 1000 characters (a fairly long paragraph) of text to read (and supposing you get to study them before you start speaking), is "guessing what the next word will be" all that relevant to decisions about intonation?
  - BoorishBears 1y ago
    They in the right direction, but wrong specifics: you can't maintain consistent prosody without context. The simplest examples are punctuation marks which change your speech before you reach the mark, but the problem extends past sentence boundaries For example: "He didn't steal the green car. He borrowed it." vs "He didn't steal the green car. He stole the red one." A natural speaker would slightly emphasize steal and borrowed in the 1st example, but emphasize green and red in the 2nd. Or like when you're building a set: "Peter called Mary." vs "John called Mary. Peter called Mary. Who didn't call Mary?" - These all sound like small nits but for naively stitched together TTS, at best they nudge the narration towards the uncanny valley (which may be acceptable for some usecases)... but at worst they make the model sound broken.

superkuh 1y ago

For local TTS for a podcast I'd try the quantized .gguf versions of Microsoft VibeVoice large in comfyui to clone my voice from a ~30 second speech sample the apply it to marked-up text of the desired podcast. But it'd be nowhere near real time and require dedicating a $300 GPU to it. And the quantized version often goes off the rails and loses consistency in voice tone or accent. So just one run often isn't enough and I you have to piece the good parts of many separate runs together. It's not set it and forget it. I do a lot of desktop screen-reader and pdf/doc/epub/etc text to speech every single day. It's been 20 years and I still use Festival 1.96 TTS with voice_nitech_us_slt_arctic_hts voice because it's so computational cheap and just slightly a step about normal festival/espeak/mbrolla/etc type TTS quality to be clear and tolerable. In terms of this local "do screenreader stuff really fast" use case I've tried modern TTS like vibevoice, kokoro tts, sherpa-onx, piper tts, orpheus tts, etc. And they all have consistency issues, many are way too slow even with a $300 GPU dedicated to them, and most output weird garble noises at unpredictable times along with the good output.

derac 1y ago
In my experience, VibeVoice is quite good as well. Even the smaller model.
OfflineSergio 1y ago
>I do a lot of desktop screen-reader and pdf/doc/epub/etc text to speech every single day. I've been working a product called WithAudio (https://with.audio https://with.audio). Are you open to me reaching out and give a free license so you can use it and let me know what you think ? I should say it only supports Windows and Mac(arm). I'm looking for people who have used similar products to get their feedback.

mcny 1y ago

Slightly off-topic but Why would a blog of all things have DRM content that I need to enable?

jsheard 1y ago
It looks like the inline Apple Podcasts player is causing that, though it's not clear why since it's loading unencrypted MP3s directly from the authors S3 bucket. I guess their player eagerly sets up DRM playback at startup rather than waiting until it's needed, or they're using EME for something else (fingerprinting?).

observationist 1y ago

Superhuman TTS is well within the capabilities of the big AI labs. Even Google had voice indistinguishable from human back in 2017, but they deliberately kneecapped it because of the potential for misuse. Boomers and older folks are not culturally or mentally equipped to handle it - even the crappy open source voice cloning we had in 2019 got used to scam people into buying gift cards. Because of the potential for abuse, nobody wants to release a truly good, general model, because it makes lawyers lose sleep. A few more generations of hardware, though, and there will be enough open data and DIY scaffolding out there to produce a superhuman model, and someone will release it. Deepfake video is already indistinguishable from real video (not oneshot prompt video generation, but deliberate skilled craft using AI tools.) Higgsfield and other tools allow for spectacular voice results, but it takes craft and care. The oneshot stuff is deliberately underpowered. OpenAI doesn't want to be responsible for a viral pitch-perfect campaign ad, or fake scandal video sinking a politician, for example. Once the lawyers calm down, or we get a decent digital bill of rights that establishes clear accountability on the user of the tool, and not the toolmaker, things should get better. Until then, look for the rogue YOLO boutique services or the ambitious open source crew to be the first to superhuman, widely available TTS.

fortran77 1y ago
Stop blaming us old people for your lack of good TTS models.
- onedognight 1y ago
  Username checks out.
bigfishrunning 1y ago
> Boomers and older folks are not culturally or mentally equipped to handle it I think a lot of younger people are also not mentally equipped to handle it. Outside of the hackernews sphere of influence, people are really bad at spotting AI slop (and also really bad at caring about it)
- 1y ago

andrewstuart 1y ago

Commercial TTS mostly sucks too. There’s flashes of brilliance but most of it is noticeably computer generated.

horhay 1y ago
The Gemini models and Eleven V3, and whatever internal audio model Sora 2 uses are about neck and neck in converging performance. They have some unexplainable flavor to them though. Especially Sora.

actuallyalys 1y ago

While it sounds like this blogger doesn't want to bother (and perhaps experimenting with AI is itself the appeal), I personally appreciate when authors read their posts instead of delegating the task to AI.

neilv 1y ago

> After filtering by my stupid rule of open models, That's a good rule. > You must enable DRM to play some audio or video on this page. Looks like `embed.podcasts.apple.com` isn't in the same spirit.

AlienRobot 1y ago

>However, like many models in this leaderboard - I can’t use it - since it doesn’t support voice cloning. That's such a strange requirement. A TTS is just that. It takes a text and speaks it out loud. The user generally doesn't care whose voice it is, and personally I think TTS's sharing the same voice is a good thing for authenticity since it lets users know that it's a TTS reading the script and not a real person. You want your voice to be reading the script, but you don't want to personally record yourself reading the text? As far as I'm concerned that's an edge case. No wonder that TTS's can't do that properly since most people don't need that in first place.

RLAIF 1y ago

[dead]

lielvilla 1y ago

I wouldn’t say suck, but it’s nowhere near plug-and-play yet. Totally agree on the pain points - I covered similar thoughts in my post: https://lielvilla.com/blog/death-of-demo/ https://lielvilla.com/blog/death-of-demo/