5 ms·

A good time to link the TTS leaderboard: https://huggingface.co/spaces/TTS-AGI/TTS-Arena https://huggingface.co/spaces/TTS-AGI/TTS-Arena Eleven Labs is still v

A good time to link the TTS leaderboard: https://huggingface.co/spaces/TTS-AGI/TTS-Arena https://huggingface.co/spaces/TTS-AGI/TTS-Arena Eleven Labs is still very far above open source models in quality. But StyleTTS2 (MIT license) is impressively good and quite fast. It'll be interesting to see where this new one ends up. The code-switching ability is quite interesting. Most open source TTS models are strictly one language per sentence, often one language per voice. In general though, TTS as an isolated system is mostly a dead end IMO. The future is in multimodal end-to-end audio-to-audio (or anything-to-audio) models, as demonstrated by OpenAI with GPT-4o's voice mode (though I've been saying this since long before their demo: https://news.ycombinator.com/item?id=38339222 https://news.ycombinator.com/item?id=38339222). Text is very useful as training data but as a way to represent other modalities like audio or image data it is far too lossy.

dartos 2y ago
Isn’t gpt 4o voice not audio to audio, but audio to text to audio?
- modeless 2y ago
  It isn't released yet, but the new one that they demoed is audio-to-audio. That's why it can do things like sing songs, and detect emotion in voices. The one that you can currently access in the ChatGPT app (with subscription) is the old one which is ASR->LLM->TTS.
  - addandsubtract 2y ago
    I'm pretty sure you can use the new GPT4o audio-to-audio model already – even without a subscription. You can even use the "Sky" model if you didn't update your app.
  - dartos 2y ago
    Are we sure it’s a single model behind the scenes doing that? Practically it doesn’t really matter, but I’d like to know for sure.
    - modeless 2y ago
      It's the second paragraph in their announcement blog post. https://openai.com/index/hello-gpt-4o/ https://openai.com/index/hello-gpt-4o/ > Prior to GPT-4o, you could use Voice Mode to talk to ChatGPT with latencies of 2.8 seconds (GPT-3.5) and 5.4 seconds (GPT-4) on average. To achieve this, Voice Mode is a pipeline of three separate models: one simple model transcribes audio to text, GPT-3.5 or GPT-4 takes in text and outputs text, and a third simple model converts that text back to audio. This process means that the main source of intelligence, GPT-4, loses a lot of information—it can’t directly observe tone, multiple speakers, or background noises, and it can’t output laughter, singing, or express emotion. > With GPT-4o, we trained a single new model end-to-end across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network.
ugh123 2y ago
> In general though, TTS as an isolated system is mostly a dead end IMO Do you mean like as a simple text to speech application? There is a huge need for better quality audiobook output.
- modeless 2y ago
  I don't think recording an audiobook with human-level quality is "simple". It's really a kind of acting. TTS models do very poorly at acting because they generally process one sentence at a time, or at most a paragraph, and have very little context or understanding. They just kind of fake it like a newscaster reading an unrehearsed script from a teleprompter. True human-level audiobook reading would require understanding the whole story, which often assumes general cultural knowledge, which you'll only get from a model trained on LLM-scale data. If you asked GPT-4o's new end-to-end voice mode to read an audiobook you'd probably get a better result than any TTS model. I bet it would even do different voices for the characters if you asked it to.
  - maxglute 2y ago
    Maybe authors can tag sentences/paragraphs with acting directions while they write to facilitate acting. Seems like there's ways for some human input to streamline process.
    - modeless 2y ago
      Approaches based on tagging and interpreting metadata are tempting. Building structured human knowledge into the system seems like a good idea. But ultimately it isn't scalable or effective compared to general learning from large amounts of data. It's exactly the famous Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html To the extent that authors provide supplemental notes or instructions to human actors reading their books, that information would be helpful to provide to an automated audiobook reading model. But there is no reason for it to be in a different form than a human actor would get. Additional structure or detail would be neither necessary nor helpful.
aprilnya 2y ago
Why isn’t Microsoft Azure’s TTS on here? (or am I missing something and it’s called something else)
- modeless 2y ago
  Many proprietary ones are missing including OpenAI. I'd guess that they don't have budget to pay for the API usage. I think the leaderboard is more focused on open source options.
presentation 2y ago
This one does seem like it does multiple languages in a sentence (at least for its currently supported languages, Chinese and English): https://www.bilibili.com/video/BV1zn4y1o7iV/?share_source=copy_web&t=26 https://www.bilibili.com/video/BV1zn4y1o7iV/?share_source=co... But it does seem like the Chinese version is better than the English one for this TTS, which would make the arena not quite as applicable to it as they're all focusing on English.
insane_dreamer 2y ago
Pi by Inflection.ai was doing audio-to-audio long before GPT-4o with the most realistic voice ever (human-like imperfections, micro-pauses, etc). I don’t know why it didn’t get more attention.
- GaggiX 2y ago
  Was it end-to-end or it was audio -> speech to text -> LLM -> text to speech -> audio? I imagine it's the latter.
  - insane_dreamer 2y ago
    It's end-to-end audio, in the sense that you speak and it will reply audibly, all without visibly transcribing your words into a prompt and submitting (it may in fact be employing STT->LLM on the backend, I don't know). Works great in the car on speaker with the kids -- endless Harry Potter trivia, science questions, etc. I was completely blown away by the voice. Truly conversational.
    - GaggiX 2y ago
      Can it understand how you feel by the intonation of your voice, can it recognize the animal by the sound? If not, then it's probably not end-to-end, ChatGPT already had this mode for months where they simply use a STT and TTS to let you converse with the AI.
    - modeless 2y ago
      > It's end-to-end audio, in the sense that you speak and it will reply audibly This is not what was meant by "audio-to-audio" or "end-to-end". It's not a statement about the UI, it's a statement about the whole system. > it may in fact be employing STT->LLM on the backend It certainly is, and additionally TTS after the LLM, all connected solely by text. This is not audio-to-audio, it's audio-to-text-to-audio, and the components are trained separately, not end-to-end. ChatGPT has already done this too, for months. See OpenAI's GPT-4 blog post: https://openai.com/index/hello-gpt-4o/ https://openai.com/index/hello-gpt-4o/ > Prior to GPT-4o, you could use Voice Mode to talk to ChatGPT with latencies of 2.8 seconds (GPT-3.5) and 5.4 seconds (GPT-4) on average. To achieve this, Voice Mode is a pipeline of three separate models: one simple model transcribes audio to text, GPT-3.5 or GPT-4 takes in text and outputs text, and a third simple model converts that text back to audio. This process means that the main source of intelligence, GPT-4, loses a lot of information—it can’t directly observe tone, multiple speakers, or background noises, and it can’t output laughter, singing, or express emotion. > With GPT-4o, we trained a single new model end-to-end across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network.
deleted 2y ago
[deleted]

dartos 2y ago

Isn’t gpt 4o voice not audio to audio, but audio to text to audio?

modeless 2y ago
It isn't released yet, but the new one that they demoed is audio-to-audio. That's why it can do things like sing songs, and detect emotion in voices. The one that you can currently access in the ChatGPT app (with subscription) is the old one which is ASR->LLM->TTS.
- addandsubtract 2y ago
  I'm pretty sure you can use the new GPT4o audio-to-audio model already – even without a subscription. You can even use the "Sky" model if you didn't update your app.
- dartos 2y ago
  Are we sure it’s a single model behind the scenes doing that? Practically it doesn’t really matter, but I’d like to know for sure.
  - modeless 2y ago
    It's the second paragraph in their announcement blog post. https://openai.com/index/hello-gpt-4o/ https://openai.com/index/hello-gpt-4o/ > Prior to GPT-4o, you could use Voice Mode to talk to ChatGPT with latencies of 2.8 seconds (GPT-3.5) and 5.4 seconds (GPT-4) on average. To achieve this, Voice Mode is a pipeline of three separate models: one simple model transcribes audio to text, GPT-3.5 or GPT-4 takes in text and outputs text, and a third simple model converts that text back to audio. This process means that the main source of intelligence, GPT-4, loses a lot of information—it can’t directly observe tone, multiple speakers, or background noises, and it can’t output laughter, singing, or express emotion. > With GPT-4o, we trained a single new model end-to-end across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network.

ugh123 2y ago

> In general though, TTS as an isolated system is mostly a dead end IMO Do you mean like as a simple text to speech application? There is a huge need for better quality audiobook output.

modeless 2y ago
I don't think recording an audiobook with human-level quality is "simple". It's really a kind of acting. TTS models do very poorly at acting because they generally process one sentence at a time, or at most a paragraph, and have very little context or understanding. They just kind of fake it like a newscaster reading an unrehearsed script from a teleprompter. True human-level audiobook reading would require understanding the whole story, which often assumes general cultural knowledge, which you'll only get from a model trained on LLM-scale data. If you asked GPT-4o's new end-to-end voice mode to read an audiobook you'd probably get a better result than any TTS model. I bet it would even do different voices for the characters if you asked it to.
- maxglute 2y ago
  Maybe authors can tag sentences/paragraphs with acting directions while they write to facilitate acting. Seems like there's ways for some human input to streamline process.
  - modeless 2y ago
    Approaches based on tagging and interpreting metadata are tempting. Building structured human knowledge into the system seems like a good idea. But ultimately it isn't scalable or effective compared to general learning from large amounts of data. It's exactly the famous Bitter Lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html To the extent that authors provide supplemental notes or instructions to human actors reading their books, that information would be helpful to provide to an automated audiobook reading model. But there is no reason for it to be in a different form than a human actor would get. Additional structure or detail would be neither necessary nor helpful.

aprilnya 2y ago

Why isn’t Microsoft Azure’s TTS on here? (or am I missing something and it’s called something else)

modeless 2y ago
Many proprietary ones are missing including OpenAI. I'd guess that they don't have budget to pay for the API usage. I think the leaderboard is more focused on open source options.

presentation 2y ago

This one does seem like it does multiple languages in a sentence (at least for its currently supported languages, Chinese and English): https://www.bilibili.com/video/BV1zn4y1o7iV/?share_source=copy_web&t=26 https://www.bilibili.com/video/BV1zn4y1o7iV/?share_source=co... But it does seem like the Chinese version is better than the English one for this TTS, which would make the arena not quite as applicable to it as they're all focusing on English.

insane_dreamer 2y ago

Pi by Inflection.ai was doing audio-to-audio long before GPT-4o with the most realistic voice ever (human-like imperfections, micro-pauses, etc). I don’t know why it didn’t get more attention.

GaggiX 2y ago
Was it end-to-end or it was audio -> speech to text -> LLM -> text to speech -> audio? I imagine it's the latter.
- insane_dreamer 2y ago
  It's end-to-end audio, in the sense that you speak and it will reply audibly, all without visibly transcribing your words into a prompt and submitting (it may in fact be employing STT->LLM on the backend, I don't know). Works great in the car on speaker with the kids -- endless Harry Potter trivia, science questions, etc. I was completely blown away by the voice. Truly conversational.
  - GaggiX 2y ago
    Can it understand how you feel by the intonation of your voice, can it recognize the animal by the sound? If not, then it's probably not end-to-end, ChatGPT already had this mode for months where they simply use a STT and TTS to let you converse with the AI.
  - modeless 2y ago
    > It's end-to-end audio, in the sense that you speak and it will reply audibly This is not what was meant by "audio-to-audio" or "end-to-end". It's not a statement about the UI, it's a statement about the whole system. > it may in fact be employing STT->LLM on the backend It certainly is, and additionally TTS after the LLM, all connected solely by text. This is not audio-to-audio, it's audio-to-text-to-audio, and the components are trained separately, not end-to-end. ChatGPT has already done this too, for months. See OpenAI's GPT-4 blog post: https://openai.com/index/hello-gpt-4o/ https://openai.com/index/hello-gpt-4o/ > Prior to GPT-4o, you could use Voice Mode to talk to ChatGPT with latencies of 2.8 seconds (GPT-3.5) and 5.4 seconds (GPT-4) on average. To achieve this, Voice Mode is a pipeline of three separate models: one simple model transcribes audio to text, GPT-3.5 or GPT-4 takes in text and outputs text, and a third simple model converts that text back to audio. This process means that the main source of intelligence, GPT-4, loses a lot of information—it can’t directly observe tone, multiple speakers, or background noises, and it can’t output laughter, singing, or express emotion. > With GPT-4o, we trained a single new model end-to-end across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network.

deleted 2y ago

[deleted]