3 ms·

Nobody has done this well enough yet. What's required: 1. Transcribe your speech using Whisper (in that case you don't have to make an effort to speak clearly

Nobody has done this well enough yet. What's required: 1. Transcribe your speech using Whisper (in that case you don't have to make an effort to speak clearly so long as you're in a relatively quiet room) 2. Get a TTS system that actually sounds good (e.g. Descript, Eleven Labs, etc.) 3. Have RAPID responses like a normal human conversation (mostly on OpenAI's side... so hopefully ChatGPT Plus fixes that)

ericlewis 4y ago
rapid responses is basically the biggest problem, it is quite hard but also because you can't stream tokens into any TTS system and have it sound good. The more "complete" the corpus the better it seems to be at using the right sort of pausing and such. So it is more like a: LLM needs to be directly connected to a TTS type of issue, somehow.
nojs 4y ago
The bottleneck is currently TTS. The best option is probably Eleven Labs, but response times are unpredictable. GPT response times can be worked around by falling back to a faster model, but you can't do that with TTS because the voice needs to be consistent. It seems like current state of the art are diffusion models ala DALL-E, see e.g. [1] (the developer, James Betker now incidentally works for OpenAI). It's nontrivial to turn this into something that works in real-time without a decent budget, though. Whisper (for transcription) is insanely fast and good. 1. https://github.com/neonbjb/tortoise-tts https://github.com/neonbjb/tortoise-tts
- ilaksh 4y ago
  The Eleven Labs guy told me they put out an update that made it more consistent, which maybe it is slightly better, but still unpredictable like you say. Maybe you could ask them about it again. I did suggest they make a higher queue priority available for higher level paid tiers.
babyshake 4y ago
If you want to have rapid responses, you basically want to have the first part of the response ready to go as soon as it is detected the user is no longer speaking. Were they asking a question? While the actual answer is being processed, start responding with "let me answer that for you..." or something like that.
standardUser 4y ago
This seems "well enough" to me! Faster responses sound like a BIG ask for a free service, and I imagine the best AI voice tech is highly proprietary.

ericlewis 4y ago

rapid responses is basically the biggest problem, it is quite hard but also because you can't stream tokens into any TTS system and have it sound good. The more "complete" the corpus the better it seems to be at using the right sort of pausing and such. So it is more like a: LLM needs to be directly connected to a TTS type of issue, somehow.

nojs 4y ago

The bottleneck is currently TTS. The best option is probably Eleven Labs, but response times are unpredictable. GPT response times can be worked around by falling back to a faster model, but you can't do that with TTS because the voice needs to be consistent. It seems like current state of the art are diffusion models ala DALL-E, see e.g. [1] (the developer, James Betker now incidentally works for OpenAI). It's nontrivial to turn this into something that works in real-time without a decent budget, though. Whisper (for transcription) is insanely fast and good. 1. https://github.com/neonbjb/tortoise-tts https://github.com/neonbjb/tortoise-tts

ilaksh 4y ago
The Eleven Labs guy told me they put out an update that made it more consistent, which maybe it is slightly better, but still unpredictable like you say. Maybe you could ask them about it again. I did suggest they make a higher queue priority available for higher level paid tiers.

babyshake 4y ago

If you want to have rapid responses, you basically want to have the first part of the response ready to go as soon as it is detected the user is no longer speaking. Were they asking a question? While the actual answer is being processed, start responding with "let me answer that for you..." or something like that.

standardUser 4y ago

This seems "well enough" to me! Faster responses sound like a BIG ask for a free service, and I imagine the best AI voice tech is highly proprietary.