11 ms·

Show HN: An open source framework for voice assistants

346 points by kwindla 2y ago

I've been obsessed for the past ~year with the possibilities of talking to LLMs. I built a bunch of one-off prototypes, shared code on X, started a Meetup group in SF, and co-hosted a big hackathon. It turns out that there are a few low-level problems that everybody building conversational/real-time AI needs to solve on the way to building/shipping something that works well: low-latency media transport, echo cancellation, voice activity detection, phrase endpointing, pipelining data between models/services, handling voice interruptions, swapping out different models/services. On the theory that something like a LlamaIndex or LangChain for real-time/conversational AI would be useful, a few of us started working on a Python library for voice (and multimodal) AI assistants/agents. So ... Pipecat: a framework for building things like personal coaches, meeting assistants, story-telling toys for kids, customer support bots, virtual friends, and snarky social bots. Most of the core contributors to Pipecat so far work together at our day jobs. This has been a kind of "20% time" thing at our company. But we're serious about welcoming all contributions. We want Pipecat to support any and all models, services, transport layers, and infrastructure tooling. If you're interested in this stuff, please check it out and let us know what you think. Submit PRs. Become a maintainer. Join the Discord. Post cool stuff. Post funny stuff when your voice agent goes completely off the rails (as mine sometimes do).

canadiantim 2y ago
Very cool, great work! I can def self using this when I start building in that direction.
bamazizi 2y ago
I wonder how the just announced "GPT-4o" with real-time voice impacts projects like this? The demo on real-time multi language translation conversation blew me away!
- avarun 2y ago
  Yeah, same question here. Building pipelines for bridging LLMs and TTS and STT models with lower latency is fine and all, but when you compare to a natively multimodal model like GPT-4o it seems strictly inferior. The future is clearly voice-native models that are able to understand nuances in voice and speech patterns, and it's not exactly a distant future.
- deleted 2y ago
  [deleted]
- kwindla 2y ago
  Here's a translation demo in Pipecat using the now ancient and arthritic GPT-4 Turbo model. :-) https://github.com/pipecat-ai/pipecat/tree/main/examples/translation-chatbot https://github.com/pipecat-ai/pipecat/tree/main/examples/tra... As soon as GPT-4o audio input is available through the APIs, we'll add 4o support to Pipecat. For bidirectional real-time audio, I think they'll need to make new WebSocket or WebRTC endpoints available.
  - jshreder 2y ago
    Just letting you know it's available right now, just specify `gpt-4o` -- for text streaming anyway. I'd hazard a guess that the audio endpoints are open now, just not documented (like most of the last launches)...
    - kwindla 2y ago
      Yeah, seems to be a drop-in replacement for the existing inference APIs. But I haven't found any docs yet for streaming audio/video input.
ilaksh 2y ago
This is great but we really need an audio-to-audio model like they demoed in the open source world. Does anyone know of anything like that? Edit: someone found one: https://news.ycombinator.com/item?id=40346992 https://news.ycombinator.com/item?id=40346992
- makeitmore 2y ago
  Most of the Pipecat examples we've been working on are focused on speech-to-speech. The examples guide you through how to do that (or you can give the hosted storytelling example a try: https://storytelling-chatbot.fly.dev/ https://storytelling-chatbot.fly.dev/) We should probably update the example in the README to better represent that, thank you!
  - ilaksh 2y ago
    Your project is amazing and I'm not trying to take away from what you have accomplished. But..I looked at the code but didn't see any audio-to-audio service or model. Can you link to an example of that? I don't mean speech to text to LLM to text to speech. I mean speech-to-speech directly, as in the ML model takes audio as input and outputs audio. As they have now in OpenAI. I am very familiar with the typical multi-model workflow and have implemented it several times.
  - TomTom101101 2y ago
    That's absolutely amazing, both visually and technically! Do you share any insights of the development process, perhaps some code?
    - TomTom101101 2y ago
      I just realized this is exactly the example provided in the repo which I haven't run yet! Thanks for adding this!
- kwindla 2y ago
  An audio-to-audio model is definitely a step forward. And I do think that's where things are going to go, generally speaking. For context relating to real-time voice AI: once you're down below ~800ms things are fast enough to feel naturally responsive for most people and use cases. The GPT-4o announcement page says they average ~320ms time to first token from an audio prompt. Which is definitely next level and is really, really exciting. You can't get to 800ms with any pipeline that includes GPT-4 Turbo today, so this is a big deal. It's possible to do ~500ms time to first token by pipelining today's fastest transcription, inference, and tts models. (For example, Deepgram transcription, Groq Llama-3, Deepgram Aura voices.)
radlad 2y ago
Siri came out in October 2011. Amazon Alexa made its debut in November 2014. Google Assistant's voice-activated speakers were released in May 2016. From what I can tell, Siri is still a dumpster fire that nobody is willing to use. And I have no personal experience with Alexa, so I can't speak to it. But I do have a few Google Home speakers and an Android phone, and I have seen no major improvements in years. In fact, it has gotten worse - for example, you can no longer add items directly to AnyList[0], only Google Keep. Or, as an incredibly simple example of something I thought we'd get a long time ago, it's still unable to interpret two-part requests, e.g. "please repeat that but louder," or "please turn off the kitchen and dining room lights." I find voice assistants very useful - especially when driving, lying in bed, cooking, or when I'm otherwise preoccupied. Yet they have stagnated almost since their debut. I can only imagine nobody has found a viable way to monetize them. What will it take to get a better voice assistant for consumers? Willow[1] doesn't seem to have taken off. [0] https://help.anylist.com/articles/google-assistant-overview/ https://help.anylist.com/articles/google-assistant-overview/ [1] https://heywillow.io/ https://heywillow.io/ edit: I realize I hijacked your thread to dump something that's been on my mind lately. Pipecat looks really cool, and I hope it takes off! I hope to get some time to experiment this weekend.
- petemir 2y ago
  > From what I can tell, Siri is still a dumpster fire that nobody is willing to use. I have no personal experience with Alexa, so I can't speak to it I use both (albeit more Alexa than Siri, both just for a really limited functionality set), and FWIW, I believe Alexa is worse than Siri. It can do two things at the same time though (just as your example: "turn on X and turn off Y", "turn on X for Y seconds", and things like that). I also feel that it has gotten worse over the years. I read about the possibility of microphones getting dust and therefore capturing worse audio, so I got a dust blower (for other reasons, too), but it didn't solve anything. After listening in the app what Alexa picks up (from an Echo and Echo Dot, both 4th. Gen), I have to say that they use really shitty microphones. Furthermore, I have been testing Whisper extensively last month, with audio coming from low-quality sources, and I think a similar model would interpret a lot better my voice than whatever Amazon is using.
- sroussey 2y ago
awenix 2y ago
Nice to see an open source implementation, i have been seeing many startups get into this space like https://www.retellai.com/ https://www.retellai.com/, https://fixie.ai/ https://fixie.ai/ etc. They always end up needing speech-to-speech models (current approach seems speech-text-text-speech with multiple agents handling 1 listening + 1 speaking), excited to see how this plays with recently announced gpt-4o
- kwindla 2y ago
  Adding to your list: https://vapi.ai https://vapi.ai -- really nice tools. (I try to keep up with all the different layers/players in this space.)
- zkoch 2y ago
  We're (fixie.ai) working on on our SLM (speech language model). We'll release something soon to play with :)
- j-bos 2y ago
  How do speech to speech models work? Do they just that many more tokens to capture nuances of spoken language?
xan_ps007 2y ago
We're also building bolna an open source voice orchestration: https://github.com/bolna-ai/bolna https://github.com/bolna-ai/bolna
russ 2y ago
LiveKit Agents, which OpenAI uses in voice mode is also open source: https://github.com/livekit/agents https://github.com/livekit/agents
orliesaurus 2y ago
The whole VAD thing is very interesting, keen to learn more about how it works and especially with multiple speakers!
userhacker 2y ago
Just made https://feycher.com https://feycher.com thats similar, but has realtime lip syncing as well. Let me know if you are interested and we can chat
35mm 2y ago
How would I go about using this to live translate phone calls?
- generalizations 2y ago
  Why would you live translate phone calls? Also, whisper.
  - 35mm 2y ago
    To talk to people in languages I am not fluent in. Whisper is too slow and doesn’t allow interruption.
- vr000m 2y ago
  Daily now supports dial-in and dial-out, https://docs.daily.co/guides/products/dial-in-dial-out#main https://docs.daily.co/guides/products/dial-in-dial-out#main Which means you can connect a bot to a call, and tell it dialout to a phone number and it will.

346 points by kwindla 2y ago

canadiantim 2y ago

Very cool, great work! I can def self using this when I start building in that direction.

bamazizi 2y ago

I wonder how the just announced "GPT-4o" with real-time voice impacts projects like this? The demo on real-time multi language translation conversation blew me away!

avarun 2y ago
Yeah, same question here. Building pipelines for bridging LLMs and TTS and STT models with lower latency is fine and all, but when you compare to a natively multimodal model like GPT-4o it seems strictly inferior. The future is clearly voice-native models that are able to understand nuances in voice and speech patterns, and it's not exactly a distant future.
deleted 2y ago
[deleted]
kwindla 2y ago
Here's a translation demo in Pipecat using the now ancient and arthritic GPT-4 Turbo model. :-) https://github.com/pipecat-ai/pipecat/tree/main/examples/translation-chatbot https://github.com/pipecat-ai/pipecat/tree/main/examples/tra... As soon as GPT-4o audio input is available through the APIs, we'll add 4o support to Pipecat. For bidirectional real-time audio, I think they'll need to make new WebSocket or WebRTC endpoints available.
- jshreder 2y ago
  Just letting you know it's available right now, just specify `gpt-4o` -- for text streaming anyway. I'd hazard a guess that the audio endpoints are open now, just not documented (like most of the last launches)...
  - kwindla 2y ago
    Yeah, seems to be a drop-in replacement for the existing inference APIs. But I haven't found any docs yet for streaming audio/video input.

ilaksh 2y ago

This is great but we really need an audio-to-audio model like they demoed in the open source world. Does anyone know of anything like that? Edit: someone found one: https://news.ycombinator.com/item?id=40346992 https://news.ycombinator.com/item?id=40346992

makeitmore 2y ago
Most of the Pipecat examples we've been working on are focused on speech-to-speech. The examples guide you through how to do that (or you can give the hosted storytelling example a try: https://storytelling-chatbot.fly.dev/ https://storytelling-chatbot.fly.dev/) We should probably update the example in the README to better represent that, thank you!
- ilaksh 2y ago
  Your project is amazing and I'm not trying to take away from what you have accomplished. But..I looked at the code but didn't see any audio-to-audio service or model. Can you link to an example of that? I don't mean speech to text to LLM to text to speech. I mean speech-to-speech directly, as in the ML model takes audio as input and outputs audio. As they have now in OpenAI. I am very familiar with the typical multi-model workflow and have implemented it several times.
- TomTom101101 2y ago
  That's absolutely amazing, both visually and technically! Do you share any insights of the development process, perhaps some code?
  - TomTom101101 2y ago
    I just realized this is exactly the example provided in the repo which I haven't run yet! Thanks for adding this!
kwindla 2y ago
An audio-to-audio model is definitely a step forward. And I do think that's where things are going to go, generally speaking. For context relating to real-time voice AI: once you're down below ~800ms things are fast enough to feel naturally responsive for most people and use cases. The GPT-4o announcement page says they average ~320ms time to first token from an audio prompt. Which is definitely next level and is really, really exciting. You can't get to 800ms with any pipeline that includes GPT-4 Turbo today, so this is a big deal. It's possible to do ~500ms time to first token by pipelining today's fastest transcription, inference, and tts models. (For example, Deepgram transcription, Groq Llama-3, Deepgram Aura voices.)

radlad 2y ago

Siri came out in October 2011. Amazon Alexa made its debut in November 2014. Google Assistant's voice-activated speakers were released in May 2016. From what I can tell, Siri is still a dumpster fire that nobody is willing to use. And I have no personal experience with Alexa, so I can't speak to it. But I do have a few Google Home speakers and an Android phone, and I have seen no major improvements in years. In fact, it has gotten worse - for example, you can no longer add items directly to AnyList[0], only Google Keep. Or, as an incredibly simple example of something I thought we'd get a long time ago, it's still unable to interpret two-part requests, e.g. "please repeat that but louder," or "please turn off the kitchen and dining room lights." I find voice assistants very useful - especially when driving, lying in bed, cooking, or when I'm otherwise preoccupied. Yet they have stagnated almost since their debut. I can only imagine nobody has found a viable way to monetize them. What will it take to get a better voice assistant for consumers? Willow[1] doesn't seem to have taken off. [0] https://help.anylist.com/articles/google-assistant-overview/ https://help.anylist.com/articles/google-assistant-overview/ [1] https://heywillow.io/ https://heywillow.io/ edit: I realize I hijacked your thread to dump something that's been on my mind lately. Pipecat looks really cool, and I hope it takes off! I hope to get some time to experiment this weekend.

petemir 2y ago
> From what I can tell, Siri is still a dumpster fire that nobody is willing to use. I have no personal experience with Alexa, so I can't speak to it I use both (albeit more Alexa than Siri, both just for a really limited functionality set), and FWIW, I believe Alexa is worse than Siri. It can do two things at the same time though (just as your example: "turn on X and turn off Y", "turn on X for Y seconds", and things like that). I also feel that it has gotten worse over the years. I read about the possibility of microphones getting dust and therefore capturing worse audio, so I got a dust blower (for other reasons, too), but it didn't solve anything. After listening in the app what Alexa picks up (from an Echo and Echo Dot, both 4th. Gen), I have to say that they use really shitty microphones. Furthermore, I have been testing Whisper extensively last month, with audio coming from low-quality sources, and I think a similar model would interpret a lot better my voice than whatever Amazon is using.
sroussey 2y ago

awenix 2y ago

Nice to see an open source implementation, i have been seeing many startups get into this space like https://www.retellai.com/ https://www.retellai.com/, https://fixie.ai/ https://fixie.ai/ etc. They always end up needing speech-to-speech models (current approach seems speech-text-text-speech with multiple agents handling 1 listening + 1 speaking), excited to see how this plays with recently announced gpt-4o

kwindla 2y ago
Adding to your list: https://vapi.ai https://vapi.ai -- really nice tools. (I try to keep up with all the different layers/players in this space.)
zkoch 2y ago
We're (fixie.ai) working on on our SLM (speech language model). We'll release something soon to play with :)
j-bos 2y ago
How do speech to speech models work? Do they just that many more tokens to capture nuances of spoken language?

xan_ps007 2y ago

We're also building bolna an open source voice orchestration: https://github.com/bolna-ai/bolna https://github.com/bolna-ai/bolna

russ 2y ago

LiveKit Agents, which OpenAI uses in voice mode is also open source: https://github.com/livekit/agents https://github.com/livekit/agents

orliesaurus 2y ago

The whole VAD thing is very interesting, keen to learn more about how it works and especially with multiple speakers!

userhacker 2y ago

Just made https://feycher.com https://feycher.com thats similar, but has realtime lip syncing as well. Let me know if you are interested and we can chat

35mm 2y ago

How would I go about using this to live translate phone calls?

generalizations 2y ago
Why would you live translate phone calls? Also, whisper.
- 35mm 2y ago
  To talk to people in languages I am not fluent in. Whisper is too slow and doesn’t allow interruption.
vr000m 2y ago
Daily now supports dial-in and dial-out, https://docs.daily.co/guides/products/dial-in-dial-out#main https://docs.daily.co/guides/products/dial-in-dial-out#main Which means you can connect a bot to a call, and tell it dialout to a phone number and it will.