4 ms·

This looks like so much fun! I have recently gotten into working with electronics, so it seems like a nice little project to undertake. I noticed that it is de

This looks like so much fun! I have recently gotten into working with electronics, so it seems like a nice little project to undertake. I noticed that it is dependent on openAIs realtime API, so it got me wondering what open alternatives there are as I would love a more realtime alexa-like device in my home that doesnt contact the cloud. I have only played with software, but the existing solutions have never felt realtime to me. I could only find <https://github.com/fixie-ai/ultravox https://github.com/fixie-ai/ultravox> that would seem to really work as realtime. It seems to be some model that wires up llama and whisper somehow, rather than treating them as separate steps which is common with other projects. What other options are available for this kind of real-time behaviour?

3D30497420 1y ago
Maybe inspiration from how Home Assistant can do local speech-to-text and vice versa? https://www.home-assistant.io/voice_control/voice_remote_local_assistant/ https://www.home-assistant.io/voice_control/voice_remote_loc... Pretty sure you'd need to host this on something more robust than an ESP32 though.
- supermatt 1y ago
  Yeah, I was looking at home assistant as well, but it doesnt feel real-time, likely due to it having the transcription stage separate from the inference.
_neil 1y ago
Not on-device but for local network I’ve been looking at Speaches[0]. Haven’t tried it yet, but I have been running kokoru-web[1] and the quality and speed is really good. [0] https://speaches.ai/ https://speaches.ai/ [1] https://huggingface.co/spaces/Xenova/kokoro-web https://huggingface.co/spaces/Xenova/kokoro-web
Sean-Der 1y ago
My plan is that Espressif’s WebRTC code[0] will hook up to pipe at [1] that gets you the freedom to do whatever you want. The design of OpenAI + WebRTC was to lean on WebRTC as much as possible to make it easier for users. [0] https://github.com/espressif/esp-webrtc-solution https://github.com/espressif/esp-webrtc-solution [1] https://github.com/pipecat-ai/pipecat https://github.com/pipecat-ai/pipecat
- supermatt 1y ago
  Fantastic! This will save a ton of work
- akadeb 1y ago
  Pipecat is awesome! is it similar to what livekit provides? I think Realtime API adoption would be higher if it is offered on Arduino rather than ESP-IDF as the latter is not very beginner friendly. That was one of the main reasons I built this repo using edge functions instead of a direct WebRTC connection.

3D30497420 1y ago

Maybe inspiration from how Home Assistant can do local speech-to-text and vice versa? https://www.home-assistant.io/voice_control/voice_remote_local_assistant/ https://www.home-assistant.io/voice_control/voice_remote_loc... Pretty sure you'd need to host this on something more robust than an ESP32 though.

supermatt 1y ago
Yeah, I was looking at home assistant as well, but it doesnt feel real-time, likely due to it having the transcription stage separate from the inference.

_neil 1y ago

Not on-device but for local network I’ve been looking at Speaches[0]. Haven’t tried it yet, but I have been running kokoru-web[1] and the quality and speed is really good. [0] https://speaches.ai/ https://speaches.ai/ [1] https://huggingface.co/spaces/Xenova/kokoro-web https://huggingface.co/spaces/Xenova/kokoro-web

Sean-Der 1y ago

My plan is that Espressif’s WebRTC code[0] will hook up to pipe at [1] that gets you the freedom to do whatever you want. The design of OpenAI + WebRTC was to lean on WebRTC as much as possible to make it easier for users. [0] https://github.com/espressif/esp-webrtc-solution https://github.com/espressif/esp-webrtc-solution [1] https://github.com/pipecat-ai/pipecat https://github.com/pipecat-ai/pipecat

supermatt 1y ago
Fantastic! This will save a ton of work
akadeb 1y ago
Pipecat is awesome! is it similar to what livekit provides? I think Realtime API adoption would be higher if it is offered on Arduino rather than ESP-IDF as the latter is not very beginner friendly. That was one of the main reasons I built this repo using edge functions instead of a direct WebRTC connection.