4 ms·

It's what Siri and Alexa should have been. I think we will see much more of this in the next years. If - and only if - it can run locally and not keep a permane

by quonn 3y ago

It's what Siri and Alexa should have been. I think we will see much more of this in the next years. If - and only if - it can run locally and not keep a permanent record then the issue of listening in the background would go away, too. This is really the biggest obstacle to a natural interaction. I want to first talk, perhaps to a friend and later ask the bot to chime in. And for that to work it really needs to listen for an extended period. This could be especially useful for home automation.

regularfry 3y ago
This is using phi-2, so the first assumption would be that it's local. It's a tiny little model in the grand scheme of things. I've been toying around with something similar myself, only I want push-to-talk from my phone. There's a route there with a WebRTC SPA, and it feels like it should be doable just by stringing together the right bits of various tech demos, but just understanding how to string everything together is more effort than it should be if you're not familiar with the tech. What's really annoying is Whisper's latency. It's not really designed for this sort of streaming use-case, they're only masking its unsuitability here by throwing (comparatively) ludicrous compute at it.
- gpderetta 3y ago
  There are people trying to frankenstain-merge Mistral and Whisper in a single multimodal model [1]. I wonder if this could improve the latency. [1] : https://paul.mou.dev/posts/2023-12-31-listening-with-llm/ https://paul.mou.dev/posts/2023-12-31-listening-with-llm/
  - huac 3y ago
    yes (you skip a decoding step) but also no (when do you start emitting?)
- pilotneko 3y ago
  This project is using Mistral, not Phi-2. However, it is clear from reading the README.MD that this runs locally, so your point still stands. That being said, it looks like all models have been optimized for TensorRT, so the Whisper component may not be as high-latency as you suggest.
  - regularfry 3y ago
    Ah, so it is. I got confused by the video, where the assistant responses are labeled as phi-2.
- renus 3y ago
  For the transcription part, we are looking into W2v-BERT 2.0 as well and will make it available in a live-streaming context. That said, Whisper, especially small (<50ms), is not as compute-heavy; right now, most of the compute is consumed by the LLM.
  - regularfry 3y ago
    No, it's not that it's compute-heavy, especially, it's that the model expects to work on 30-second samples. So if you want sub-second latency, you have to do 30 seconds worth of processing more than once a second. It just multiplies the problem up. If you can't offload it to a gpu it's painfully inefficient. As to why that might matter: my single 4090 is occupied with most of a Mixtral instance, and I don't especially want to take any compute away from that.
    - intalentive 3y ago
      For minimum latency you want a recurrent model that works in the time domain. A Mamba-like model could do it.
deleted 3y ago
[deleted]

by quonn 3y ago

regularfry 3y ago

This is using phi-2, so the first assumption would be that it's local. It's a tiny little model in the grand scheme of things. I've been toying around with something similar myself, only I want push-to-talk from my phone. There's a route there with a WebRTC SPA, and it feels like it should be doable just by stringing together the right bits of various tech demos, but just understanding how to string everything together is more effort than it should be if you're not familiar with the tech. What's really annoying is Whisper's latency. It's not really designed for this sort of streaming use-case, they're only masking its unsuitability here by throwing (comparatively) ludicrous compute at it.

gpderetta 3y ago
There are people trying to frankenstain-merge Mistral and Whisper in a single multimodal model [1]. I wonder if this could improve the latency. [1] : https://paul.mou.dev/posts/2023-12-31-listening-with-llm/ https://paul.mou.dev/posts/2023-12-31-listening-with-llm/
- huac 3y ago
  yes (you skip a decoding step) but also no (when do you start emitting?)
pilotneko 3y ago
This project is using Mistral, not Phi-2. However, it is clear from reading the README.MD that this runs locally, so your point still stands. That being said, it looks like all models have been optimized for TensorRT, so the Whisper component may not be as high-latency as you suggest.
- regularfry 3y ago
  Ah, so it is. I got confused by the video, where the assistant responses are labeled as phi-2.
renus 3y ago
For the transcription part, we are looking into W2v-BERT 2.0 as well and will make it available in a live-streaming context. That said, Whisper, especially small (<50ms), is not as compute-heavy; right now, most of the compute is consumed by the LLM.
- regularfry 3y ago
  No, it's not that it's compute-heavy, especially, it's that the model expects to work on 30-second samples. So if you want sub-second latency, you have to do 30 seconds worth of processing more than once a second. It just multiplies the problem up. If you can't offload it to a gpu it's painfully inefficient. As to why that might matter: my single 4090 is occupied with most of a Mixtral instance, and I don't especially want to take any compute away from that.
  - intalentive 3y ago
    For minimum latency you want a recurrent model that works in the time domain. A Mamba-like model could do it.

deleted 3y ago

[deleted]