17 ms·
Show HN: WhisperFusion – Low-latency conversations with an AI chatbot
WhisperFusion builds upon the capabilities of open source tools WhisperLive and WhisperSpeech to provide a seamless conversations with an AI chatbot.
- asynchronous 3y agoVery neat capability. Need to see more of hyper-optimizing models to one specific use case, this is a great example of doing so.
- quonn 3y agoIt's what Siri and Alexa should have been. I think we will see much more of this in the next years. If - and only if - it can run locally and not keep a permanent record then the issue of listening in the background would go away, too. This is really the biggest obstacle to a natural interaction. I want to first talk, perhaps to a friend and later ask the bot to chime in. And for that to work it really needs to listen for an extended period. This could be especially useful for home automation.
- regularfry 3y agoThis is using phi-2, so the first assumption would be that it's local. It's a tiny little model in the grand scheme of things. I've been toying around with something similar myself, only I want push-to-talk from my phone. There's a route there with a WebRTC SPA, and it feels like it should be doable just by stringing together the right bits of various tech demos, but just understanding how to string everything together is more effort than it should be if you're not familiar with the tech. What's really annoying is Whisper's latency. It's not really designed for this sort of streaming use-case, they're only masking its unsuitability here by throwing (comparatively) ludicrous compute at it.
- gpderetta 3y agoThere are people trying to frankenstain-merge Mistral and Whisper in a single multimodal model [1]. I wonder if this could improve the latency. [1] : https://paul.mou.dev/posts/2023-12-31-listening-with-llm/ https://paul.mou.dev/posts/2023-12-31-listening-with-llm/
- huac 3y agoyes (you skip a decoding step) but also no (when do you start emitting?)
- pilotneko 3y agoThis project is using Mistral, not Phi-2. However, it is clear from reading the README.MD that this runs locally, so your point still stands. That being said, it looks like all models have been optimized for TensorRT, so the Whisper component may not be as high-latency as you suggest.
- regularfry 3y agoAh, so it is. I got confused by the video, where the assistant responses are labeled as phi-2.
- renus 3y agoFor the transcription part, we are looking into W2v-BERT 2.0 as well and will make it available in a live-streaming context. That said, Whisper, especially small (<50ms), is not as compute-heavy; right now, most of the compute is consumed by the LLM.
- regularfry 3y agoNo, it's not that it's compute-heavy, especially, it's that the model expects to work on 30-second samples. So if you want sub-second latency, you have to do 30 seconds worth of processing more than once a second. It just multiplies the problem up. If you can't offload it to a gpu it's painfully inefficient. As to why that might matter: my single 4090 is occupied with most of a Mixtral instance, and I don't especially want to take any compute away from that.
- intalentive 3y agoFor minimum latency you want a recurrent model that works in the time domain. A Mamba-like model could do it.
- deleted 3y ago[deleted]
- yieldcrv 3y agoI like how Chat GPT 4 will stammer, stutter and pause. This would be even better with a little "uhm" right when the speaker finishes talking, or even a chat bot that interrupts you a little bit, predicting when you're finishing - even incorrectly. like an engaged but not-most-polite person does
- pyryt 3y agoKnowing when to speak is actually a prediction task in itself. See eg https://arxiv.org/abs/2010.10874 https://arxiv.org/abs/2010.10874 Would be indeed great to get something like this integrated with whisper, LLM and TTS
- zachthewf 3y agoHard for me to imagine that this could be solved in text space. I think the prediction task needs to be done on the audio.
- stiffler01 3y agoWe thought about doing this in Whisper itself, since its already working in the audio space.
- stiffler01 3y agoYes, this is something we want to look into in more detail, really appreciate sharing the research.
- cristyansv 3y agoImagine porting this to a dedicated app that can access the context of the open window and the text on the screen, providing an almost real-time assistant for everything you do on screen.
- column 3y agoAutomatically take a screenshot and feed it to https://github.com/vikhyat/moondream https://github.com/vikhyat/moondream or similar? Doable. But while very impressive, the results are a bit of mixed bag (some hallucinations)
- cristyansv 3y agoI'm sure something like the accessibility API will have a smaller latency. https://developer.apple.com/library/archive/samplecode/UIElementInspector/Introduction/Intro.html https://developer.apple.com/library/archive/samplecode/UIEle...
- summarity 3y agorewind.ai seems to be moving in this direction
- cristyansv 3y agothis looks equally scary and incredible, especially the "summarize what I worked on today" examples.
- fragmede 3y agoit works really well, and locally too!
- albertzeyer 3y agoSee also the blog post: https://www.collabora.com/news-and-blog/news-and-events/whisperfusion-ultra-low-latency-conversations-with-an-ai-chatbot.html https://www.collabora.com/news-and-blog/news-and-events/whis... WhisperFusion, WhisperLive, WhisperSpeech, those are very interesting projects. I'm curious about latency (of all those 3 systems individually, and also the LLM), and WER numbers of WhisperLive. I did not really find any numbers on that? This is a bit strange, as those are the most crucial information about such models? Maybe I just looked at the wrong places (the GitHub repos).
- renus 3y agoWhisperLive builds upon the Whisper model; for the demo, we used small.en, but you can also use large without introducing a bigger latency for the overall pipeline since the transcription process is decoupled from the LLM and text-to-speech process.
- albertzeyer 3y agoYes, but when you change Whisper to make it live, to get WhisperLive, surely this has an effect on the WER, it will get worse. The question is, how much worse? And what is the latency? Depending on the type of streaming model, you might be able to control the latency, so you get a graph, latency vs WER, and in the extreme (offline) case, you have the original WER. How exactly does WhisperLive work actually? Did you reduce the chunk size from 30 sec to something lower? To what? Is this fixed or can it be configured by the user? Where can I find information on those details, or even a broad overview on how WhisperLive works?
- renus 3y agohttps://github.com/collabora/WhisperLive https://github.com/collabora/WhisperLive
- albertzeyer 3y agoYes I have looked there. I did not find any WER numbers and latency numbers (ideally both together in a graph). I also did not find the model being described. *Edit* Ah, when you write faster_whisper, you actually mean https://github.com/SYSTRAN/faster-whisper https://github.com/SYSTRAN/faster-whisper? And for streaming, you use https://github.com/ufal/whisper_streaming https://github.com/ufal/whisper_streaming? So, the model as described in http://www.afnlp.org/conferences/ijcnlp2023/proceedings/main-demo/cdrom/pdf/2023.ijcnlp-demo.3.pdf http://www.afnlp.org/conferences/ijcnlp2023/proceedings/main...? There, for example in Table 1, you have exactly that, latency vs WER. But the latency is huge (2.85 sec the lowest). Usually, streaming speech recognition systems have latency well beyond 1 sec. But anyway, is this actually what you use in WhisperLive / WhisperFusion? I think it would be good to give a bit more details on that.
- pizzathyme 3y agoWhenever I walk my dog I find myself wanting a conversationalist LLM layer to exist in the best form. LLM's now are great at conversation, but the connective tissue between the LLM and natural dialog needs a lot of work. Some of the problems: - Voice systems now (including ChatGPT mobile app) stop you at times when a human would not, based on how long you pause. If you said, "I think I'm going to...[3 second pause]" then LLM's stop you, but a human would wait - No ability to interrupt them with voice only - Natural conversationalists tend to match one another's speed, but these system's speed are fixed - Lots of custom instructions needed to change from what works in written text to what works in speech (no bullet points, no long formulas) On the other side of this problem is a super smart friend you can call on your phone. That would be world changing.
- renus 3y agoGood point; another area we are currently looking into is predicting intention; often, when talking to someone, we have a good idea of what that person might say next. That would not only help with latency but also, allow us to give better answers, and load the right context.
- hombre_fatal 3y agoYeah. While I like the idea of live voice chat with an LLM, it turns out I’m not so good at getting a thought across without pauses, and that gets interpreted as the LLM’s turn to respond. I’d need to be able to turn on a magic spoken word like “continue” for it to be useful. I do like the interface though.
- renus 3y agopyryt posted https://arxiv.org/abs/2010.10874 https://arxiv.org/abs/2010.10874, which might be helpful here, but we probably end off with personalized models that learned from conversation styles. A magic stop/processing word would be the easiest to add since you already have the transcript, but it's taking the natural feel of a conversation.
- visarga 3y agoI think the Whisper models need to predict end-of-turn based on content. And if it still gets input after the EOT, it can just drop the LLM generation and start over at the next EOT.
- pyryt 3y agoHas anyone experimented with integrating real-time lipsync into a low-latency audio bot? I saw some demos with d-id but their pricing was closer to $1/minute which makes it rather prohibitive
- tasty_freeze 3y agoI'm aching for someone to come up with a low latency round trip voice recognition, LLM, speech generation tuned to waste the time of phone scammers. There is one famous youtube guy who has tried this exact thing, but the one video I saw was very, very primitive and unconvincing.
- jsheard 3y agoOTOH the technology which allows that would just as easily, and more likely be used by the scammers themselves to fully automate robocalling rather than having to outsource to call centres like they currently do. Your time wasting robot would just be wasting the time of another robot that's simultaneously on the line with a thousand other people.
- lxe 3y agoOh this is neat! I was wondering how to get whisper to stream-transcribe well. I have a similar project using whisper + styletts with the similar goal to gave minimal delay: https://github.com/lxe/llm-companion https://github.com/lxe/llm-companion
- dmw_ng 3y agoThere must have been 100 folk with the same idea at the same time, I'm very excited for having something like this running mics in my home so long as it's running locally (and not costing $30/mo. in electricity to operate). Lots of starter projects, feels like a polished solution (e.g. easy maintainability and good home assistant integration etc) is right around the corner now Have been tempted to try and build something out myself, there are tons of IP cameras around with 2-way audio. If the mic was reasonable enough quality, the potential for a multimodal LLM to comment contextually on the scene as well as respond through the speaker in a ceiling-mounted camera appeals to me a lot. "Computer, WTF is this old stray component I found lying under the sink?"
- fragmede 3y agoWhat is SOTA for model-available vision systems? If there's a camera, can it track objects so it can tell me where I put my keys in the room without having to put an $30 airtag on them?
- dmw_ng 3y agoI think good in-home vision models are probably still a little bit away yet, but it seems already the case you could start to plan for their existence. It would also be possible to fine-tune a puny model to trigger a function to pass the image to a larger hosted model if explicitly requested to, there are a variety of ways things could be tiered to keep processing that can be done practically at home at home, and still make it possible to automatically (or on user's request) defer the query to a larger model operated by someone else
- monkeydust 3y agoThis post reminded me of Vocode: https://github.com/vocodedev/vocode-python https://github.com/vocodedev/vocode-python Discussion on them here from 10 months ago: https://news.ycombinator.com/item?id=35358873 https://news.ycombinator.com/item?id=35358873 I tried the demo back then and was very impressed. Anyone using it in dev or production?
- domrdy 3y agoI think they did a pivot to LLM phone calls? I've tried their library the other day and it works quite well. It even has the "interrupt feature" that is being talked about a few threads up. Supports a ton of backends for transcribe/voice/LLM.
- monkeydust 3y agoYea the I interrupt worked well, would guess (?) this could be deployed for local conversation without need for phone.
- wruza 3y agoCould someone please summarize the differences (or similarities) of the LLM part against TGWUI+llama.cpp setup with offloading layers to tensor cores? Asking because 8x7B Q4_K_M (25GB, GGUF) doesn't seem to be "ultra-low latency" on my 12GB VRAM + RAM. Like, at all. I can imagine running 7-13GB sized model with that latency (cause I did, but... it's a small model), or using 2x P40 or something. Not sure what the assumptions they make in the README. Am I missing something? Can you try it without TTS part?
- freeqaz 3y agoThe video example is using Phi-2 which is a 2.7bn param network. I think that's part of how they're achieving the low latency here! Has anybody fine-tuned Phi-2? I haven't found any good resources for that yet.
- renus 3y agoWe tested https://huggingface.co/cognitivecomputations/dolphin-2_6-phi-2 https://huggingface.co/cognitivecomputations/dolphin-2_6-phi... as well, in some tasks it performs better. That said, you can use Mistral as well, we support a few models through TensorRT-LLM.
- ramon156 3y agoGreat to hear its seamless real-time ultra low-latency. Hopefully the next iteration is blazingly fast too!
- localhost 3y agoThere are two things that I think are needed and that I'm not sure if anyone provides yet to make this scenario work well: 1. Interruption - I need to be able to say "hang on" and have the LLM pause. 2. Wait for a specific cue before responding. I like "What do you think?" That + low latency are crucial. It needs to feel like talking to another person.
- philsnow 3y ago> 2. Wait for a specific cue before responding. I like "What do you think?" "Over."
- dr_kiszonka 3y ago"Over and out" closes the app. ;) Saying "Go" to indicate it's the bot's turn would work for me. (Or maybe pressing a button.) The bot should always stop wherever I start speaking.
- m463 3y ago> I like "What do you think?" I like it too! And I can't help but think getting into the habit of saying it -- it would help us get along much better with other people in our lives.
- bilsbie 3y agoIt would be cool if the Ai could interrupt too.
- andai 3y ago"Imma let you finish, but..."
- plufz 3y agoI agree, it is unnatural and a little stressful with current implementations. It feels like I first need to figure out what to say and than say it so I don’t pause and mess up my input. I hope the new improved Siri and Google assistant will be able to chain actions as well. “Ok Google, turn off the lights. Ok Google, stop music.” Feels a bit cumbersome.
- bilsbie 3y agoMy dream is to do pair coding (and pair learning) with an AI. It would be a live conversation and it can see whatever I’m doing on my screen. We’re gradually getting closer.
- andai 3y agoI wanted to say that's Copilot, but you meant speaking instead of typing?
- bilsbie 3y agoI envision it feeling like you're pair programming with a person. (I have problems staying motivated.) But that might be a good place to start.
- andai 3y agoInteresting! I do find projects with friends very engaging. They always seem to lose interest long before I do, though. That really bums me out, and used to make me lose steam too. My current approach is, "I'm going to do it no matter what, and if you want to join that's cool too." -- That's the main advantage of GPT for me... not infinite wisdom, but infinite willingness to listen, and infinite enthusiasm! That is in very short supply among humans. Which is probably why it costs $200/hr in human form, heh. Unconditional positive regard. Pi.ai is also surprisingly good for that, better than GPT in some aspects (talking out "soft problems" — not as good for technical stuff).
- pksebben 3y agoI have a similar dream; with one major caveat - it must unequivocally be a local model. "See whatever I'm doing on my screen" comes with "leaks information to the model" and that could go real-bad-wrong real fast.
- bilsbie 3y agoI have a feeling people will really demand local only for AI. I’m not sure why the demand never materialized for other highly personal services like search, photos, medical, etc. But I just have this hunch we all really want it for AI.
- doctorpangloss 3y agoThis is an excellent project with excellent packaging. It is primarily a packaging problem. Why does every Python application on GitHub have its own ad-hoc, informally specified, bug ridden, slow implementation of half of setuptools? Why does TensorRT distribute the most essential part of what it does in an "examples" directory? huggingface_cli... man, I already have a way to download something by a name, it's a zip file. In fact, why not make a PyPi index that facades these models? We have so many ways already to install and cache read only binary blobs...
- traverseda 3y agoWell the huggingface one is obvious enough, they want to encourage vendor lock-in, make themselves the default. Same reason why docker downloads from dockerhub unless you explicitly request a full url.
- abdullahkhalids 3y agoWe see these tools such as this posted several times a week. Is there any expectation they will be installable by the common person? Where is the setup.exe, .deb, .rpm, .dmg?
- renus 3y agoWe are going to put the sample interface into the Docker, so it's more mainly: > docker run --gpus all --shm-size 64G -p 80:80 -it ghcr.io/collabora/whisperfusion:latest instead of: > docker run --gpus all --shm-size 64G -p 6006:6006 -p 8888:8888 -it ghcr.io/collabora/whisperfusion:latest > cd examples/chatbot/html > python -m http.server
- codethief 3y agoSeeing that this uses TensorRT (i.e. seems well optimized), what GPUs are supported? Could I run this on a Jetson?
- stiffler01 3y agoThat is something we are looking forward to as well. Stay tuned for updates on Jetson support. We tested it on 3090 and 4090 works as expected.
- WhackyIdeas 3y agoI had a quick read through, maybe I missed something but is this all local run or does it need api access to OpenAI’s remote system? The reason I ask is that I’m building something that does both TTS and STT using OpenAI, but I do not want to be sending a never ending stream of audio to OpenAI just for it to listen for a single command I will eventually give it. If I can do all of this local and use Mistral instead, then I’d give it a go too.
- renus 3y agoEverything runs locally, we use: - WhisperLive for the transcription - https://github.com/collabora/WhisperLive https://github.com/collabora/WhisperLive - WhisperSpeech for the text-to-speech - https://github.com/collabora/WhisperSpeech https://github.com/collabora/WhisperSpeech and an LLM (phi-2, Mistral, etc.) in between
- WhackyIdeas 3y agoThank you! When I read OpenAI I was thinking would be going through them. This revelation is perfect timing for me… keeping user data even more private. Excellent!
- jdkee 3y agoDoes anyone know if you can replace TensorRT with a similar call to Apple's CoreML for the same functionality?