6 ms·
Jarvis: A Voice Virtual Assistant in Python (OpenAI, ElevenLabs, Deepgram)
- Alyx1337 3y agoHere is a video demo of the project: https://youtu.be/aIg4-eL9ATc?si=66ynl4Mlci9v76rU https://youtu.be/aIg4-eL9ATc?si=66ynl4Mlci9v76rU
- alchemist1e9 3y agoNice work! Very impressed. Do you happen to know anything about any open source voice identification software? I’ve noticed with ChatGPT voice and any other voice driven assistant that a massive problem is the background voices and noise. One solution could be advanced pre-processing to ID your voice only. Another idea I’ve had is using something professional with PTT: https://sheepdogmics.com/products/quick-disconnect-mic-tubeless-earpiece-kenwood https://sheepdogmics.com/products/quick-disconnect-mic-tubel...
- Alyx1337 3y agoThanks! I don't know a lot about this but someone shared this local voice assistant in the comments: https://github.com/KoljaB/LocalAIVoiceChat https://github.com/KoljaB/LocalAIVoiceChat Could be a good lead
- alchemist1e9 3y agoYeah github.com/KoljaB is quite a collection of stuff! I agree. It all seems your vision of JARVIS, which I share completely but haven't accomplished what you have, again excellent work and thank you for sharing, is very attainable. Probably combining your work along with KoljaB is very promising.
- Alyx1337 3y agoThank you very much!
- Jayakumark 3y agoCheck whether this can help https://github.com/resemble-ai/resemble-enhance/tree/main https://github.com/resemble-ai/resemble-enhance/tree/main
- visarga 3y agoGoogle Gemini was trained on audio and can generate audio directly. Whatever you build now will be replaced by a much better version soon.
- bitsandbooks 3y ago"Jarvis" is a trademark of Marvel, so that name will definitely not work. https://trademarks.justia.com/862/94/jarvis-86294162.html https://trademarks.justia.com/862/94/jarvis-86294162.html
- djoldman 3y agoAnd just for those who felt the need to check: > Computer application software that may be downloaded via global computer networks and electronic communication networks for use in connection with mobile computers, mobile phones, and tablet computers, namely, software for use as a voice controlled personal digital assistant
- deleted 3y ago[deleted]
- beardyw 3y agoBut there are many apps called Jarvis, so I am not sure how that is supposed to work?
- petemir 3y agoSo many that it is actually quite counterproductive to call it that way. I honestly have lost track of how many AI-based assistants named JARVIS I have encountered already =/.
- torstenvl 3y agoIANAIPL but I find it difficult to believe that trademark is valid, considering it's never been used in trade. There is no Jarvis digital assistant software sold either fictionally or IRL. Even if the trademark were somehow upheld, I don't see how there could be any damages.
- bsenftner 3y agoTrademarks are industry specific. If you made a fictional AI character named Jarvis and tried selling media based on that character THEN Marvel has a case. Creating a talking AI Assistant named Jarvis would be an expensive court case, which Marvel/Disney has to cash to pursue, but it would be a legal stretch with a lot of moneyed interests willing to back the non-Marvel/Disney side.
- vessenes 3y agoI'm curious - what made you choose deepgram over just running whisper? I don't have any experience with deepgram but whisper has worked so well in my own tests that I didn't even ever consider there might be API speech recognition-only companies.
- Alyx1337 3y agoDeepgram advertised itself as being the fastest, and I wanted to focus on limiting response delay so I chose it. I hope I did not get misled.
- cloudking 3y agoWonderful hack, the overall response latency is the only thing that hurts the UX, if you can get the response time down would be epic. Nice work.
- Alyx1337 3y agoThanks! There are ways to shave off the latency: hosting locally, using quantized/smaller models, streaming data instead of doing the tasks sequentially
- java_beyb 3y agowell, deepgram might be the fastest among cloud-dependent APIs, like Speechmatics and Assembly AI mentioned above. -but- it cannot be faster than local or smaller models as you mentioned. Among local solutions, Whisper SDK doesn't support streaming, I haven't seen any good workarounds or successfully implemented it. VOSK, DeepSpeech, Kaldi, et al were good once upon a time... Picovoice seems to be doing well. I was planning to work on this: https://picovoice.ai/blog/chatgpt-ai-virtual-assistant-in-python/ https://picovoice.ai/blog/chatgpt-ai-virtual-assistant-in-py... using Eleven Labs and Cheetah. Hope I can crave some time
- jkachmar 3y agounless i'm misunderstanding `whisper.cpp` seems to support streaming & the repository includes a native example[0] and a WASM example[1] with a demo site[2]. [0]: https://github.com/ggerganov/whisper.cpp/tree/master/examples/stream https://github.com/ggerganov/whisper.cpp/tree/master/example... [1]: https://github.com/ggerganov/whisper.cpp/blob/master/examples/stream.wasm https://github.com/ggerganov/whisper.cpp/blob/master/example... [2]: https://whisper.ggerganov.com/stream/ https://whisper.ggerganov.com/stream/
- Spiwux 3y agoI wonder if we're at a point where you could build a voice assistant like that, except almost-realtime and streamed end to end: User speaks and speech to text starts streaming text while the user is still speaking. That text stream is piped into a LLM, which also streams its output text. That output text is streamed to text-to-speech, which also generates audio in a streaming manner.
- WiSaGaN 3y agoAny existing stream api for llm input?
- everforward 3y agoYou can do the "almost-realtime" part, all locally. I tinkered with a Python script for a few hours that used Whisper to speech-to-text, fed that into a local Mistral model (don't recall which), and then piped the output into text-to-speech. It wasn't really streamed, though. Audio input was buffered, fully evaluated to a string, then fed into the LLM and the full text was converted back to audio. The Whisper speech-to-text was pretty real-time, the LLM was not. I was barely scraping by on hardware specs, though.
- canadiantim 3y agoyou try using ESP box?
- evilantnie 3y agoTTS and STT models have decent support for streaming in chunks, but the accuracy drops the smaller the chunk size. Current state of LLMs are pretty limited in their ability to handle streaming inputs due to attention window constraints. There is some emerging research into attention sinks and caching initial tokens that look promising. I don't think we're quite there yet though.
- modeless 3y agoI implemented this! All local models. And I packaged it up so people can install it with one click: https://apps.microsoft.com/detail/9NC624PBFGB7 https://apps.microsoft.com/detail/9NC624PBFGB7 The speech recognition part needs work for sure, but when it works you can see the potential. It's very different from the way it feels to talk to Siri or even ChatGPT's voice mode. It won't be long before we are having real conversations with our computers.
- Jayakumark 3y agoFound Similar project but fully local for anyone interested. https://github.com/KoljaB/LocalAIVoiceChat https://github.com/KoljaB/LocalAIVoiceChat This one looks similar to this but also does internet search, mail, music, smart home etc.. Hope that there is a standard interface that gets plugged into it. so anyone can develop addons. https://github.com/KoljaB/Linguflex https://github.com/KoljaB/Linguflex
- Alyx1337 3y agoHow did you find these? I was literally looking for tutorials all day long and could not find something. These projects look insane!
- Jayakumark 3y agoYes, was searching for Realtime STT and got a hit on GitHub , then looked at his other projects and found he builts up on his STT and TTS projects, it’s just 2 second latency on Local voice chat. Which is very good .
- qainsights 3y agoGreat tool. I also created Kel - AI assistant for terminal. Please check https://kel.qainsights.com https://kel.qainsights.com
- chankstein38 3y agoI built this kind of thing for GPT-3 way back and then repurposed for 3.5 when I got API access to that. Though I used Whisper. I was hoping this would have wake word handling because that was what I struggled with but it appears that it just starts listening when you click a button or something.
- Alyx1337 3y agoYeah I had the same issue so I used (stole) this answer on StackOverflow: https://stackoverflow.com/questions/46734345/python-record-on-loop-stop-recording-when-silent https://stackoverflow.com/questions/46734345/python-record-o... Basically there's a library that records until it detects a silence
- TuringNYC 3y agoWe use this exact stack at work (OpenAI, ElevenLabs, Deepgram) for some exploratory use cases. The key issue we have now is latency with the LLM. Deepgram and Elevanlabs work brilliantly!
- Alyx1337 3y agoGreat! What do you guys have in mind in terms of products using these tools. Yeah unfortunately it's hard to shave on latency.
- TuringNYC 3y agoWe do live in-studio briefings 3x/wk. These are both in-person and live-broadcast. The first thing we did was add an AI Co-Briefer who sits on the panel. The LLM latency makes it a bit hard, but it was a good experiment. The Deepgram worked brilliantly well with transcription across the entire studio, even for un-microphoned guest participants. That live broadcast created a lot of buzz and numerous other use cases have popped up across the company. I'm working on a tech blog showcase next week to show it off on HN hopefully!
- ty00001 3y agoProblem with this is Deepgram's accuracy (but agree their speed/latency is excellent). We used to use them too, but eventually we got so frustrated with poor accuracy we switched to Speechmatics - would definitely recommend checking them out.
- dukeofdoom 3y agoIf you're really sitting at the computer for 24 hours...I have a family member that died from blood clots that formed when he sitting at his computer for too long.
- bloopernova 3y agoSomewhat amusing to consider that the (in-character) Marvel Cinematic Universe JARVIS could have been an LLM! And of course Ultron is an asshole, it was trained on input from Tony Stark! Back in 2008/9 I wondered just what would be required to run JARVIS, something you could converse with naturally, would understand what you meant, and be able to take care of complex mechanical tasks. The Iron Man suits have always been mostly Do-What-I-Mean (DWIM) managed by JARVIS or other AI agents, and now all of that seems to be attainable. It's going to be an interesting time discovering just how well a human and AI agent can work together. I could see a military personal spotter, keeping track of enemy combatants, managing larger awareness of the battlefield, etc. I wonder how much a soldier could safely offload?
- Alyx1337 3y agoExactly my thought, I was like "Jarvis has got to be just a 2030 version of an LLM". Yeah I actually considered making a spotter AI using computer vision in a game like ARMA 3 or Squad but kind of difficult. I made a spotter for ground vehicles on aerial imagery using YOLOv5 here: https://github.com/AlexandreSajus/Military-Vehicles-Image-Recognition https://github.com/AlexandreSajus/Military-Vehicles-Image-Re... There's a French defense company, Preligens, that actually does this currently
- bloopernova 3y agoI imagine that within the next couple of years there's going to be a "general purpose vision" model (GPV? :) More of a framework to perform the general purpose task of "recognize things in 30 (60? 120?) frames per second video and act on events in the video"
- Jean-Papoulos 3y agoHere I was thinking about putting something like this in my home, and jokingly calling it Jarvis. This will be a great starting point, shame you can't choose the models you want to talk to (ie use local models instead of OpenAI), but great nonetheless !
- Alyx1337 3y agoThat was exactly my thought haha, I want Jarvis at home. You could easily modify my code to run a local LLM instead
- ohthehugemanate 3y agoActually there are a few LLM wrappers around that use the openai API spec (localai is a good one)... so you could just allow a configurable openai endpoint URI and technically users can swap in any model.
- khaki54 3y agoThere is another one (Also Jarvis) that's been around for a while and is more useful, wonder if they can combine forces? https://github.com/ggeop/Python-ai-assistant https://github.com/ggeop/Python-ai-assistant Not sure if anyone has noticed but OpenAI now has a mobile app (I've been using the PWA all this time) and the voice assistant on there is really strong. Sounds good, fast, and seems to even run a pass on my voice before it submits the query.
- mfld 3y agoI assume it could be made more responsive by using a streaming text-to-speech synthesis like ElevenLabs Cheetah. This approach was taken by the RoboDad recently discussed on HN. Btw, is there a streaming text-to-speech tool that supports languages other than english?
- pogitalonx 3y agoAlso check out Willow- https://heywillow.io https://heywillow.io It doesn’t synthesize voice back (yet) but open source and runs all offline on ESP32-based hardware and works with HomeAssistant!