4 ms·

Is there a good project out there that pairs whisper with something like llama.cpp to create a private local voice assistant? Llama2 isn't as good as GPT-4 but

by api 3y ago

Is there a good project out there that pairs whisper with something like llama.cpp to create a private local voice assistant? Llama2 isn't as good as GPT-4 but it's a hell of a lot smarter at Q&A than Siri or Alexa or any of those things. PSA: I will pay for such a thing if it's really good, privacy respecting, local-first, and preferably at least source available.

Ono-Sendai 3y ago
I have something pretty rudimentary here: https://github.com/Ono-Sendai/project-2501 https://github.com/Ono-Sendai/project-2501 Whisper.cpp + chatGPT + windows text-to-speech.
gpderetta 3y ago
I literally played with cat'ing the output of one into the input of the other and it worked better than I had any reason to expect. edit: in my 30minutes of playing with it, I didn't find a good sounding open-source text-to-speech model for the final stage of the pipeline.
- cooper_ganglia 3y ago
  It’s not open-source, but Play.HT has a new “Turbo model” that can begin generating text-to-speech within 150ms. I’ve tried it out, and it’s pretty impressive in terms of both quality and speed. There’s an API, so perhaps that would be worth looking into!
- regularfry 3y ago
  This lack of a decent open text to speech is really frustrating, because some of the closed ones are just scary good.
  - rickydroll 3y ago
    I've used various forms of Dragon since 1994 to help me deal with hand/arm problems. I would love to tell Windows to go take a flying leap but I can't because I need to use Dragon. If I had real-time or near real-time recognition AND the ability to edit by voice text in any field or application that I would be in a very good place. A better place would also include adding per application/global commands to drive the application. If you haven't lived with speech recognition, it's not apparent that the command space for speech environment is significantly different from the command space for mouse and hands. In order to make the command space for speech work well, the speech recognition environment the application needs to present to the API with access to all functionality and data within the application.
    - maeil 3y ago
      > I would love to tell Windows to go take a flying leap but I can't because I need to use Dragon. What does Dragon do you for that Talon can't?
      - rickydroll 3y ago
        Talon has advanced quite a bit since I last I looked but it looks like it still falls short. However, I will give it a trial run again. It looks quite useful if you need command and control. for general diction, not so much. To be fair, Dragon and other SR systems fail at speech driven editing. 99.9% of my dragon use is plane text dictation and when if the app is select-and -say enabled,edit and make corrections by voice. Speech commands as they are implemented are rarely useful mostly because there are too many of them to remember. Fortunately, my hands have recovered enough it is faster to type and mouse that it is to silently try to remember commands, construct what I want to say and then say it without stumbling or pausing then, undoing what was recognized, and trying again until I get it right. one thing I'd like to do is difficult; have the same command give the same results in different contexts. For example: say "make me root" and have the command recognize what machine on what network I'm connected to, sent the command "sudo su -<enter>" and then send the right password from my password manager without me having to type anything. example: tail [forever] <service log> work the same no matter what distribution, of logging method (syslog vrs journald) and priv level. if I need a sudo before accessing log data put one in and give it the password if needed without any action om my part. another should be possible is dictating into an app (like thunderbird) with text boxes and when focus is outside of the box, turn off recognition output. if you leave recognition output active outside of a text box then speech is typing hotkey commands from the letters in the words recognized. disaster example: hotkey-stroking your way through your email, phone rings, you forgot to turn your mic off. at the end of the all, you recognize your out-of-context recognition has destroyed your mailbox is and you have no idea how to recover it. The biggest thing is dragon is an out-of-the box solution. 15 mins (or less) install and I am dictating at high accuracy and a large vocabulary.
  - kkielhofner 3y ago
    If you think open source speech recognition is behind don't even look at text to speech synthesis. It's not even in the same galaxy.
regularfry 3y ago
https://github.com/ggerganov/whisper.cpp/tree/master/examples/talk-llama https://github.com/ggerganov/whisper.cpp/tree/master/example... is worth a poke. llama.cpp supports llama2 on CPU.
kkielhofner 3y ago
Shameful plug for my project Willow: https://heywillow.io/ https://heywillow.io/ Note that it's important to understand the realities here - short of something like an RTX 3090/4090 with LLama and every performance optimization available when it comes to responsiveness and accuracy competitive with commercial solutions this is a big challenge. Even with the potential improvements of this work and optimizations like Ctranslate (used by our Willow Inference Server and faster-whisper) getting sub one second response times like Echo/Alexa more-or-less automatically calls for GPU even with every performance trick available. As I like to say when it comes to ML/AI/speech rec/speech synthesis if you bring a CPU to a GPU fight you're going to lose - and all of the commercial implementations are certainly using GPU/TPU on top of who knows what else they've come up with over the years and their immense spending. To get an idea of how dramatic this is you can see the benchmarks with Ctranslate2/faster-whisper and our Willow Inference Server here[0]. Looking at those real-world numbers even at a claimed 6x performance improvement a mighty Threadripper PRO 5955WX can barely meet this goal with the models needed for voice assistant use cases under real world conditions (medium/large). Throw an LLM in the mix and you're sitting around waiting at least several seconds for a response, even with ridiculous hardware. On anything less than ridiculous hardware (including GPU) that becomes at least 10s of seconds very quickly. At the fundamental level a seven year old $100 used GTX 1070 is approximately 5x faster than a monster CPU like the Threadripper PRO 5955WX - at a fraction of the cost and power. That's just for the first step (speech rec), to get something approaching Alexa-level you're in RTX 3090 territory because performance and VRAM. Amazon has spent hundreds of millions of dollars (minimum) over the better part of a decade developing Echo/Alexa. The open source world has a long way to go to catch up. [0] - https://heywillow.io/components/willow-inference-server/#benchmarks https://heywillow.io/components/willow-inference-server/#ben...

by api 3y ago

Ono-Sendai 3y ago

I have something pretty rudimentary here: https://github.com/Ono-Sendai/project-2501 https://github.com/Ono-Sendai/project-2501 Whisper.cpp + chatGPT + windows text-to-speech.

gpderetta 3y ago

I literally played with cat'ing the output of one into the input of the other and it worked better than I had any reason to expect. edit: in my 30minutes of playing with it, I didn't find a good sounding open-source text-to-speech model for the final stage of the pipeline.

cooper_ganglia 3y ago
It’s not open-source, but Play.HT has a new “Turbo model” that can begin generating text-to-speech within 150ms. I’ve tried it out, and it’s pretty impressive in terms of both quality and speed. There’s an API, so perhaps that would be worth looking into!
regularfry 3y ago
This lack of a decent open text to speech is really frustrating, because some of the closed ones are just scary good.
- rickydroll 3y ago
  I've used various forms of Dragon since 1994 to help me deal with hand/arm problems. I would love to tell Windows to go take a flying leap but I can't because I need to use Dragon. If I had real-time or near real-time recognition AND the ability to edit by voice text in any field or application that I would be in a very good place. A better place would also include adding per application/global commands to drive the application. If you haven't lived with speech recognition, it's not apparent that the command space for speech environment is significantly different from the command space for mouse and hands. In order to make the command space for speech work well, the speech recognition environment the application needs to present to the API with access to all functionality and data within the application.
  - maeil 3y ago
    > I would love to tell Windows to go take a flying leap but I can't because I need to use Dragon. What does Dragon do you for that Talon can't?
    - rickydroll 3y ago
      Talon has advanced quite a bit since I last I looked but it looks like it still falls short. However, I will give it a trial run again. It looks quite useful if you need command and control. for general diction, not so much. To be fair, Dragon and other SR systems fail at speech driven editing. 99.9% of my dragon use is plane text dictation and when if the app is select-and -say enabled,edit and make corrections by voice. Speech commands as they are implemented are rarely useful mostly because there are too many of them to remember. Fortunately, my hands have recovered enough it is faster to type and mouse that it is to silently try to remember commands, construct what I want to say and then say it without stumbling or pausing then, undoing what was recognized, and trying again until I get it right. one thing I'd like to do is difficult; have the same command give the same results in different contexts. For example: say "make me root" and have the command recognize what machine on what network I'm connected to, sent the command "sudo su -<enter>" and then send the right password from my password manager without me having to type anything. example: tail [forever] <service log> work the same no matter what distribution, of logging method (syslog vrs journald) and priv level. if I need a sudo before accessing log data put one in and give it the password if needed without any action om my part. another should be possible is dictating into an app (like thunderbird) with text boxes and when focus is outside of the box, turn off recognition output. if you leave recognition output active outside of a text box then speech is typing hotkey commands from the letters in the words recognized. disaster example: hotkey-stroking your way through your email, phone rings, you forgot to turn your mic off. at the end of the all, you recognize your out-of-context recognition has destroyed your mailbox is and you have no idea how to recover it. The biggest thing is dragon is an out-of-the box solution. 15 mins (or less) install and I am dictating at high accuracy and a large vocabulary.
- kkielhofner 3y ago
  If you think open source speech recognition is behind don't even look at text to speech synthesis. It's not even in the same galaxy.

regularfry 3y ago

https://github.com/ggerganov/whisper.cpp/tree/master/examples/talk-llama https://github.com/ggerganov/whisper.cpp/tree/master/example... is worth a poke. llama.cpp supports llama2 on CPU.

kkielhofner 3y ago

Shameful plug for my project Willow: https://heywillow.io/ https://heywillow.io/ Note that it's important to understand the realities here - short of something like an RTX 3090/4090 with LLama and every performance optimization available when it comes to responsiveness and accuracy competitive with commercial solutions this is a big challenge. Even with the potential improvements of this work and optimizations like Ctranslate (used by our Willow Inference Server and faster-whisper) getting sub one second response times like Echo/Alexa more-or-less automatically calls for GPU even with every performance trick available. As I like to say when it comes to ML/AI/speech rec/speech synthesis if you bring a CPU to a GPU fight you're going to lose - and all of the commercial implementations are certainly using GPU/TPU on top of who knows what else they've come up with over the years and their immense spending. To get an idea of how dramatic this is you can see the benchmarks with Ctranslate2/faster-whisper and our Willow Inference Server here[0]. Looking at those real-world numbers even at a claimed 6x performance improvement a mighty Threadripper PRO 5955WX can barely meet this goal with the models needed for voice assistant use cases under real world conditions (medium/large). Throw an LLM in the mix and you're sitting around waiting at least several seconds for a response, even with ridiculous hardware. On anything less than ridiculous hardware (including GPU) that becomes at least 10s of seconds very quickly. At the fundamental level a seven year old $100 used GTX 1070 is approximately 5x faster than a monster CPU like the Threadripper PRO 5955WX - at a fraction of the cost and power. That's just for the first step (speech rec), to get something approaching Alexa-level you're in RTX 3090 territory because performance and VRAM. Amazon has spent hundreds of millions of dollars (minimum) over the better part of a decade developing Echo/Alexa. The open source world has a long way to go to catch up. [0] - https://heywillow.io/components/willow-inference-server/#benchmarks https://heywillow.io/components/willow-inference-server/#ben...