8 ms·

QuiLLMan: Voice chat with Vicuna-13B

174 points by pierremenard 3y ago

wongarsu 3y ago
That's a pretty cool showcase of modal [1]. From a marketing perspective I have to congratulate, this is a really well done way to get people to check out your platform. 1: https://modal.com/ https://modal.com/
- fredliu 3y ago
  +1 modal.com is the first thing I checked after reading the readme.
- npace12 3y ago
  +1, props to modal.com, well done and their site is nice
- turnsout 3y ago
  Is it basically Vercel for ML?
- bredren 3y ago
  I also clicked through but immediately abandoned wondering: What’s the state of container based ML deployments? Can I take a container orchestration if these services and just put them on a vps w a GPU and run this? Is there secret or just special sauce in ML infra?
sramam 3y ago
Very cool - the demo was simple, functional and clear. It was a bit laggy, but for a free demo from an open source project, I should be the one being shamed! Well done.
IronWolve 3y ago
Been using textgen and downloading tons of models, the models are all over the place. The problems of accuracy and short term memory are major issues that people are trying to implement work arounds. Check out textgen, it has voice in/out, graphics in/out, memory plugin, api, plugins, etc, all running locally. https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui
- bioemerl 3y ago
  > it has voice in/out, Do you know how to get this working? I looked through the read-me and didn't see any options for it.
  - IronWolve 3y ago
    https://github.com/oobabooga/text-generation-webui/blob/main/docs/Extensions.md https://github.com/oobabooga/text-generation-webui/blob/main... You need to enable the extensions. I only did voice out locally with silero_tts, it also supports voice out with eleven labs api. Voice input is via whisper tts.
- CrazyStat 3y ago
  Any models and/or plugins you recommend?
forgingahead 3y ago
Nice to see Tortoise being used - I still think it's the best TTS system out there now. Generation time is slow, but quality is incredible. I wonder if the code can be optimised to speed up the generation, but I don't think the author is maintaining it any longer.[0] [0]https://github.com/neonbjb/tortoise-tts https://github.com/neonbjb/tortoise-tts
- discardedrefuse 3y ago
  You might want to check out Mimic 3 by the Mycroft team. Its also open source and runs offline. https://mycroft.ai/mimic-3/ https://mycroft.ai/mimic-3/
  - mdaniel 3y ago
    It took quite a bit of digging to find the repo link https://github.com/MycroftAI/mimic3#readme https://github.com/MycroftAI/mimic3#readme and it's AGPL-3 for those interested in such things
- turnsout 3y ago
  Tortoise looks really nice! The output is very "polished" and audiobook-like. It's a contrast to Bark[0] which is far more expressive but unpredictable. [0]: https://github.com/suno-ai/bark https://github.com/suno-ai/bark
- Tepix 3y ago
  Coqui also looks interesting. https://github.com/coqui-ai/TTS https://github.com/coqui-ai/TTS Support for it was recently added to vocode: https://github.com/vocodedev/vocode-python/pull/56 https://github.com/vocodedev/vocode-python/pull/56
- bondsynth 3y ago
  If you're generally interested in TTS check out Bondsynth AI, it's a product I've been working on for long-form text-to-speech (think ebook to audiobook or website to audio). It's still in beta, so it's free, but I'm looking for feedback. Demo at: https://www.youtube.com/watch?v=OmQup3kst5s https://www.youtube.com/watch?v=OmQup3kst5s Signup at: https://signup.bondsynth.ai https://signup.bondsynth.ai
breakingrules 3y ago
[dead]
juliennakache 3y ago
Interesting. How does local development work or remote debugging work if the entire production toolchain is abstracted away with proprietary software?
- thundergolfer 3y ago
  This docs guide has some answers: https://modal.com/docs/guide/developing-debugging https://modal.com/docs/guide/developing-debugging
teacpde 3y ago
Why is it serverless? It clearly has an API server.
- pierremenard 3y ago
  The "API server" only runs when requests come in. Otherwise Modal scales everything down to 0, and you don't pay for anything. It's similar to putting Lambda behind an API gateway.
- SparkyMcUnicorn 3y ago
  Because the resources are on demand, and the provider takes care of all the server management. https://en.wikipedia.org/wiki/Serverless_computing https://en.wikipedia.org/wiki/Serverless_computing
- b33j0r 3y ago
  I think serverless just means that you hire less devops people for your specific team. It’s a cost-saving measure that costs you mortal wounds in integration expense, and eventually building out a real infrastructure plan.
- ZephyrBlu 3y ago
  Is this the first time you've heard the term serverless? It means your code is plug & play, you don't have to manage the infra (I.e. a server) yourself.
  - andybak 3y ago
    I know the term serverless - but it was fairly ambiguous in this context. I presumed the same as the person you're replying to.
  - teacpde 3y ago
    I understand the concept of serverless as in AWS Lambda, but my first impression reading the title is serverless as in no backend. That would be a lot more amazing.
lxe 3y ago
For "actually serverless" voice chat, check out https://whisper.ggerganov.com/ https://whisper.ggerganov.com/
- EGreg 3y ago
  what is the point when the Web browsers mostly all already support it https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_...
  - lxe 3y ago
    Engineering curiosity and fun
  - hombre_fatal 3y ago
    When you want much better, state of the art quality. Compare it yourself.
    - EGreg 3y ago
      The browser seems to do very well for me, and in multiple languages. What about this, can it do languages?
      - RockRobotRock 3y ago
        Whisper supports many different languages.
      - hombre_fatal 3y ago
        Whisper can do you one better: you don't need to know the spoken language ahead of time, and you can mix languages in the same sentence. It does all sorts of things better like upgrading words into proper nouns, and doing it based on context. Honestly it's the first time speech recognition finally feels "production ready" after years of Siri/Alexa.
  - easrng 3y ago
loudmax 3y ago
Sorry if this is off-topic, but those are some really good book recommendations in the demonstration image! If those are coming from Vicuna, that speaks well of it.
yewenjie 3y ago
Does anybody know if there is an easy-to-deploy setup for tortoise-tts only?
- pierremenard 3y ago
  Yup, the code in the above repo is based on https://github.com/metavoicexyz/tortoise-tts-modal-api https://github.com/metavoicexyz/tortoise-tts-modal-api
deleted 3y ago
[deleted]
tasty_freeze 3y ago
I pitched this on a recently thread, but it was 12+ hours after it was posted, so I'll try again here. What I really want is a program to waste the time of phone calls making unsolicited sales pitches. It would do voice to text, run a simple language model to generate responses, then synthesize the voice back. It doesn't need to be a sophisticated model, not much more sophisticated than the classic "Eliza" program. A few years back someone did this with a canned loop of vague responses and it fooled the sales people for surprisingly long: https://www.youtube.com/watch?v=XSoOrlh5i1k https://www.youtube.com/watch?v=XSoOrlh5i1k It seems like it could all run locally for low latency. Probably the most important part to get right would be a TTS system that isn't immediately pegged as a robot.
- deleted 3y ago
  [deleted]
- Zetaphor 3y ago
  Have you seen Lenny? It's a much lower tech solution that seems to work quite well. https://lennytroll.com/ https://lennytroll.com/
  - tasty_freeze 3y ago
    Yes, that was the youtube video I linked to. If that can be successful with some telemarketers, I'm sure a chat-gpt aided program could be even more successful at frustrating telemarketers.
- mdaniel 3y ago
  just in case someone finds that YT video entertaining, you'll really love https://www.youtube.com/@KitbogaShow https://www.youtube.com/@KitbogaShow
syntaxing 3y ago
Off tangent but can someone at Apple please just replace the Siri word recognition with whisper. We can finally have multi language support and not dogshit recognition.
lee101 3y ago
[dead]
testernews 3y ago
How about a self hosted version?

174 points by pierremenard 3y ago

That's a pretty cool showcase of modal [1]. From a marketing perspective I have to congratulate, this is a really well done way to get people to check out your platform. 1: https://modal.com/ https://modal.com/

fredliu 3y ago
+1 modal.com is the first thing I checked after reading the readme.
npace12 3y ago
+1, props to modal.com, well done and their site is nice
turnsout 3y ago
Is it basically Vercel for ML?
bredren 3y ago
I also clicked through but immediately abandoned wondering: What’s the state of container based ML deployments? Can I take a container orchestration if these services and just put them on a vps w a GPU and run this? Is there secret or just special sauce in ML infra?

sramam 3y ago

Very cool - the demo was simple, functional and clear. It was a bit laggy, but for a free demo from an open source project, I should be the one being shamed! Well done.

IronWolve 3y ago

Been using textgen and downloading tons of models, the models are all over the place. The problems of accuracy and short term memory are major issues that people are trying to implement work arounds. Check out textgen, it has voice in/out, graphics in/out, memory plugin, api, plugins, etc, all running locally. https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui

bioemerl 3y ago
> it has voice in/out, Do you know how to get this working? I looked through the read-me and didn't see any options for it.
- IronWolve 3y ago
  https://github.com/oobabooga/text-generation-webui/blob/main/docs/Extensions.md https://github.com/oobabooga/text-generation-webui/blob/main... You need to enable the extensions. I only did voice out locally with silero_tts, it also supports voice out with eleven labs api. Voice input is via whisper tts.
CrazyStat 3y ago
Any models and/or plugins you recommend?

forgingahead 3y ago

Nice to see Tortoise being used - I still think it's the best TTS system out there now. Generation time is slow, but quality is incredible. I wonder if the code can be optimised to speed up the generation, but I don't think the author is maintaining it any longer.[0] [0]https://github.com/neonbjb/tortoise-tts https://github.com/neonbjb/tortoise-tts

discardedrefuse 3y ago
You might want to check out Mimic 3 by the Mycroft team. Its also open source and runs offline. https://mycroft.ai/mimic-3/ https://mycroft.ai/mimic-3/
- mdaniel 3y ago
  It took quite a bit of digging to find the repo link https://github.com/MycroftAI/mimic3#readme https://github.com/MycroftAI/mimic3#readme and it's AGPL-3 for those interested in such things
turnsout 3y ago
Tortoise looks really nice! The output is very "polished" and audiobook-like. It's a contrast to Bark[0] which is far more expressive but unpredictable. [0]: https://github.com/suno-ai/bark https://github.com/suno-ai/bark
Tepix 3y ago
Coqui also looks interesting. https://github.com/coqui-ai/TTS https://github.com/coqui-ai/TTS Support for it was recently added to vocode: https://github.com/vocodedev/vocode-python/pull/56 https://github.com/vocodedev/vocode-python/pull/56
bondsynth 3y ago
If you're generally interested in TTS check out Bondsynth AI, it's a product I've been working on for long-form text-to-speech (think ebook to audiobook or website to audio). It's still in beta, so it's free, but I'm looking for feedback. Demo at: https://www.youtube.com/watch?v=OmQup3kst5s https://www.youtube.com/watch?v=OmQup3kst5s Signup at: https://signup.bondsynth.ai https://signup.bondsynth.ai

breakingrules 3y ago

[dead]

juliennakache 3y ago

Interesting. How does local development work or remote debugging work if the entire production toolchain is abstracted away with proprietary software?

thundergolfer 3y ago
This docs guide has some answers: https://modal.com/docs/guide/developing-debugging https://modal.com/docs/guide/developing-debugging

teacpde 3y ago

Why is it serverless? It clearly has an API server.

pierremenard 3y ago
The "API server" only runs when requests come in. Otherwise Modal scales everything down to 0, and you don't pay for anything. It's similar to putting Lambda behind an API gateway.
SparkyMcUnicorn 3y ago
Because the resources are on demand, and the provider takes care of all the server management. https://en.wikipedia.org/wiki/Serverless_computing https://en.wikipedia.org/wiki/Serverless_computing
b33j0r 3y ago
I think serverless just means that you hire less devops people for your specific team. It’s a cost-saving measure that costs you mortal wounds in integration expense, and eventually building out a real infrastructure plan.
ZephyrBlu 3y ago
Is this the first time you've heard the term serverless? It means your code is plug & play, you don't have to manage the infra (I.e. a server) yourself.
- andybak 3y ago
  I know the term serverless - but it was fairly ambiguous in this context. I presumed the same as the person you're replying to.
- teacpde 3y ago
  I understand the concept of serverless as in AWS Lambda, but my first impression reading the title is serverless as in no backend. That would be a lot more amazing.

lxe 3y ago

For "actually serverless" voice chat, check out https://whisper.ggerganov.com/ https://whisper.ggerganov.com/

EGreg 3y ago
what is the point when the Web browsers mostly all already support it https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_...
- lxe 3y ago
  Engineering curiosity and fun
- hombre_fatal 3y ago
  When you want much better, state of the art quality. Compare it yourself.
  - EGreg 3y ago
    The browser seems to do very well for me, and in multiple languages. What about this, can it do languages?
    - RockRobotRock 3y ago
      Whisper supports many different languages.
    - hombre_fatal 3y ago
      Whisper can do you one better: you don't need to know the spoken language ahead of time, and you can mix languages in the same sentence. It does all sorts of things better like upgrading words into proper nouns, and doing it based on context. Honestly it's the first time speech recognition finally feels "production ready" after years of Siri/Alexa.
- easrng 3y ago

loudmax 3y ago

Sorry if this is off-topic, but those are some really good book recommendations in the demonstration image! If those are coming from Vicuna, that speaks well of it.

yewenjie 3y ago

Does anybody know if there is an easy-to-deploy setup for tortoise-tts only?

pierremenard 3y ago
Yup, the code in the above repo is based on https://github.com/metavoicexyz/tortoise-tts-modal-api https://github.com/metavoicexyz/tortoise-tts-modal-api

deleted 3y ago

[deleted]

tasty_freeze 3y ago

I pitched this on a recently thread, but it was 12+ hours after it was posted, so I'll try again here. What I really want is a program to waste the time of phone calls making unsolicited sales pitches. It would do voice to text, run a simple language model to generate responses, then synthesize the voice back. It doesn't need to be a sophisticated model, not much more sophisticated than the classic "Eliza" program. A few years back someone did this with a canned loop of vague responses and it fooled the sales people for surprisingly long: https://www.youtube.com/watch?v=XSoOrlh5i1k https://www.youtube.com/watch?v=XSoOrlh5i1k It seems like it could all run locally for low latency. Probably the most important part to get right would be a TTS system that isn't immediately pegged as a robot.

deleted 3y ago
[deleted]
Zetaphor 3y ago
Have you seen Lenny? It's a much lower tech solution that seems to work quite well. https://lennytroll.com/ https://lennytroll.com/
- tasty_freeze 3y ago
  Yes, that was the youtube video I linked to. If that can be successful with some telemarketers, I'm sure a chat-gpt aided program could be even more successful at frustrating telemarketers.
mdaniel 3y ago
just in case someone finds that YT video entertaining, you'll really love https://www.youtube.com/@KitbogaShow https://www.youtube.com/@KitbogaShow

syntaxing 3y ago

Off tangent but can someone at Apple please just replace the Siri word recognition with whisper. We can finally have multi language support and not dogshit recognition.

lee101 3y ago

[dead]

testernews 3y ago

How about a self hosted version?