6 ms·

I've looked into open source voice assistants before. I found mycroft, Jarvis and a few others, but either got bogged down in dependencies or configuration. Man

I've looked into open source voice assistants before. I found mycroft, Jarvis and a few others, but either got bogged down in dependencies or configuration. Many supported shipping your data to Google or Amazon if you configured it, or an open source voice recognition tool. I hate this idea that our voice has to be shipped somewhere to be processed. I remember a lot of the speech-to-text tools in the early 2000s weren't all that great (they needed a lot of training), but why haven't we been able to advance on-device processing? Why is everything done in "the cloud." So the only way to semi-accurately do voice recognition is to source algorithms that re-train off of millions of people? We have processors in our desktops and laptops that dwarf that compute power by leaps and bounds. We should be looking to Star Trek TNG level voice processing, on each individual device, without some central mainframe. But marketing, advertising revenue, data mining, free (as in beer) software that pumps your data like an oil rig, efficiency in data centre (cloud) design .. all these factors have led to these powerful little Intel/ARM/Ryzen chips to be nothing more than thin clients when they're not playing games. If Mozilla really wanted to make something amazing and in the spirit of Firefox, give us an experiment where voice processing is done on our devices. Even if it meant I needed to download a 230GB data set, I'd gladly do it, if it could remotely help in getting away from these data silos.

deleted 6y ago
[deleted]
Vinnl 6y ago
That is what they are working on, but that needs high-quality training data: https://commonvoice.mozilla.org/en https://commonvoice.mozilla.org/en This is just another way of gathering that data. (If consented to.)
- posguy 6y ago
  Mozilla DeepSpeech is trained on about 2000 hours of audio that is mostly spoken by American males. It has little ability to handle noise or accents and has a 5.97% Word Error Rate on LibriVox (which is noiseless, plain spoken english). Meanwhile, Google, Microsoft & IBM have tons of fresh audio coming in constantly to use in augmenting their models. Baidu was able to build a competitive English Speech to Text model with 5000 hours of quality audio to train against. Mozilla did create Common Voice to address this serious data gap, but it has only collected 1492hrs of validated English audio: https://commonvoice.mozilla.org/en/datasets https://commonvoice.mozilla.org/en/datasets
  - deleted 6y ago
    [deleted]
  - scrollaway 6y ago
    There's tens of millions of hours of video content out there that has been subtitled pretty well, and I'd wager a lot of it is under usable licenses for Mozilla. Has that been considered?
    - est31 6y ago
      If you know such sources, file an issue, and better yet, download the video content yourself and publish a dataset. But note that raw video content is not training data. It has to be segmented to be in short enough parts for training (few seconds), the subtitles have to be aligned to match what's said precisely, and one needs to balance the data, e.g. when 90% of speakers are men and 10% are women, you have a problem.
      - scrollaway 6y ago
        Gotcha, all excellent points.
bartvk 6y ago
Even Apple, with its focus on privacy and local AI, needs an internet connection for most Siri queries.
- AsyncAwait 6y ago
  The HomePod has a relatively powerful CPU for what is doing, I'd assume there's lots of headroom to do local intent recognition on device, but Apple likes to get new training data constantly too, so they haven't really tried.
- specialist 6y ago
  Drives me nuts. "Hey Siri, Pause" shouldn't require an internet connection. I often have bad or no connection (eg podcasts while hiking). Voice commands must be ubiquitous. If I have to fail-revert back to touch, what's the point?
Erlich_Bachman 6y ago
Try Nuance Dragon options (no affiliation, just a good tool). They have software that works offline, without cloud. They have a very competitive voice recognition quality for the supported languages. There are some products that are explicitly cloud-based (like mobile ones), but there are also desktop versions that do all the processing, inferring and even learning of your voice and new dictionaries on your local machine. You will need a somewhat powerful PC, but nothing crazy. (PS but to be fair, it's a dictation tool with some extra commands, even if a very good one. It is not really a voice assistant. But I'm not sure if that is harder or easier to do. There are open-source offline assistant tools too that work quite well already, because once you have a set of pre-determined phrases/formulas, they are much easier to parse.)
- solidasparagus 6y ago
  Speech-to-text is relatively easy (not to discount the decades of work that went into it, lots of which was done by Dragon), but extracting intent and sending the user to the correct place is what is really difficult. IMO it is essentially an unsolved problem.
bostik 6y ago
> why haven't we been able to advance on-device processing? Probably because getting the payloads shipped for server side processing provides a constant stream of training material updates? That's my cynical take, at least. But there's got to be more. Paraphrasing H.L. Mencken[0], for every complex problem there is an answer that is straightforward, easy to accept - and wrong. I remember how in the late naughties Nokia had an early version of on-device speech recognition: long-press the trigger key and you could call people by uttering out a saved "voice tag". IIRC the feature picked the right entry roughly 1/3 of the time. For what comes next I'm relying on hearsay, but from what I heard at the time, the feature was originally developed at MIT. Nokia then financed the team to optimise their code and underlying detection model to be ported to ARM and to fit the constrained memory/CPU envelope. Because Nokia definitely had collaboration with MIT at the time, this is at least plausible. If people are expected to use speech-to-text in real life, it has to work within very strict boundaries. Low latency and high accuracy are table stakes. At least some level of contextual awareness would be nice. As long as predictive text input routinely provides us with meme-worthy failures, I won't expect anything better from (fundamentally noisier) speech inputs. And if server-side processing is the only way to get performance from dismal to somewhat functional, practical applications don't have much of a choice. Plus, you don't have to ship your model to end user devices. For what it's worth, I dislike voice interfaces. But when they do work, I dislike them less than Byzantine and user-hostile phone menu systems. I guess that qualifies as progress. 0: https://quoteinvestigator.com/2016/07/17/solution/ https://quoteinvestigator.com/2016/07/17/solution/ 1: https://nokia-e71.helpdoc.net/en/nokia-e71-user-guide/phone/voice-commands/ https://nokia-e71.helpdoc.net/en/nokia-e71-user-guide/phone/...
synesthesiam 6y ago
> I found mycroft, Jarvis and a few others, but either got bogged down in dependencies or configuration. More recently, there is also Rhasspy (https://rhasspy.readthedocs.io https://rhasspy.readthedocs.io) and voice2json (https://voice2json.org https://voice2json.org). I'm the author of both, if you have questions. > Why is everything done in "the cloud." Besides being a way of collecting data and ultimately making money, it avoids some of the "bogged down in dependencies or configuration" problem. My voice projects need to run entirely offline on a variety of hardware and operating systems. If each client was just a little app piping audio data to a cloud service, it would be way easier to write and maintain. > So the only way to semi-accurately do voice recognition is to source algorithms that re-train off of millions of people? Nope. You can absolutely tune a speech model locally on your own samples and get great accuracy. The trouble comes with open-ended speech: people expect the voice assistant to recognize that new artist or movie they heard about yesterday. That doesn't work without upkeep somewhere. Rhasspy/voice2json are intended for pre-defined voice commands using a template language. You can get almost perfect accuracy with this approach, even with millions of possible commands. Re-training only takes a minute, so personal upkeep isn't bad. > Even if it meant I needed to download a 230GB data set, I'd gladly do it, if it could remotely help in getting away from these data silos. It's a lot less than that; at most 1-2GB for a given language, usually a few 100MB: https://github.com/synesthesiam/voice2json-profiles/ https://github.com/synesthesiam/voice2json-profiles/
- binwiederhier 6y ago
  Very cool projects. Thanks for sharing (and creating them!).
- vongomben 6y ago
  @synesthesiam your work is wonderful and I am willing to test both ASAP. It kind of fills a gap left from snips.ai, will check and compare with what's on the market now. Super recent (less than a year for both projects, but may be wrong) and apparently well documented. Hats off
marcinzm 6y ago
Let's say you have two companies A and B. A does on-device only and B does in the cloud. Even if their systems begin being just as accurate, B has an ever growing stream of new training data with which to improve their system. A does not. Over time B will become better and better while A does not. Worse, by downloading the model onto devices the model can be easily copied, cloned, reverse engineered, interrogated and otherwise used by competitors.
heavyset_go 6y ago
Kalliope[1] is a no code, but Python-based, voice assistant. I like it because you can switch out voice backends and it is designed to be modular with node-like constructs to build signals and responses. I stuck it in a container and can control Home Assistant with it, run scripts, etc. [1] https://github.com/kalliope-project/kalliope https://github.com/kalliope-project/kalliope

deleted 6y ago

[deleted]

Vinnl 6y ago

That is what they are working on, but that needs high-quality training data: https://commonvoice.mozilla.org/en https://commonvoice.mozilla.org/en This is just another way of gathering that data. (If consented to.)

posguy 6y ago
Mozilla DeepSpeech is trained on about 2000 hours of audio that is mostly spoken by American males. It has little ability to handle noise or accents and has a 5.97% Word Error Rate on LibriVox (which is noiseless, plain spoken english). Meanwhile, Google, Microsoft & IBM have tons of fresh audio coming in constantly to use in augmenting their models. Baidu was able to build a competitive English Speech to Text model with 5000 hours of quality audio to train against. Mozilla did create Common Voice to address this serious data gap, but it has only collected 1492hrs of validated English audio: https://commonvoice.mozilla.org/en/datasets https://commonvoice.mozilla.org/en/datasets
- deleted 6y ago
  [deleted]
- scrollaway 6y ago
  There's tens of millions of hours of video content out there that has been subtitled pretty well, and I'd wager a lot of it is under usable licenses for Mozilla. Has that been considered?
  - est31 6y ago
    If you know such sources, file an issue, and better yet, download the video content yourself and publish a dataset. But note that raw video content is not training data. It has to be segmented to be in short enough parts for training (few seconds), the subtitles have to be aligned to match what's said precisely, and one needs to balance the data, e.g. when 90% of speakers are men and 10% are women, you have a problem.
    - scrollaway 6y ago
      Gotcha, all excellent points.

bartvk 6y ago

Even Apple, with its focus on privacy and local AI, needs an internet connection for most Siri queries.

AsyncAwait 6y ago
The HomePod has a relatively powerful CPU for what is doing, I'd assume there's lots of headroom to do local intent recognition on device, but Apple likes to get new training data constantly too, so they haven't really tried.
specialist 6y ago
Drives me nuts. "Hey Siri, Pause" shouldn't require an internet connection. I often have bad or no connection (eg podcasts while hiking). Voice commands must be ubiquitous. If I have to fail-revert back to touch, what's the point?

Erlich_Bachman 6y ago

Try Nuance Dragon options (no affiliation, just a good tool). They have software that works offline, without cloud. They have a very competitive voice recognition quality for the supported languages. There are some products that are explicitly cloud-based (like mobile ones), but there are also desktop versions that do all the processing, inferring and even learning of your voice and new dictionaries on your local machine. You will need a somewhat powerful PC, but nothing crazy. (PS but to be fair, it's a dictation tool with some extra commands, even if a very good one. It is not really a voice assistant. But I'm not sure if that is harder or easier to do. There are open-source offline assistant tools too that work quite well already, because once you have a set of pre-determined phrases/formulas, they are much easier to parse.)

solidasparagus 6y ago
Speech-to-text is relatively easy (not to discount the decades of work that went into it, lots of which was done by Dragon), but extracting intent and sending the user to the correct place is what is really difficult. IMO it is essentially an unsolved problem.

bostik 6y ago

> why haven't we been able to advance on-device processing? Probably because getting the payloads shipped for server side processing provides a constant stream of training material updates? That's my cynical take, at least. But there's got to be more. Paraphrasing H.L. Mencken[0], for every complex problem there is an answer that is straightforward, easy to accept - and wrong. I remember how in the late naughties Nokia had an early version of on-device speech recognition: long-press the trigger key and you could call people by uttering out a saved "voice tag". IIRC the feature picked the right entry roughly 1/3 of the time. For what comes next I'm relying on hearsay, but from what I heard at the time, the feature was originally developed at MIT. Nokia then financed the team to optimise their code and underlying detection model to be ported to ARM and to fit the constrained memory/CPU envelope. Because Nokia definitely had collaboration with MIT at the time, this is at least plausible. If people are expected to use speech-to-text in real life, it has to work within very strict boundaries. Low latency and high accuracy are table stakes. At least some level of contextual awareness would be nice. As long as predictive text input routinely provides us with meme-worthy failures, I won't expect anything better from (fundamentally noisier) speech inputs. And if server-side processing is the only way to get performance from dismal to somewhat functional, practical applications don't have much of a choice. Plus, you don't have to ship your model to end user devices. For what it's worth, I dislike voice interfaces. But when they do work, I dislike them less than Byzantine and user-hostile phone menu systems. I guess that qualifies as progress. 0: https://quoteinvestigator.com/2016/07/17/solution/ https://quoteinvestigator.com/2016/07/17/solution/ 1: https://nokia-e71.helpdoc.net/en/nokia-e71-user-guide/phone/voice-commands/ https://nokia-e71.helpdoc.net/en/nokia-e71-user-guide/phone/...

synesthesiam 6y ago

> I found mycroft, Jarvis and a few others, but either got bogged down in dependencies or configuration. More recently, there is also Rhasspy (https://rhasspy.readthedocs.io https://rhasspy.readthedocs.io) and voice2json (https://voice2json.org https://voice2json.org). I'm the author of both, if you have questions. > Why is everything done in "the cloud." Besides being a way of collecting data and ultimately making money, it avoids some of the "bogged down in dependencies or configuration" problem. My voice projects need to run entirely offline on a variety of hardware and operating systems. If each client was just a little app piping audio data to a cloud service, it would be way easier to write and maintain. > So the only way to semi-accurately do voice recognition is to source algorithms that re-train off of millions of people? Nope. You can absolutely tune a speech model locally on your own samples and get great accuracy. The trouble comes with open-ended speech: people expect the voice assistant to recognize that new artist or movie they heard about yesterday. That doesn't work without upkeep somewhere. Rhasspy/voice2json are intended for pre-defined voice commands using a template language. You can get almost perfect accuracy with this approach, even with millions of possible commands. Re-training only takes a minute, so personal upkeep isn't bad. > Even if it meant I needed to download a 230GB data set, I'd gladly do it, if it could remotely help in getting away from these data silos. It's a lot less than that; at most 1-2GB for a given language, usually a few 100MB: https://github.com/synesthesiam/voice2json-profiles/ https://github.com/synesthesiam/voice2json-profiles/

binwiederhier 6y ago
Very cool projects. Thanks for sharing (and creating them!).
vongomben 6y ago
@synesthesiam your work is wonderful and I am willing to test both ASAP. It kind of fills a gap left from snips.ai, will check and compare with what's on the market now. Super recent (less than a year for both projects, but may be wrong) and apparently well documented. Hats off

marcinzm 6y ago

Let's say you have two companies A and B. A does on-device only and B does in the cloud. Even if their systems begin being just as accurate, B has an ever growing stream of new training data with which to improve their system. A does not. Over time B will become better and better while A does not. Worse, by downloading the model onto devices the model can be easily copied, cloned, reverse engineered, interrogated and otherwise used by competitors.

heavyset_go 6y ago

Kalliope[1] is a no code, but Python-based, voice assistant. I like it because you can switch out voice backends and it is designed to be modular with node-like constructs to build signals and responses. I stuck it in a container and can control Home Assistant with it, run scripts, etc. [1] https://github.com/kalliope-project/kalliope https://github.com/kalliope-project/kalliope