3 ms·

Shameless plug, but I have been working on an open source IDE plugin [1] for the IntelliJ Platform which attempts to do this. Previously, we used an older HMM-b

Shameless plug, but I have been working on an open source IDE plugin [1] for the IntelliJ Platform which attempts to do this. Previously, we used an older HMM-based speech toolkit called CMUSphinx [2], but are currently transitioning to a deep speech recognition system. We also tried a number of cloud APIs including Amazon Lex and Google Cloud Speech, but they were too slow -- offline STT is really important for low latency UX applications. For navigation and voice typing, we need something customizable and fairly responsive. Custom grammars would be nice for various contexts and programming languages. There are a few good OSS offline deep speech libraries including Mozilla DeepSpeech [3], but their resource footprint is too high. We settled on the currently less mature vosk [4], which is based on Kaldi [5] (a more popular deep speech pipeline), and includes a number of low-footprint, pretrained language models for real-time streaming inference. Research has shown how to deploy efficient deep speech models on CPUs [6], so we're hoping those gains will translate to faster performance on commodity laptops soon. You can follow this issue [7] for updates on our progress. Contributions are welcome! [1]: https://github.com/OpenASR/idear/ https://github.com/OpenASR/idear/ [2]: https://cmusphinx.github.io/ https://cmusphinx.github.io/ [3]: https://github.com/mozilla/DeepSpeech https://github.com/mozilla/DeepSpeech [4]: https://github.com/alphacep/vosk-api https://github.com/alphacep/vosk-api [5]: https://github.com/kaldi-asr/kaldi https://github.com/kaldi-asr/kaldi [6]: https://ai.facebook.com/blog/a-highly-efficient-real-time-text-to-speech-system-deployed-on-cpus/ https://ai.facebook.com/blog/a-highly-efficient-real-time-te... [7]: https://github.com/OpenASR/idear/issues/52 https://github.com/OpenASR/idear/issues/52

daanzu 6y ago
I agree with everything you said, but I would add that a critical component of voice command and control is strict grammars. There is so much structure and context in what we speak, and being able to limit what can be recognized to only what can be reasonably spoken (based on the current context) can allow massive increases in accuracy. (EDIT: ah, you edited to add a mention of this as well.) And one shameless plug deserves another! Vosk is a great project, but my kaldi-active-grammar [0] (mentioned in another comment here) also uses the same Kaldi engine, but extends it and is designed specifically for this use case. It supports defining many grammars, in any combination, and activating/deactivating them at will instantly per-utterance. I think it's probably a better fit as a backend for your project than vosk. My work focuses on the backend technology, so it would be great to have more front ends using it to put it within users' reach (so to speak). [0] https://github.com/daanzu/kaldi-active-grammar https://github.com/daanzu/kaldi-active-grammar

daanzu 6y ago

I agree with everything you said, but I would add that a critical component of voice command and control is strict grammars. There is so much structure and context in what we speak, and being able to limit what can be recognized to only what can be reasonably spoken (based on the current context) can allow massive increases in accuracy. (EDIT: ah, you edited to add a mention of this as well.) And one shameless plug deserves another! Vosk is a great project, but my kaldi-active-grammar [0] (mentioned in another comment here) also uses the same Kaldi engine, but extends it and is designed specifically for this use case. It supports defining many grammars, in any combination, and activating/deactivating them at will instantly per-utterance. I think it's probably a better fit as a backend for your project than vosk. My work focuses on the backend technology, so it would be great to have more front ends using it to put it within users' reach (so to speak). [0] https://github.com/daanzu/kaldi-active-grammar https://github.com/daanzu/kaldi-active-grammar