12 ms·
Voice2json: Offline speech and intent recognition on Linux
- marcodiego 5y agoFor those who care: MIT license.
- deleted 5y ago[deleted]
- marcodiego 5y agoGood FLOSS speech recognition and TTS is badly needed. Such interaction should not be left to an oligoply with bad history of not respecting users freedoms and privacy.
- sodality2 5y agoMozilla CommonVoice is definitely trying. I always do a few validations and a few clips if I have a few minutes to spare, and I recommend everyone does. They need volunteers to validate and upload speech clips to create a dataset. https://commonvoice.mozilla.org/en https://commonvoice.mozilla.org/en
- jfarina 5y agoI wonder if they use movies and tv; recordings where the script is already available.
- wongarsu 5y agoThat's fine for training your own model, but I don't think you could distribute the training set. That seems like a clear copyright violation, against one of the groups that cares most about copyright. Maybe you could convince a couple of indie creators or state-run programs to licence their audio? But I'm not sure if negotiating that is more efficient than just recording a bit more audio, or promoting the project to get more volunteers.
- sodality2 5y agoDoes using copyrighted works to train a machine learning model make that model infringing?
- marcodiego 5y agoGP is not talking about the model but about the training data set.
- sodality2 5y agoI am aware, I'm asking if the model, however, is infringing. Surely you can't distribute them in a dataset but is training on copyrighted data legal, and can you distribute that model?
- _jal 5y agoAll text written by a human in the US is automatically copyright the author. So if an engine trained on works under copyright is a derivative work, GPT3 and friends have serious problems.
- wongarsu 5y agoGenerally a ML model transforms the copyrighted material to the point where it isn't recognizable, so it should be treated as its own unrelated work that isn't infringing or derivative. But then you have e.g. GPT that is reproducing some (largeish) parts of the training set word-for-word, which might be infringing. Also I don't think there have been any major court cases about this, so there's no clear precedent in either direction.
- kelnos 5y agoI expect that wouldn't be perfect, though. Sometimes the cut that makes it into the final product doesn't exactly match the script. Sometimes it's due to an edit, other times it's due to an actor saying something similar to but not exactly what the script says, but the director deciding to just go with it. What might work better is using closed captions or subtitles, but I've also seen enough cases where those don't exactly match the actual speech either.
- wcarss 5y agoI've used the deepspeech project a fair amount and it is good. It's not perfect, certainly, and it honestly isn't good enough yet for an accurate transcription in my mind, but it's good. Easy to work with, pretty good results, and all the right kinds of free. Thanks for taking time to contribute!
- tootie 5y agoIf you read the doc, it says voice2json is layer on top of the actual voice recognition engine. And it supports mozilla deep speech, pocket sphinx and a few others as the underlying engine.
- teraflop 5y agoI like the idea, and decided to try doing some validation. The first thing I noticed is that it asks me to make a yes-or-no judgment of whether the sentence was spoken "accurately", but nowhere on the site is it explained what "accurate" means, or how strict I should be. (The first clip I got was spoken more or less correctly, but a couple of words are slurred together and the prosody is awkward. Without having a good idea of the standards and goals of the project, I have no idea whether including this clip would make the overall dataset better or worse. My gut feeling is that it's good for training recognition, and bad for training synthesis.) This seems to me like a major issue, since it should take a relatively small amount of effort to write up a list of guidelines, and it would be hugely beneficial to establish those guidelines before asking a lot of volunteers to donate their time. I don't find it encouraging that this has been an open issue for four years, with apparently no action except a bunch of bikeshedding: https://github.com/common-voice/common-voice/issues/273 https://github.com/common-voice/common-voice/issues/273
- deleted 5y ago[deleted]
- xwx 5y agoI downloaded the (unofficial) Common Voice app [1] and it provides a link to some guidelines [2], which also aren't official but look sensible and seem like the best there is at the moment. [1] https://f-droid.org/packages/org.commonvoice.saverio/ https://f-droid.org/packages/org.commonvoice.saverio/ [2] https://discourse.mozilla.org/t/discussion-of-new-guidelines-for-recording-validation/36465 https://discourse.mozilla.org/t/discussion-of-new-guidelines...
- cptskippy 5y agoAfter listening to about 10 clips your point becomes abundantly clear. One speaker, who sounded like they were from the mid-west United States, was dropping the S off words in a couple clips. I wasn't sure if it was misreads or some accent I'd never heard. Another speaker, with a thick accent that sounded European, sounded out all the vowels in circuit. Had I not had the line being read, I don't think I'd have understood the word. I heard a speaker with an Indian accent who added a preposition to the sentence that was inconsequential but incorrect none the less. I hear these random prepositions added as flourishes frequently with some Indian coworkers, does anyone know the a reason? It's kind of like how American's interject "Umm..." or drop prepositions (e.g. "Are you done your meal?") and I almost didn't pick up on it. For that matter where did the American habit of dropping prepositions come from? It seems like it's people in the North East primarily.
- cerved 5y agoWeird sentences
- deleted 5y ago[deleted]
- londons_explore 5y agoGood speech recognition generally requites massive mountains of training data, both labelled and unlabelled. Massive mountains of data tends to be incompatible with opensource projects. Even Mozilla collecting user statistics is pretty controversial. Imagine someone like Mozilla trying to collect hundreds of voice clips from each of tens of millions of users!!
- marcodiego 5y agoReally complicated question, but considering the free world got wikipedia and openstreetmaps, I'd bet we'll find a way.
- JadeNB 5y ago> Really complicated question, but considering the free world got wikipedia and openstreetmaps, I'd bet we'll find a way. Both of those involve entering data about external things. Asking people to share their own data is another thing entirely—I suspect most people, me included, are much more suspicious about that.
- posmonerd 5y agoNot an expert on any of this, but wouldn’t already published content (public or proprietary) such as Youtube videos, audiobooks, tv interviews, movies, tv programs, radio programs, podcasts, etc. be useful and exempt from privacy concerns? Do user collected clips have soemthing so special to the point that it’s critical to collect them?
- eliaspro 5y agoMovies etc would need to be transcribed accurately to be useful for training and even then just provide a single sample for the specific item.
- sodality2 5y ago> Imagine someone like Mozilla trying to collect hundreds of voice clips from each of tens of millions of users!! They do, and it's working! https://commonvoice.mozilla.org/en https://commonvoice.mozilla.org/en
- londons_explore 5y agoSpeech recognition algorithms today require lots of data, lots of training computation, and a decent design. Decent designs are in published papers all over the place, so thats a solved issue. Lots of compute requires lots of $$$, which isn't opensource-friendly. Lots of data also isn't really opensource friendly. Sadly this is a niche that the opensource business model doesn't really fit.
- marcodiego 5y agoPeople would probably say the same about wikipedia 20 years ago. People said similar things about gnu, gcc and linux 30 years ago.
- catblast01 5y agoGNU is heavily skewed to developer tools and infrastructure, and gcc is no counterexample. There are so many classes of software where this does not work. Pretty much anything for heavily regulated industries is not well served by FLOSS. There are few open source insurance software or medical records systems (the few that exist are highly niche and/or limited), EDA/CAD is not well served by FLOSS (I’ve toyed with FreeCAD, but even hobbyists gravitate to Fusion). Outside of developer tooling and infrastructure: commercial, generally closed source, closed development software is king. * besides the hard part of standing up an EMR is not installing a prepackaged software.
- sildur 5y ago> Lots of compute requires lots of $$$, which isn't opensource-friendly. Not really, look up BOINC.
- jauer 5y agoThere's more involved in than just raw CPU cycles. It's not something that is easily adapted to BOINC, but trying to offload things to BOINC to free up clusters better suited to training models might make sense.
- airstrike 5y ago
- asdfman123 5y ago> should not be left to an oligoply with bad history of not respecting users freedoms and privacy So companies with a lot of data, then.
- Animats 5y agoThat's not what this is. This is more like the system you use for phone-answering systems ("Do you want help with a bill, payment, order, or refund?")
- GekkePrutser 5y agoIndeed, this is what I got from it too. It seems an alternative to VoiceXML used by companies like Nuance.
- GekkePrutser 5y agoIndeed, and it doesn't have to be as "machine learning" as the big ones. A FLOSS system would only have my voice to recognise and I would be willing to spend some time training it. Very different usecase from a massive cloud that should recognise everyone's voice and accent.
- cf 5y agoI'd check out coqui https://coqui.ai/ https://coqui.ai/ It's well-documented and works basically out of box. I wish the STT models bundled were closer to the quality of Kaldi but the ease-of-use has no comparisons. And maybe with time it will surpass Kaldi in quality too.
- hardwaresofton 5y agoThere are a bunch of good libraries that work offline out there for speech recognition -- CMUSphinx[0] has been around a long time and work seems to have shifted a little bit to Kaldi[1] and Vosk[2] (?). Julius is still going strong as well[3]. CMUSphinx and Julius have been around for ~10+ years at this point. [EDIT] - there's even a useful Quora post[4] [0]: https://cmusphinx.github.io/ https://cmusphinx.github.io/ [1]: https://www.kaldi-asr.org/doc/about.html https://www.kaldi-asr.org/doc/about.html [2]: https://github.com/alphacep/vosk-server https://github.com/alphacep/vosk-server [3]: https://github.com/julius-speech/julius https://github.com/julius-speech/julius [4]: https://www.quora.com/Are-there-any-open-source-APIs-for-speech-recognition https://www.quora.com/Are-there-any-open-source-APIs-for-spe...
- smcameron 5y agopico2wav with the en-gb voice seems not too bad for TTS. I had reasonable luck in limited domain speech recognition with pocketsphinx, but it does need some custom vocabulary. Example: https://www.youtube.com/watch?v=tfcme7maygw https://www.youtube.com/watch?v=tfcme7maygw Granted, maybe this is "not good enough", but I feel like I got pretty far with pico2wave, pocketsphinx plus 1980's Zork level "comprehension" technology. And the open source status of pico2wave is a bit questionable, I'll grant you that. A bit more detail about the implementation here: https://scaryreasoner.wordpress.com/2016/05/14/speech-recognition-and-natural-language-processing-in-space-nerds-in-space/ https://scaryreasoner.wordpress.com/2016/05/14/speech-recogn...
- yewenjie 5y agoHow does it compare to Vosk and other open source models/APIs?
- robmsmt 5y agoI am working on something to compare at least 10 different ASRs both open source and production ones.
- synesthesiam 5y agoI plan to add Vosk support soon. The goal of voice2json is to provide a common layer on top of existing open source engines. This common layer lets you train custom speech/intent models with having to know the details of each engine.
- hirundo 5y agoI wonder if it would be possible to map vim keybindings to sounds and effectively drive the editor with the mouth when the hands are otherwise occupied. It might be possible to use sounds that compose into pronounceable words with minimal syllables for combinations. What would vim bindings look like as a concise command language suited to human vocalization? E.g. maybe "dine" maps to d$ and "chine" to c$. So as in keyboard vim you can guess what "dend" and "chend" do.
- deleted 5y ago[deleted]
- twobitshifter 5y agohttps://youtu.be/8SkdfdXWYaI?t=600 https://youtu.be/8SkdfdXWYaI?t=600 this guy is already there: Slurp slap scratch buff yank
- skratlo 5y agoI now get the joke about Emacs and OS
- krysp 5y agoI do this successfully for work using https://talonvoice.com/ https://talonvoice.com/ - initial learning curve is steep, but once you learn how to configure and hack on the commands, you can be very effective. I use it maybe half the day to combat lingering RSI symptoms, and with some work I could probably use it for 98% of input for the computer. Some people do use it for 100% afaik
- intrepidhero 5y agoWould love to see a demo integrating this with an IDE for either voice to code or for voice commands to navigate menus. I think the killer application would layer voice and traditional input rather than replace.
- offtop5 5y agoFantastic. Might use this with a Raspberry pi to set up some projects around the house. Is it possible to buy higher quality voice data ?
- nmstoker 5y agoIf you're interested in projects on a Pi then you might just be interested in this: https://github.com/rhasspy/rhasspy https://github.com/rhasspy/rhasspy It's from the same author.
- GekkePrutser 5y agoI like rhasspy but the problem I have with it is that it's too much of a toolkit and less of an application. There's too many choices to pick for the different components.. I think they should pick one of each and really tune them so it works really well. This way they'd take a lot of complexity away from the user.
- synesthesiam 5y agoAgreed. I've at least added a "Recommended" option in the web UI that's language-specific. Part of the problem is that language support varies dramatically between components. There's usually a pretty obvious "best" set for English, but it gets more difficult with other languages.
- bmn__ 5y agoHas anyone had any success getting the software to work? It's entirely unpackaged: https://repology.org/projects/?search=voice2json https://repology.org/projects/?search=voice2json https://pkgs.org/search/?q=voice2json https://pkgs.org/search/?q=voice2json Docker image is broken, how'd that happen? $ voice2json --debug train-profile ImportError: numpy.core.multiarray failed to import Traceback (most recent call last): File "/usr/lib/voice2json/.venv/lib/python3.7/site-packages/deepspeech/impl.py", line 14, in swig_import_helper return importlib.import_module(mname) File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 670, in _load_unlocked File "<frozen importlib._bootstrap>", line 583, in module_from_spec File "<frozen importlib._bootstrap_external>", line 1043, in create_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed ImportError: numpy.core.multiarray failed to import
- nerdponx 5y agoThe source package does have installation instructions and appears to use Autotools: https://voice2json.org/install.html#from-source https://voice2json.org/install.html#from-source. Hopefully at least building from source works.
- mdaniel 5y agoBuilding the v2.0 tag (or even master) using docker does not: E: The repository 'http://security.ubuntu.com/ubuntu eoan-security Release' does not have a Release file. And just bumping the image tag to ":groovy" caused subsequent silliness, so this project is obviously only for folks who enjoy fighting with build systems (and that matches my experience of anything in the world that touches Numpy and friends)
- disgruntledphd2 5y ago
- jrm4 5y agoExcellent! I just installed MyCroft the other day to play around with it; while it looks like a great start, two odd things. The first is obvious, which is the online/offline thing. The second was a little surprising (and maybe I missed it?) There was not much in the way of easily accessing transcribed output to and from shell scripts?
- nwalker85 5y agoReally interesting use of intents and entities. I feel like some of this is reinventing the wheel, since there is already a grammar specification, but novel use of intents/entities. https://www.w3.org/TR/speech-grammar/ https://www.w3.org/TR/speech-grammar/
- Edman274 5y agoYeah, in my experience no one uses or supports that specification, which is a shame because if you're using something like AWS Connect with AWS Lex for telephony IVR, you can't just create a grammar and then have AWS Lex figure out how to turn its recognized speech-to-text into something that matches a grammar rule. Thus, Lex will return speech-to-text results that are according to general English grammar rules, rather than what you might have prompted the user to reply with. You'll be unpleasantly surprised if you think that defining a custom entity as alphanumeric always prevents the utterance "[wʌn]" as sometimes matching "won" instead of "one" or "1". Edit - Sorry, I realize that's a tangent. What I'm saying is that when I was evaluating speech to text engines for things like IVR systems using AWS and Google, neither of them supported SRGS. Microsoft does, I think, but they didn't have a telephony component, and IBM was ignored from the get go, so "no one" really means "two very large companies."
- nwalker85 5y agoSome do, some don't, sure. Google STT for example supports class tokens natively. There are also services like uniMRCP that allow for certain SRGS grammar features to be used with Google STT, but they are limited in what constructs they support. I've worked pretty extensively with a platform called Verbio, and they fully support the SRGS grammar specification. I work in conversational AI, and when I do implementations, I have to evaluate the complexity of the use case and whether or not a full grammar will be needed and choose a STT provider based on that.
- synesthesiam 5y agoMy templating language was inspired by JSGF, which seems to have informed the ABNF version of the W3C Speech Grammars. I don't support probabilities, though, since those are derived during the n-gram model generation. I would have preferred to use a standard. Perhaps this is something for a future version.
- bachmitre 5y agoThis should come with pre-trained templates to create new templates via voice commands ;)
- a-dub 5y agoneat. would be even neater if it used state to provide a prior on likely intents. (ie. in its most simple form, if you know the light is on, "turn on the light" has a prior of 0)
- jedimastert 5y agoThings like state would probably be under the scope of whatever you're feeding the intents into
- a-dub 5y agoyes, but by the time you've generated an intent it's too late to improve recognition accuracy using the prior.
- varispeed 5y agoIt's not quite clear, but do you need to sacrifice your privacy in any way to use it? E.g. sending the data to some service in order to get trained model?
- nmstoker 5y agoThe description clarifies the underlying systems: >> Supported speech to text systems include: >> CMU’s pocketsphinx >> Dan Povey’s Kaldi >> Mozilla’s DeepSpeech 0.6 >> Kyoto University’s Julius In case you're not aware, those are all locally run (thus not sending data off, not sacrificing privacy as you mention)
- synesthesiam 5y agoAuthor here. Thanks to everyone for checking out voice2json! The TLDR of this project is: a unified command-line interface to different offline speech recognition projects, with the ability to train your own grammar/intent recognizer in one step. My apologies for the broken packages; I'll get those fixed shortly. My focus lately has been on Rhasspy (https://github.com/rhasspy/rhasspy https://github.com/rhasspy/rhasspy), which has a lot of the same ideas but a larger scope (full voice assistant). Questions, comments, and suggestions are welcomed and appreciated!
- Jeaye 5y agoHave you seen anyone using this for vim? Do you have any example of how that might look, or insight into whether it would work?
- synesthesiam 5y agoI haven't seen this yet, but I imagine it would involve running at least "voice2json record-command | voice2json transcribe-wav | jq .text". This will record a single command (until silence), and output the text transcription.
- markjgx 5y agoHey there, this looks great. I was wondering why Deepspeech 0.6? Why not the latest version DeepSpeech 0.9?
- synesthesiam 5y agoI need to cycle back and update voice2json. Rhasspy (the full voice assistant) supports DeepSpeech 0.9.3.
- markjgx 5y agoAwesome, thanks.
- tootie 5y ago
- dokem 5y agoI can finally build the Jarvis home assistant I dreamt of when first learning coding in high school. To bad now I know voice assistant widgets generally are useless.
- tootie 5y agoSpeech recognition is actually orthogonal to AI. In my day the AI prototypes (like ELIZA) were basically chat bots. Speech recognition is now very sophisticated and accurate. Determining meaning from human language (spoken or written) is far more advanced than it used to be but still kinda sucks.
- dokem 5y agoYes, text to speech has become impressively good and can basically be relied upon. But siri/alexa still really disappoint me and dont seem to be much better than a list of basic rules one could program up.
- slmjkdbtl 5y agoNot familiar with any of these tech, but would it be better to get the intent by voice2txt command.wav | txt2intent ? Or the intent analyzation actually requires the sound data (what are the cases of the same phrase expressing different intent, or how do we even define / categorize intent in this context)
- pabs3 5y agoI wonder if this would pass the Debian Deep Learning Team's Machine Learning policy, which requires public data under a libre license that is retrainable using software under a libre license, without any proprietary drivers: https://salsa.debian.org/deeplearning-team/ml-policy https://salsa.debian.org/deeplearning-team/ml-policy
- synesthesiam 5y agoI think many of the available Kaldi/DeepSpeech models would pass, at least with the "Type-F Reproducibility". The pocketsphinx models would not, however, since they were trained on private datasets. My aim has been to train "good enough" models for any public/free data I can get my hands on.
- drno123 5y agoI would really like this technology to takeoff in mobile apps. When interacting with my mobile phone often it is more convenient to do navigation by voice than by using my finger.
- Mitch104 5y agoJust posted a medium article on using this tool to execute terminal commands with your voice https://mitchellharle.medium.com/how-to-execute-terminal-commands-with-your-voice-on-linux-eb1f6de58bff https://mitchellharle.medium.com/how-to-execute-terminal-com... it was surprisingly straight forward!
- redsummer 5y agoCan I transcribe audio files into plain text with this? Or is this overkill? Don't need json, just plain text.