11 ms·
Initial Release of Mozilla’s Open Source Speech Recognition Model and Voice Data
- tmzt 9y agoHow does this compare with Snips.co which can do offline speech recognition on a Rapberry Pi 3? Coyld this be used to train a model/engine that can be used that way?
- reubenmorais 9y agoThe model we released today is not yet optimized for smaller devices like that, but our plan is to make it usable on targets like the RPi3.
- RubenSandwich 9y agoAre you releasing any prebuilt models, I searched but couldn't find any, so people can go and play with your work without training? Edit: NM found it under releases: https://github.com/mozilla/DeepSpeech/releases https://github.com/mozilla/DeepSpeech/releases.
- daveevad 9y agoThank you for finding and linking the prebuilt model. It was eluding me.
- knocte 9y agoOMG, C++??? I thought Mozilla folks would give Rust a spin.
- TD-Linux 9y agoIt is Tensorflow based which has a C++ API. Though, it looks like they provide Rust bindings for the Deep Speech library itself.
- Cyphase 9y agoThe URL is https://snips.ai/ https://snips.ai/ .
- tmzt 9y agoThanks, going by memory which is apparently faulty.
- skoocda 9y agoI can't find exact numbers on Snips.ai, but generally there's a linear relation between the size of the inference model in RAM and the accuracy it can obtain. I'd have to assume DeepSpeech outperforms anything running on a RasPi3, at least for LVCSR. It hits 93.5% accuracy on Librispeech, which I've never seen from any offline recognition models.
- woodson 9y agoKaldi has 4.14% WER (95.86% accuracy) on the same test dataset (test-clean) [1] using a model that runs faster than real time on CPU. You would have to make the model smaller to run it in real time on a RasPi3, but according to this [2], you can get decent WERs for read speech even then. [1] https://github.com/kaldi-asr/kaldi/blob/master/egs/librispeech/s5/local/chain/tuning/run_tdnn_1b.sh https://github.com/kaldi-asr/kaldi/blob/master/egs/librispee... [2] https://groups.google.com/d/msg/kaldi-help/Pr6jPH1Qshg/kn8df0JmBQAJ https://groups.google.com/d/msg/kaldi-help/Pr6jPH1Qshg/kn8df...
- TD-Linux 9y agoThey are listed as a contributor to the Common Voice dataset: https://medium.com/mozilla-open-innovation/sharing-our-common-voice-mozilla-releases-second-largest-public-voice-data-set-e88f7d6b7666 https://medium.com/mozilla-open-innovation/sharing-our-commo... I think their product is higher level - it has things like recognizing commands and sentence structures, and hotword detection.
- mediocrejoker 9y agoI don't think snips is open source, for one.
- bobajeff 9y agoAwesome, we're so close to having a speech-to-text system I can trust. I really wish Mozilla would release a keyboard app for Android. It would instantly be the single most trusted keyboard available.
- wslh 9y agoRegarding Android keyboards, it is horrific that Google keyboard sends all your key presses, except passwords, to them.
- yjftsjthsd-h 9y ago... it does what? Even if you disable the "Share Snippets" option?
- jasonkostempski 9y agoI hope not but I only noticed the opt-out feature just a few weeks ago. It's sickening that Google thinks it's acceptable. Opt-in or opt-out, it never should have been considered a viable "feature" to include.
- wslh 9y agoThere is no official statement of Google about this but there is a generic Android warning saying "This method can collect all of the text that you enter except passwords including personal data and credit card numbers.". I wonder why Google doesn't explicitly say what they are doing. I don't want to distribute FUD but this is a critical component.
- beojan 9y agoThat's because any keyboard could, theoretically, be a keylogger.
- api 9y agoEverything by Google sends everything to Google.
- muxator 9y agoI am very grateful for this release from Mozilla, and more generally for the broad vision of their effort. As time passes, the quest for openness and freedom in software moves higher up in the stack. Thanks to the latest ~30 years of effort, we basically came to a point in which we have free OSes, basic infrastructure, building tools, end-user applications. In the last ~10 years we changed paradigm: autonomous desktop computing progressively transitioned to mobile, with a lot of functionality offloaded to "the cloud". What I feel is needed, going forward, is working towards building a viable free replacement for these distributed services. DeepSpeech is a step in the right direction. Edit: just speaking about SW, here. HW is worth a different topic, and probably poses even more challenges.
- tonmoy 9y agoIt’s exciting to see open source RISC-V gaining momentum on the HW world
- TaylorAlexander 9y agoAgreed!
- TaylorAlexander 9y agoThanks for the note about HW. In my field in robotics openness is not common (ROS being a major exception). I hold a rather uncommon view that a path to a more free society requires abundance of open source tools. In the software world we are getting closer, but the whole hardware world is still one big unmoving binary blob. I make open source robots and I’m hoping to help in that area. But we need open source machine tools, open source factories, and open manufacturing processes. There’s lots of work do be done in hardware land.
- 24gttghh 9y agoI would think 3D printing, portable CNC machines[0], and custom PCB fabrications[1] are making headway into the openness of the HW space. I think that even before the tools, or maybe alongside of, there needs to be an abundance of cheap, clean energy to power the open source tools for the real revolution in manufacturing to take off. [0]http://www.goliathcnc.com/ http://www.goliathcnc.com/ [1]https://oshpark.com/ https://oshpark.com/
- pmoriarty 9y agoDoes this foreshadow a day when Firefox starts spying on what I'm talking about?
- khedoros1 9y agoIf they wanted to do that, there are easier ways to do it than to release a speech recognition system and the speech data that makes it work.
- jimktrains2 9y agoWouldn't this be the opposite? By bundling the speech model so that it's on the client device instead of their server, your speech need not leave your device.
- asabjorn 9y agoDoes this relate or help at all with speaker identification? The Microsoft speech API also provides speaker recognition which is useful for many applications: https://azure.microsoft.com/en-us/services/cognitive-services/speaker-recognition/ https://azure.microsoft.com/en-us/services/cognitive-service...
- woodson 9y agoNo, this is for speech recognition only.
- ocdtrekkie 9y agoReally happy to see someone major looking at open source speech recognition. Due to the lack of a self-hosted or on-device solution, my assistant/automation software is basically designed for speech, but not currently doing recognition because I haven't found a non-cloud option that does what I need it to.
- kljuka 9y agoHow does Mozilla's 6.5% error rate on LibriSpeech’s test-clean dataset compare to Google's, Apple's, Amazon's and other's voice recognition? I couldn't easily find any chart of comparisons.
- TD-Linux 9y agoIn the Baidu Deep Speech 2 paper, the Baidu implementation is able to get 5.33%, and a human 5.83%. https://arxiv.org/pdf/1512.02595v1.pdf https://arxiv.org/pdf/1512.02595v1.pdf
- Dowwie 9y agothe world’s second largest publicly available voice dataset, which was contributed to by nearly 20,000 people globally Well done, people!
- gok 9y agoThis is super cool, but I'd be cautious about the usefulness of this data set. Both this data set and LibriSpeech are read speech, where the speaker was prompted with a transcription and asked to say it out loud. In practice it's very rare that you're trying to transcribe speech that's already been transcribed. Speech patterns for computer-directed speech (e.g. for voice activated user interfaces) or human-to-human speech (e.g. for meeting transcription) are quite different.
- punchingwater 9y agoYup, this is an excellent point. We have, and will continue to explore ways to allow Common Voice users to speak more organically (for instance by answering a question, or responding free-form to some other sort of prompt). The problem with this approach is that it requires an extra step, transcription, which at the scale we are trying to achieve is pretty costly in either money or time (ie. tedium for our users). Eventually we hope that speech engines can take care of the transcription part, but for now we need people. That said, we will definitely be exploring ways to build in organic speech and perhaps transcriptions to the Common Voice app. This will solve another problem for us too, which is getting public domain material for people to read. Doing this obviously requires a much more complex user experience, and we have more work to figure out how to make something that people will want to use and contribute to. Stay tuned for that :) On the flip side, we hope that these datasets, models, and the tools (ie. DeepSpeech) can get more people (researchers, start-ups, hobbyist) over the hump of building an MVP of something useful in voice. Once you have people using your products, collecting useful in-context voice data becomes much easier. On that note, another approach we are working on is partnering with universities and socially-aware startups like MyCroft, SNIPS, and Mythic. Imagine if voice products in market allowed their users to opt-in to contributing their utterances to an open resource similar to Common Voice. Of course, sharing your voice publicly is not for everyone, or every product scenario. But it does work for some. And if we pool our resources, our hope is to indeed commoditize speech-to-text so that we can focus on more interesting challenges like building voice experiences people want to use. (For instance, could voice somehow be a "progressive enhancement" to the web?).
- 9y ago
- richdougherty 9y agoIf you want to help them out you can visit https://voice.mozilla.org/ https://voice.mozilla.org/ and record some sentences.
- melling 9y agoThey have an iOS app too: https://itunes.apple.com/us/app/project-common-voice-by-mozilla/id1240588326?mt=8 https://itunes.apple.com/us/app/project-common-voice-by-mozi... It’s a great idea to crowd source this. Wonder if this project can turn voice recognition into a solved problem. I just set a daily reminder so I can do 10 minutes a day.
- punchingwater 9y agoThank you so much! I also want to emphasize the importance of listening (validating) as well as recording. Validation is an big part of the puzzle for building machine learning viable data.
- RubenSandwich 9y agoCan I suggest is encourging user's to get recordings from their children as well, as most speech recognition libraries are pretty poor with children's voices. (IMO Alexa Voice Service is by far the best with children voices.)
- yjftsjthsd-h 9y agoIs that okay legally? Maybe parental permission is enough
- Vinnl 9y agoOne thing that wasn't entirely clear to me is how strict you have to be when validating? i.e. I encountered one recording that was completely silent - I figured that had to be marked as invalid. However, another one was barely audible, but by intently listening I did recognise it pronounced the right words - is that OK? And should we validate whether they match the proper accents as well? e.g. if I hear a clear Dutch accent, I presume you wouldn't want that labelled "native British speaker"?
- jononor 9y agoWowow. Time to build an open microphone array for impeoved speech pickup. That can be connected to RPi for voice control that respects privacy.
- mediocrejoker 9y agoSort of like this one? https://www.matrix.one/products/voice https://www.matrix.one/products/voice
- jononor 9y agoThat is a very nice board! The inclusion of a FPGA and ESP32 makes it very capable, and 65USD is a good price for such a package. And I found beamforming code for the microphone array (running on host computer) at https://github.com/matrix-io/matrix-creator-hal/blob/master/cpp/driver/microphone_array.cpp https://github.com/matrix-io/matrix-creator-hal/blob/master/...
- hardwaresofton 9y agoFor those who don't know that open source speech recognition that does not depend on AI already exists: http://cmusphinx.sourceforge.net/ http://cmusphinx.sourceforge.net/ http://julius.osdn.jp/en_index.php http://julius.osdn.jp/en_index.php Maybe with this data set released eventually all that additional data will be used to improve those tools as well
- punchingwater 9y agoDon't forget Kaldi! https://github.com/kaldi-asr/kaldi https://github.com/kaldi-asr/kaldi
- skykooler 9y agoThe problem with Kaldi is that it's virtually impossible to get a dictation model working with Kaldi unless you have a doctorate in speech recognition. There is no "I know basic programming, but little about speech recognition" documentation for Kaldi.
- yjftsjthsd-h 9y agoBetween learning curve and dependency hell, I've never managed good results with Kaldi, Simon, or Sphinx. It's unfortunate; hopefully we'll get an easy to use option soon.
- hardwaresofton 9y agoWhen was the last time you tried sphinx? The library has changed a LOT. Their guides, new website and other resources basically walk you from zero knowledge to working demo.
- yjftsjthsd-h 9y agoOn, I will have to try again:) Thanks for the tip; I always thought Sphinx should be ideal, it was just too much work to get it working.
- monsieurgaufre 9y agoI don't see why Mozilla would do this kind of things except to spread ressources. I know that it is an anecdote, but I don't know anyone who uses any kind of speech to text in part because they all suck if you don't speak english and even then..
- elaus 9y agoReally? I know a ton of people who talk to their phones (Google Assistant, Siri, navigation apps, ...) or have something like Alexa in their homes and I don't live in an english speaking country.
- monsieurgaufre 9y agoI live in Montreal and don't know anyone who bought one or speak to their phone. Might be a cultural bias or just an anecdata as stated above.
- khedoros1 9y agoPersonal assistant devices are selling like hotcakes, and there are tons of options for creating smart lighting and such. Maybe I hang around tinkerers too much, but I think it would be nice to have a method of speech control that doesn't rely on someone's remote speech recognition API.
- monsieurgaufre 9y agoAs i stated above, it might be an anecdata or a cultural bias as I live in Montreal. While i do believe that's interesting, I don't understand why that would be Mozilla's job.
- khedoros1 9y agoWhy not? They build free software and promote open standards; speech recognition is a prominent area that doesn't have any really good open solutions right now.
- deleted 9y ago[deleted]
- tardygrad 9y agoThis is tangential, but I wonder if something like this could be (mis)used to break captcha - by feeding in the disabled-friendly audio captcha and passing the results back to the captcha server. As voice recognition becomes more sophisticated I think captchas are going to have to evolve to kjeep up as well.
- yjftsjthsd-h 9y agoPretty sure that's been done with existing voice to text systems. So, yes :)
- candiodari 9y agoCaptchas are utterly beat, and more so, it's not the technology or difficulty : they are a lost cause. Pretty much any problem you might present in a captcha, machine learning performs better than humans. So today, failure to captcha is actually an indication that the other end is human.
- StavrosK 9y agoThis is very timely, as today I was thinking of caving in and getting an Echo Dot so I can control my smart home devices by voice. I would love an open-hardware microphone array that I could use with a Pi or something similar, to write my own Alexa. Not only would I love this, I would store all my commands and send them to Mozilla to help with their speech recognition models. I don't want to be the guy who wishes someone else would do all the work and then give it away for free, so I'll do what I can to help (which is probably limited to writing a bunch of code and documentation on how people can set this up more easily). Congratulations and thanks to Mozilla for this.
- bronco21016 9y agoNot affiliated in any way but have you seen the MATRIX Creator or MATRIX Voice? Both contain microphone arrays that can interface with a Raspberry Pi. I’d love to see one of these meshed together with this new Mozilla voice project for an open source Echo or Google Home. The only missing piece it seems at this point is NLP and all of the glue that converts commands to API calls.
- StavrosK 9y agoOh I hadn't, this is fantastic (and it integrates with an ESP32), thanks! I wonder if it includes software to do some live DSP to reduce noise... The fact that it can just connect to the Raspberry Pi's GPIOs and provide great sound is ideal, though. I'm very glad someone has made this, I wish I had known about it before so I could back it.
- veli_joza 9y agoThere are already some open source personal assistants, like Mycroft and Jasper. Integrating Mozilla's DeepSpeech into one of them would be fantastic.