12 ms·
Facebook open-sources a speech-recognition system and a machine learning library
- sfilargi 8y agoInteresting, why does FB need a speech recognition system?
- leesec 8y agoBecause of their many products that support voice features? Messenger, whatsapp, etc...
- sfilargi 8y agoWhy would you need voice recognition for them? What is the use-case?
- qeternity 8y agoSpeech-to-text i.e. dictation? One of the most common use-cases.
- deleted 8y ago[deleted]
- hnauz 8y agoThat's handled by the system on both Android and iOS. I remain unconvinced.
- pavlov 8y agoFacebook delivers software on many other platforms than just iOS and Android. There’s also the new Portal hardware.
- baq 8y agoyeah well facebook wants to know what you said and not what android told it you said.
- computerex 8y agoThe built in offline Android speech recognizer is really bad. Giants like Google and Facebook are blessed with data, and so they can train state of the art speech recognition models (much much better than what you get out of the built in Android recognizer) and then provide speech recognition as a service. They can control the recognition because it happens on their servers and is independent of Android or any other OS. And so FB for instance can send some voice data to their servers and get a text output. And then FB can use text sentiment analysis to get further context about the message. Sadly, most people don't have the speech data to train their own recognizers on large vocabulary systems, and that's even harder for languages that are not English. With exception of Google/Amazon/FB/Microsoft/Baidu/etc other people have to use the API's offered by the above companies to do high fidelity recognition. Which sucks because there is a cost to each recognition. You have to pay someone else to do it. Whereas FB/Amazon/MS/Baidu/etc can do high fidelity recognition offline on large vocabulary and offer it as a service. THIS is why FB wants to make speech recognition systems.
- rhizome 8y agoThe built in offline Android speech recognizer is really bad. Giants like Google and Facebook are blessed with data, and so they can train state of the art speech recognition models (much much better than what you get out of the built in Android recognizer) and then provide speech recognition as a service. They can control the recognition because it happens on their servers and is independent of Android or any other OS. Is the implication that offline Android recognition does not train on the owner's voice at all? I imagine a lot of phones these days are at least as powerful as the Pentium 200s used to train (successfully!) Dragon Dictate et al 20+ years ago.
- computerex 8y agoWell let's forget the offline android recognizer. That's the one that's built in and doesn't go to Google via the internet to get a better accuracy transcription. It's fairly good for what it is but doesn't come close to the accuracy when you go to the google recognition servers via their API's. That's because the models they offer via the recognition services are much larger, robust and better than what you get straight out of Android. These services offered by companies such as Google do not adapt the acoustic model to individual speakers and are therefore known to be speaker independent. Secondly, when I say "train", it is in a totally different context than how you seem to be using the term. You are using it in the context of adapting an acoustic model to a individual speaker to improve the performance. I am talking about building the initial model. Typical RNN or even convolution based algorithms require a lot of time and processing power to train. What's even harder to get than the processing power though is of course, data to train off of.
- donjoe 8y agoAdvertisement based on keywords.
- srbijadotokija 8y agothis was never confirmed afaik, but it looks like they are doing just that
- djohnston 8y agodata input that doesn't involve keyboards..
- underwater 8y agoI think Portal has voice control, but is currently using Alexa for most functionality.
- rurban 8y agoTheir mobile app most likely records everything being said and transmits it encrypted to their servers, so that they can improve ad targeting.
- pavlov 8y agoThat’s a nonsense conspiracy theory. Thousands of people have internal access to the full source code for the mobile app. “Ah, but there’s an undetectable binary blob that gets linked in and called without being detected by anyone working on the code,” you say. In that case consider the battery life impact. The power consumption of the Facebook app compares favorably to its social media peers. Is everybody else also recording, encoding and encrypting all the time?
- sfilargi 8y agoI am not saying that the grand-parent is on to something, but I would be careful when calling anything done by them as "nonsense conspiracy theory". They have proved these "conspiracy theories" to be true quite a few times.
- traek 8y ago> They have proved these "conspiracy theories" to be true quite a few times. Such as? Facebook has explicitly denied spying on people's conversations. I can't think of any situations where they have flat out lied about something like that, so I'm curious to hear what conspiracy theories they have proven true.
- cwkoss 8y agoThey lied to congress about selling access to users' private messages.
- pavlov 8y agoYou mean this week’s news that Spotify and Netflix had experimental messaging client integrations years ago? It’s a complete mischaracterisation that Facebook shared private messages with Spotify. By the same logic, you could say Google is sharing your emails with Apple when you access Gmail using the iOS Mail client. Spotify was offering an integrated client to FB’s chat service. This UI integration was a market failure and was discontinued years ago. Of all the things wrong with Facebook, this wasn’t worth the noise.
- karmasimida 8y agoWhy not? Facebook as research arm dedicated to play Go/Starcraft also, what do you think their reason is for doing that? They can use a speech recognition system to transcribe videos, just like what Youtube is doing to improve ad targeting and recommendation. Why is this hard to understand?
- sfilargi 8y agoNo need to be aggressive, as I am not being cynical. I am genuinely asking, because I couldn't see any immediate need for it in their main product.
- dnautics 8y agoBut also remember that accessibility for visually impaired folk is often. A regulatory imperative in those product lines.
- danso 8y agoFacebook has its own Alexa/Echo like offering. Not sure if Messenger also has voice functionality.
- moneil971 8y agoIt does, you can do video and voice calling from the Messenger app
- nurettin 8y agothat would be a bit creepy and opportunistic if fb did that. Yt videos are by and large uploaded for profit. Fb videos are uploaded for storing birthdays and family vacations.
- dqpb 8y agoCould be used for indexing videos for search
- bad_good_guy 8y agoPotential Oculus control option?
- matz1 8y agospeech recognition system is becoming fundamental for all kind of software these days. speech to text input, speech navigation, etc
- 52358 8y agoisn't Portal supposed to have it's own voice assistant, Aloha or whatnot?
- jka 8y agoSending voice messages is a pretty common form of user-to-user messaging in Asia - i.e. record a brief audio snippet instead of tapping out a sentence on a phone keyboard
- amelius 8y agoInterpret the spoken text in posted videos? E.g. for automatic subtitling?
- xpaulbettsx 8y agohttps://portal.facebook.com https://portal.facebook.com for one
- ArtDev 8y agoPerhaps to listen in on your voice conversations and serve you ads so relevant its creepy? From a software standpoint, this has never been proven. However, its super weird when it happens to you. For example, I traveled to meet a coworker who was playing a mobile game I had never seen before and we talked about it. I never Googled it or anything like that. Hours later I checked Instagram and the first ad was for the same mobile game. Coincidence? Perhaps the game was simply advertised more in his city than my own? Perhaps our phones being near each other prompted a "friend request suggestion" and then a took that to another level with installed apps? Or just a coincidence and I am thinking too much about it. lol.
- Spooky23 8y agoTotally not because they listen to anyone to profile users or target ads.
- gaius 8y agoSo their app can eavesdrop on your conversations to target more ads. Facebook does nothing without an ulterior motive.
- IshKebab 8y agoThis is excellent. Modern free speech recognition software is hard to come by. Everything except Kaldi has laughable error rates, and Kaldi is a huge pain to set up. Will be interesting to see what people can do with this and the available data sets.
- faitswulff 8y ago> available data sets Reminder that Mozilla's Common Voice project accepts voice donations! https://voice.mozilla.org/ https://voice.mozilla.org/
- darkpuma 8y agoGiven the value of open source training data (or scarcity of it), has anybody attempted to use LibriVox for training? https://en.wikipedia.org/wiki/LibriVox https://en.wikipedia.org/wiki/LibriVox https://librivox.org/ https://librivox.org/ The recordings are public domain audio books of public domain books, so the licensing should be fine. The audio isn't annotated, but given the value involved I think it would be worth attempting to use forced alignment to annotate the recordings with their public domain source texts. Forced alignment using the sort of speech recognizer you're trying to train in the first place may be a bit "chicken and the egg", but from some experiments I've run myself existing open source speech recognizers can do it reasonably well. Humans could manually tune up the alignment to improve the quality if necessary. As for motivating people to actually do that mundane work... well these are audio books so maybe the work isn't so mundane after all! The LibriVox recording of Tom Sawyer (read by John Greenman: https://librivox.org/tom-sawyer-by-mark-twain/ https://librivox.org/tom-sawyer-by-mark-twain/) is pretty great and has been listened to by millions of people. If somebody created a "read along" web app that showed you the text of the book from Project Gutenberg getting highlighted as the audiobook from LibriVox was played, users who have an interest in reading/hearing the book could have their attention held by Mark Twain and with the right UI provide fine tuning for the forced alignment at the same time.
- lingz 8y agoLibrivox is commonly used as a training corpus however it's main weakness is that reading speech differs quite a lot from conversational speech.
- maddyboo 8y agoLink to the GitHub repo, for the lazy: https://github.com/facebookresearch/wav2letter/ https://github.com/facebookresearch/wav2letter/
- motivated_gear 8y agoAs someone behind a corporate firewall on a Friday, thank you
- vbuwivbiu 8y agoIs it ethical to use their code ?
- darkpuma 8y agoI think you should consider it ethical to do ethical things using a tool created by unethical people. Consider for instance Fritz Haber, the so called "father of chemical warfare" who contributed to the development of the Haber–Bosch process for artificial nitrogen fixation, that facilitated the mass production of explosives by Germany during the Great War, but also provides as much as 50% of the nitrogen in the body of an average human today due to it's role in the production of fertilizer. Whether it's ethical to contribute back to the project knowing that the unethical creator might derive unethical utility from your contributions, is perhaps slightly more complicated. However the same could be said of any open source project, you could create something new wholly from scratch and if you release it publicly, somebody else could use it for something unethical. I commend your consideration of ethical concerns, which I think is lacking in the tech industry today. But in this particular case I don't believe there is too much cause for concern.
- adjkant 8y agoI think your points generally stand, but I think Facebook open source raises some specific issues at least worth consideration: If your use of it contributes to its popularity, perhaps becoming the standard of X area, does that give Facebook the company more power and possibly enable other unethical actions? I think it's probably not as much of a worry given the narrowness of the area, but I do think this is something to consider when it comes to React for example.
- ProAm 8y agoNo
- srbijadotokija 8y agowhy not ?
- 8y ago
- Yoric 8y agoDoes anyone know ho this compares with Mozilla's DeepSpeech? https://github.com/mozilla/DeepSpeech https://github.com/mozilla/DeepSpeech
- yorwba 8y agoFrom https://arxiv.org/abs/1812.06864 https://arxiv.org/abs/1812.06864 "On Librispeech, we report state-of-the-art performance among end-to-end models, including Deep Speech 2 trained with 12 times more acoustic data and significantly more linguistic data." Specifically they claim word error rates that are 1 to 2 percentage points lower, 3.44% on "clean" and 11.24% on "other".
- StudentStuff 8y agoIt'll be interesting to see if anyone can reproduce their results, thus far its been troublesome: https://github.com/facebookresearch/wav2letter/issues/88 https://github.com/facebookresearch/wav2letter/issues/88
- m0zg 8y agoProbably positively. I brought up DeepSpeech in a docker container the other day, it couldn't understand me at all, using Mozilla's pre-trained model. _Some_ words were right, but the output did not make sense.
- StudentStuff 8y agoDid you use one of the examples (eg the Python GUI or CLI client)? WebRTCVad (which is used in most of those) made mincemeat of my audio files, versus feeding them into DeepSpeech got me usable results. Just need VAD for files nearing or over 1 minute in length (unless you have 4+GB of RAM to spare).
- dlojudice 8y agoI wonder how all these open-source releases happening this month [1][2][3] are related to improve team morale internally... [1] https://github.com/facebookresearch/DeepFocus https://github.com/facebookresearch/DeepFocus [2] https://github.com/facebookresearch/nevergrad https://github.com/facebookresearch/nevergrad [3] https://github.com/facebookresearch/pytext https://github.com/facebookresearch/pytext
- rock_hard 8y agoProbably just a end of year push to get things out
- benbayard 8y agoOr they're in a production code freeze so engineers and spend more time pushing these final projects to finally get them released.
- xpaulbettsx 8y agoIt's Perf Review time, gotta get that Impact without affecting production
- honkycat 8y agoI like to take recordings of my thoughts on my cellphone similar to Dale Cooper. Unfortunately, I do not have a Diane on the other end to translate my thoughts to text, I have to do that myself. I've been looking into things like mozilla/deepspeech and other open source libraries for automatically converting my messages to text. I'll have to take a look at this project as well!
- halfjew22 8y agoHey me too! A while ago I was looking st trying to figure out how to hack something like this together myself when I came across what is now one of my top thre apps: Otter. Sounds like a shill and I don’t really care. I’m a premium member with 6,000 minutes of transcript time per month (and sometimes I’ve used almost all of it) and I couldn’t be happier. You can export everything, support and head of product are kind and responsive, and you can click in the transcription anywhere and it will play the audio at that point. Exactly what I need. My main complain is that it’s geared towards corporate environments for conferences, meetings, etc, and so the grouping isn’t exactly what I like, but I was my text editor to keep the links more to my liking. Being able to search by word hundreds of hours of my thoughts is a fantastically empowering experience and I hope you find the same. Let me know what you think! Shoot me an email if you want to chat about it ever. If you can't tell I'm a pretty big fan.
- ocdtrekkie 8y agoAlways excited to see more speech recognition releases. Way too many solutions are "just point your microphone's feed at our cloud service", and a lot of those that aren't have somewhat lagged behind.
- EGreg 8y agoIs it better than openEars? I want something that will work onboard without sending to a server.
- mark_l_watson 8y agoVery nice. In my work, as much as I love working with RNNs, convolutional models are faster to train and use. Years ago, FB blew away text modeling speed with fasttext and it is good see these projects made publicly available also. As much as I sometimes criticize FB and Google over privacy issues, they also do a lot of good by realeasing open source systems. Most of my work involves using TensorFlow and Keras, and a little over two years ago I replaced a convolutional text classification model with fasttext, with good results.
- stochastic_monk 8y agoI haven’t done any significant projects using neural networks for sequence modeling or analysis, but when I do, I plan to start with a temporal convolutional network, based on [0]. They argue that RNNs being standard is likely an artefact of the history of the field, while they get superior performance from TCNs. [0] https://github.com/locuslab/TCN https://github.com/locuslab/TCN
- yablak 8y agoThe truth is that the best architecture depends on the problem, and you get a lot out of "graduate student gradient descent", i.e., hyperparameter search and fiddling with the architecture fine grained details. You'll want to experiment with rnn, tcn, and transformer (t2t) models to find the best one for your problem. Disclaimer, I wrote the TF RNN api.
- alexcnwy 8y agoTotally agree - it's hard to say which sequence architecture works best for your problem (are there long term dependencies, do they exist at multiple levels of scale, etc) and dataset size. Convolutions are cheap to compute and can be more efficient on smaller data but it's also possible that a CNN outperforms when you have a small dataset and an RNN when you have a larger dataset.
- Zephyr314 8y agoFor graduate students out there that would rather be doing research than "graduate student gradient descent" (or, high dimensional, non-convex optimization in your head), SigOpt (YC W15) is a SaaS Optimization platform that is completely free for academic research [0]. Hundreds of researchers around the world use it for their projects. Disclaimer: I co-founded SigOpt and wasted way too much of my PhD on "graduate student gradient descent" [0]: https://sigopt.com/solution/for-academia https://sigopt.com/solution/for-academia
- nixostap 8y agohi from hacker.
- universe_lol 8y agospeech system ?? why
- ZainRiz 8y agoDid they release a model to go with this (so that the average dev can actually use this in their app) or is this just a tool for researchers?
- dindresto 8y agoI was wondering the same. Also, it's weird the issues on the repository are all older than the first commit.
- ISV_Damocles 8y agoThey probably squashed the history before release because they only got legal's approval on the current state of the repo and not all prior history (which may or may not have had hacks referencing internal systems at some time).
- nh2 8y agohttps://github.com/facebookresearch/wav2letter/issues/88 https://github.com/facebookresearch/wav2letter/issues/88 mentions a "pre-trained model named librispeech-glu-highdropout.bin", but I couldn't find it anywhere. https://github.com/facebookresearch/wav2letter/issues/93 https://github.com/facebookresearch/wav2letter/issues/93 also mentions a pre-trained model but without any reference which one or where to find it. Googling "librispeech-glu-highdropout.bin" still shows the text "luajit ~/wav2letter/test.lua ~/librispeech-glu-highdropout.bin -progress -show -test dev-clean -save -datadir ~/librispeech-proc/ -dictdir ~/librispeech-proc/ -gfsai ..." for https://github.com/facebookresearch/wav2letter/blob/master/README.md https://github.com/facebookresearch/wav2letter/blob/master/R..., but clicking it, it's gone. But the Google Cache still has the result, including 3 pre-trained models: wget https://s3.amazonaws.com/wav2letter/models/librispeech-glu-highdropout.bin wget https://s3.amazonaws.com/wav2letter/models/librispeech-glu-highdropout-cpu.bin wget https://s3.amazonaws.com/wav2letter/models/librispeech-glu-lowdropout.bin The cache also includes a much more detailed README on how to use the software.
- taf2 8y agoThank you! I found that some of the forks also have the more detailed README files for example: https://github.com/19ai/wav2letter https://github.com/19ai/wav2letter
- ngngngng 8y ago> developing in modern C++ is not much slower than in a scripting language. Is this accurate? I haven't written C++ since freshmen year of college and it was very cumbersome then.
- computerex 8y agoNot in my opinion. It's still an unmanaged language at the end of the day and has to be compiled. Secondly, it's still a very technical language.
- AlexCoventry 8y agoIt's improved a lot in terms of expressiveness, but it's still plagued by memory errors and obscure error messages. Who said that, anyway? I don't see it in the text linked by the OP link.
- ngngngng 8y agoSorry I should have mentioned, it's part of the research paper announcing the release. It's on the right column of the first page. https://arxiv.org/pdf/1812.07625.pdf https://arxiv.org/pdf/1812.07625.pdf
- AlexCoventry 8y agoThanks.
- jimmychangas 8y agoPeople are quick to criticize Facebook here on HN, but this release is awesome. I believe open source speech recognition is still lacking, and any contribution is very welcome. CMU Sphinx and Kaldi are great, but it feels like the most recent advances in the field are still hidden behind paid services.
- oliwarner 8y agoWhat worries me —very much in-keeping with existing criticisms— is what Facebook wants to do with this tech. Voice is not a core business for them. Yet. They are demonstrably very good at using technology and their omnipresence against their users (to monetize them without their knowledge).
- pbalau 8y agoYou would be amazed about how little comments does an article about FB doing "this tech thing" attracts, vs a generic FB is bad. There are terribly few people that can comment on a tech subject, but everyone and their dog has an opinion about how FB destroys humanity
- stevenjohns 8y agoSo what you're saying is that there are more people who can associate with social ethics vs. specialised areas of programming? And this is a revelation? Perhaps it might help you if you look at it from the perspective that what Facebook has open-sourced here isn't affecting a billion people's privacy and it's not being willfully used as a tool of intimidation and propaganda by governments. That's just a couple thoughts on why other posts might attract more comments.
- mempko 8y agoPeople confuse Facebook the org with Facebook the workers. Don't confuse the great work it's workers do with the what management decides.
- chrisbrandow 8y ago
- g_delgado14 8y agoManagement: "Let's keep open sourcing all of the things so that the informed community will overlook our transgressions!"
- ToFab123 8y agoSo how good is that speech recognition system in, let's say, to listen in on a phone conversation and use that information in, let's say, the Facebook news feed. You know the precise thing many, me including, suspect Facebook is actively doing. And if you say they don't do any things like that, why do they need a speech recognition system in the first given that Facebook is a text based system.
- varenc 8y agoThey need speech recognition for their Alexa/Google Home like hardware product: https://portal.facebook.com https://portal.facebook.com (Notice how much they emphasize privacy and security in their marketing...)
- ToFab123 8y agoOh my... Some people are actually going to let Facebook listen in on everything. Wow. Not in a million years in my house.
- akhilcacharya 8y agoFor the record, I do think it's telling that they currently use Amazons's Alexa Voice Service for this (or at least most of it after the hotword recognition) instead of building on Project M.
- varenc 8y agoAlexa is optional and Facebook has their own voice recognition for Portal features, You can use the wake word “Hey Portal” for a limited set of portal commands mostly around calling and messaging, or “Alexa” for all the Alexa capabilities. When using Alexa functionality Facebook isn’t suppose to “listen” at all. (Though not sure if these queries still flow through their servers or not?) https://portal.facebook.com/help/2149102838698668/ https://portal.facebook.com/help/2149102838698668/
- modwilliam 8y agoSeems that they have an automatic captioning feature for videos/audio on the site
- SpaceManNabs 8y agoFrom what I understand, this is less of a machine learning advancement and more of an engineering advancement? Trying to see if any of the bleeding edge stuff has been implemented. Still waiting for SeLUs to be standard!
- m0zg 8y agoWhere are the pre-trained models? It's worthless without them. Edit: NM, someone hunted down the AWS links wget https://s3.amazonaws.com/wav2letter/models/librispeech-glu-highdropout.bin wget https://s3.amazonaws.com/wav2letter/models/librispeech-glu-highdropout-cpu.bin wget https://s3.amazonaws.com/wav2letter/models/librispeech-glu-lowdropout.bin
- freyir 8y agoAny idea why they developed Flashlight for this, instead of using PyTorch?
- wpdev_63 8y agoIt would be interesting to see this benchmarked against mozilla's deepspeech.
- buboard 8y agoHow good is this library? Is this good and fast enough to see wider adoption in embedded devices or phones? Would be awesome to be able to voice-enable apps without the need for a cloud provider. How would this compare with a C++ pytorch-based approach?
- woodson 8y agoIt has dependencies on CUDA and/or Intel MKL, so not really suitable as-is for embedded/phones.
- suyash 8y agoIs there a demo of a working example app using this library?
- nshm 8y agoI like the way they politely skipped Kaldi WER on test-clean (4.31) in results table. Their best WER (4.91) will not look so attractive.
- woodson 8y agoEven more so if they compared it to Kaldi TDNN-LSTM with RNNLM lattice rescoring (test-clean 3.22%, apparently): https://github.com/kaldi-asr/kaldi/blob/master/egs/librispeech/s5/local/chain/tuning/run_tdnn_lstm_1b.sh https://github.com/kaldi-asr/kaldi/blob/master/egs/librispee...
- Nimitz14 8y agoAnd kaldi gets <8% on test-other other while this gets over 11%!