29 ms·

Speech Recognition Is Not Solved

435 points by allenleein 9y ago

theresistor 9y ago
Speech recognition for multi-lingual speakers is another pain point. I live with a native French speaker, so my conversations naturally include a lot of French proper names, as well sometimes switching languages mid conversation or even mid sentence. Lots of recognition engines can handle English and French, but treat them as mutually exclusive. It frustrates me to no end when I know that Siri can recognize a French proper name just fine if I switch it modally into French, but will botch it horribly in English.
- rav 9y ago
  As a multi-lingual speaker I've even had problems with speech synthesis. For instance, Google Maps insists on narrating driving directions in the system language, which is set to English; this makes the narrated local non-English place names incomprehensible. If I succumb and change the system language I can understand the narrated place names, but in return I have to put up with the poor speech synthesis of non-English languages.
  - theresistor 9y ago
    I was just in France last week suffering through the same problem. I’m American, so I want to keep my system set to English both due to familiarity of the interface and because I want data in familiar units (miles, degrees Fahrenheit), but it kills me when every street and place name is absolutely butchered.
    - theresistor 9y ago
      An interesting observation I made last week was that the poor synthesis of French street names in English mode was worse than American-mispronouncing-French level. I wonder if that exposes some level of mismatch between how the synthesis engine models language and how humans do.
      - ghaff 9y ago
        In the case of French names it probably depends on if you have any exposure to French pronunciation rules at all. I have a pretty crappy level of high school French but I can usually manage not to butcher pronunciation. I was with a friend in Montreal though and she had absolutely no idea how to pronounce many things.
pjmlp 9y ago
As Portuguese native speaker forced to use foreign languages to talk to devices, it is not solved at all.
- reaperducer 9y ago
  Even English doesn't work the way it should. Remember Apple had to rush out a patch shortly after Siri was released because it couldn't understand Australians. Siri still doesn't understand the majority of my Minnesota relatives.
cscurmudgeon 9y ago
Just the other day I was arguing (pleasantly) with someone here on HN saying that AI has solved speech rec (among other tasks). https://news.ycombinator.com/item?id=15429287 https://news.ycombinator.com/item?id=15429287 Wish this article was written a few days earlier. An analogue of this article exists for most other domains claimed to have been solved.
- throwmenow_0140 9y ago
  Sorry to bother you, but the translation of your comment is pretty good (English -> Spanish -> French -> English): No, it's not learning new classes of objects from a single image or a few images is very difficult. See http://www.sciencemag.org/content/350/6266/1332.short The machine translation is a joke. Put comments on this page by translating Google into another language and go back to English and see what you get. I did a little part of you. Just human level for just a simple little prayer. > But even if you do not make this assumption, identifying the object involves spitting the distance from the performance of the human level. source: https://news.ycombinator.com/item?id=15429862 https://news.ycombinator.com/item?id=15429862 Not to debunk your claims, just interesting to see how good translation works (even if it's not human-level, I can understand what you say).
  - cscurmudgeon 9y ago
    I am not sure that is a good translation because it completely misses a common idiom: "within spitting distance" "spitting the distance from the performance of the human level" If this were spoken out loud, it might get mistaken for "splitting the distance" which seems to have a different meaning (half as good as humans). But the point is subtle differences matter a lot in human communication. But, I do agree with your point that in a lot of cases even without human-level performance one can get things done.
    - throwmenow_0140 9y ago
      After reading the translated version again, I'm not sure I can support my statement anymore. I think a big part of my impression was based on the fact the I read the original comment and knew what you were talking about. > I did a little part of you. Just human level for just a simple little prayer. It's just bad. It sounds like it's based on a n-gram model - about something religious, strangely enough.
xigency 9y ago
A big thing for me is the difference between offline and online speech recognition. This really ties in with the last point, advancements need to be efficient in order to be used. I'm not sure how much work it would take to scale down Apple's voice recognition to run on the device or if it is feasible with their model, but currently it can take 5-10 seconds longer to get an answer from Siri during peak times.
jtth 9y ago
Also, water is wet and rocks are hard.
- dang 9y ago
  Would you please stop posting unsubstantive and/or snarky comments? https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
johnramsden 9y ago
What I would like to see you would be an open source speech recognition system that is easy to use, and works. There is really no match for the proprietary solutions in the open source world right now, and that is disappointing. Not to say that there aren't open source speech recognition systems, they just aren't completely usable in the way proprietary solutions are. A lot of research goes into open source speech recognition, where they are lacking is in the datasets and user experience. Hopefully, Mozilla's Common Voice project https://voice.mozilla.org/ https://voice.mozilla.org/ will be successful in producing an open dataset that everyone has access to and will spur on innovation.
- LandoCalrissian 9y ago
  I recently have been trying out open source solutions for voice recognition for a personal project and you are very correct that it lags very far behind proprietary solutions. Pocketsphinx is still very limited and Kaldi takes quite a bit to setup in a usable fashion. There were a few other options I looked at that I can't think of from the top of my head but were all in similar condition. As the article says latency is still a problem and it's a huge problem in current open source solutions, some stuff I was testing was easily 5 seconds. I know that can be improved with configuration, but when dealing with libraries of 10 words or so, that's pretty bad. I feel like anyone who is seriously interested in this space has been scooped up by all the big companies and the open source solutions have really seemed to linger because of it. It's one of the first areas I've seen where open source alternatives are really behind the proprietary solutions. Kind of bummed me out.
  - dharma1 9y ago
    Kaldi is the best, there was just Tensorflow integration added which will hopefully speed up development (though I haven't seen any pretrained models for that yet). Here's a blog post - http://www.googblogs.com/kaldi-now-offers-tensorflow-integration/ http://www.googblogs.com/kaldi-now-offers-tensorflow-integra... The easiest way to deploy Kaldi is this - https://github.com/alumae/kaldi-gstreamer-server https://github.com/alumae/kaldi-gstreamer-server (or a docker image of the that)
salqadri 9y ago
I remember using the Merlin Microsoft Agent to create speech-based "apps" in 2001, including text-to-speech as well as speech recognition. I made a SAT preparation that would test me by presenting a word and I had to say the meaning. I also made a Spanish learning app using Merlin, where it would read out the spanish word (in spanish!) and I would have to say the english one. It worked really well. Its been 16 years since then, and I have to say that I had expected this area to have been completely "solved" by now.
canjobear 9y ago
It's always been a puzzle to me that published WER is so low, and yet when I use dictation (which is a lot--I use it for almost all text messages) I always have to make numerous corrections. This sentence is crucial and suggests a way to understand how WER can be apparently lower than human levels, yet ASR is still obviously imperfect: > When comparing models to humans, it’s important to check the nature of the mistakes and not just look at the WER as a conclusive number. In my own experience, human transcribers tend to make fewer and less drastic semantic errors than speech recognizers. This suggests that humans make mistakes specifically on semantically unimportant words, while computers make mistakes more uniformly. That is, humans are able to allocate resources to correct word identification for the important words, with less resources going to the less important ones. So maybe the way to improve speech recognition is not to focus on WER, but on WER weighted by word importance, or to train speech recognition systems end-to-end with some end goal language task so that the DNN or whatever learns to recognize the important words for the task.
- IanCal 9y ago
  This is, to me, one of the major problems with many algorithmic solutions to problems. An x% increase does in precision, F measure or any other score does in no way mean that the results are better. I've repeatedly seen improvements to traditional measures that make the subjective result worse. It's incredibly hard to measure and solve (if anyone has good ideas please let me know). I check a lot of sample data manually when we make changes, doing that (with targeting at important cases) is really the only way I think to do things.
  - canjobear 9y ago
    If you've got a dictation system on a phone, wouldn't a very good metric be the corrections people make after dictating? I guess a problem would be if people become so used to errors that they send messages without corrections. I have some friends who do this: they send garbled messages that I have to read out loud to understand. But there will always be a subset of people who want to get it right.
- adrianbg 9y ago
saosebastiao 9y ago
I would argue that it can't be solved without severe privacy implications. Apple's talk-to-text, for example, should know my son's name by now, but it doesn't. And while I'm mildly frustrated at the fact that I have to go back and edit text messages on a regular basis, I'm pretty glad that Apple doesn't know my son's name. I'd hate for a company like Facebook to have access to all of the proper nouns in my everyday life.
- macintux 9y ago
  You can teach your iPhone to know your son's name: http://appleinsider.com/articles/17/10/19/how-to-teach-siri-to-pronounce-names-correctly-on-your-iphone http://appleinsider.com/articles/17/10/19/how-to-teach-siri-... You'd need to create a contact if he doesn't already have one.
twothamendment 9y ago
It is one thing to hear and correctly identify the words. It is another to understand the meaning. I've been thinking about this because my son has Auditory Processing Disorder, APD. He can hear great, even a whisper across the house. The trouble is the words don't always make sense. He can tell me the words he heard and they are correct, but assigning a meaning to them doesn't work like it does for most people. After playing him a bunch of audio with missing words, the lady who tested him was blown away by how smart he is - explaining that he is quickly filling in almost every missing word based by guessing from all thing possible things it could be, narrowing it down on context and getting it right. I guess that is a normal, all day long task for him. Since then I've thought about voice recognition differently. The AI to understand the context or fill in the blanks is what will make or break it.
- deleted 9y ago
  [deleted]
- noir_lord 9y ago
  I'd never heard of APD. Interesting, all my life I've struggled to follow a conversation in a crowded environment, so much so that I actively avoid background noise with words in, I work with silicon ear plugs in or headphones and music with no lyrics. Looking at NHS symptoms they describe me as a child, didn't learn to read until I was 8. I nearly ended up in the remedial track but for a single awesome teacher who spotted that it wasn't that I was incapable but that I was struggling to understand in a noisy classroom, she gave up her lunch time for 6mths and taught me one on one by the end of that year I could read much better.
  - fillskills 9y ago
    Wow, I wish there was an easy way to pay back star teachers or mentors.
    - soperj 9y ago
      Is it really that they were a star? Or just that they cared.
nblavoie 9y ago
[Slightly off-topic] And what about people with minor to major stuttering troubles like I do ? It is so frustrating to tell Alexa, Siri or Google (even Cortana) what I want because the timer delay expiration is so short.
ballenf 9y ago
> Latency: With latency, I mean the time from when the user is done speaking to when the transcription is complete. ... While this may sound extreme, remember that producing the transcript is usually the first step in a series of expensive computations. For many applications, making a transcription seems like an unnecessary step and source of errors. Skipping transcription when the user doesn't need it (most cases where I use it) would seem like a way to get some gain, but perhaps at the reduction of debuggability. > For example in voice search the actual web-scale search has to be done after the speech recognition. That's an area where literal exact transcription is usually required. But even then, Siri/Cortana/Alexa might be better off trying to figure out the meaning of what someone's asking rather than figuring out the exact words spoken in order to return the best results. Most people are quite bad at formulating good internet searches without a lot of trial and error. Let Google listen to a person talk about the problem they have or issue and then formulate the best results for that instead of forcing us to come up with the exact right phrasing to get appropriate results. It would help tremendously with the synonym and homophone issues that are so annoying now.
- johnmcd3 9y ago
  > For many applications, making a transcription seems like an unnecessary step and source of errors. Skipping transcription when the user doesn't need it (most cases where I use it) would seem like a way to get some gain, but perhaps at the reduction of debuggability. Agree. What's really needed is research into (and development supporting) how to combine the expertise from a speech recognition layer with the next layer in a machine learning process. That higher layer contains the domain specific knowledge needed for the problem at hand, and still leverage a speech layer focused on a broad speech data set and speech-specific learning (from Google, Microsoft, the community, etc.) Today, how richly can information be shared? I see with Google's speech API you can only share a very finite list of domain-specific expected vocabulary. Why not have speech tools at least output sets of possible translations with associated probabilities? Do any of the top tools allow this? Then you could at least train your next level models with the knowledge of where ambiguity most exists, and what a couple of options might have been for certain words or phrases...
Radim 9y ago
Just the other day, I participated in a discussion about how "language identification" is a solved problem -- in fact, hasn't it been solved for a decade? As anyone who's had to use langid in practice will testify, it's solved only as long as: A) you want to identify 1 out of N languages (reality: a text can be in any language, outside your predefined set) B) you assume the text is in exactly one language (reality: can be 0, can be multiple, as is common with globalized English phrases) C) you don't need a measure of confidence (most algos give an all-or-nothing confidence score [0]) D) the text isn't too short (twitter), too noisy (repeated sections ala web pages, boilerplate), too regional/dialect, etc. In other words, not solved at all. In my experience, the same is true for any other ML task, once you want to use it in practice (as opposed to "write an article about"). The amount of work to get something actually working robustly is still non-trivial. In some respects, things have gotten worse over the past years due to a focus on cranking up the number of model parameters, at the expense of a decent error analysis and model interpretability. [0] https://twitter.com/RadimRehurek/status/872280794152054784 https://twitter.com/RadimRehurek/status/872280794152054784
- notahacker 9y ago
  I'm reminded of how often I see Twitter offer to translate English-language tweets containing a proper noun or two from absurdly unconnected languages. And that's with text containing mostly common and distinctive English words.
  - abecedarius 9y ago
    I'm pretty sure Twitter's langid uses character n-grams. You'll see a tweet that's plain English that happens to match n-gram statistics unusually well with some other language, which pushes the likelihood score to just above English. (I checked this by running an example or two through my own langid code.) It shouldn't be hard to improve on by treating the score on a tweet as bayesian evidence to combine with a prior from preceding tweets.
- jdmichal 9y ago
  In other words, we have some systems that are great at processing spherical cows in a vacuum...
joemag 9y ago
Mistakes in understanding speech are common even among humans. My wife and I have to repeat and clarify ourselves fairly frequently in our day to day conversations. She even jokes at times that I must check my hearing, because I often mishear what she said, while she thought she was being perfectly clear. I think where computers fall short is in two areas: 1) The rate of errors hasn’t hit the inflection point of being comparable to day-to-day intra-human interaction, and 2) There is no good mechanism for detecting and correcting errors. At least, none that I’ve seen. That second one is important. When I hear my wife ask me “Please, hand me a tractor”, I realize that I must’ve misheard, and ask her to clarify “what?” With speech recognition, I either have to manually re-read and modify recognized text, or cancel and repeat the entire request. Both take time, and negate some of the efficiencies of using speech recognition.
- twothamendment 9y ago
  "I want some taquitos" - https://www.youtube.com/watch?v=CA-LGL9iMh4 https://www.youtube.com/watch?v=CA-LGL9iMh4
- throwmenow_0140 9y ago
  "Please, hand me a tractor" - Did she say tracker [1]? Something like Rhymezone (searching for similar sounding words) would be a solution. It would still need a human who's deciding which word to choose, but after some time it could learn which words you prefer. [1] https://www.rhymezone.com/r/rhyme.cgi?Word=tractor&typeofrhyme=sim https://www.rhymezone.com/r/rhyme.cgi?Word=tractor&typeofrhy...
- pbhjpbhj 9y ago
  One needs to know your wife to know that "hand me a tractor" is wrong. Perhaps she makes model farms as a hobby, perhaps you work in plant rental and that means "pass me the keys to one of the tractors", the possibilities are endless. You need vision, memory, profiling, etc., to even begin to properly contextualise day-to-day conversation.
  - Myrmornis 9y ago
    Hm, I think I disagree with you and GP. The answer is Bayesian inference to both of you isn't it? The prior is going to give very low weight to your domestic partner asking you to "hand them a tractor", despite the fact that it's not impossible that it's the correct words.
morinted 9y ago
I'm part of the stenographer/captioner community as a hobby and you'd be surprised at how many people think that all captions are automatically computer-generated. In reality the practical limitations of autogenerated captions are demonstrated by YouTube's caption system. It's good but even a mistake every other sentence (95%+ accuracy) can completely obfuscate the meaning behind the video.
- tyingq 9y ago
  I've seen some pretty terrible human produced captions on television shows as well.
  - morinted 9y ago
    Big time. There are various levels of caption quality and part of it is the tools used. 1. QWERTY transcriptionists working brief, 5-10 minute shifts typing as fast as they can and rotating. 2. Voice-mask reporters using Dragon with a voice mask (more commonly phonetic typos) 3. Stenographers using a steno machine, some of them were not trained to do realtime and lack the ability to edit as they go. It's all up to the individual, unfortunately. Furthermore, the budget for live captions is sometimes smaller than it should be and so the service purchased is the cheapest, not the best. I pay attention to captions now. I've watched TSN (Canada) and it seems that past 10 p.m. they put in someone who isn't fully trained or graduated from school and they most definitely shouldn't be providing captions as they are near unreadable. Good captions are priceless for the access that they offer.
aidenn0 9y ago
I think some people overestimate how good humans are at speech recognition. Unfamiliar accents and noisy environments cause havoc with many people. I had a friend who learned English in India when I was in High School, so I was used to that accent; many of my classmates in College could not understand anything our Indian T/A said freshmen year. Similarly I had friends for whom English was a second language who had lived in the US for years and were definitely fluent, but had to enable subtitles for a movie in which the characters had a strong southern accent; in general non-rhotic accents were very troublesome for them having only spoken english with midwesterners. The article mentions the Scottish accent, and I would call that the hardest accent of native English speakers for those in the US to understand.
- twothamendment 9y ago
  I had a chance to live in the middle of the "hood" in a large American city for about 6 months. I was one of the few white guys around. I spoke English, they spoke English - but it was about a week before I could understand people, especially on a crowded city bus. The first few days felt like a foreign country and it was the same country I grew up in! I wonder how well Siri and friends work with Ebonics?
  - aidenn0 9y ago
    That's a double whammy because diction is so different as well; our brain fills in the gaps in phoneme recognition by pattern matching and the differing dialect makes that much harder as well.
  - deleted 9y ago
    [deleted]
- joemag 9y ago
  Agree with this. I’m currently traveling to a country where English is not the primary language, but is still spoken universally. I am having a hell of a time understanding people - the unfsmikiar accent is wrecking havoc with my brain’s speech recognition centers. I’m currently where speech recognition software was in the 90s.
PaulHoule 9y ago
One thing I see missing is "conversational smarts". People mishear a certain fraction of what they hear and they'll ask you to clarify what you said. You could have superhuman performance at tasks like Switchboard and still have something embarrassingly bad in the field because the real task (having a conversation) is not trained for in the competitions with canned training sets.
deleted 9y ago
[deleted]
dmreedy 9y ago
Of course it's not solved. We don't even know how to define the problem. Speech is[1] a fundamental component of Language, which is a fundamental component of Intelligence[2]. This is addressed somewhat in the conversation around semantic error rate; that there is more to processing raw audio speech than the calculus of mapping signals to tokens; some understanding of semantics and context is required to differentiate between otherwise indistinguishable surface forms. I find it doubtful that there's a clean interface that separates the 'intelligent' parts of the brain from the 'language' parts of the brain from the 'speech' parts of the brain. This leakiness (or richness, really) means that you can't neatly solve any one part of this chain to the level of competence that the brain solves it. That means to 'solve' speech recognition, you have to 'solve' language, and thus 'solve' general intelligence. And to 'solve' general intelligence, you have to understand it, in a theoretical sense, which we don't. Indeed, it will likely involve solving all the other modalities of sensation as well. It's definitely the case that you need to have a model for prosody to understand speech. It is entirely possible that vision is a large factor as well, in the form of body language, lip reading, eye contact, and so on. Speech recognition is quite good for what it is. For many practical applications, especially to do with young, white, newspeaker-accented English speakers who sound the most like the people who develop it, and the data sets used to develop it, it is good enough in the 80/20 sense. But it is nowhere near solved by even the least rigorous definition of the word. ---- [1] according to the philosophy I subscribe to, at any rate [2] according to the philosophy I subscribe to, at any rate
- baddox 9y ago
  Problem definition really is the issue. Even appealing to the Turing test is less than satisfying. I’m fairly sure I’m a human, but Siri and similar will likely outperform me in certain categories like place names and popular music.
  - jdmichal 9y ago
    Don't forget that being too good can also cause you to fail the Turing test. I would not expect a human to be able to answer some questions that may be trivial to a computer. Things like, "What's the square-root of 137?" Or, "Identify this obscure song within 5 seconds of listening from a random starting point."
ccozan 9y ago
Speech Recognition, beside transcribing phonemes and match to a NN of possible words, is not solved because speech is highly integrated with the human context: who is speaking, to whom is speaking, where is the speech happening, why is the speech initiated, and so on. My kids needed 5-6 years of continous daily talking until I could say they understand a conversation almost completly. Every single word or phrase I directed at them was spoken in a certain context and had a certain role in communicating with them when from the context it wasn't clear what my intentions were. Of course, they had their fun, throwing words away and repeating endlesly some funny word or phrase or terribly spelling them. It is interesting that you as an adult need too learn their own prononciation, at some moment in time I even wrote a small dictionary. Again, speech recognition will simply stay at recognizing phonemes/words only for a very long time, until we have a true AI Assistant that walks with us, sees with us, and hears with us in the same time. Then it can apply some semantic and other context based related NN.
billfruit 9y ago
I heard even voice synthesis, ie text to speech, is not solved yet.
ChuckMcM 9y ago
Too many people equate speech to text as 'speech recognition' and it clearly isn't. What is not a solved problem is Natural Language Processing. That NLP isn't solved, is a fertile source of papers. What is 'kinda' solved is the voice CLI, that is where speech is converted to text, that text is inserted in a class command interpreter, and if recognized the command is executed, otherwise it isn't. Nobody expects to be able to type 'find me the files that changed yesterday' into a computer shell and have it do something sensible. But 'find ./ -type -f -mtime yesterday' sure we can expect that to work. Getting from the first one to the second one requires something called an ontology. I worked on a team at IBM that was building this sort of capability. The plan was that in conjunction with an excellent tool kit for disassembling a sentence into its component parts (verbs, nouns, adjectives etc) and then matching the elements recognized with an onotology associated with that action. 'find' being the verb would select the search action ontology, 'files' as an object of the verb would select the computer files object ontology, and then 'changed' and 'yesterday' would match 'modification' in that ontology and 'yesterday' would match one of the two sub children of 'modification' (time and change-actor). Then you would go back with what was the compiler equivalent of an abstract syntax tree which the 'command generator' would use to emit the necessary command actions and flags. I have no idea why they cancelled it. It was an inscrutable place to work at in many ways :-).
- gok 9y ago
  "Speech Recognition" is an accepted term of art for "speech to text" or "transcription." "Speech Understanding" is a related but different problem.
  - astrodust 9y ago
    I'd challenge you to transcribe a casual conversation without understanding. Too many words sound very similar, if not identical, and the context of the conversation dictates which word is in play. Even something simple like "They're unhappy, Ness" could be interpreted as "Their unhappiness" unless you know Ness is a person in the converation.
    - 9y ago
turbohedgehog 9y ago
As a deaf person I am using Ava [0] which uses IBM's speech to text service [1] as its backend AFAIK. I am always impressed by how it picks up on context clues to make corrections in realtime and capitalizing proper nouns (Incredible Pizza for example). However, it does not work with multiple speakers on a single microphone. [0] https://www.ava.me/ https://www.ava.me/ [1] https://www.ibm.com/watson/services/speech-to-text/ https://www.ibm.com/watson/services/speech-to-text/
ksec 9y ago
Then there is Chinese, which the same "sound" can have many different words. And we constantly make up new short form or words, a lot more so then English means recognizing these are much harder. ( Then there is Cantonese... ) Sometimes I wonder, if Apps could add vocabulary to devices dictionary, so certain terms in Gaming, Fashion, or Tech or other domain, which are rarely used in normal conversation would be recognized in speech to text as well as Auto Correct.
notadoc 9y ago
Siri reminds me of that every time I attempt to use it.
- philfrasty 9y ago
  Baffled every time that Siri has actually been shipped by Apple to be used in production and not as a pre-pre-pre-pre-alpha.
artellectual 9y ago
Do you guys thing “understanding” needs more that what we have right now in terms of AI, does understanding require intelligence that is self aware? I mean when we talk to each other, we understand in spoken words, little expressions, even when we talk over the phone. I feel like without the ability to process emotion and self-aware AI we wont really achieve “full understanding” only simulated ones.
singularity2001 9y ago
Speech Recognition (STT) used to be completely monopolized by Nuance. Slowly the gates are opening (thanks to google and DL), but I guess it is still a minefield of patents.
amorphid 9y ago
I am a native English speaker from California. It is hard to get my Google Personal Assistant to understand the difference between desert and dessert. It's also hard to request music by an artist with name similar to another artist. "Hey Google, play Mika radio" has a 50/50 chance of starting music by Meiko. The additional "problem" is that I like Meiko, too, but I feel obligated to cancel the Meiko music & re-request Mika so that Google (hopefully) learns to recognize the difference between Mika and Meiko. Maybe our spoken language will start to transform into distinctly unique sounds so we can verbally interact with computers with relative ease. When I was in the US Army, I was trained to speak in a certain manner to help my communication to be more clear. [1] I don't see a reason humans and computers can't each make reasonable compromises to make verbal communication easier. [1] https://en.wikipedia.org/wiki/Voice_procedure https://en.wikipedia.org/wiki/Voice_procedure
pkpp1233 9y ago
It's pretty good right now for English speakers. At least usable. We've tried https://www.btwtalk.com https://www.btwtalk.com for transcribing our internal calls, and while it does mess up a bit, it's good enough for later reference.
nkkollaw 9y ago
I would also like to confirm that speech recognition in language other than English is completely not solved. In Italian, for instance, any device will barely be able to recognize what you're saying, and will almost always do something completely different from what you've asked.
notlarrypage 9y ago
I think that some speech recognition systems depend too much on language model priors. This works well for routine tasks where most of the speaker's words are easy to predict. It fails when the speaker's words are specific or unusual. For example, try speaking the words: "OK Google, the thick round box jumped over the hazy bog."
makecheck 9y ago
While understanding “any” speech is a great long-term goal, I wish they’d allow these systems to be a little dumber when I want them to. One example: I often give the same commands to Alexa every day, even at similar times. Yet every few days it just utterly misunderstands those commands, picking words I have never even used before. It doesn’t even offer to choose similar commands I’ve used recently. Worse, the misunderstood command might trigger a paragraph of senseless babble about the misunderstood command, forcing me to shout over Alexa to make it hear the command I really wanted. So please, please, add some “dumb” options. I want to pick some words, have the system learn them, and just obey commands 90% of the time.
trengrj 9y ago
We recently did a comparative analysis of cloud speech-to-text providers for a project. We looked at: 1. Google Cloud Speech API 2. Microsoft Bing Speech API 3. IBM Watson Speech to Text The ranking was as listed above but we had real challenges working with call-center audio recordings. The quality was less than idea but still very clear. We saw a huge reduction in accuracy compared to in-browser testing. Additionally Australian English is particularly not solved. Because Google's API isn't currently doing speaker-detection, we looked at using Watson's speaker-detection as a secondary step but found it too complex and error prone. There is definitely room for a startup in this area and it also needs continued investment from the bigger cloud providers.
- splatcollision 9y ago
  For Watson Speech to Text - Did you choose the correct model to match your source audio quality? They default to a "Broadband" model intended for high quality audio sources, but you can also select "Narrowband" for things like phone quality. Not guaranteeing a difference, but in my experience, matching the source quality to the correct model makes some difference. I've not compared them extensively but for streaming realtime, I found that Watson beat the Google api for a specific use-case. Your mileage may vary! They also provide a handy Mic / File reader interface for browsers: https://github.com/watson-developer-cloud/speech-javascript-sdk/ https://github.com/watson-developer-cloud/speech-javascript-...
- adrianbg 9y ago
  Here are some benchmarks on telephone speech, including both APIs and human transcription services: https://remeeting.com/app/benchmarks https://remeeting.com/app/benchmarks Google actually did pretty badly for us on extended telephone speech. Not sure why.
WalterBright 9y ago
> It’s the only way to go from ASR which works for some people, most of the time to ASR which works for all people, all of the time. All people all the time doesn't remotely work for people, either. That's why the Air Force, for example, uses the Phonetic Alphabet, and aviation in general uses special jargon that is harder to misinterpret, such as "affirmative" for "yes". http://militarytimechart.com/military-phonetic-alphabet/ http://militarytimechart.com/military-phonetic-alphabet/ Radio DJs and actors tend to enunciate more clearly than regular people do. You'll notice this more if you speed up the sound - the professionals are still clear at higher speeds where non-professionals become unintelligible.
hzhou321 9y ago
I suspect we (human) are very bad at hearing, computers already are doing are much better jobs than us. However, I suspect for most our communication, we do not actually hear the conversation -- we guess the conversation. Only when evidence such as facial response or out-of-context words is caught then we actually try to hear. Even then, we are still trying to second guess. Computers cannot compete with our guessing ability, not until they are trained with our life experiences.
- hprotagonist 9y ago
  we are exceptionally good at hearing. Our ears are truly exquisitely precise things.Where "hearing" ends and "understanding" begins is, shall we say, a point of contention. My vote is for the auditory nerve as the inflection point. Anything above the cochlear nucleus on the auditory pathway is black voodoo magic and anyone who claims they understand it needs to reevaluate their kool-aid intake.
sytelus 9y ago
Can this be attributed to lack of good challenging open and large enough dataset that can be used as benchmark? The switchboard dataset was "challanging" in its time but pointless from current standards. One doesn't have to look beyond trying to do voice dictation in car to see how horrible current speed recognition systems are compared to humans.
- woodson 9y ago
  There are the CHiME challenges (http://spandh.dcs.shef.ac.uk/chime_challenge/ http://spandh.dcs.shef.ac.uk/chime_challenge/) that have test data with distant microphone speech and noisy speech data, and IARPA ran its Automatic Speech recognition In Reverberant Environments (ASpIRE) Challenge in 2014/15 (https://www.iarpa.gov/index.php/working-with-iarpa/prize-challenges/306-automatic-speech-in-reverberant-environments-aspire-challenge https://www.iarpa.gov/index.php/working-with-iarpa/prize-cha...).
tmulc18 9y ago
In the caption for figure 2 there is this quote, "Notice the humans are worse at transcribing the non-American accents." But when I look at the bar graph, I see the word error rates of the model is much higher than the word error rates of humans for most of the non-American accents. Am I missing something?
kazinator 9y ago
Did you say ... "speed trek ignition involved?" If yes, say "yes" or press 1, otherwise say "no" or press 2 to speak your request again, star to hear the options, or stay on the line for the next available agent ...
cairo_x 9y ago
Speech recognition is such a gimmick. In practice the applications are quite narrow. The truth is, most people with properly functioning social faculties feel like complete dicks when talking to a machine.
charlysl 9y ago
It seems to me that much of this can be explained by one of the fundamental principles of machine learning: Training and test samples must be iid (ie independently sampled from the same distribution) to get good performance. Otherwise there is bias. This applies not only to speakers as individuals, but to all the other factors mentioned, like context, bg noise, gender, age, accent, education, mike, emotion, etc etc. Many of the issues described can be traced down to this principle. This is why, possibly, the core issue is data. Although models and *PUs matter too, the theoretical performance boundary dependends on data quality only (with a model to match).
tanilama 9y ago
Technically yes. But in the realm of application, it is mature enough that people can use some drop-in solution, without too much tuning, and get passable result. It is like say Web Application is not solved. Surely, it is not. But the remaining part falls into the arts category, while the hard barrier of the technologies itself is no longer there.
rocky1138 9y ago
Seriously, who thought it was solved? Anyone who has used any voice interaction can tell that it's just not there yet.
garfieldnate 9y ago
Don't forget speech of children and the elderly, which are also ifficult. I had to laugh a little at Indian accent recognition being so low in humans. I like Indian people but I've always hated the accent because it's so difficult to understand.
brw12 9y ago
I'm constantly surprised by the poor contextual quality of speech recognition. I think the basic audio recognition does well, but when there is ambiguity, it seems like systems that are popularly considered high-performing degrade drastically. For instance, I'm using Dragon NaturallySpeaking to dictate this right now, but if I say a certain punctuation mark at the end of a sentence, half the time it's going to say excavation mark! Ditto with Google's Google Now assistant, or whatever the heck it's called these days. I have a Pixel 2 phone (Dragon heard "pixel to phone" -- it doesn't have up-to-date context on proper nouns in the news), but when I tried to create a calendar event using "Create calendar event... meet Bruno for pizza", it heard "MIT pronoun for pizza". It has hundreds of samples of my voice, and it already knew I was creating an event! "Meet" has to be one of the most common first words used in events. It seems to me like there is pretty low hanging fruit, and that we need more focus on flexibility and resourcefulness rather than acting as though we're moving from 99.5% accuracy to 99.6%.

435 points by allenleein 9y ago

Speech recognition for multi-lingual speakers is another pain point. I live with a native French speaker, so my conversations naturally include a lot of French proper names, as well sometimes switching languages mid conversation or even mid sentence. Lots of recognition engines can handle English and French, but treat them as mutually exclusive. It frustrates me to no end when I know that Siri can recognize a French proper name just fine if I switch it modally into French, but will botch it horribly in English.

rav 9y ago
As a multi-lingual speaker I've even had problems with speech synthesis. For instance, Google Maps insists on narrating driving directions in the system language, which is set to English; this makes the narrated local non-English place names incomprehensible. If I succumb and change the system language I can understand the narrated place names, but in return I have to put up with the poor speech synthesis of non-English languages.
- theresistor 9y ago
  I was just in France last week suffering through the same problem. I’m American, so I want to keep my system set to English both due to familiarity of the interface and because I want data in familiar units (miles, degrees Fahrenheit), but it kills me when every street and place name is absolutely butchered.
  - theresistor 9y ago
    An interesting observation I made last week was that the poor synthesis of French street names in English mode was worse than American-mispronouncing-French level. I wonder if that exposes some level of mismatch between how the synthesis engine models language and how humans do.
    - ghaff 9y ago
      In the case of French names it probably depends on if you have any exposure to French pronunciation rules at all. I have a pretty crappy level of high school French but I can usually manage not to butcher pronunciation. I was with a friend in Montreal though and she had absolutely no idea how to pronounce many things.

pjmlp 9y ago

As Portuguese native speaker forced to use foreign languages to talk to devices, it is not solved at all.

reaperducer 9y ago
Even English doesn't work the way it should. Remember Apple had to rush out a patch shortly after Siri was released because it couldn't understand Australians. Siri still doesn't understand the majority of my Minnesota relatives.

cscurmudgeon 9y ago

Just the other day I was arguing (pleasantly) with someone here on HN saying that AI has solved speech rec (among other tasks). https://news.ycombinator.com/item?id=15429287 https://news.ycombinator.com/item?id=15429287 Wish this article was written a few days earlier. An analogue of this article exists for most other domains claimed to have been solved.

throwmenow_0140 9y ago
Sorry to bother you, but the translation of your comment is pretty good (English -> Spanish -> French -> English): No, it's not learning new classes of objects from a single image or a few images is very difficult. See http://www.sciencemag.org/content/350/6266/1332.short The machine translation is a joke. Put comments on this page by translating Google into another language and go back to English and see what you get. I did a little part of you. Just human level for just a simple little prayer. > But even if you do not make this assumption, identifying the object involves spitting the distance from the performance of the human level. source: https://news.ycombinator.com/item?id=15429862 https://news.ycombinator.com/item?id=15429862 Not to debunk your claims, just interesting to see how good translation works (even if it's not human-level, I can understand what you say).
- cscurmudgeon 9y ago
  I am not sure that is a good translation because it completely misses a common idiom: "within spitting distance" "spitting the distance from the performance of the human level" If this were spoken out loud, it might get mistaken for "splitting the distance" which seems to have a different meaning (half as good as humans). But the point is subtle differences matter a lot in human communication. But, I do agree with your point that in a lot of cases even without human-level performance one can get things done.
  - throwmenow_0140 9y ago
    After reading the translated version again, I'm not sure I can support my statement anymore. I think a big part of my impression was based on the fact the I read the original comment and knew what you were talking about. > I did a little part of you. Just human level for just a simple little prayer. It's just bad. It sounds like it's based on a n-gram model - about something religious, strangely enough.

xigency 9y ago

A big thing for me is the difference between offline and online speech recognition. This really ties in with the last point, advancements need to be efficient in order to be used. I'm not sure how much work it would take to scale down Apple's voice recognition to run on the device or if it is feasible with their model, but currently it can take 5-10 seconds longer to get an answer from Siri during peak times.

jtth 9y ago

Also, water is wet and rocks are hard.

dang 9y ago
Would you please stop posting unsubstantive and/or snarky comments? https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

johnramsden 9y ago

What I would like to see you would be an open source speech recognition system that is easy to use, and works. There is really no match for the proprietary solutions in the open source world right now, and that is disappointing. Not to say that there aren't open source speech recognition systems, they just aren't completely usable in the way proprietary solutions are. A lot of research goes into open source speech recognition, where they are lacking is in the datasets and user experience. Hopefully, Mozilla's Common Voice project https://voice.mozilla.org/ https://voice.mozilla.org/ will be successful in producing an open dataset that everyone has access to and will spur on innovation.

LandoCalrissian 9y ago
I recently have been trying out open source solutions for voice recognition for a personal project and you are very correct that it lags very far behind proprietary solutions. Pocketsphinx is still very limited and Kaldi takes quite a bit to setup in a usable fashion. There were a few other options I looked at that I can't think of from the top of my head but were all in similar condition. As the article says latency is still a problem and it's a huge problem in current open source solutions, some stuff I was testing was easily 5 seconds. I know that can be improved with configuration, but when dealing with libraries of 10 words or so, that's pretty bad. I feel like anyone who is seriously interested in this space has been scooped up by all the big companies and the open source solutions have really seemed to linger because of it. It's one of the first areas I've seen where open source alternatives are really behind the proprietary solutions. Kind of bummed me out.
- dharma1 9y ago
  Kaldi is the best, there was just Tensorflow integration added which will hopefully speed up development (though I haven't seen any pretrained models for that yet). Here's a blog post - http://www.googblogs.com/kaldi-now-offers-tensorflow-integration/ http://www.googblogs.com/kaldi-now-offers-tensorflow-integra... The easiest way to deploy Kaldi is this - https://github.com/alumae/kaldi-gstreamer-server https://github.com/alumae/kaldi-gstreamer-server (or a docker image of the that)

salqadri 9y ago

I remember using the Merlin Microsoft Agent to create speech-based "apps" in 2001, including text-to-speech as well as speech recognition. I made a SAT preparation that would test me by presenting a word and I had to say the meaning. I also made a Spanish learning app using Merlin, where it would read out the spanish word (in spanish!) and I would have to say the english one. It worked really well. Its been 16 years since then, and I have to say that I had expected this area to have been completely "solved" by now.

canjobear 9y ago

It's always been a puzzle to me that published WER is so low, and yet when I use dictation (which is a lot--I use it for almost all text messages) I always have to make numerous corrections. This sentence is crucial and suggests a way to understand how WER can be apparently lower than human levels, yet ASR is still obviously imperfect: > When comparing models to humans, it’s important to check the nature of the mistakes and not just look at the WER as a conclusive number. In my own experience, human transcribers tend to make fewer and less drastic semantic errors than speech recognizers. This suggests that humans make mistakes specifically on semantically unimportant words, while computers make mistakes more uniformly. That is, humans are able to allocate resources to correct word identification for the important words, with less resources going to the less important ones. So maybe the way to improve speech recognition is not to focus on WER, but on WER weighted by word importance, or to train speech recognition systems end-to-end with some end goal language task so that the DNN or whatever learns to recognize the important words for the task.

IanCal 9y ago
This is, to me, one of the major problems with many algorithmic solutions to problems. An x% increase does in precision, F measure or any other score does in no way mean that the results are better. I've repeatedly seen improvements to traditional measures that make the subjective result worse. It's incredibly hard to measure and solve (if anyone has good ideas please let me know). I check a lot of sample data manually when we make changes, doing that (with targeting at important cases) is really the only way I think to do things.
- canjobear 9y ago
  If you've got a dictation system on a phone, wouldn't a very good metric be the corrections people make after dictating? I guess a problem would be if people become so used to errors that they send messages without corrections. I have some friends who do this: they send garbled messages that I have to read out loud to understand. But there will always be a subset of people who want to get it right.
adrianbg 9y ago

saosebastiao 9y ago

I would argue that it can't be solved without severe privacy implications. Apple's talk-to-text, for example, should know my son's name by now, but it doesn't. And while I'm mildly frustrated at the fact that I have to go back and edit text messages on a regular basis, I'm pretty glad that Apple doesn't know my son's name. I'd hate for a company like Facebook to have access to all of the proper nouns in my everyday life.

macintux 9y ago
You can teach your iPhone to know your son's name: http://appleinsider.com/articles/17/10/19/how-to-teach-siri-to-pronounce-names-correctly-on-your-iphone http://appleinsider.com/articles/17/10/19/how-to-teach-siri-... You'd need to create a contact if he doesn't already have one.

twothamendment 9y ago

It is one thing to hear and correctly identify the words. It is another to understand the meaning. I've been thinking about this because my son has Auditory Processing Disorder, APD. He can hear great, even a whisper across the house. The trouble is the words don't always make sense. He can tell me the words he heard and they are correct, but assigning a meaning to them doesn't work like it does for most people. After playing him a bunch of audio with missing words, the lady who tested him was blown away by how smart he is - explaining that he is quickly filling in almost every missing word based by guessing from all thing possible things it could be, narrowing it down on context and getting it right. I guess that is a normal, all day long task for him. Since then I've thought about voice recognition differently. The AI to understand the context or fill in the blanks is what will make or break it.

deleted 9y ago
[deleted]
noir_lord 9y ago
I'd never heard of APD. Interesting, all my life I've struggled to follow a conversation in a crowded environment, so much so that I actively avoid background noise with words in, I work with silicon ear plugs in or headphones and music with no lyrics. Looking at NHS symptoms they describe me as a child, didn't learn to read until I was 8. I nearly ended up in the remedial track but for a single awesome teacher who spotted that it wasn't that I was incapable but that I was struggling to understand in a noisy classroom, she gave up her lunch time for 6mths and taught me one on one by the end of that year I could read much better.
- fillskills 9y ago
  Wow, I wish there was an easy way to pay back star teachers or mentors.
  - soperj 9y ago
    Is it really that they were a star? Or just that they cared.

nblavoie 9y ago

[Slightly off-topic] And what about people with minor to major stuttering troubles like I do ? It is so frustrating to tell Alexa, Siri or Google (even Cortana) what I want because the timer delay expiration is so short.

ballenf 9y ago

> Latency: With latency, I mean the time from when the user is done speaking to when the transcription is complete. ... While this may sound extreme, remember that producing the transcript is usually the first step in a series of expensive computations. For many applications, making a transcription seems like an unnecessary step and source of errors. Skipping transcription when the user doesn't need it (most cases where I use it) would seem like a way to get some gain, but perhaps at the reduction of debuggability. > For example in voice search the actual web-scale search has to be done after the speech recognition. That's an area where literal exact transcription is usually required. But even then, Siri/Cortana/Alexa might be better off trying to figure out the meaning of what someone's asking rather than figuring out the exact words spoken in order to return the best results. Most people are quite bad at formulating good internet searches without a lot of trial and error. Let Google listen to a person talk about the problem they have or issue and then formulate the best results for that instead of forcing us to come up with the exact right phrasing to get appropriate results. It would help tremendously with the synonym and homophone issues that are so annoying now.

johnmcd3 9y ago
> For many applications, making a transcription seems like an unnecessary step and source of errors. Skipping transcription when the user doesn't need it (most cases where I use it) would seem like a way to get some gain, but perhaps at the reduction of debuggability. Agree. What's really needed is research into (and development supporting) how to combine the expertise from a speech recognition layer with the next layer in a machine learning process. That higher layer contains the domain specific knowledge needed for the problem at hand, and still leverage a speech layer focused on a broad speech data set and speech-specific learning (from Google, Microsoft, the community, etc.) Today, how richly can information be shared? I see with Google's speech API you can only share a very finite list of domain-specific expected vocabulary. Why not have speech tools at least output sets of possible translations with associated probabilities? Do any of the top tools allow this? Then you could at least train your next level models with the knowledge of where ambiguity most exists, and what a couple of options might have been for certain words or phrases...

Radim 9y ago

Just the other day, I participated in a discussion about how "language identification" is a solved problem -- in fact, hasn't it been solved for a decade? As anyone who's had to use langid in practice will testify, it's solved only as long as: A) you want to identify 1 out of N languages (reality: a text can be in any language, outside your predefined set) B) you assume the text is in exactly one language (reality: can be 0, can be multiple, as is common with globalized English phrases) C) you don't need a measure of confidence (most algos give an all-or-nothing confidence score [0]) D) the text isn't too short (twitter), too noisy (repeated sections ala web pages, boilerplate), too regional/dialect, etc. In other words, not solved at all. In my experience, the same is true for any other ML task, once you want to use it in practice (as opposed to "write an article about"). The amount of work to get something actually working robustly is still non-trivial. In some respects, things have gotten worse over the past years due to a focus on cranking up the number of model parameters, at the expense of a decent error analysis and model interpretability. [0] https://twitter.com/RadimRehurek/status/872280794152054784 https://twitter.com/RadimRehurek/status/872280794152054784

notahacker 9y ago
I'm reminded of how often I see Twitter offer to translate English-language tweets containing a proper noun or two from absurdly unconnected languages. And that's with text containing mostly common and distinctive English words.
- abecedarius 9y ago
  I'm pretty sure Twitter's langid uses character n-grams. You'll see a tweet that's plain English that happens to match n-gram statistics unusually well with some other language, which pushes the likelihood score to just above English. (I checked this by running an example or two through my own langid code.) It shouldn't be hard to improve on by treating the score on a tweet as bayesian evidence to combine with a prior from preceding tweets.
jdmichal 9y ago
In other words, we have some systems that are great at processing spherical cows in a vacuum...

joemag 9y ago

Mistakes in understanding speech are common even among humans. My wife and I have to repeat and clarify ourselves fairly frequently in our day to day conversations. She even jokes at times that I must check my hearing, because I often mishear what she said, while she thought she was being perfectly clear. I think where computers fall short is in two areas: 1) The rate of errors hasn’t hit the inflection point of being comparable to day-to-day intra-human interaction, and 2) There is no good mechanism for detecting and correcting errors. At least, none that I’ve seen. That second one is important. When I hear my wife ask me “Please, hand me a tractor”, I realize that I must’ve misheard, and ask her to clarify “what?” With speech recognition, I either have to manually re-read and modify recognized text, or cancel and repeat the entire request. Both take time, and negate some of the efficiencies of using speech recognition.

twothamendment 9y ago
"I want some taquitos" - https://www.youtube.com/watch?v=CA-LGL9iMh4 https://www.youtube.com/watch?v=CA-LGL9iMh4
throwmenow_0140 9y ago
"Please, hand me a tractor" - Did she say tracker [1]? Something like Rhymezone (searching for similar sounding words) would be a solution. It would still need a human who's deciding which word to choose, but after some time it could learn which words you prefer. [1] https://www.rhymezone.com/r/rhyme.cgi?Word=tractor&typeofrhyme=sim https://www.rhymezone.com/r/rhyme.cgi?Word=tractor&typeofrhy...
pbhjpbhj 9y ago
One needs to know your wife to know that "hand me a tractor" is wrong. Perhaps she makes model farms as a hobby, perhaps you work in plant rental and that means "pass me the keys to one of the tractors", the possibilities are endless. You need vision, memory, profiling, etc., to even begin to properly contextualise day-to-day conversation.
- Myrmornis 9y ago
  Hm, I think I disagree with you and GP. The answer is Bayesian inference to both of you isn't it? The prior is going to give very low weight to your domestic partner asking you to "hand them a tractor", despite the fact that it's not impossible that it's the correct words.

morinted 9y ago

I'm part of the stenographer/captioner community as a hobby and you'd be surprised at how many people think that all captions are automatically computer-generated. In reality the practical limitations of autogenerated captions are demonstrated by YouTube's caption system. It's good but even a mistake every other sentence (95%+ accuracy) can completely obfuscate the meaning behind the video.

tyingq 9y ago
I've seen some pretty terrible human produced captions on television shows as well.
- morinted 9y ago
  Big time. There are various levels of caption quality and part of it is the tools used. 1. QWERTY transcriptionists working brief, 5-10 minute shifts typing as fast as they can and rotating. 2. Voice-mask reporters using Dragon with a voice mask (more commonly phonetic typos) 3. Stenographers using a steno machine, some of them were not trained to do realtime and lack the ability to edit as they go. It's all up to the individual, unfortunately. Furthermore, the budget for live captions is sometimes smaller than it should be and so the service purchased is the cheapest, not the best. I pay attention to captions now. I've watched TSN (Canada) and it seems that past 10 p.m. they put in someone who isn't fully trained or graduated from school and they most definitely shouldn't be providing captions as they are near unreadable. Good captions are priceless for the access that they offer.

aidenn0 9y ago

I think some people overestimate how good humans are at speech recognition. Unfamiliar accents and noisy environments cause havoc with many people. I had a friend who learned English in India when I was in High School, so I was used to that accent; many of my classmates in College could not understand anything our Indian T/A said freshmen year. Similarly I had friends for whom English was a second language who had lived in the US for years and were definitely fluent, but had to enable subtitles for a movie in which the characters had a strong southern accent; in general non-rhotic accents were very troublesome for them having only spoken english with midwesterners. The article mentions the Scottish accent, and I would call that the hardest accent of native English speakers for those in the US to understand.

twothamendment 9y ago
I had a chance to live in the middle of the "hood" in a large American city for about 6 months. I was one of the few white guys around. I spoke English, they spoke English - but it was about a week before I could understand people, especially on a crowded city bus. The first few days felt like a foreign country and it was the same country I grew up in! I wonder how well Siri and friends work with Ebonics?
- aidenn0 9y ago
  That's a double whammy because diction is so different as well; our brain fills in the gaps in phoneme recognition by pattern matching and the differing dialect makes that much harder as well.
- deleted 9y ago
  [deleted]
joemag 9y ago
Agree with this. I’m currently traveling to a country where English is not the primary language, but is still spoken universally. I am having a hell of a time understanding people - the unfsmikiar accent is wrecking havoc with my brain’s speech recognition centers. I’m currently where speech recognition software was in the 90s.

PaulHoule 9y ago

One thing I see missing is "conversational smarts". People mishear a certain fraction of what they hear and they'll ask you to clarify what you said. You could have superhuman performance at tasks like Switchboard and still have something embarrassingly bad in the field because the real task (having a conversation) is not trained for in the competitions with canned training sets.

deleted 9y ago

[deleted]

dmreedy 9y ago

Of course it's not solved. We don't even know how to define the problem. Speech is[1] a fundamental component of Language, which is a fundamental component of Intelligence[2]. This is addressed somewhat in the conversation around semantic error rate; that there is more to processing raw audio speech than the calculus of mapping signals to tokens; some understanding of semantics and context is required to differentiate between otherwise indistinguishable surface forms. I find it doubtful that there's a clean interface that separates the 'intelligent' parts of the brain from the 'language' parts of the brain from the 'speech' parts of the brain. This leakiness (or richness, really) means that you can't neatly solve any one part of this chain to the level of competence that the brain solves it. That means to 'solve' speech recognition, you have to 'solve' language, and thus 'solve' general intelligence. And to 'solve' general intelligence, you have to understand it, in a theoretical sense, which we don't. Indeed, it will likely involve solving all the other modalities of sensation as well. It's definitely the case that you need to have a model for prosody to understand speech. It is entirely possible that vision is a large factor as well, in the form of body language, lip reading, eye contact, and so on. Speech recognition is quite good for what it is. For many practical applications, especially to do with young, white, newspeaker-accented English speakers who sound the most like the people who develop it, and the data sets used to develop it, it is good enough in the 80/20 sense. But it is nowhere near solved by even the least rigorous definition of the word. ---- [1] according to the philosophy I subscribe to, at any rate [2] according to the philosophy I subscribe to, at any rate

baddox 9y ago
Problem definition really is the issue. Even appealing to the Turing test is less than satisfying. I’m fairly sure I’m a human, but Siri and similar will likely outperform me in certain categories like place names and popular music.
- jdmichal 9y ago
  Don't forget that being too good can also cause you to fail the Turing test. I would not expect a human to be able to answer some questions that may be trivial to a computer. Things like, "What's the square-root of 137?" Or, "Identify this obscure song within 5 seconds of listening from a random starting point."

ccozan 9y ago

Speech Recognition, beside transcribing phonemes and match to a NN of possible words, is not solved because speech is highly integrated with the human context: who is speaking, to whom is speaking, where is the speech happening, why is the speech initiated, and so on. My kids needed 5-6 years of continous daily talking until I could say they understand a conversation almost completly. Every single word or phrase I directed at them was spoken in a certain context and had a certain role in communicating with them when from the context it wasn't clear what my intentions were. Of course, they had their fun, throwing words away and repeating endlesly some funny word or phrase or terribly spelling them. It is interesting that you as an adult need too learn their own prononciation, at some moment in time I even wrote a small dictionary. Again, speech recognition will simply stay at recognizing phonemes/words only for a very long time, until we have a true AI Assistant that walks with us, sees with us, and hears with us in the same time. Then it can apply some semantic and other context based related NN.

billfruit 9y ago

I heard even voice synthesis, ie text to speech, is not solved yet.

ChuckMcM 9y ago

Too many people equate speech to text as 'speech recognition' and it clearly isn't. What is not a solved problem is Natural Language Processing. That NLP isn't solved, is a fertile source of papers. What is 'kinda' solved is the voice CLI, that is where speech is converted to text, that text is inserted in a class command interpreter, and if recognized the command is executed, otherwise it isn't. Nobody expects to be able to type 'find me the files that changed yesterday' into a computer shell and have it do something sensible. But 'find ./ -type -f -mtime yesterday' sure we can expect that to work. Getting from the first one to the second one requires something called an ontology. I worked on a team at IBM that was building this sort of capability. The plan was that in conjunction with an excellent tool kit for disassembling a sentence into its component parts (verbs, nouns, adjectives etc) and then matching the elements recognized with an onotology associated with that action. 'find' being the verb would select the search action ontology, 'files' as an object of the verb would select the computer files object ontology, and then 'changed' and 'yesterday' would match 'modification' in that ontology and 'yesterday' would match one of the two sub children of 'modification' (time and change-actor). Then you would go back with what was the compiler equivalent of an abstract syntax tree which the 'command generator' would use to emit the necessary command actions and flags. I have no idea why they cancelled it. It was an inscrutable place to work at in many ways :-).

gok 9y ago
"Speech Recognition" is an accepted term of art for "speech to text" or "transcription." "Speech Understanding" is a related but different problem.
- astrodust 9y ago
  I'd challenge you to transcribe a casual conversation without understanding. Too many words sound very similar, if not identical, and the context of the conversation dictates which word is in play. Even something simple like "They're unhappy, Ness" could be interpreted as "Their unhappiness" unless you know Ness is a person in the converation.
  - 9y ago

turbohedgehog 9y ago

As a deaf person I am using Ava [0] which uses IBM's speech to text service [1] as its backend AFAIK. I am always impressed by how it picks up on context clues to make corrections in realtime and capitalizing proper nouns (Incredible Pizza for example). However, it does not work with multiple speakers on a single microphone. [0] https://www.ava.me/ https://www.ava.me/ [1] https://www.ibm.com/watson/services/speech-to-text/ https://www.ibm.com/watson/services/speech-to-text/

ksec 9y ago

Then there is Chinese, which the same "sound" can have many different words. And we constantly make up new short form or words, a lot more so then English means recognizing these are much harder. ( Then there is Cantonese... ) Sometimes I wonder, if Apps could add vocabulary to devices dictionary, so certain terms in Gaming, Fashion, or Tech or other domain, which are rarely used in normal conversation would be recognized in speech to text as well as Auto Correct.

notadoc 9y ago

Siri reminds me of that every time I attempt to use it.

philfrasty 9y ago
Baffled every time that Siri has actually been shipped by Apple to be used in production and not as a pre-pre-pre-pre-alpha.

artellectual 9y ago

Do you guys thing “understanding” needs more that what we have right now in terms of AI, does understanding require intelligence that is self aware? I mean when we talk to each other, we understand in spoken words, little expressions, even when we talk over the phone. I feel like without the ability to process emotion and self-aware AI we wont really achieve “full understanding” only simulated ones.

singularity2001 9y ago

Speech Recognition (STT) used to be completely monopolized by Nuance. Slowly the gates are opening (thanks to google and DL), but I guess it is still a minefield of patents.

amorphid 9y ago

I am a native English speaker from California. It is hard to get my Google Personal Assistant to understand the difference between desert and dessert. It's also hard to request music by an artist with name similar to another artist. "Hey Google, play Mika radio" has a 50/50 chance of starting music by Meiko. The additional "problem" is that I like Meiko, too, but I feel obligated to cancel the Meiko music & re-request Mika so that Google (hopefully) learns to recognize the difference between Mika and Meiko. Maybe our spoken language will start to transform into distinctly unique sounds so we can verbally interact with computers with relative ease. When I was in the US Army, I was trained to speak in a certain manner to help my communication to be more clear. [1] I don't see a reason humans and computers can't each make reasonable compromises to make verbal communication easier. [1] https://en.wikipedia.org/wiki/Voice_procedure https://en.wikipedia.org/wiki/Voice_procedure

pkpp1233 9y ago

It's pretty good right now for English speakers. At least usable. We've tried https://www.btwtalk.com https://www.btwtalk.com for transcribing our internal calls, and while it does mess up a bit, it's good enough for later reference.

nkkollaw 9y ago

I would also like to confirm that speech recognition in language other than English is completely not solved. In Italian, for instance, any device will barely be able to recognize what you're saying, and will almost always do something completely different from what you've asked.

notlarrypage 9y ago

I think that some speech recognition systems depend too much on language model priors. This works well for routine tasks where most of the speaker's words are easy to predict. It fails when the speaker's words are specific or unusual. For example, try speaking the words: "OK Google, the thick round box jumped over the hazy bog."

makecheck 9y ago

While understanding “any” speech is a great long-term goal, I wish they’d allow these systems to be a little dumber when I want them to. One example: I often give the same commands to Alexa every day, even at similar times. Yet every few days it just utterly misunderstands those commands, picking words I have never even used before. It doesn’t even offer to choose similar commands I’ve used recently. Worse, the misunderstood command might trigger a paragraph of senseless babble about the misunderstood command, forcing me to shout over Alexa to make it hear the command I really wanted. So please, please, add some “dumb” options. I want to pick some words, have the system learn them, and just obey commands 90% of the time.

trengrj 9y ago

We recently did a comparative analysis of cloud speech-to-text providers for a project. We looked at: 1. Google Cloud Speech API 2. Microsoft Bing Speech API 3. IBM Watson Speech to Text The ranking was as listed above but we had real challenges working with call-center audio recordings. The quality was less than idea but still very clear. We saw a huge reduction in accuracy compared to in-browser testing. Additionally Australian English is particularly not solved. Because Google's API isn't currently doing speaker-detection, we looked at using Watson's speaker-detection as a secondary step but found it too complex and error prone. There is definitely room for a startup in this area and it also needs continued investment from the bigger cloud providers.

splatcollision 9y ago
For Watson Speech to Text - Did you choose the correct model to match your source audio quality? They default to a "Broadband" model intended for high quality audio sources, but you can also select "Narrowband" for things like phone quality. Not guaranteeing a difference, but in my experience, matching the source quality to the correct model makes some difference. I've not compared them extensively but for streaming realtime, I found that Watson beat the Google api for a specific use-case. Your mileage may vary! They also provide a handy Mic / File reader interface for browsers: https://github.com/watson-developer-cloud/speech-javascript-sdk/ https://github.com/watson-developer-cloud/speech-javascript-...
adrianbg 9y ago
Here are some benchmarks on telephone speech, including both APIs and human transcription services: https://remeeting.com/app/benchmarks https://remeeting.com/app/benchmarks Google actually did pretty badly for us on extended telephone speech. Not sure why.

WalterBright 9y ago

> It’s the only way to go from ASR which works for some people, most of the time to ASR which works for all people, all of the time. All people all the time doesn't remotely work for people, either. That's why the Air Force, for example, uses the Phonetic Alphabet, and aviation in general uses special jargon that is harder to misinterpret, such as "affirmative" for "yes". http://militarytimechart.com/military-phonetic-alphabet/ http://militarytimechart.com/military-phonetic-alphabet/ Radio DJs and actors tend to enunciate more clearly than regular people do. You'll notice this more if you speed up the sound - the professionals are still clear at higher speeds where non-professionals become unintelligible.

hzhou321 9y ago

I suspect we (human) are very bad at hearing, computers already are doing are much better jobs than us. However, I suspect for most our communication, we do not actually hear the conversation -- we guess the conversation. Only when evidence such as facial response or out-of-context words is caught then we actually try to hear. Even then, we are still trying to second guess. Computers cannot compete with our guessing ability, not until they are trained with our life experiences.

hprotagonist 9y ago
we are exceptionally good at hearing. Our ears are truly exquisitely precise things.Where "hearing" ends and "understanding" begins is, shall we say, a point of contention. My vote is for the auditory nerve as the inflection point. Anything above the cochlear nucleus on the auditory pathway is black voodoo magic and anyone who claims they understand it needs to reevaluate their kool-aid intake.

sytelus 9y ago

Can this be attributed to lack of good challenging open and large enough dataset that can be used as benchmark? The switchboard dataset was "challanging" in its time but pointless from current standards. One doesn't have to look beyond trying to do voice dictation in car to see how horrible current speed recognition systems are compared to humans.

woodson 9y ago
There are the CHiME challenges (http://spandh.dcs.shef.ac.uk/chime_challenge/ http://spandh.dcs.shef.ac.uk/chime_challenge/) that have test data with distant microphone speech and noisy speech data, and IARPA ran its Automatic Speech recognition In Reverberant Environments (ASpIRE) Challenge in 2014/15 (https://www.iarpa.gov/index.php/working-with-iarpa/prize-challenges/306-automatic-speech-in-reverberant-environments-aspire-challenge https://www.iarpa.gov/index.php/working-with-iarpa/prize-cha...).

tmulc18 9y ago

In the caption for figure 2 there is this quote, "Notice the humans are worse at transcribing the non-American accents." But when I look at the bar graph, I see the word error rates of the model is much higher than the word error rates of humans for most of the non-American accents. Am I missing something?

kazinator 9y ago

Did you say ... "speed trek ignition involved?" If yes, say "yes" or press 1, otherwise say "no" or press 2 to speak your request again, star to hear the options, or stay on the line for the next available agent ...

cairo_x 9y ago

Speech recognition is such a gimmick. In practice the applications are quite narrow. The truth is, most people with properly functioning social faculties feel like complete dicks when talking to a machine.

charlysl 9y ago

It seems to me that much of this can be explained by one of the fundamental principles of machine learning: Training and test samples must be iid (ie independently sampled from the same distribution) to get good performance. Otherwise there is bias. This applies not only to speakers as individuals, but to all the other factors mentioned, like context, bg noise, gender, age, accent, education, mike, emotion, etc etc. Many of the issues described can be traced down to this principle. This is why, possibly, the core issue is data. Although models and *PUs matter too, the theoretical performance boundary dependends on data quality only (with a model to match).

tanilama 9y ago

Technically yes. But in the realm of application, it is mature enough that people can use some drop-in solution, without too much tuning, and get passable result. It is like say Web Application is not solved. Surely, it is not. But the remaining part falls into the arts category, while the hard barrier of the technologies itself is no longer there.

rocky1138 9y ago

Seriously, who thought it was solved? Anyone who has used any voice interaction can tell that it's just not there yet.

garfieldnate 9y ago

Don't forget speech of children and the elderly, which are also ifficult. I had to laugh a little at Indian accent recognition being so low in humans. I like Indian people but I've always hated the accent because it's so difficult to understand.

brw12 9y ago

I'm constantly surprised by the poor contextual quality of speech recognition. I think the basic audio recognition does well, but when there is ambiguity, it seems like systems that are popularly considered high-performing degrade drastically. For instance, I'm using Dragon NaturallySpeaking to dictate this right now, but if I say a certain punctuation mark at the end of a sentence, half the time it's going to say excavation mark! Ditto with Google's Google Now assistant, or whatever the heck it's called these days. I have a Pixel 2 phone (Dragon heard "pixel to phone" -- it doesn't have up-to-date context on proper nouns in the news), but when I tried to create a calendar event using "Create calendar event... meet Bruno for pizza", it heard "MIT pronoun for pizza". It has hundreds of samples of my voice, and it already knew I was creating an event! "Meet" has to be one of the most common first words used in events. It seems to me like there is pretty low hanging fruit, and that we need more focus on flexibility and resourcefulness rather than acting as though we're moving from 99.5% accuracy to 99.6%.