13 ms·
Researchers reach human parity in conversational speech recognition
- grzm 10y agoI don't have a background in this area, so I'm likely easily impressed, but this seems really impressive. And the acknowledgement that there's a lot of work to be done, such as discriminating between speakers and recognition in adverse environments. Yeah, it's Microsoft writing on their own technology, but they addressed in the text the questions I had already in mind from just reading the title. It didn't leave me with feeling that it's just a marketing piece. > Still, he cautioned, true artificial intelligence is still on the distant horizon It's frustrating when technologies like image and speech recognition and robotics are conflated with AI.
- gremlinsinc 10y agoI don't get why that's frustrating, without image and speech recognition AI isn't possible. -- How intelligent could people be without sensory perception? Hellen Keller is the exception, take away the ability for humans to process sound, and images as a species and we wouldn't be nearly as advanced as we are today even if we still had the same brain structure and mental capacity. Ray Kurzweil understood this and is why he paved the way for some of the first speech / image recognition platforms like ocr/fax/etc... To me speech/image recognition is a precursor/adjunct of AI. You can have the former without the latter, but the latter will never be realized without those.
- Retric 10y agoThe touring test shows AI has little to do with image or speech recognition. An AI that operates in a purely virtual environment would still be revolutionary.
- gingerrr 10y agoThe Turing test doesn't test for intelligence, simply human-like communication. Things like lying, typing errors, or reaction to insult aren't indicators of an artificial intelligence, but are requirements for passing the Turing test. One of the biggest criticisms of the Turing test is, because it's a behavioral or functional test, there is no way to ensure that the computer is actually thinking intelligently at all, rather than following a very-well-articulated rules engine.
- wbl 10y agoThis is also true for pwople. How do you know humans are thinking and not following lots of social rules they learn thanks to special neural hardware for learning social interaction? Think about the intelligence conveyed in ritualized greetings.
- JumpCrisscross 10y agoYou are recapitulating Descartes. Consciousness appears to be immeasurable. You can only know that you are conscious. It is polite to give others the benefit of doubt.
- pharrington 10y agoThe commenter said nothing about consciousness. He simply said that just being able to hold a conversation is not a particularly useful measure of intelligence. The broader form of that is that we don't even know if humans in general are intelligent.
- Scarblac 10y agoYou can know that you are conscious, but how do you know you are intelligent?
- ghurtado 10y agoI can think of no better test of whether some specific being is intelligent than fooling another human into thinking they are. Can you think of a better one? > because it's a behavioral or functional test What else would you test for in an AI other than its output and behavior? Conversely: how would you test a human for intelligence other than through its output and behavior?
- duaneb 10y agoPerhaps because "true AI" is an illusion. There is always a better term that has meaning. And yet people like Ray Kurzweil toss the term around as if it does have meaning. What's the point unless you define it? You might as well reference anti gravity or teleportation.
- nix0n 10y agoIt's something to work towards, always on the horizon. Kurzweil was a great inventor once but he's a prophet now and he needs to speak prophetically, not scientifically.
- grzm 10y agoLots of good questions about what I mean by AI. I agree my statement wasn't very precise. And to be honest, that's a reflection of me not having a precise definition of what it is, exactly. However, I do think there's a distinction between speech/image recognition and what's commonly understood as AI, with speech and image recognition being sensory input systems, and AI in a more colloquial sense, such as learning and problem solving on at approximately human level. My statement was imprecise, and given I was expressing frustration at imprecision, my bad :) Thanks for the feedback! Gives me more to think about.
- ant512 10y agoOn the other hand, if you had true AI first, you'd just say "here's an input device; figure out how to parse the data."
- ghurtado 10y ago> I don't get why that's frustrating, without image and speech recognition AI isn't possible Wait, what? The most commonly known (and perhaps oldest) AI test in no way requires either image nor speech recognition. Millions of blind and deaf human beings would like to disagree with your claim that they are not intelligent or sentient beings. Seriously, what is the basis of this claim?
- gremlinsinc 10y agoI'm not saying one can't be smart/intelligent without being able to see/hear, I'm saying if you take away all senses from ALL human beings there won't be any way to communicate at all, or know that others exist, and to learn from them. Learning is what makes intelligence possible--and the passing of information through generations. Hellen Keller fought extremely hard to overcome her sensory issues, but she had a good teacher, who presumably had someone else help them or guide them some. -- But to organically learn to speak when nobody can hear you, or to read when you can't even feel the braile, etc..would be nearly impossible.
- nercht12 10y ago> Learning is what makes intelligence possible Intelligence (at least in this case) is being able to take data and draw conclusions from it, but you just can't see such potential when you don't have input. Is a computer not a calculator when there's no software installed? No. It's still a calculator, but it just doesn't have inputs. One day, we may be able to give sight to the blind, but for now, considering such people without ANY learning capacity as "unintelligent" is still wrong.
- Lxr 10y agoI'm curious, what is your definition of AI?
- PravlageTiem 10y agoA machine process qualifies as AI if it believes in a god. EDIT: I see the nuances of epistemological problems are lost to HN and knee-jerk culture war atheism still rules supreme.
- jcoffland 10y agoELIZA could be made to believe in God.
- PravlageTiem 10y agoBeing made to believe in a god isn't the same as believing in one. Silicon Valley thinks AI is not an epistemological problem, as if neurons can be perfectly simulated atomically and that all intelligence processes can be categorized as structured vs. unstructured. Very naive conclusions. Ironically, they BELIEVE if you simulate the axiomatic neuron perfectly, emergent properties of intelligence will mystically emerge after some undefined threshold of complexity. They are permanently unable to simulate the two billion years of neurological evolution and natural selection for such a belief to hold. If you must emulate human intelligence, you must be able to navigate the realm of distilling reality from the overlap between truth and belief. Mocking how humans observe reality isn't enough. Abrahamic religion explored the alternative intelligence problem in great depth over two thousand years ago. It's a pity the results have been lost to Progressive axe grinding.
- ralusek 10y agoFor Artificial Intelligence to believe in God would be for it to believe in its makers, which are human. A machine would be intelligent to be aware that it was made by man. A man doesn't need to believe in God to be intelligent. In fact, the two are pretty much inversely related.
- psyc 10y agoI prefer a conceptualization where they are AI, AI is a gradient, and our current tech level is somewhere on the gradient.
- madenine 10y agoIts frustrating when people assume any reference to AI means AGI or strong AI.
- AndrewKemendo 10y agoIt's not their fault really. Tv and movies, especially ones like the matrix, AI and Irobot call AGI, AI. Doesn't make it any less frustrating, but also distinguishes the people in it versus those who aren't.
- jcoffland 10y agoI'm not so sure anyone was using the term AGI when the first Matrix movie came out. Anyway, you cannot expect Hollywood to always use the precisely correct term for things scientific without first pushing up your glasses.
- tree_of_item 10y ago> It's frustrating when technologies like image and speech recognition and robotics are conflated with AI. Are you kidding? Of course these things are examples of artificial intelligence. I don't understand why people keep moving the goalposts wrt "AI".
- arstin 10y agoIf you take the goalposts to have been set by Turing's 1950 "Computing Machinery and Intelligence", the only people moving them are researchers who want to trump up their own work or marketers who want to sell things. My own taste uses the word "AI" as you do in a permissive way to include simpler tasks (which in themselves are more elemental than useful) like identifying an object in an image and presenting a few straightforward interpretations of sentences. But what Turing stipulated an actual AI would be able to do is "reach parity" with a fully human conversation, with all of our knowledge and values and desires and subtlety of motivation. When you really dig into what goes on in a real conversation---the joking, the shades of meaning, the individual quirks and tribal patterns, the lying, the compassion, ambition, insecurity---I think it's hard not to admit that we're still quite far from that. We may not even want it! But exactly the genius of the Turing test for setting a rough standard for AI is that it demands competences far beyond mere language processing.
- abecedarius 10y agoTuring wasn't trying there to articulate a single target for AI research, he was responding to the claim that artificial minds were not possible even in principle. His response was basically, try this, if in open-ended conversation you couldn't tell one from a human, would you really still say there's no mind there? As you say, that's a really high bar from where we are, then or now. I'm sure Turing's own goals for AI were broader than eventually passing the Turing test. He worked on neural nets himself and would've considered progress in perception as partial progress in AI.
- arstin 10y agoYep! The assertion that Turing was setting a "single target for AI research" in the way you're using the phrase was clearly not what he was doing or, I hope, what I said. My intent was the opposite: to draw attention the massive range of "targets for research" which are already implicit in the original Turing Test in order to note that saying "there's still lots more to do" is hardly "moving the goalpost". Just to link up the way you put things with the way I chose to here, his argument for how an artificial mind is possible proposes a reasonable, minimal standard which different sides can agree to---a criterion of, "well if it can do that then sure it's a mind!". And the choice of a open and unbounded conversation as the standard was brilliant because of the massive range of subcompetencies which are required for actually executing it (including, obviously, perception). Which, of course, we gradually continue to plod through in AI research.
- Eridrus 10y agoThe actual paper has a section on error analysis that is particularly enlightening: https://arxiv.org/abs/1610.05256 https://arxiv.org/abs/1610.05256 On the CallHome dataset humans confuse words 4.1% of the time, but delete 6.5% of words, most commonly deleting the word "I". Their ASR system confuses 6.5% of words on this dataset, but only deletes 3.3% of words, so depending on how you view this their claim about being better than humans isn't definitely true, if you consider the task to be speech recognition, rather than transcription. Also, while the overall "word error rate" is lower than humans it's not clear if this is because the transcription service they used is not seeking perfect output, but rather good enough output and the errors the transcription service makes may not be as bad as the errors the ASR system makes in terms of how well you can recover the original meaning from the transcription. It's clearly great work, but reaching human parity is marketing fluff.
- AnimalMuppet 10y agoI'm not an expert on this at all, but my suspicion is that humans are more tolerant of missing words than we are of mistaken words. Especially a word like "I", we just assume it if it's missing. So if this speech recognition is for creating a transcript for humans, this (in my uneducated opinion) isn't as good as humans do, at least not yet.
- rhizome 10y agoIf nothing's changed in the past 15 years, anything under 95% accuracy is not accurate enough for automation without human intervention (copy editing).
- joe_the_user 10y agoI was recently at a bar where they showed a movie with incomprehensible subtitles (English to English). I assume this was because they skimped and bought automatic subtitling. I think one important aspect is while humans miss words, they often get the sentence meaning correct. When computers miss words, they tend to substitute words that sound similar. That's readable if you have time but not necessarily as a stream of text going by...
- uvesten 10y agoGood for them! I'm a bit surprised that the researchers didn't already possess human-level speech recognition, though.
- jameshart 10y agoI wasn't sure which way round to parse it (ironically). Are they saying that Microsoft researchers are now almost as good at recognizing conversational speech as humans? Or that Microsoft researchers can now almost pass as human in conversational speech? Either way, good news for Microsoft Research, I think.
- cptskippy 10y agoPlot twist, Microsoft's researchers aren't human.
- deleted 10y ago[deleted]
- Maarten88 10y agoNice to read this, being someone who uses lots of Microsoft products, but I have mixed feelings: after all these years Cortana still understands 0.0% of my native language (Dutch). Very disappointing, especially seeing that Google has no problems understanding Dutch.
- jcoffland 10y agoWhat do you think the chances are that Philips is working on this?
- cellis 10y agoOk, when's the next 2GB Xbox One update and will this fix the problem of me saying "Xbox watch NBC", and it 'hearing' "Xbox watch TV"?
- Animats 10y agoVery nice. How long before something this good is available as open source? A tough test would be to hook this up to a police/fire scanner, or air traffic control radio.
- ChuckMcM 10y agoThey did post the code on github (https://github.com/Microsoft/CNTK https://github.com/Microsoft/CNTK) with the Microsoft open source license. Presumably you could feed it speech from a running instance of gnu-radio.
- ar15saveslives 10y agoCNTK is just a toolkit, like tensorflow or theano. Code for the paper was not published.
- azinman2 10y agoLet alone the datasets used to train, which are worth a lot of money.
- zump 10y agoAussie company (Appen) provides the datasets :D
- mtrimpe 10y agoThat's the one that seems to be using illegal GCHQ wiretaps ... see the (Dutch) article below about an Appen translator getting a private voicemail of her ex to translate. http://www.volkskrant.nl/tech/privegesprekken-van-duizenden-nederlanders-in-handen-van-tech-bedrijf~a4386302/ http://www.volkskrant.nl/tech/privegesprekken-van-duizenden-...
- zump 10y agoNo, it's GCHQ contracting Appen. They only deliver the platform for distributed outsourced transcription.
- wbhart 10y agoSo Microsoft finally have an AI that can "wreck a nice beach". Along with text autocompletion, we are all set for a decade of irritating miscommunication.
- _kst_ 10y agoI thought it was "wreck a nice peach".
- jpm_sd 10y agoI look forward to being able to converse with Microsoft's research team as easily as I can with humans. I hope that one day, journalists can learn to write headlines with similarly low rates of error.
- hashkb 10y agoIt's on purpose... journalists learned to do it this way.
- avodonosov 10y agohttp://www.smbc-comics.com/comics/20090830.gif http://www.smbc-comics.com/comics/20090830.gif
- Namrog84 10y agoLinking the image instead of the site shall more people to miss out on the red button extra comic frame. And the meta hover text(xkcd style)(which there usually is) . http://www.smbc-comics.com/?id=1623 http://www.smbc-comics.com/?id=1623
- mastazi 10y agoWait a second... I just realised I've been reading SMBC for years now and I've never noticed the red button! I'm mad at how much I must have missed out on but, at the same time, I'm glad you pointed that out!
- radarsat1 10y agoThe term "human parity" refers to a comparison of the error rate, which is a single scalar summarizing performance in terms of mistakes made. It says nothing about the kind of mistakes, and I can easily imagine that machines qualitatively do not make at all the same kind of mistakes as humans. I'd be curious to know if the kind of mistakes machines make might strike human listeners as quite stupid, but maybe not.. many algorithms are getting better at taking into context and prior knowledge into account.
- szupie 10y agoSection 9 in the paper[1] is all about comparing these mistakes between the system and humans. The most common mistakes for humans and the system are in tables 9—11. We find that the artificial errors are substantially the same as human ones with one large exception confusions between backchannel words [acknowledgment words like “uh-huh”] and hesitations. The difference they found, but suspect might be a result of the different transcription guidelines of the training corpus: we see that by far the most common error in the ASR system is the confusion of a hesitation in the reference for a backchannel in the hypothesis. People do not seem to have this problem. [1]: https://arxiv.org/abs/1610.05256 https://arxiv.org/abs/1610.05256
- radarsat1 10y agoVery cool that they investigated this! Thanks, I hadn't read the paper (obviously)
- bendykstra 10y agoThat reminds me of the photocopier that used a very poorly chosen image compression algorithm. It would sometimes substitute similar looking characters to save space. A $7000 figure in a budget might become $9000 and a person looking over the document would have no indication that the '9' had been cut and pasted from somewhere else on the page. Google Voice transcription often makes easily detectable mistakes that I can mentally correct using context. Smarter, context-aware mistakes might be worse, if they are also more difficult to detect.
- eb0la 10y agoKnowing Microsoft this will be part of Cortana in a few weeks. I hope it will be integrated soon with the Speech API as well ( https://msdn.microsoft.com/en-us/library/hh361633(v=office.14).aspx https://msdn.microsoft.com/en-us/library/hh361633(v=office.1... ).
- bpicolo 10y agoIt's not necessarily the case that they can do it in near-real-time (or near real time at reasonable scale). I would expect the true state of the art to take a while? Maybe part of the definition is real-time, but the article doesn't specify that.
- ArkyBeagle 10y agoThey'll still call it Cortana? I rather like Cortana, but it seems to get a lot of hate - comparisons to MS BOB and what not.
- plussed_reader 10y agoDo I have to use Windows to leverage this new software setup?
- raimue 10y ago> [...] a speech recognition system that makes the same or fewer errors than professional transcriptionists. How low would the error rate be for humans that can fully concentrate on listening instead of writing at the same time? Unfortunately, that cannot be tested.
- Kenji 10y agoI have read too many human parity claims that left me disappointed to believe this one. Call me a pessimist or a cynic. I'll be very excited when I have the code running on my machine and when I can compose this comment verbally without a hassle.
- loup-vaillant 10y agoGreat. Now Microsoft has the means to store every Skype conversations indefinitely —it's only text, now. Seriously, great work, but just like facial recognition, this will cut both ways.
- dlubarov 10y agoCompressed speech doesn't take much space anyway. Narrowband AMR uses around 7kbit/s (depending on the desired quality), or ~1 megabyte for a 20 minute call. The quality isn't great, but it's adequate for most purposes, including speech recognition with reasonable accuracy.
- nicklovescode 10y agoIs there a demo or video of them using this? Would enjoy playing with it.
- deleted 10y ago[deleted]
- dfgonzalez 10y agoDid someone put it as SaaS already?
- skoocda 10y agoWe're close to an alpha release of Spreza, which might be relevant to your question. Look us up! dm me if you've got questions
- braindead_in 10y agoVery cool. How do you compare to speechmatics?
- skoocda 10y agoVery similar accuracy, timing and alignment. We don't do speaker diarization at all because the results seem consistently weak, even among the competitors such as Speechmatics. I'd hazard to say our web editor is much better for providing an end-to-end solution, where accuracy and verification matters. If I may ask, what software does your team use at Scribie?
- windlep 10y agoI'll admit I'm not very interested in speech recognition of this nature when it can't disambiguate the speaker. ie. the way Amazon Echo and other voice recognition systems can't tell the difference between a human in the room and the TV. Even when one might be clearly a female voice vs. a male. None of the voice recognition systems on the market learn my voice distinctly from my wife's or sons, and I don't want their speech triggering things on accident (especially my son's), so I don't use any of them. I'll be more impressed when I can restrict Amazon Echo or one of these assistants to ignoring any voice that isn't at least rather similar to my own, not merely recognizing the words I'm speaking.
- gusmd 10y agofor what it's worth, the "OK Google" functionality in my Android phone is trained against my voice and does a pretty good job at rejecting my wife's and coworkers commands.
- sean2 10y agoFor what it's worth, even after some training none of the phones in my office seem to pick up their owner's voice rather than someone else telling their own phone to run a search (we're all mid 30s male). My phone rarely listens to me until I hold down the home button but one guy I know, who has a slow, deep voice, triggers Google to start listening in normal conversation all the time.
- ankitbko 10y agoWell they do have speaker identification. But I don't know how good it is. You may try them out. https://www.microsoft.com/cognitive-services/en-us/speaker-recognition-api https://www.microsoft.com/cognitive-services/en-us/speaker-r... They also are having a private preview of CRIS which supports custom acoustic models https://www.cris.ai/ https://www.cris.ai/
- swagtricker 10y agoTime files like an arrow, but fruit flies like a banana. Wake me up when they can match human recognition of context.
- tempestn 10y agoAt first I wondered whether you meant to type it that way to make a deeper point, but I'm assuming it's just a typo.
- nattyice 10y agoMeaning will always escape us when it comes to language. Not only will there always be a disconnect between the speaker and his or her audience, there will always be a subjective perspective that cannot be tapped into. Can AI ever really be compared to a subjective perspective? Although the article recognizes that perfection has not been assumed, parity might not even be a capacity. Conversation is difficult to measure. Take a look at the philosophical viewpoint of Deconstruction. Food for thought. http://www.iep.utm.edu/deconst/ http://www.iep.utm.edu/deconst/
- aisofteng 10y agoIn that case, it's already 100% accurate. Don't confuse handwaving with science.
- ahmetyas01 10y agoany video or audio to get the idea how close they are?
- EGreg 10y agoIn English, probably.
- chris_st 10y agoPerhaps the folks at Microsoft's Lync (named after this gentleman, [1], no doubt) or maybe it's Skype for Business now could get some of this research. We have this at work (alas), and it does "transcription" of voicemail, which it sends as an email. It's easily 90% wrong, regardless of speaker, unless it's a slightly bad connection, when it's worse. [1] https://www.youtube.com/watch?v=NV9fKUkx76Q https://www.youtube.com/watch?v=NV9fKUkx76Q
- andulus 10y agoI wonder if this success has any help in the advancement of another application neural networks? Do these achievements translate easily to other domains, or it's just an isolated case?
- braindead_in 10y agoI run an human powered transcription service and I get really excited on such news. Typing is the first step of our process (of four) and any ASR system which can generate even around 80% accurate transcript of a file will be incredibly useful. We have tried several systems but unfortunately none have been able to get there yet.
- InYan 10y agoIt is so far only on a marketing par with the ргьфт - in fact the guys from Microsoft received WER (word error rate) comparable with the number of errors that would make a person. But these are inherently different mistakes - people often pass unimportant words while algorithm does not recognize some not-so running or vernacular words.
- jarboot 10y agoHow long do you think it is until captioning companies / TRSs such as Captel downsize significantly because of tech like this?
- aab0 10y agoSupply generates its own demand; by making captioning even cheaper, it can increase the demand for transcription services and people to check it over. There are a lot of podcasts and YT videos that could benefit from transcriptions but it's too expensive now.
- dalys 10y agoI think it's more impressive when you actually hear a sample from the switchboard task: https://catalog.ldc.upenn.edu/desc/addenda/LDC97S62.wav https://catalog.ldc.upenn.edu/desc/addenda/LDC97S62.wav From https://catalog.ldc.upenn.edu/LDC97S62 https://catalog.ldc.upenn.edu/LDC97S62
- mirekrusin 10y ago...still waiting for english to/from dolphin translator.
- mirekrusin 10y agoWouldn't just simple "word after list of words" probability help?
- botw 10y agooff topic but related, is speech-to-text engine in android open sourced? can it work offline entirely?