4 ms·
The actual paper has a section on error analysis that is particularly enlightening: https://arxiv.org/abs/1610.05256 https://arxiv.org/abs/1610.05256 On the Ca
by Eridrus 10y ago
The actual paper has a section on error analysis that is particularly enlightening: https://arxiv.org/abs/1610.05256 https://arxiv.org/abs/1610.05256
On the CallHome dataset humans confuse words 4.1% of the time, but delete 6.5% of words, most commonly deleting the word "I".
Their ASR system confuses 6.5% of words on this dataset, but only deletes 3.3% of words, so depending on how you view this their claim about being better than humans isn't definitely true, if you consider the task to be speech recognition, rather than transcription.
Also, while the overall "word error rate" is lower than humans it's not clear if this is because the transcription service they used is not seeking perfect output, but rather good enough output and the errors the transcription service makes may not be as bad as the errors the ASR system makes in terms of how well you can recover the original meaning from the transcription.
It's clearly great work, but reaching human parity is marketing fluff.
- AnimalMuppet 10y agoI'm not an expert on this at all, but my suspicion is that humans are more tolerant of missing words than we are of mistaken words. Especially a word like "I", we just assume it if it's missing. So if this speech recognition is for creating a transcript for humans, this (in my uneducated opinion) isn't as good as humans do, at least not yet.
- rhizome 10y agoIf nothing's changed in the past 15 years, anything under 95% accuracy is not accurate enough for automation without human intervention (copy editing).
- joe_the_user 10y agoI was recently at a bar where they showed a movie with incomprehensible subtitles (English to English). I assume this was because they skimped and bought automatic subtitling. I think one important aspect is while humans miss words, they often get the sentence meaning correct. When computers miss words, they tend to substitute words that sound similar. That's readable if you have time but not necessarily as a stream of text going by...
- cloudjacker 10y ago> I was recently at a bar where they showed a movie with incomprehensible subtitles (English to English). I assume this was because they skimped and bought automatic subtitling. I don't think that is possible. Like, worse subtitling isn't even an option.
- john_oshea 10y ago"Trainspotting" would be an interesting challenge: <https://www.theguardian.com/books/2008/may/31/irvinewelsh> https://www.theguardian.com/books/2008/may/31/irvinewelsh>
- WildUtah 10y agoDon' You Go Rounin' Roun To Re Ro: http://www.nbc.com/saturday-night-live/video/british-movie/n13054 http://www.nbc.com/saturday-night-live/video/british-movie/n...
- ChrisClark 10y agoIt might have also been a 'bootleg' DVD from China or downloaded a version from there. I've had quite a few where they had done a horrible job translating into Chinese for the subtitles already, and then did a literal machine translation back to English. The character on the screen said "Hello", the English subtitle said "You good." Which would be the literal translation of nihao.
- Arkaad 10y agosee also the infamous Star Wars Episode 3 Chinese bootleg "Do no want".
- joe_the_user 10y agoIt English subtitling on an movie in English, as they often have at bars.
- codekansas 10y ago
- nshm 10y agoFrom the paper eight different recognizers combined together approached results of a single person reviewed once. Not really a parity.
- rattray 10y agoIsn't it substantially simpler + cheaper to combine 8 computer programs than to combine >1 people?
- djsumdog 10y agoLooks like they used various neural networks and trained based on data they currently already transcribe professionally. So it had to be trained and only represents a subset of people who speak and have their works transcribed at Microsoft events. Being Microsoft, I'm sure the place is diverse, but there's no mention in the paper on accents or dialects that I can see (might just be missing it).
- deleted 10y ago[deleted]
- xdh168 10y agoThe conversational speech has been very difficult. The switchboard task has been around for over 20 years. We used the exact same published data and evaluation protocols. The Human Parity claim on the switchboard task is as scientific as we can be based on the best knowledge of ours.
- derekja 10y agoAh, you finally made a hackernews account! Congrats, XD, that's a great milestone you guys hit.