24 ms·
For those not familiar with the NIST 2000 Switchboard evaluation[1] it is a series of 8kHz audio recordings (ie. crappy phone quality samples) of a conversation
by josho 10y ago
For those not familiar with the NIST 2000 Switchboard evaluation[1] it is a series of 8kHz audio recordings (ie. crappy phone quality samples) of a conversation, including things like "uhhuh" and other pause words. So, 6% seems pretty good.
[1] http://www.itl.nist.gov/iad/mig/tests/ctr/2000/h5_2000_v1.3.html http://www.itl.nist.gov/iad/mig/tests/ctr/2000/h5_2000_v1.3....
- rhizome 10y agoWhen I worked at a speech recognition startup in the late 90s I was told that anything over 5% is essentially useless for no-touch recognition.
- M_Grey 10y agoThat's very technically impressive, but 6% is still a long way from where it needs to be as anything like a primary interface. Even 1% is pretty high when you consider how many words you can utter in a few minutes, and how many errors that would generate. Edit: For comparison: http://www.utdallas.edu/~assmann/hcs6367/lippmann97.pdf http://www.utdallas.edu/~assmann/hcs6367/lippmann97.pdf
- mintplant 10y agoHow does it compare to the typical human error rate?
- M_Grey 10y agoBest I can find is section four here: http://www.utdallas.edu/~assmann/hcs6367/lippmann97.pdf http://www.utdallas.edu/~assmann/hcs6367/lippmann97.pdf It seems to be significantly less than 1% - 1.6%
- funkymike 10y agoRecognizing the words is just the first step. Getting meaning out of those words is what really counts. When on a telephone call I may miss a percentage of what was said, due to poor audio quality, the speaker's accent, etc., but based on context I can still understand the message the speaker is trying to convey most of the time. Transcribing each word with near-perfect accuracy is unnecessary if the layers above that can handle it.
- M_Grey 10y agoThat's true, but I wonder if computers will exceed out technical accuracy first, or begin to actually "understand" things first? I suspect the former.
- icebraining 10y agoThey don't need to actually understand - whatever that means - they can apply statistical models based on phonetic distances and large corpora of dialogue.
- M_Grey 10y agoHas that worked so far?
- copperx 10y agoOf course it has. We went down from 40% to 6.9% error rate in 26 years. It may take a couple of decades to get to 0.1%.
- Fricken 10y agoGoogle voice search will make several guesses as to what I said. It will sometimes enter a nonsensical, but similar sounding phrase to what I intended into the text box, but will then figure out that it's first guess is nonsense and return the correct search result. So just based on observation I would say it works pretty good. On mobile I almost always use voice search, and unless I'm searching for an italian name or something weird it almost always hears me correctly. Even in a noisy pub.
- pesenti 10y agoThe human error rate on this task is 4% (https://pdfs.semanticscholar.org/387e/7349b8e31e316c2a7380603d1b6c0b1417c8.pdf https://pdfs.semanticscholar.org/387e/7349b8e31e316c2a738060...). So you are basically saying that telephones are a useless interface... What is amazing is how consistently people overestimate human performance.
- Swizec 10y agoCompared to voice over internet? Yeah, it's pretty terrible.
- M_Grey 10y agohttp://www.utdallas.edu/~assmann/hcs6367/lippmann97.pdf http://www.utdallas.edu/~assmann/hcs6367/lippmann97.pdf I did say "primary interface" as well, which definitely rules out a mediocre phone connection.
- contextfree 10y agoDoes using speech multimodally with other input methods (touch, gesture, pen, clicker, game controller, even keyboard - may seem silly but speech is potentially faster at some tasks) still count as "primary" if the other input method is used supplementarily to help disambiguate?
- M_Grey 10y agoI'd say, and let be clear that I know and accept this to be a relatively arbitrary distinction on my part, that for an input to be "primary" it needs to be able to stand alone (touchscreen input, keyboard and mouse, etc). That's not to say that it must stand alone, but that when all other options are off the table, that would be your preferred method for text/data entry. That's a troublesome definition though at least in part for reasons you've brought up or alluded to, which is that a multimodal approach is pretty clearly going to dominate. That said, speech recognition at least stands to replace the keyboard for say, the author of a book or article, if it's good enough.
- 10y ago
- fareesh 10y agoDoes the word error rate account for context? For example, if I heard "clown" instead of "cloud", I would probably treat it as something I misheard it in the context of the sentence and correct my understanding accordingly. I guess what I am trying to ask is that if the system reaches 4% (human error rate for same sample), can it really be considered to be on-par?
- nl 10y agoIt measures against what humans write down when they listen. Humans generally use context, so I'd expect that they would record "cloud" if that was appropriate in context. The Microsoft system also does this: it uses language modelling to attempt to model what word is more likely in a given context. This gives them a 18% word error rate reduction (see section 7 of the paper: http://arxiv.org/pdf/1609.03528v1.pdf http://arxiv.org/pdf/1609.03528v1.pdf).
- gok 10y agoAll language models attempt to model what word is more likely in a given context :) What is somewhat unusual about this approach is they use recurrent neural networks for the language modeling, as opposed to more traditional approached like a backoff n-gram model.
- nl 10y agoMost modern language modelling is RNN based now; eg: https://arxiv.org/abs/1602.02410 https://arxiv.org/abs/1602.02410 (which I think is SOTA).
- cs702 10y agoJudging by my everyday interactions, a 6% error rate is lower than human error rates in casual conversation. People regularly ask each other, "sorry, what did you say?", "wait, what did she say?", "would you repeat that please?", "huh?", etc.
- chipperyman573 10y agoThe difference is that people are OK with a human asking for clarification, but systems like Siri need to have a near-zero error rare before people will consider them good (a person who has to repeat themselves once every 20 times will consider it bad, or at least not good enough)
- a3_nm 10y agoI'm not sure people expect super-human performance out of Siri. An important difference is that a human who doesn't understand will say so, and ask to repeat the relevant part (or to choose between two alternatives), conversationally; or it will pick an interpretation which is not the intended one but was an understandable misunderstanding. Contrast this with speech recognition, which will often substitute words that are nonsensical in context, making it look silly from a human perspective...
- cptskippy 10y agoExactly, when SR has a low confidence level it needs to ask for you to repeat yourself. Not just choose the highest confidence match and hope for the best.
- copperx 10y ago> a series of 8kHz audio recordings (ie. crappy phone quality samples) I worked in speech recognition for a bit and 8kHz was the standard audio rate for recordings. No one saw this as an impediment; and it really isn't. 8kHz is able to capture up to 4kHz frequencies and the fundamental frequencies of speech are MUCH lower than 4kHz.
- pps43 10y agoAt 8 kHz sampling rate, you can tell f from s only by context (they sound the same).
- YeGoblynQueenne 10y ago>> 6% seems pretty good. That's 6% on the NIST dataset. Typically, results get much worse on real-world datasets, not least because trying to get good results on the same dataset year after year leads to subtle bias. Don't forget this is a dataset that's been around for 16 years, now.