5 ms·

> The software apparently has an 85 per cent accuracy rate. That number seems really low to me for a task specific system. i.e. it doesn't need to understand

> The software apparently has an 85 per cent accuracy rate. That number seems really low to me for a task specific system. i.e. it doesn't need to understand every possible thing someone could say; just the subset of language used to place orders at McDonald's. For reference (https://paperswithcode.com/sota/speech-recognition-on-librispeech-test-other https://paperswithcode.com/sota/speech-recognition-on-libris...) SOTA models are at ~5% WER for general speech. And besides, who wants to order from an AI that's going to fuck up 15% of the time? Compared to a human that error rate is perhaps on-par, but a human has an internal confidence measurement. We know when we've heard something wrong. Speech systems don't really have that (1). So the AI will just blunder forward with your order. I'd much rather interact with a system that says "Sorry, what was that?" 15% of the time (i.e. the current meat based voice recognition that fast food restaurants use) versus a system where I constantly have to check the screen and tell it "Oh, no, sorry, can you fix the, ummm, uhh, we don't want 20 orders of ketchup packets... Oh, god, no we don't want 40 ketchup packets! No our order isn't done! WAIT!" (1) Yes, you can guess at confidence by measuring the logits, but that doesn't work in practice. It's nowhere near a human's capability to self-measure confidence in our predictions.

sumnuyungi 5y ago
Using your own link, SOTA is 2.6%. WER could be lower for the much smaller vocabulary used in a drive-thru. I'm not sure what you mean by "that doesn't work in practice" re: using logits. Word-level confidence is pretty useful with GCP Speech-to-Text [1] [1] https://cloud.google.com/speech-to-text/docs/word-confidence#:~:text=When%20the%20Speech%2Dto%2DText,number%20between%200.0%20and%201.0 https://cloud.google.com/speech-to-text/docs/word-confidence....
- version_five 5y ago
  > I'm not sure what you mean by "that doesn't work in practice" re: using logits. I suspect they mean they ML models are usually poorly calibrated and that the softmax-over-logits probabilities generally don't reflect actually error rates, so they're tough to use meaningfully for asking people to repeat themselves. Personally, if I have to deal with an automated order system, I'd rather some kind of search tree that let's me traverse it using three (left, right, back) well separated noises and a "dumb" back-end instead of having to pretend a ML system and I are having the meeting of the minds that a voice based discussion implies. I understand that such a system would be hard or impossible to train lay-people to use, but it would be nice to have a "cut the crap" option to let people interface more effectively with the order system and not take part in the charade of a "discussion"
- fpgaminer 5y ago
  > I'm not sure what you mean by "that doesn't work in practice" re: using logits. Like the other comment points out, models today aren't "calibrated" to give that kind of information. More precisely, we aren't training our models to explicitly tell us how confident they are in their predictions. They're simply trained to give the predictions that result in the lowest average error over the training dataset. For example, we can consider the simple task of recognizing the words "yes" or "no". A naive model could return (0%, 100%) all the time (always guess "no") and, if the dataset is balanced, would get a score of 50%. Another naive model could return (50%, 50%) all the time and get the same score, 50%. Yet in practice we'd rather have the latter model because it better expresses that model's level of confidence. The former model, even though it gets the same average error rate, expresses a level of confidence in its answers that isn't there. As of today, we only train models on the overall error rate, so our training methods don't prefer one kind of output over the other. That's why measuring the logits to guesstimate confidence isn't actually a good metric. It just happens to accidentally be one sometimes. A speech recognition model might get to the same WER as a human, but humans are keenly aware of when they didn't hear a word right. That's invaluable information to a food ordering system which can then respond by asking for clarification, rather than blindly following its "best guess" which results in the aforementioned ordering of 40 ketchup packets. And as far as I'm aware there are no loss functions for training confidence measurements into a system, so this is very much an unsolved problem in speech recognition systems.
thrower123 5y ago
15% error is wildly less than I normally get ordering at the McDonalds drive-through. Not quite half the time I have to give my order again at the second window, then again at the third window.
- gambiting 5y ago
  Here in the UK when you go through the drive through at McDs the order appears on the screen next to the microphone, they ask you to confirm that's what you ordered. Never had a wrong order this way. Seems like a very simple solution if something this is isn't standard everywhere.
jeeeb 5y ago
> SOTA models are at ~5% WER for general speech. Do SOTA results measure performance in difficult environments though? Presumably dealing with people talking out of their car window next to a busy road would be a lot more difficult than dealing with a relatively clear audio recording. EDIT: It looks like the linked results are for an audio book dataset. That seems like an optimal environment where you’re going to get clear enunciation with minimal background noise.
ampdepolymerase 5y ago
I never understood why the speakers in drive-throughs are so shitty. It is like the low resolution CCTV camera recordings released by police that are close to useless for identifying perpetrators. What is the point? This is 2021, DSP technology is a commodity on the orders of a few cents to a few dollars per chip. There is absolutely no reason for an intercom to sound so bad, ditto for CCTVs. A commercial grade surveillance/intercom system that cost 3-4 figures should at the very least perform better than consumer tech like Amazon Ring.
brokenmachine 5y ago
All that matters is what is cheaper for McDs. Meat-based order processors are relatively expensive. Like, way more expensive than 15% incorrect orders.

sumnuyungi 5y ago

Using your own link, SOTA is 2.6%. WER could be lower for the much smaller vocabulary used in a drive-thru. I'm not sure what you mean by "that doesn't work in practice" re: using logits. Word-level confidence is pretty useful with GCP Speech-to-Text [1] [1] https://cloud.google.com/speech-to-text/docs/word-confidence#:~:text=When%20the%20Speech%2Dto%2DText,number%20between%200.0%20and%201.0 https://cloud.google.com/speech-to-text/docs/word-confidence....

version_five 5y ago
> I'm not sure what you mean by "that doesn't work in practice" re: using logits. I suspect they mean they ML models are usually poorly calibrated and that the softmax-over-logits probabilities generally don't reflect actually error rates, so they're tough to use meaningfully for asking people to repeat themselves. Personally, if I have to deal with an automated order system, I'd rather some kind of search tree that let's me traverse it using three (left, right, back) well separated noises and a "dumb" back-end instead of having to pretend a ML system and I are having the meeting of the minds that a voice based discussion implies. I understand that such a system would be hard or impossible to train lay-people to use, but it would be nice to have a "cut the crap" option to let people interface more effectively with the order system and not take part in the charade of a "discussion"
fpgaminer 5y ago
> I'm not sure what you mean by "that doesn't work in practice" re: using logits. Like the other comment points out, models today aren't "calibrated" to give that kind of information. More precisely, we aren't training our models to explicitly tell us how confident they are in their predictions. They're simply trained to give the predictions that result in the lowest average error over the training dataset. For example, we can consider the simple task of recognizing the words "yes" or "no". A naive model could return (0%, 100%) all the time (always guess "no") and, if the dataset is balanced, would get a score of 50%. Another naive model could return (50%, 50%) all the time and get the same score, 50%. Yet in practice we'd rather have the latter model because it better expresses that model's level of confidence. The former model, even though it gets the same average error rate, expresses a level of confidence in its answers that isn't there. As of today, we only train models on the overall error rate, so our training methods don't prefer one kind of output over the other. That's why measuring the logits to guesstimate confidence isn't actually a good metric. It just happens to accidentally be one sometimes. A speech recognition model might get to the same WER as a human, but humans are keenly aware of when they didn't hear a word right. That's invaluable information to a food ordering system which can then respond by asking for clarification, rather than blindly following its "best guess" which results in the aforementioned ordering of 40 ketchup packets. And as far as I'm aware there are no loss functions for training confidence measurements into a system, so this is very much an unsolved problem in speech recognition systems.

thrower123 5y ago

15% error is wildly less than I normally get ordering at the McDonalds drive-through. Not quite half the time I have to give my order again at the second window, then again at the third window.

gambiting 5y ago
Here in the UK when you go through the drive through at McDs the order appears on the screen next to the microphone, they ask you to confirm that's what you ordered. Never had a wrong order this way. Seems like a very simple solution if something this is isn't standard everywhere.

jeeeb 5y ago

> SOTA models are at ~5% WER for general speech. Do SOTA results measure performance in difficult environments though? Presumably dealing with people talking out of their car window next to a busy road would be a lot more difficult than dealing with a relatively clear audio recording. EDIT: It looks like the linked results are for an audio book dataset. That seems like an optimal environment where you’re going to get clear enunciation with minimal background noise.

ampdepolymerase 5y ago

I never understood why the speakers in drive-throughs are so shitty. It is like the low resolution CCTV camera recordings released by police that are close to useless for identifying perpetrators. What is the point? This is 2021, DSP technology is a commodity on the orders of a few cents to a few dollars per chip. There is absolutely no reason for an intercom to sound so bad, ditto for CCTVs. A commercial grade surveillance/intercom system that cost 3-4 figures should at the very least perform better than consumer tech like Amazon Ring.

brokenmachine 5y ago

All that matters is what is cheaper for McDs. Meat-based order processors are relatively expensive. Like, way more expensive than 15% incorrect orders.