3 ms·

I wonder why Google is doing it like this. Sending an audio file of your voice over the wire seems like unnecessary overhead. From an API perspective I'd much

by dstein 16y ago

I wonder why Google is doing it like this. Sending an audio file of your voice over the wire seems like unnecessary overhead. From an API perspective I'd much rather have client-side speech-to-text built into the browser (accessible via a JavaScript API). I think they're already doing STT client-side in Android, so what's the hold-up with embedding it in Chrome and letting web developers go nuts with voice-enabling their web apps?

tel 16y ago
I'm not familiar with Android, so I'm not sure about the capabilities of its client side speech recognition algorithms, but the short answer is that competent speech to text is currently a very, very intensive problem. SR is roughly divided into acoustic and language modeling. The acoustic model proposes words that might have been said given some chunk of speech and the language model tells you what the most likely actual word is given what's been said. The acoustic model can be solved in a large number of ways — though production technologies use very large hidden Markov models — but decoding a word sequence from speech might scale like O(knm^2) with n being the size of your vocabulary (often large), m the complexity of the acoustic model (# of phonemes modelled, perhaps) and k the number of acoustic frames. The n at the very least can be parallelized (embarrassingly), but the m and k cannot. The language model involves a search through an exponential search space of orderings of words in the vocabulary (n^l choices, but l is also unknown). Anything sophisticated also will have an incredibly large (in memory) model as it has to have parameters across words, pairs of words, triples of words, grammatical categories, topics, etc. etc. Solving both of these problems well simultaneously is not a task for a consumer computer. Speedy algorithms with small vocabularies and simple models exist and are implemented (Dragon Naturally Speaking, for instance) but Google didn't go and record a million hours of GOOG-411 to produce Dragon's technology over again. --- Finally, there's a lot of work on front end signal processing in SR. Before you get into acoustic and language modeling, you often transform your input into another representation (often spectral components from sliding 10ms frames). A growing camp in SR research involves finding sparse front end representations of speech though. If the client-side software is capable of quickly computing a sparse representation of the speech, that could dramatically reduce the latency and bandwidth issues.
- tel 16y ago
  By the way, the reason Google is willing to spend this computational effort for your convenience is probably the same as GOOG-411. They are definitely recording every translation they do in order to use as a huge training corpus later. The best data is more data.
jerf 16y ago
We don't know what algorithm they are using or how computationally intensive it is. It may be the case that it would choke anything but the most powerful modern desktops/laptops; they can afford to throw 32 cores at the problem for a fraction of a second where you can't, or it may be backing to an arbitrarily large pre-computed lookup table of some type stored in memory on these machines. I doubt it's that bad, I just made a number up, but it could be bad. And cell phones may not be able to do anything with it at all. Using a server gives them the freedom to use more power, automatically collect the results for more learning, and not need to worry about compatibility. (Or reverse engineering.)
- tel 16y ago
  To throw out another interesting number, I often hear people complaining about finding ways to fit hundreds of gigabytes of language models into memory in order to make search complete before heat death. That's really just not a cell phone level of computation yet.
Groxx 16y ago
Easier possible-answer than the others so far: if they have the audio, they can train their engine.
MatthewPhillips 16y ago
Android STT isn't client side, that's why you get a "connection error" message if you turn data off and try it. STT is a problem best solved with big iron.
ezy 16y ago
There's actually two main reasons, related to each other: In order to do the kind of open domain recognition google (and others) do here, you need data. Lots of data. That implies that the models themselves are very large, and (generally) require more CPU to recognize with. You can't do that on a mobile device, mainly because of the space issue, but fancy new algorithms probably consume more CPU than even the latest mobile hw can manage[1] The second, related issue is that, again, in order to do this kind of open domain recognition, you need to constantly improve the models[2]. Even a moderately sized set of models would be a pain in the rear to send back up to every android device using the system. [1] You can typically do the front-end feature processing (and adaptation) on the device, and most vendor's solutions end up doing a bit of that. [2] Even for speaker dependent modelling (dictation), you'd probably want to associate a key with the model and adapt that model in the cloud rather than sending model updates back to the device because of the space and transmit times.
anigbrowl 16y ago
I think they're already doing STT client-side in Android Not client-side, or I would use it more often (lot of dead spots near my house due to the peculiar geography).

by dstein 16y ago

tel 16y ago

I'm not familiar with Android, so I'm not sure about the capabilities of its client side speech recognition algorithms, but the short answer is that competent speech to text is currently a very, very intensive problem. SR is roughly divided into acoustic and language modeling. The acoustic model proposes words that might have been said given some chunk of speech and the language model tells you what the most likely actual word is given what's been said. The acoustic model can be solved in a large number of ways — though production technologies use very large hidden Markov models — but decoding a word sequence from speech might scale like O(knm^2) with n being the size of your vocabulary (often large), m the complexity of the acoustic model (# of phonemes modelled, perhaps) and k the number of acoustic frames. The n at the very least can be parallelized (embarrassingly), but the m and k cannot. The language model involves a search through an exponential search space of orderings of words in the vocabulary (n^l choices, but l is also unknown). Anything sophisticated also will have an incredibly large (in memory) model as it has to have parameters across words, pairs of words, triples of words, grammatical categories, topics, etc. etc. Solving both of these problems well simultaneously is not a task for a consumer computer. Speedy algorithms with small vocabularies and simple models exist and are implemented (Dragon Naturally Speaking, for instance) but Google didn't go and record a million hours of GOOG-411 to produce Dragon's technology over again. --- Finally, there's a lot of work on front end signal processing in SR. Before you get into acoustic and language modeling, you often transform your input into another representation (often spectral components from sliding 10ms frames). A growing camp in SR research involves finding sparse front end representations of speech though. If the client-side software is capable of quickly computing a sparse representation of the speech, that could dramatically reduce the latency and bandwidth issues.

tel 16y ago
By the way, the reason Google is willing to spend this computational effort for your convenience is probably the same as GOOG-411. They are definitely recording every translation they do in order to use as a huge training corpus later. The best data is more data.

jerf 16y ago

We don't know what algorithm they are using or how computationally intensive it is. It may be the case that it would choke anything but the most powerful modern desktops/laptops; they can afford to throw 32 cores at the problem for a fraction of a second where you can't, or it may be backing to an arbitrarily large pre-computed lookup table of some type stored in memory on these machines. I doubt it's that bad, I just made a number up, but it could be bad. And cell phones may not be able to do anything with it at all. Using a server gives them the freedom to use more power, automatically collect the results for more learning, and not need to worry about compatibility. (Or reverse engineering.)

tel 16y ago
To throw out another interesting number, I often hear people complaining about finding ways to fit hundreds of gigabytes of language models into memory in order to make search complete before heat death. That's really just not a cell phone level of computation yet.

Groxx 16y ago

Easier possible-answer than the others so far: if they have the audio, they can train their engine.

MatthewPhillips 16y ago

Android STT isn't client side, that's why you get a "connection error" message if you turn data off and try it. STT is a problem best solved with big iron.

ezy 16y ago

There's actually two main reasons, related to each other: In order to do the kind of open domain recognition google (and others) do here, you need data. Lots of data. That implies that the models themselves are very large, and (generally) require more CPU to recognize with. You can't do that on a mobile device, mainly because of the space issue, but fancy new algorithms probably consume more CPU than even the latest mobile hw can manage[1] The second, related issue is that, again, in order to do this kind of open domain recognition, you need to constantly improve the models[2]. Even a moderately sized set of models would be a pain in the rear to send back up to every android device using the system. [1] You can typically do the front-end feature processing (and adaptation) on the device, and most vendor's solutions end up doing a bit of that. [2] Even for speaker dependent modelling (dictation), you'd probably want to associate a key with the model and adapt that model in the cloud rather than sending model updates back to the device because of the space and transmit times.

anigbrowl 16y ago

I think they're already doing STT client-side in Android Not client-side, or I would use it more often (lot of dead spots near my house due to the peculiar geography).