14 ms·
Google’s new voice recognition system works instantly and offline (Pixel only)
- melling 8y ago"But it’s sort of funny considering hardly any of Google’s other products work offline. Are you going to dictate into a shared document while you’re offline? Write an email? Ask for a conversion between liters and cups? You’re going to need a connection for that!" While offline, you might write email drafts, your blog, or even a book: https://medium.com/@augustbirch/what-i-learned-writing-an-entire-novel-on-my-phone-f1655d09b00b https://medium.com/@augustbirch/what-i-learned-writing-an-en... What's missing is the ability to make edits using your phone. You can probably speak at over 100 words a minute but then you need to stop to bring up the software keyboard.
- semi-extrinsic 8y agoFWIW Google Translate (including the "translate from picture" feature) is an example of a product that has had offline option for quite some time. You have to tell it to download for each language pair IIRC.
- maxxxxx 8y agoGoogle Maps has an excellent offline mode on iOS and Android. I wish Apple Maps had that too.
- int_19h 8y agoIf only it didn't force-expire downloaded maps after a while...
- maxxxxx 8y agoI get updates every 30 days. Do they expire if you don’t update?
- gruez 8y agoYes
- maxxxxx 8y agoThat's stupid. I do offline maps mainly for the times when I don't have a good connection. It makes no sense to lose the maps when you are spending an extended amount of time off-grid. That's exactly when you need offline maps.
- toomuchtodo 8y agoLicense issues. Wouldn't happen if all the data was OpenStreetMap and Google data only.
- sliken 8y agoI suspect very very few people pay to have a smart phone and don't get it on WAN or wifi at least once a month.
- int_19h 8y agoIf I remember correctly, the maps aren't auto-updated by default; you do get notifications, but those are too easy to ignore (esp. since Maps is chatty in general). And then for someone on an expensive metered or slow connection - which is a lot of people in the developing countries - they might not want auto-update at all. So if they didn't notice expiration, they'll find out that their maps aren't there next time they try to navigate offline.
- mellow-lake-day 8y agoFor the record it wasn't always this way, the last couple of years though they have made a lot of improvements on this front. I think it may have something to do with Google's "next billion of devices" being in countries with bad connectivity. With that said I especially like the Google Maps offline features which have been added recently. You can even have it calculate driving directions completely offline if you have the starting and ending addresses.
- ehsankia 8y agoThe offline aspect is hardly the main draw here though. As mentioned earlier in the article, the latency reduction is huge. Another aspect they didn't really cover is privacy implications. Lastly, you may not be offline, but dodgy connections can also be a pain if you need a stable stream of packets going back and forth.
- ksec 8y ago>As mentioned earlier in the article, the latency reduction is huge. Well, on macOS offline voice recognition is actually much slower than online. Not to mention the choice of words and Vocab is quite limited. I love to get an offline version, but so far every online version seems to be better.
- i_cant_speel 8y agoI don't think that means this one will be slower. Google has always been the leader in the voice recognition field. I'll bet they'll do offline processing better than Apple.
- ksec 8y ago>Google has always been the leader in the voice recognition field Interestingly may be only for English. In my experience Apple is doing far better in Japanese, Chinese ( Both Mandarin and Cantonese )
- pault 8y agoI refuse to put an amazon/apple/google surveillance device in my home, so I am very interested in a DIY digital assistant device. I'm aware of a few options but it seems like offline voice recognition is always a little sub-par. I am really looking forward to the day when an offline, open source digital assistant can compare in quality to a proprietary/cloud device.
- snaky 8y agoIt shouldn't be offline, using recognition online but at your own cloud would be the way too.
- bufferoverflow 8y agoIt means google doesn't need to pay for all the servers busy doing speech recognition. They shifted that work to the user's device.
- jasonvorhe 8y agoIt's hilarious how they can't do anything right. If it's in the cloud it's evil because Google, if it happens on device it's evil because Google.
- bufferoverflow 8y agoWho said it's evil?
- jfoster 8y agoMost of their products work offline and sync when a connection is regained. That includes Google Docs and Gmail.
- berbec 8y agoThis will be great when ported to Lineage!
- Someone1234 8y agoJust to be clear: This has nothing to do with "Wake Words" (e.g. OK Google, Alexa, Hey Siri, etc) which have always been handled offline/locally. This is translating what you said after the wake word from voice to text on the local [Pixel] hardware rather than sending it into Google's Cloud. The biggest benefits here are speed and reliability. It could also handle some actions offline.
- penagwin 8y agoAnother benefit is privacy, this eliminates an entire set of potentially personal data from being handed off to Google.
- Someone1234 8y agoI'm unclear on if this moves the privacy needle. It says they do offline translation, but they still may attempt to send the audio clip to compare with the resulting text translation. It could be used to improve privacy, I just don't know if it will be used that way.
- penagwin 8y agoTo me it's clear that this is in it's early phases, and as it's only available on Pixel devices and not to the general public I think it's safe to say this is part of it's testing. However, as you said, if this is always a requirement then it doesn't affect privacy at all, which to me would be a real shame but this is Google after all. We just have to wait and see for now.
- usrusr 8y agoOn the other hand, when you can transcribe locally, uploading whole days worth of eavesropping would not cause a noticeable spike in traffic. I'd consider it more a lateral change than an improvement.
- gbrown 8y agoMy first thought as well. Typed from my s̶u̶r̶v̶e̶i̶l̶l̶a̶n̶c̶e̶ ̶d̶e̶v̶i̶c̶e̶ smartphone.
- dragonwriter 8y ago> But it’s sort of funny considering hardly any of Google’s other products work offline. I dunno, Android and a lot of Google's mobile apps that aren't about online communication work fine offline. Actually, a lot of the online communications ones do too, as much as is even conceivable, they just don't transmit and receive offline, because, how would they?
- modeless 8y agoGoogle AI blog: https://ai.googleblog.com/2019/03/an-all-neural-on-device-speech.html https://ai.googleblog.com/2019/03/an-all-neural-on-device-sp... arXiv: https://arxiv.org/abs/1811.06621 https://arxiv.org/abs/1811.06621
- gok 8y agoThese are much much better links
- p1esk 8y agoInteresting that they're using RNN transducer. I thought everyone's moved to CNN lately.
- pakl 8y agoFeedforward CNNs cannot tolerate as much noise and real-world variability as RNNs.
- p1esk 8y agoCitation?
- robbomacrae 8y agoThere is this new type of trolling where someone just puts in bare minimal counterarguments and asks for citations knowing full well it's labour intensive to do so. If your not trolling.. your not doing a good job of avoiding these comparisons. The onus should be on you here to provide citations first that dissprove the person your replying to.
- fwip 8y agoClosely related to sea-lioning, in my opinion.
- 8y ago
- dotdi 8y agoCall me cynical but I cannot picture Google not tapping into everything you run through their voice recognition software, even if it does work offline. Doesn't mean it won't phone home later.
- penagwin 8y agoRight, but once this is implemented there is no excuse for audio to be "sent home". Of course there will still be usage analytics, etc. But it does increase privacy to some degree, especially when compared to sending all audio after a certain phrase is mentioned.
- snaky 8y agoGoogle does not need any excuse.
- penagwin 8y agoThey do need an excuse, however most likely it will be "to further improve our models with more data", which could certainly be true, much to all of our distaste. We have to wait and see. I'm sure we all look forward to a completely offline solution.
- lallysingh 8y agoFor what? They only really make their money on ads for things you're actively searching for. Everything else they have in ads works rather poorly in the text world. Trying to interpret interests out of task-driven voice commands is way beyond their capabilities. But, enough of that. I'm holding out until decent voice dictation is standard everywhere and a well understood engineering problem with good open source implementations. Mostly so I don't have to type address into my car's GPS.
- intopieces 8y agoYou might be waiting a while. The problem you describe - with GPS - is an acoustic modeling problem, and a very difficult one at that. Cars come in many shapes and make lots of odd noises and various speeds, not to mention these devices are mounted in many places in relation to the speaker.
- jsight 8y agoDidn't they advertise something like this a few years ago? I seem to remember trying it and finding that it didn't really work as well as the online recognition at the time. EDIT: Looks like something was added in Jelly Bean: https://stackoverflow.com/questions/17616994/offline-speech-recognition-in-android-jellybean https://stackoverflow.com/questions/17616994/offline-speech-...
- tonywastaken 8y agoDictation works offline on iPhone since iOS10
- tacomonstrous 8y agoOffline transcription has been available on Android as well. This is an announcement of a faster, more accurate model.
- hathawsh 8y agoI just switched my Pixel 1 to airplane mode and tried voice input. Sure enough, it worked offline and it was fast! Very impressive work. (I've tried that before, but in the past it could only understand a few special phrases.) I suppose this new feature came with the security update my phone downloaded a few days ago. There are lots of ways to spin this, but I see it as a significant improvement for any app that could benefit from voice input. It's immediate and not susceptible to network glitches. The benefit for Google, IMHO, is primarily more sales of updated Android devices.
- tacomonstrous 8y agoUnless you very recently (meaning today) accepted a download of a new language pack for English, it's likely just the old model, which is perfectly functional, while not being as accurate as the online version.
- legohead 8y agobeen using Google Voice for several years now for most of my communications in text, email, slack, whatever (only on phone, of course). it is quite good, and very fast. but it's still not there. it has trouble with nuances like "call" vs "called" -- can't hear that suffix very well in regular speech. for me, it also has a really hard time with pronouns. many times I'll start off with regular speech, go to look at what was transcribed and notice a couple errors that would make me look like a fool, backspace the whole thing, and then repeating it all gain in a very robot like voice. it's almost there.
- reificator 8y ago> been using Google Voice for several years now for most of my communications in text, email, slack, whatever (only on phone, of course). Just a heads up, Google Voice is the name of a product that offers telephony service including SMS, and has been around for a decade or so. https://www.google.com/voice https://www.google.com/voice
- lostmsu 8y agoAt the same time pre-pixel phones get features stripped. "OK Google" now requires phone to be awake and unlocked, or plugged in to work.
- Causality1 8y agoFinally. Over the past year or so I've noticed significant increases in the voice recognition lag across a handful of devices and across multiple wireless carriers.
- shereadsthenews 8y agoHrmm, Gboard only? Does it mean they don't/can't use this model for voice commands? I do sometimes dictate messages to my phone but my main use of Android voice recognition is Android Auto commands to navigate or play music.
- moron4hire 8y agoThis is great. I've been working on voice systems for VR and AR applications. On the HoloLens, it's a dream once you have your entire interface speech enabled. Can't wait to start porting to Android. Daydream and ARCore apps are going to see a huge improvement.
- openloop 8y agoCool.
- jacksmith21006 8y agoImpressive. Never thought the slight lag was an issue. But bet I now will. It is a different feel to have real time recognition.
- adzm 8y agoDoes the Pixel have some specific hardware that this uses, or is it simply limited to Pixel to limit the rollout? I am curious if I should get my hopes up to see this on gboard with non-Pixel Android devices.
- joshvm 8y agoThe Pixel 2+ does have a coprocessor for compute workloads (the Visual Core). However users here have reported this working on a Pixel 1, which doesn't have that chip. The Verge says it may reach other devices later. It sounds like it's both better than the old dictation model, and significantly smaller.
- davidw 8y agoCall me when it can figure out my wife's Italian name, pronounced correctly :-(
- davidw 8y agoDownvoter(s): sorry, but it's true. Google can't figure out my wife's name, which is pretty freaking lame, as she's the person I interact with the most in terms of sending messages and emails and whatnot.
- convivialdingo 8y agoPerhaps check if this could work for you? https://www.google.com/amp/s/www.howtogeek.com/340108/how-to-add-phonetic-names-to-contacts-in-android/amp/ https://www.google.com/amp/s/www.howtogeek.com/340108/how-to...
- flukus 8y agoFinally google has caught up to 1997: https://en.wikipedia.org/wiki/Dragon_NaturallySpeaking https://en.wikipedia.org/wiki/Dragon_NaturallySpeaking Sure it might work better now, but that's expected when computers are much more powerful than a pentium 100 with 32MB of RAM. Uploading voice to google servers for processing was always just a data grab.
- gattilorenz 8y agoWell, try to train your Dragon (no pun intended) and then let another person speak, possibly someone with a different accent. GBoard works somewhat reliably with 0 training and my Italian accent, that's an apple to oranges comparison.
- flukus 8y ago> Well, try to train your Dragon (no pun intended) The training was in part due to the hardware limitations. Some versions let you skip the training or have only a small training session and let it learn as it went, I'd be surprised if googles systems weren't doing some sort of continuous learning too. > and then let another person speak, possibly someone with a different accent. It had profiles to handle that, although it needed training for each profile. With google does it automatically switch profiles or does it mess up what it's learned about you're voice? It's very opaque. > GBoard works somewhat reliably with 0 training and my Italian accent, that's an apple to oranges comparison. I haven't tried the new system but all of google voice recognition has struggled with my Australian accent to the point of being unusable.
- DSingularity 8y agois more than just newer hardware. These CNN/RNN based methods are vastly more robust than older bayessian/markov based methods.
- flukus 8y ago> These CNN/RNN based methods are vastly more robust than older bayessian/markov based methods. They were invented in the 80's, they weren't used because of limited processing power.
- tlepsh 8y agoWhat's so special about it? Just tried this on the BlackBerry keyboard and there it works instantly without being connected to the internet as well.
- bad_user 8y agoAI systems that are able to work offline are great for privacy. The thought that every interaction with my phone is being streamed in realtime to a third party server freaks me out. Kudos to Google for working on this.
- eternal_virgin 8y agoI hope they add this to the Web Speech API in Chrome. It doesn't have punctuation right now and that's a killer for me.
- amelius 8y agoThey can still send the information they collected from your microphone later, when you connect to the internet ... You want an open source solution, not just an offline solution.
- bad_user 8y agoEven so, offline AI solutions have been piss poor and Google moving the state of the art in spite of their vested interest in keeping people online is a good thing. Yes, we want an open source solution, but I'm not going to work on it. So who's going to work on it? Are you? In absence of resources working towards the ideal, I'll applaud any step in the right direction.
- hipitihop 8y agoTo be fair there are others that have been pushing the needle in this space for considerable time. The standout for me is https://snips.ai https://snips.ai Offline, multiple platforms, multiple languages, many parts open source and more oss parts in the pipeline. While certainly not currently aimed at dictation, but instead assistant building and automation. In this space on device speed, privacy & offline are critical. In the case of Snips "piss poor" falls short from the reality of what I have experienced YMMV. Nonetheless we all benefit from this progress
- dep_b 8y agoIt would be nice if Siri would at least allow me to turn cellular data back on with a voice command. Turn-to-turn navigation tends to consume a lot of data when I'm abroad using a temporary SIM so I drive without network connection on offline maps but that kills Siri meaning I can't do anything anymore without touching my phone.
- zulln 8y agoDisable mobile netork for Maps specificially, while keeping it on for everything else.
- stanley 8y agoAt the risk of being downvoted, any Pixel users enable "Hey Google" recognition on their phones only to regret it? I'm constantly dealing with the phone interpreting commands intended for a Google Home speaker, which sometimes results in both the speaker and the phone acting on the same command. To my dismay, there's no way to disable Hey Google recognition on the phone after it's been enabled. Perhaps someone here has run into this issue as well? It's a huge pain point for me.
- _asummers 8y agoYou can register different trigger words, or at least you used to be able to. I had my phone wired to OK Google and the Home wired to Hey Google. Wasn’t an issue once I made the distinction. I no longer have an Android so I can’t comment on this still working. If you have external parties at your home regularly, that would obviously complicate things.
- 40acres 8y agoHmm, is this how the "what song is playing" feature works? Google claims it works offline (I haven't tested it) but I have a hard time believing that Google is storing information related to every song out there. What about new songs?
- shereadsthenews 8y agoYou never heard of Google Music? I believe getting fingerprints of all published music is probably one of the least complex tasks Google faces.
- mynameisvlad 8y agoIt was covered on HN when this feature was released. There's a database of >10,000 song fingerprints on-device (IIRC, people found it and it was ~100MB) that's updated based on the most popular songs from GPM/Youtube Music. https://venturebeat.com/2017/10/19/how-googles-pixel-2-now-playing-song-identification-works/ https://venturebeat.com/2017/10/19/how-googles-pixel-2-now-p...
- davidy123 8y agoThe other, gigantic shoe that will someday drop will be Google transcribing every incidental conversation. It can already do that, on-device, for every song that's heard, ever. It's a super power, being able to remember every word spoken around you, time and place, but of course it has privacy implications even if all the work is done without their cloud.
- rocky1138 8y agoI've noticed a trend in older folk where they get halfway into a sentence knowing they don't know the name of what they are referencing only to ask those around them for help identifying the reference on the way to making some other point, e.g., "That's like that time in It's a Wonderful Life where _______, gosh, I can't think of his name. What was the guy that was in that movie? Oh yeah, James Stewart. That's like the time in It's a Wonderful Life where yada yada yada" I'm hopeful that voice recognition assistants will help the burden during Christmas visits :D
- firefoxd 8y agoI can't pinpoint when exactly, but on windows XP, there used to be a speech to text engine that worked locally. When you set it up, you had to read some text to train it with your voice. You could constantly train it to improve it. This was before the cloudamagig, so I wonder it ran on. Edit: found the link https://www.techrepublic.com/article/solutionbase-using-speech-recognition-in-windows-xp/ https://www.techrepublic.com/article/solutionbase-using-spee...
- gok 8y agoThese end-to-end speech recognition systems are very intriguing. One major limitation is that since they don't model phonetics, they have no great way to deal with highly irregular orthography that doesn't show up in the training data. For example, there is no great way for the system to learn that the pronunciation "black" can be spelled "6LACK" sometimes. The paper on arXiv goes into how they deal with this. Basically they run a traditional WFST decoder over the output of the RNN-T to take spelling context into account. Still, it's impressive how far the neural system can get with no explicit lexicon or acoustic modeling in general.
- fouronnes3 8y agoI'm wondering if we'll see it write emoji at some point.
- sidcool 8y agoThis is an impressive engineering feat. Imagine the applications at edge devices! Microsoft is also trying hard to get their "Intelligent Edge" right.
- thrax 8y agoVoice on my pixel 3 is incredible. I normally have problems with voice recognition but this understands me better than some friends I have. It really is magical.
- beatle_sauce 8y agoThere is an excellent overview over their speech recognition system. http://iscslp2018.org/images/T4_Towards%20end-to-end%20speech%20recognition.pdf http://iscslp2018.org/images/T4_Towards%20end-to-end%20speec...
- nukeop 8y agoNow they can save valuable CPU time and your phone will extract advertising keywords from your conversations for them, even without an internet connection. It's way more efficient to cache speech converted to text while offline rather than audio clips. The servers get cleaned up text data, saving bandwidth and storage.
- jcelerier 8y agoI never understood the need for server-side speech recognition. Did an internship in 2013 for speech recognition on a BeagleBoard with Julius (https://github.com/julius-speech/julius https://github.com/julius-speech/julius), the thing worked with ninety-ish % accuracy (japanese language) and delay comparable to what my tablet gives - but locally.
- foobarbecue 8y agoI always figured doing it server-side was just to capture the users' data,either because the company wants a big training set, or for more nefarious purposes like tagged advertising.
- jdc0589 8y agogot it this morning on the way in to work. already used it a bunch and its GREAT.
- nojvek 8y agoGoogle and its dominance in both AI and reach into everyone’s private lives really scares me. There is a machine that can work totally offline, listen to audio, transcribe it, have a basic understanding and blast me with ads everywhere I go in the digital universe. It can then psycologically slowly manipulate my behavior via ads making us buy/do things that we don’t even realize it. It’s gonna be a scary world for my kids.
- camkego 8y agoFor application purposes where you don't want to source the audio from the microphone, is it possible for an Android application to feed audio to Gboard in order to source audio from other sources than the microphone? Maybe the Pixel has a mixer which allows audio from sources other than the microphone?