7 ms·

Google Assistant Can Now Translate Languages on Your Phone

87 points by jonbaer 7y ago

jph 7y ago
"However, there's always a chance Assistant could accidentally start recording snippets of conversations and therefore potentially sensitive and identifiable information. " - This is why security people want a hardware microphone switch.
- s3r3nity 7y ago
  Funny enough, Facebook added one for both the camera and microphone on Portal.
  - ma2rten 7y ago
    Alexa and Google Home have them too.
why-oh-why 7y ago
Isn’t the title wrong? The audio is sent to the server so it’s not “on my phone.” This type of translation was already available “on my phone” completely offline (written, with Google Translate) I use translation often and I was hoping to finally have an easy way to have a written conversation, but this still doesn’t show the right keyboard when picking the language in “Keyboard” mode.
- elcomet 7y ago
  Google assistant runs on the phone, even if the processing doesn't.
  - OrgNet 7y ago
    but the processing is the important part
    - elcomet 7y ago
      Not really for users. They care about doing the request and seeing the results on their phones.
      - OrgNet 7y ago
        that will change when they figure out the privacy implications
        shadowgovt 7y ago
        Evidence suggests it does not.
        OnlineGladiator 7y ago
        It's depressing how true this is.
        diffserv 7y ago
        Can you expand on the the privacy implications? How is this different than picking up the phone and having your convo go through ATT/Verizon networks? or using your ISP? Both parties can "legally" work with authorities to wiretap you? Are you worried that the (training) algorithms that run on your voice somehow end up leaking your identity? Or are you worried that someone at Google knows your voice? Also, not sure if Google has this fact in their ToS but if they do, what is the issue?
        klyrs 7y ago
        Phone networks are not primarily advertising agencies, and they're regulated as utilities, so the difference is nontrivial. And the personalized risk isn't from random strangers knowing your voice -- it's your stalker ex, or other bad actors, who might get way more insight into your life than you want
        OrgNet 7y ago
        > Or are you worried that someone at Google knows your voice? Google already knows who you are, now they know a bit more about you, including what you talk about, and of course your voice which probably can be used to locate you with all those "smart" speakers around.
- izacus 7y ago
  The title is about expanding this feature to the phones (it was before available only on Home devices) so in that context, it did start working on the phones. Assistant (just like the counterparts Siri and Alexa) is always using server for processing and users of them are aware of that.
reaperducer 7y ago
This sort of thing is surprisingly useful. A couple of weeks ago I was going to Human Resources on the other side of campus and there was a Chinese family wandering around, obviously lost. The mother showed me her phone with some Chinese-language map app that I'd never seen before. It indicted that there was a shopping mall where we were standing. Obviously, her map app was wrong since the company has been at this location for 30 years. But I was able to say to my phone, "Hey, Siri. How to you say 'I'm sorry, there is no shopping center here.' In Chinese?" And then I held my phone for her to see while Siri both printed out the translation on the screen and spoke it to her. I said a few other hopefully helpful phrases to her, but she seemed happy with my guidance and did lots of smiling and nodding. (I assume the article is about the Google version of this. I wasn't able to read the article because Wired popped up so many ads and DIV modals on the screen that there wasn't any actual story text.)
- AlotOfReading 7y ago
  Perhaps an issue with her maps app applying GCJ-02 or BD-09? Apparently the "in china" check for the noise function is a simple bounding box, which includes much of the surrounding countries.
- stinos 7y ago
  It's useful, but after so many years the state of machine translations and speech recognition in general is still not exactly reliable. It's like it doesn't have context or doesn't know how to apply it. I've heard success stories like this before, experienced them a few times as well, but most of the times the experience for me is subpar to the point it gets so annoying and needs so much manual intervention I gave up, thinking I'll just try again in 5 or 10 years and see if it's any beter. Maybe my accent or pronounciation sucks but I tried getting Siri to write down text messages about 10 times. Most of the times it was close, but none of the times the words were 100% correct and in more than 50% of cases that led to the produced sentences not conveying the original meaning. Same for navigation. Names of cities (in Europe) seem problematic, like confusing Miltenberg (DE) with Milton in Canada or so. Similar for Google Translate. Our Portugese taxi driver didn't speak English and was worried about getting us to the airport in time. His phone showed us he was worried about the weather. I get 'tempo' can mean both, but it's these subtle differences technology still is lacking.
OldGuyInTheClub 7y ago
I saw this video before its public release circa 1993. Autotranslation, virtual agents, realtime video conferencing on handheld devices, virtual reality... all there. I was finishing my postdoc at Bell Labs where it was shown to us as a glimpse into the company's future plans. I didn't know where the bandwidth would come from and neither did the presenter when I asked except to say that "It will have to be built, won't it?." Needless to say I didn't have the foresight to invest in it, either https://www.youtube.com/watch?v=yFWCoeZjx8A&feature=emb_logo https://www.youtube.com/watch?v=yFWCoeZjx8A&feature=emb_logo "AT&T's vision of the future, circa 1993 - AT&T Archives"
- rcpt 7y ago
  2min in and you can see they also predicted NIMBYs
  - OldGuyInTheClub 7y ago
    Oh, we've been around a lot longer than that.
- kleiba 7y ago
  Interestingly, 1993 is also the year a rather large research endeavor began in Europe, paving the way for the kind of on-the-fly speech translation technology that is now built into Google Assistant: https://www.youtube.com/watch?v=noZBab-Lmss https://www.youtube.com/watch?v=noZBab-Lmss
jzwinck 7y ago
The Google Translate app has had this feature for years. Near real time audio, and video translation too! It works very well for many languages but not especially well for Chinese (admittedly a harder problem than Spanish).
steelframe 7y ago
I recently vacationed in Japan, and I found that using Google Translate on the phone wasn't practical because the other person would have no clue what was going on or how they are supposed to work with the app. The language auto-detect was abysmal; half the time it thought Japanese was English. They would start responding to the translated statement before the app played the chime, and I'd have to rudely interrupt them and fumble with the app to get it to start listening again. They were never sure when it was "okay" to start talking again. In the end I tended to communicate everything I really needed by holding up the number of things I wanted on my fingers and pointing and smiling, caveman-style. They would usually know just enough English to say, "Two? Ok." Then the price of whatever I was buying would show up on the register surrounded by mystery Japanese characters, but the numbers were Western Arabic, so that was all I needed to know. One thing I learned is that you can get by with very little knowledge of the local language to successfully travel and eat.
- missosoup 7y ago
  I had the opposite experience. Everyone I tried to communicate with via google translate responded positively except one disgruntled train station attendant. I would say excuse me, I don't speak Japanese can you please help me. And then I'd show them my phone with the translated version of whatever my query is and then press the mic button and they'd get the point that they can speak now. Worked great.
  - j88439h84 7y ago
    That's a nice workflow.
  - noja 7y ago
    Did it work for a conversation, rather than a question and answer?
    - missosoup 7y ago
      For face-to-face it wasn't great because speech recognition is error prone. Usually I would always type and the person I'm interacting with would speak and if they saw the text was wrong they'd run it again. For chatting over sms/whatever it was great. Enough to have complex conversations and get to know people that I couldn't possibly communicate with otherwise. It's not perfect, but it's a lot better than nothing if you aren't proficient in the local language. Also your mileage will vary by language. English-Japanese seems to be a pretty good combination. English-Chinese and English-Russian are a lot more flaky.
whiddershins 7y ago
When I was in Tokyo a bartender chatted with me for a while using a hand held stand-alone device that did voice (audio without relying on text) two way near-real-time translation. It worked really well and I was confused I have never seen one before.
asdff 7y ago
It would be great if OCR got to the point where you could just point the phone camera at a sign and have it output 1:1 what is in front of you with translated text, like a magic little window frame. Shit still struggles with parsing PDFs though so I'm not holding out too much hope, though.
- wetpaws 7y ago
  Google translate kinda does this already, with a mixed success
- bradfitz 7y ago
  Have you seen https://www.youtube.com/watch?v=06olHmcJjS0&feature=emb_title https://www.youtube.com/watch?v=06olHmcJjS0&feature=emb_titl... ? That was 2015 and it's improved a bit since. I've used it in a number of countries with great success, especially reading menus. (Disclaimer: Googler, but unrelated to those teams)
- dreamcompiler 7y ago
  The Google Translate app does exactly that in its camera mode, and has been doing so for at least 3 years.
tsimionescu 7y ago
It's interesting that all the positive Google Translate experiences shared here have to do with Chinese and Japanese. In general, my experience with translations of European languages has been abysmal, for anything but the simplest expressions. The resulting expressions are often so ungrammatical that they were basically unintelligible. I remember a Greek taxi driver who picked me up for a long trip, and seeing the distance, initially assumed I was going to the airport - and asked about it, prompting a flurry of No no nos from me and pointing on a map. He later tried to use Google translate to explain why he had assumed this (Greek to English) , but what came out was so garbled I only understood that he was saying something about distance and airport, prompting another confused flurry of map pointing (he abandoned the hope of explaining the initial confusiom and resigned to just driving...) . It was only minutes later, trying to think about what had happened, that I finally puzzled out what that translation must have meant.
- TazeTSchnitzel 7y ago
  For translations where one side is not English, Google Translate does a horrible job, because it will incorrectly translate to English first, then incorrectly translate from English to the target language. This means you end up with translation errors only comprehensible to someone who speaks all three languages! But even without that, even for what should be simple translations from English to e.g. Swedish, it makes so many nonsensical errors. Not just misunderstanding context, but fabricating novel and absurd translations of common words. I think it's gotten worse since they switched to their whole-sentence neural net system. At least in the past, the individual words made some sense, and you could click on them individually to see other (sometimes more accurate) alternatives.
kwhitefoot 7y ago
The HN title is misleading. The Wired title says "through" not "on". And the article makes it clear that an internet connection is required. Surely "on" would mean that no connection to the internet would be needed.
forgetfulusr 7y ago
Ah, from the title I thought they figured out how to do it locally on your phone, without being connected to Google. Knowing they are only into making Ad products aka tracking, it would be a shame to get used to such a nice free offering. Thanks but no thanks, I guess us suckers will have to take a few hrs to learn a few of the local language phrases.

87 points by jonbaer 7y ago

jph 7y ago

"However, there's always a chance Assistant could accidentally start recording snippets of conversations and therefore potentially sensitive and identifiable information. " - This is why security people want a hardware microphone switch.

s3r3nity 7y ago
Funny enough, Facebook added one for both the camera and microphone on Portal.
- ma2rten 7y ago
  Alexa and Google Home have them too.

why-oh-why 7y ago

Isn’t the title wrong? The audio is sent to the server so it’s not “on my phone.” This type of translation was already available “on my phone” completely offline (written, with Google Translate) I use translation often and I was hoping to finally have an easy way to have a written conversation, but this still doesn’t show the right keyboard when picking the language in “Keyboard” mode.

elcomet 7y ago
Google assistant runs on the phone, even if the processing doesn't.
- OrgNet 7y ago
  but the processing is the important part
  - elcomet 7y ago
    Not really for users. They care about doing the request and seeing the results on their phones.
    - OrgNet 7y ago
      that will change when they figure out the privacy implications
      - shadowgovt 7y ago
        Evidence suggests it does not.
        OnlineGladiator 7y ago
        It's depressing how true this is.
      - diffserv 7y ago
        Can you expand on the the privacy implications? How is this different than picking up the phone and having your convo go through ATT/Verizon networks? or using your ISP? Both parties can "legally" work with authorities to wiretap you? Are you worried that the (training) algorithms that run on your voice somehow end up leaking your identity? Or are you worried that someone at Google knows your voice? Also, not sure if Google has this fact in their ToS but if they do, what is the issue?
        klyrs 7y ago
        Phone networks are not primarily advertising agencies, and they're regulated as utilities, so the difference is nontrivial. And the personalized risk isn't from random strangers knowing your voice -- it's your stalker ex, or other bad actors, who might get way more insight into your life than you want
        OrgNet 7y ago
        > Or are you worried that someone at Google knows your voice? Google already knows who you are, now they know a bit more about you, including what you talk about, and of course your voice which probably can be used to locate you with all those "smart" speakers around.
izacus 7y ago
The title is about expanding this feature to the phones (it was before available only on Home devices) so in that context, it did start working on the phones. Assistant (just like the counterparts Siri and Alexa) is always using server for processing and users of them are aware of that.

reaperducer 7y ago

This sort of thing is surprisingly useful. A couple of weeks ago I was going to Human Resources on the other side of campus and there was a Chinese family wandering around, obviously lost. The mother showed me her phone with some Chinese-language map app that I'd never seen before. It indicted that there was a shopping mall where we were standing. Obviously, her map app was wrong since the company has been at this location for 30 years. But I was able to say to my phone, "Hey, Siri. How to you say 'I'm sorry, there is no shopping center here.' In Chinese?" And then I held my phone for her to see while Siri both printed out the translation on the screen and spoke it to her. I said a few other hopefully helpful phrases to her, but she seemed happy with my guidance and did lots of smiling and nodding. (I assume the article is about the Google version of this. I wasn't able to read the article because Wired popped up so many ads and DIV modals on the screen that there wasn't any actual story text.)

AlotOfReading 7y ago
Perhaps an issue with her maps app applying GCJ-02 or BD-09? Apparently the "in china" check for the noise function is a simple bounding box, which includes much of the surrounding countries.
stinos 7y ago
It's useful, but after so many years the state of machine translations and speech recognition in general is still not exactly reliable. It's like it doesn't have context or doesn't know how to apply it. I've heard success stories like this before, experienced them a few times as well, but most of the times the experience for me is subpar to the point it gets so annoying and needs so much manual intervention I gave up, thinking I'll just try again in 5 or 10 years and see if it's any beter. Maybe my accent or pronounciation sucks but I tried getting Siri to write down text messages about 10 times. Most of the times it was close, but none of the times the words were 100% correct and in more than 50% of cases that led to the produced sentences not conveying the original meaning. Same for navigation. Names of cities (in Europe) seem problematic, like confusing Miltenberg (DE) with Milton in Canada or so. Similar for Google Translate. Our Portugese taxi driver didn't speak English and was worried about getting us to the airport in time. His phone showed us he was worried about the weather. I get 'tempo' can mean both, but it's these subtle differences technology still is lacking.

OldGuyInTheClub 7y ago

I saw this video before its public release circa 1993. Autotranslation, virtual agents, realtime video conferencing on handheld devices, virtual reality... all there. I was finishing my postdoc at Bell Labs where it was shown to us as a glimpse into the company's future plans. I didn't know where the bandwidth would come from and neither did the presenter when I asked except to say that "It will have to be built, won't it?." Needless to say I didn't have the foresight to invest in it, either https://www.youtube.com/watch?v=yFWCoeZjx8A&feature=emb_logo https://www.youtube.com/watch?v=yFWCoeZjx8A&feature=emb_logo "AT&T's vision of the future, circa 1993 - AT&T Archives"

rcpt 7y ago
2min in and you can see they also predicted NIMBYs
- OldGuyInTheClub 7y ago
  Oh, we've been around a lot longer than that.
kleiba 7y ago
Interestingly, 1993 is also the year a rather large research endeavor began in Europe, paving the way for the kind of on-the-fly speech translation technology that is now built into Google Assistant: https://www.youtube.com/watch?v=noZBab-Lmss https://www.youtube.com/watch?v=noZBab-Lmss

jzwinck 7y ago

The Google Translate app has had this feature for years. Near real time audio, and video translation too! It works very well for many languages but not especially well for Chinese (admittedly a harder problem than Spanish).

steelframe 7y ago

I recently vacationed in Japan, and I found that using Google Translate on the phone wasn't practical because the other person would have no clue what was going on or how they are supposed to work with the app. The language auto-detect was abysmal; half the time it thought Japanese was English. They would start responding to the translated statement before the app played the chime, and I'd have to rudely interrupt them and fumble with the app to get it to start listening again. They were never sure when it was "okay" to start talking again. In the end I tended to communicate everything I really needed by holding up the number of things I wanted on my fingers and pointing and smiling, caveman-style. They would usually know just enough English to say, "Two? Ok." Then the price of whatever I was buying would show up on the register surrounded by mystery Japanese characters, but the numbers were Western Arabic, so that was all I needed to know. One thing I learned is that you can get by with very little knowledge of the local language to successfully travel and eat.

missosoup 7y ago
I had the opposite experience. Everyone I tried to communicate with via google translate responded positively except one disgruntled train station attendant. I would say excuse me, I don't speak Japanese can you please help me. And then I'd show them my phone with the translated version of whatever my query is and then press the mic button and they'd get the point that they can speak now. Worked great.
- j88439h84 7y ago
  That's a nice workflow.
- noja 7y ago
  Did it work for a conversation, rather than a question and answer?
  - missosoup 7y ago
    For face-to-face it wasn't great because speech recognition is error prone. Usually I would always type and the person I'm interacting with would speak and if they saw the text was wrong they'd run it again. For chatting over sms/whatever it was great. Enough to have complex conversations and get to know people that I couldn't possibly communicate with otherwise. It's not perfect, but it's a lot better than nothing if you aren't proficient in the local language. Also your mileage will vary by language. English-Japanese seems to be a pretty good combination. English-Chinese and English-Russian are a lot more flaky.

whiddershins 7y ago

When I was in Tokyo a bartender chatted with me for a while using a hand held stand-alone device that did voice (audio without relying on text) two way near-real-time translation. It worked really well and I was confused I have never seen one before.

asdff 7y ago

It would be great if OCR got to the point where you could just point the phone camera at a sign and have it output 1:1 what is in front of you with translated text, like a magic little window frame. Shit still struggles with parsing PDFs though so I'm not holding out too much hope, though.

wetpaws 7y ago
Google translate kinda does this already, with a mixed success
bradfitz 7y ago
Have you seen https://www.youtube.com/watch?v=06olHmcJjS0&feature=emb_title https://www.youtube.com/watch?v=06olHmcJjS0&feature=emb_titl... ? That was 2015 and it's improved a bit since. I've used it in a number of countries with great success, especially reading menus. (Disclaimer: Googler, but unrelated to those teams)
dreamcompiler 7y ago
The Google Translate app does exactly that in its camera mode, and has been doing so for at least 3 years.

tsimionescu 7y ago

It's interesting that all the positive Google Translate experiences shared here have to do with Chinese and Japanese. In general, my experience with translations of European languages has been abysmal, for anything but the simplest expressions. The resulting expressions are often so ungrammatical that they were basically unintelligible. I remember a Greek taxi driver who picked me up for a long trip, and seeing the distance, initially assumed I was going to the airport - and asked about it, prompting a flurry of No no nos from me and pointing on a map. He later tried to use Google translate to explain why he had assumed this (Greek to English) , but what came out was so garbled I only understood that he was saying something about distance and airport, prompting another confused flurry of map pointing (he abandoned the hope of explaining the initial confusiom and resigned to just driving...) . It was only minutes later, trying to think about what had happened, that I finally puzzled out what that translation must have meant.

TazeTSchnitzel 7y ago
For translations where one side is not English, Google Translate does a horrible job, because it will incorrectly translate to English first, then incorrectly translate from English to the target language. This means you end up with translation errors only comprehensible to someone who speaks all three languages! But even without that, even for what should be simple translations from English to e.g. Swedish, it makes so many nonsensical errors. Not just misunderstanding context, but fabricating novel and absurd translations of common words. I think it's gotten worse since they switched to their whole-sentence neural net system. At least in the past, the individual words made some sense, and you could click on them individually to see other (sometimes more accurate) alternatives.

kwhitefoot 7y ago

The HN title is misleading. The Wired title says "through" not "on". And the article makes it clear that an internet connection is required. Surely "on" would mean that no connection to the internet would be needed.

forgetfulusr 7y ago

Ah, from the title I thought they figured out how to do it locally on your phone, without being connected to Google. Knowing they are only into making Ad products aka tracking, it would be a shame to get used to such a nice free offering. Thanks but no thanks, I guess us suckers will have to take a few hrs to learn a few of the local language phrases.