6 ms·
For the past 5 years or so, as a kind of benchmark, I've been checking how Google translates a particular word from Greek, to French. The word is "χελιδόνι", m
by YeGoblynQueenne 9y ago
For the past 5 years or so, as a kind of benchmark, I've been checking how Google translates a particular word from Greek, to French.
The word is "χελιδόνι", meaning "swallow" (the bird) in Greek. For as long as I've been trying this tiny little experiment, Google has been translating it to the French word "avaler"- the verb "to swallow".
https://translate.google.com/#el/fr/%CF%87%CE%B5%CE%BB%CE%B9%CE%B4%CF%8C%CE%BD%CE%B9 https://translate.google.com/#el/fr/%CF%87%CE%B5%CE%BB%CE%B9...
Once in a while, a different translation appears- "machaon", which is a kind of butterfly, the Old World swallow-tail. This is slightly closer but still absurd. If GT can make the connection to "swallow-tail", how can it not see the connection to the bird, "swallow"?
The problem seems to be that, in order to translate from Greek to French, Google goes via English. A great big chunk of context is lost in the process, especially since now the translation between two languages that have different forms for male and female nouns goes through a third language that does not.
So for example, I've seen the same gender-reversal as Hofstadter reports in his article. The following string in Greek means "I saw my teacher and she said hello to me".
Είδα τη δασκάλα μου και μου είπε γειά.
The following are the French and English translations by GT:
J'ai vu mon professeur et il m'a dit bonjour.
I saw my teacher and he said hello.
This seems to happen because Google's English language model has learned that "he" is found in the context of "teacher" and "said" more often than "she" is. But of course, such a statistical association is, well, meaningless and for that useless when it comes to translation, where you actually need to know when the least likely case is correct.
Generally, it looks like the complete abandonment of any attempt at representing meaning, and relying instead on text statistics to do meaning-intensive work like translation, is producing a lot of nonsense. And that's not a criticism of Google Translate only. I think professor Chomsky might "win" that old debate with prof. Norvig, after all.
- dragandj 9y agoAre you sure that this particular translation is "nonsense"? To me, it seems almost perfect. The only mistake in the French version is the gender of the teacher, which is a mistake that many humans can and do make, in addition to many other mistakes...
- puszczyk 9y agoWell, the Greek version says she-teacher and it translates to French he-teacher. Def a big mistake. Note this is not the same as messing up if eg some random butterfly you see is he or she in French. The context is there, but google translate seems to convert everything to English, loosing some meaning in the process. It don’t want to say google translate is a complete garbage. On the contrary, I use it often e.g to plan my holiday trips or shop online in other EU countries. Nevertheless, I rarely translate anything to my native language, but rather to English. It the auto translation made a great progress and it is very useful, but it has a long way to go still.
- 1024core 9y ago> Well, the Greek version says she-teacher and it translates to French he-teacher. Def a big mistake. I have Mandarin speaking friends (graduates of US universities), for whom English is a second language. They often use "she" instead of "he" (or vice versa) when referring to others. If humans, who have been using English for 7-8 years, can't get this right, then I wouldn't call it a "big mistake". It is a mistake, for sure; but it's not the end of the world.
- YeGoblynQueenne 9y agoYour friends however will be able to keep track of the references to the same person in a conversation, even if they get the gender wrong. In GT's case it's evidence that it can't do that, and therefore will mangle longer passages that require this level of understanding.
- 1024core 9y agoActually, no. Just a couple of days ago, when talking about a female colleague, they kept switching from "he" to "she" and back and forth. It's almost as if Mandarin has no "he" or "she" equivalents.
- 9y ago
- fenomas 9y ago> ..I've been checking how Google translates a particular word from Greek, to French. I have a couple of these in Japanese. One is "七輪" (brazier), which Google Translate thinks is "tambourine"; the other is "ちゃぶ台" (tea table), for which it returns "Shabu-bashi". No idea what's going on in either case - the latter doesn't even mean anything in Japanese. It's always nice to read Hofstadter, but when Google Translate trips up over single words, delving into subtleties of contextual grammar seems like overkill.
- YeGoblynQueenne 9y ago>> It's always nice to read Hofstadter, but when Google Translate trips up over single words, delving into subtleties of contextual grammar seems like overkill. Ah, but see, the very counterintuitive reasoning is that it's actually easier to translate a sentence than a single word- because the context provided by the other tokens in the sentence can help determine the intended usage of the word. For instance, the following sentence is translated pretty much 100% correctly by GT: Το χελιδόνι είναι αποδημητικό πτηνό. The swallow is a migratory bird. Which in a way makes the mistake on the single word even harder to understand (I mean, for users who don't know anything about how machine translation works). Of course, when sentences start getting more complex, looking at the near context, or even the far context (that LSTM nets are good at) is not going to help you. That's even more so when you have no way to tell which of multiple possible meanings of a word you should use to derive the correct context and find the correct translation. So, yes, the inability of GT to discern between meaning at the word-level affects its ability to translate more complex passages, but you may miss that with simpler sentences- which I think may explain why many people think that GT translations are actually pretty good, when they mostly really suck.
- fenomas 9y agoI understand what you're saying, but there's no subtlety or dual meaning in my examples - Google Translate is just taking in a word with a single, unambiguous meaning, and returning a wholly unrelated word (or in the latter case gibberish). In other words they're cases where GT is less useful than a plain lookup database - which makes comparisons to human translators seem rather beside the point!
- braindongle 9y ago"...such a statistical association is, well, meaningless..." Good point, but it's even worse than meaningless. It's stereotype-reinforcing. That particular inference, that the role in society is such-and-such, so presume that the subject is male, is to be fought at every turn. I would much prefer the awkwardness of "he/she" when the machine is uncertain rather than some "this is just the way things are" presumption.
- JoeAltmaier 9y agoBut be honest, language has this odd 'gender' thing going on. Orthogonal to any interests about equality of the sexes. Not saying its right or helpful, but its there and a responsible translation algorithm has to take it into account.
- braindongle 9y agoAgreed. I edited my comment to add what I think would be a reasonable way to handle it. "he/she" is just silly for humans to say, but is seems appropriate when the text is coming from a machine and there is uncertainty.
- JoeAltmaier 9y agoGood solution!
- ghaff 9y agoSingular they seems to be the general approach that people are moving towards. It's now accepted in AP style under some circumstances--the some circumstances being basically if you can't figure out a way to cleanly write around it. And AP style is one of the stylebooks a lot of organizations look to for their own house styles. He/she has always been pretty awkward and silly. But in the eyes of at least some people, it also has the failing that even after all that it still assumes binary genders.
- graeme 9y agoMakes it hard to paste though. Then again, so does an error in gender. But, I don't think he/she is viable because a major use case is people pasting quickly on their phones. You can see this because the app has automatic clipboard copy built in.
- charlescearl 9y ago“Chomsky might win...with Norvig” ...better to see Schank over Cherniak
- jessriedel 9y agoA bit more info, including a test where Google doesn't lose gender information when the English word happens to have it (actor/actress): https://www.quora.com/Does-Google-Translate-use-English-as-an-intermediary-step-language-Spanish-English-Italian https://www.quora.com/Does-Google-Translate-use-English-as-a... It appears that they have been trying to construct (with ML, not by hand) a universal intermediary language since at least 2016 to avoid this sort of thing, but that it still isn't as good as just using English as the intermediary. https://www.newscientist.com/article/2114748-google-translate-ai-invents-its-own-language-to-translate-with/ https://www.newscientist.com/article/2114748-google-translat...
- username223 9y ago> It appears that they have been trying to construct (with ML, not by hand) a universal intermediary language... Oops. Google has enough money to hire some linguists, so they hopefully have heard of the debates over Chomsky's Universal Grammar. "Universal human language" is highly unlikely to work; it doesn't even work well for color names.
- YeGoblynQueenne 9y agoMy understanding is that English is the intermediary language, but not on purpose- they tried to create an "interlingua" but they trained with too many English-to-X and X-to-English data and ended up with a model of English as their intermediary language model. And it's hard even for them to know because their models are opaque and hardly amenable to principled examination. (I mean, you can't prove anything about them, so you can't really know what they're doing in there).
- russdill 9y agoJust put it context and it's fine: το χελιδόνι πέταξε μακριά l'hirondelle s'envola
- tmoravec 9y agoNice examples. But the problem with "swallow" and similar lies somewhere else. It's that translating a single word into a single word is a ridiculously stupid idea. As a person whose mother tongue is not English, I occasionally stumble on a word that I need to translate. GT is utterly useless for me. A random free online dictionary gives me 20 different translations for "swallow". I am honestly surprised that a company with such talent creates a tool that is this useless; it's literally about doing something like this: 1) Check if the word or phrase is in a dictionary. If it is, return all results from the dictionary. 2) If it's not, try something smarter. But no, this is not Artificial Intelligence enough! /s
- pawelk 9y ago> to translate from Greek to French, Google goes via English. I found the English as an intermediary language amusing when translating a word between two Slavic languages, Polish and Russian. Wanted to look up something about Russian hand planes (the wood working tool). Typed "strug" and "hebel" into Google translate, and got... "самолет", which, being able to parse some Cyrillic, reads suspiciously like Polish "samolot" (airplane). Same the other way around: "рубанки" (hand planes) -> "samoloty" (airplanes). It kept the plural form, but of the wrong word :) The "hand" part of "hand plane" is getting lost in translation, both ways. Edit: similar between Polish and German, "hebel" -> "Flugzeug", all the more amusing considering hebel is a loanword from German Hobel :) When I need to translate a single word I go to its Wikipedia page in language A, then look for language B in the "in other languages" list. (That's how I finally came to "рубанки").
- visarga 9y ago> Google goes via English. No, it doesn't. It goes via an intermediate representation which is common for all the languages they translate. The meaning of a phrase is represented as a "meaning vector", or more recently, a matrix. The problem is that it is hard to translate rare words, because when the neural net decodes the next word, it needs to consider the whole dictionary at once (the problem of too large softmax function slowing down the process). So they need to focus on a subset of words. There are various solutions, such as hierarchical softmax, but they are still difficult to train. Google translate uses a ton of compute to create a model as is. They probably can't offer everyone a version that is 2% better but uses 100x more resources.
- pawelk 9y agoHow do you get from Polish "hebel" to German "Flugzeug", if not by dropping the "hand" part of "hand plane"? Especially when the Polish is a loanword from German Hobel...
- visarga 9y agoWe've become too entitled and can't appreciate the huge amount of work that went into automatic translation any more. If GT doesn't cut it, we can always take a ... you know ... dictionary and do it by hand. Google search isn't perfect either, Wikipedia has errors, Twitter has bots, why are we bothering to use them? Because they are better than not having them.
- pawelk 9y agoOr maybe one could reasonably expect that Google, with all that money and engineering power, would implement something like a dictionary lookup to short-circuit single word translations? Edit: I did not want to complain about the translation quality nor show my sense of entitlement. You have authoritatively stated and used a some jargon to tell us it doesn't use English when translating between languages, so I simply wanted to understand more about this process because, on the surface, it looks like English was involved in my specific example.