3 ms·

Neural translation has been really amazing and impressive. I stumbled into a way to be mean to Google translate this week: https://twitter.com/driainmurray/stat

by imurray 10y ago

Neural translation has been really amazing and impressive. I stumbled into a way to be mean to Google translate this week: https://twitter.com/driainmurray/status/818934530207862784 https://twitter.com/driainmurray/status/818934530207862784 OpenNMT can also run amok given totally unreasonable input too (example below). I'm being unfair, but the more serious point is that translation systems might go crazy in less obvious ways. While fluency is greatly improved, they certainly don't always get the meaning correct. I'm excited to see what people will do to better diagnose when we can trust the translation. Input: "Toll ach toll toll ach toll toll toll toll toll toll toll toll toll toll toll toll toll toll toll toll toll toll toll toll Deutsch." Output: "Tell me great great, great great, great great great great great great great great great love great ? ? ? great love ? ? ? ? ? ? ? great ? ? ? ? great love ? ? ? great great ? ? ? great great ? ? ? great great ? ? ? great great ? ? great great ? ? ? great great ? ? great great ? ? great love ? ? Good ? ? ? great love ? ? great love ? ? Good love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? great love ? ? Good ?" [Each ? was a musical note glyph, but HN ate the unicode character.]

srush 10y ago
Unfortunately we do run into these types of "robustness issues" in our experiments. They're rare enough that they don't really change many of the performance metrics of the system, but they are quite embarrassing. It is currently a bit of a mystery as to why they go so wrong, better diagnostic tools are sorely needed. Informally, these mistakes similar to the more well-known mis-classification of noise issues in cnn's for vision.
rvense 10y ago
As someone whose interest in AI and related tech stems as much from a desire to learn about our own brains as it does about problem-solving technologies, the failure modes of these systems are as almost as interesting as the good results - and also the part that tells me we've quite a ways to go. I love your example too. I studied linguistics, and I always wondered about this idea that a sentence consisting of a determiner and an inifinite number of repetitions of the same adjective followed by a noun was supposed to be a valid sentence of any language. There has to be a little more to that story... Back on topic, I really love the thought of running my own Google Translate server.

by imurray 10y ago

srush 10y ago

Unfortunately we do run into these types of "robustness issues" in our experiments. They're rare enough that they don't really change many of the performance metrics of the system, but they are quite embarrassing. It is currently a bit of a mystery as to why they go so wrong, better diagnostic tools are sorely needed. Informally, these mistakes similar to the more well-known mis-classification of noise issues in cnn's for vision.

rvense 10y ago

As someone whose interest in AI and related tech stems as much from a desire to learn about our own brains as it does about problem-solving technologies, the failure modes of these systems are as almost as interesting as the good results - and also the part that tells me we've quite a ways to go. I love your example too. I studied linguistics, and I always wondered about this idea that a sentence consisting of a determiner and an inifinite number of repetitions of the same adjective followed by a noun was supposed to be a valid sentence of any language. There has to be a little more to that story... Back on topic, I really love the thought of running my own Google Translate server.