3 ms·

This post seems to over-exaggerate a commonly known mathematical property. Suppose I have languages X, Y, and Z. My machine currently knows how to translate be

by spynxic 10y ago

This post seems to over-exaggerate a commonly known mathematical property. Suppose I have languages X, Y, and Z. My machine currently knows how to translate between X->Y and X->Z. The goal is to turn Y into Z without direct training. The process would be to translate Y into X and X into Z.. effectively Y->Z. This isn't really transfer learning as much as it is logical induction...Or am I missing something?

railorsi 10y ago
Could we say logical induction is the principle behind transfer learning then?
- spynxic 10y ago
  Emergent induction via high dimensional structure composition.. maybe. I haven't read any whitepages about the underlying processes P.S. note the hand-waviness when I use the word "emergent"
eridius 10y ago
Presumably what you're missing is the fact that this is translating from Y->Z in a single step, rather than doing two separate translations.
- spynxic 10y ago
  Then the high dimensional representation of the language that allows such a translation to be possible is profound
- mattkrause 10y ago
  They're certainly not doing English to Portuguese by way of Spanish or anything like that. However, you could almost read the paper as "We can translate something from a natural language into a high-dimensional representation, then turn that high-dimensional representation back into (another, possibly different) natural language. "
  - empath75 10y ago
    It seems similar to automatically captioning images.
    - mattkrause 10y ago
      Very much so! The two tasks are surprisingly interchangable. I once worked on a project where we used a statistical MT approach to "translate" between image features and captions--and I don't think we were the only ones trying such things. In a pleasing bit of symmetry, the attentional network used here looks like it was initially developed for image captioning.
mattkrause 10y ago
Well.... It is one of the oldest ideas in machine translation. In the late 1950s, Richens proposed an MT system that used an intermediate representation called an "interlingua." For each language, the system has two components, one that maps a natural language onto the interlingua, and a second that converts the interlingua back into a natural language. Use different source and destination pairs and--bam! you've got a multi-way translator. However, this only works if a) the interlingua is rich enough to capture the semantics of the to-be-translated text and b) the conversions between natural languages and interlingua also preserves those semantics. In practice, neither has worked pretty well. In the 1960s, the interlingua was typically hand-crafted and rules were either hand written, or later, induced, to convert natural languages to/from it. Since people have been working on this for 60 years, you can probably image how well that works[]. The clever bit here is that you don't really need to do that. If you have enough data, the LSTMs can learn it on their own (see Figure 2, for example, where semantically-related sentences from multiple languages end up in the same neighborhoods). [] Actually, somewhat better than you'd think. It certainly wouldn't have done Pushkin any justice, but it was surprisingly decent on weather reports (etc).
- csydas 10y ago
  I was going to comment on the article and idea of multilingual NN's in general since, as I'm learning Russian currently, I'm having a hard time seeing how you can get a good automated translation without some editorializing or awareness of what's going on in both sentences. Your comment on NN's "... not doing Pushkin justice" stood out since it's something I've been wondering how NN could handle a language like Russian that traditionally doesn't use the verb "to be" in sentences, much like the E-Prime idea that exists in English for clarity of description. Just trying to re-think in terms of a Russian sentence instead of an English sentence has also proven difficult, and I'd be curious how you can automate it on NN's well. My colleagues (russian natives) are all able to spot auto-translated messages from me nearly instantly since it messes up pretty rudimentary grammar rules or just doesn't phrase the sentence like you would in native Russian.
nl 10y ago
No one has managed to get induction working reliably for natural language tasks in any way that is generalizable enough to rely on. So that is new.

by spynxic 10y ago

railorsi 10y ago

Could we say logical induction is the principle behind transfer learning then?

spynxic 10y ago
Emergent induction via high dimensional structure composition.. maybe. I haven't read any whitepages about the underlying processes P.S. note the hand-waviness when I use the word "emergent"

eridius 10y ago

Presumably what you're missing is the fact that this is translating from Y->Z in a single step, rather than doing two separate translations.

spynxic 10y ago
Then the high dimensional representation of the language that allows such a translation to be possible is profound
mattkrause 10y ago
They're certainly not doing English to Portuguese by way of Spanish or anything like that. However, you could almost read the paper as "We can translate something from a natural language into a high-dimensional representation, then turn that high-dimensional representation back into (another, possibly different) natural language. "
- empath75 10y ago
  It seems similar to automatically captioning images.
  - mattkrause 10y ago
    Very much so! The two tasks are surprisingly interchangable. I once worked on a project where we used a statistical MT approach to "translate" between image features and captions--and I don't think we were the only ones trying such things. In a pleasing bit of symmetry, the attentional network used here looks like it was initially developed for image captioning.

mattkrause 10y ago

Well.... It is one of the oldest ideas in machine translation. In the late 1950s, Richens proposed an MT system that used an intermediate representation called an "interlingua." For each language, the system has two components, one that maps a natural language onto the interlingua, and a second that converts the interlingua back into a natural language. Use different source and destination pairs and--bam! you've got a multi-way translator. However, this only works if a) the interlingua is rich enough to capture the semantics of the to-be-translated text and b) the conversions between natural languages and interlingua also preserves those semantics. In practice, neither has worked pretty well. In the 1960s, the interlingua was typically hand-crafted and rules were either hand written, or later, induced, to convert natural languages to/from it. Since people have been working on this for 60 years, you can probably image how well that works[]. The clever bit here is that you don't really need to do that. If you have enough data, the LSTMs can learn it on their own (see Figure 2, for example, where semantically-related sentences from multiple languages end up in the same neighborhoods). [] Actually, somewhat better than you'd think. It certainly wouldn't have done Pushkin any justice, but it was surprisingly decent on weather reports (etc).

csydas 10y ago
I was going to comment on the article and idea of multilingual NN's in general since, as I'm learning Russian currently, I'm having a hard time seeing how you can get a good automated translation without some editorializing or awareness of what's going on in both sentences. Your comment on NN's "... not doing Pushkin justice" stood out since it's something I've been wondering how NN could handle a language like Russian that traditionally doesn't use the verb "to be" in sentences, much like the E-Prime idea that exists in English for clarity of description. Just trying to re-think in terms of a Russian sentence instead of an English sentence has also proven difficult, and I'd be curious how you can automate it on NN's well. My colleagues (russian natives) are all able to spot auto-translated messages from me nearly instantly since it messes up pretty rudimentary grammar rules or just doesn't phrase the sentence like you would in native Russian.

nl 10y ago

No one has managed to get induction working reliably for natural language tasks in any way that is generalizable enough to rely on. So that is new.