3 ms·

"NNs that rely on backprop only really work on continuous functions and many interesting things like language[0] are not products of those" is real high on my l

"NNs that rely on backprop only really work on continuous functions and many interesting things like language[0] are not products of those" is real high on my list of limits. [0]: https://www.linkedin.com/pulse/google-hyping-why-deep-learning-cannot-applied-easily-berkan-ph-d https://www.linkedin.com/pulse/google-hyping-why-deep-learni...

evrydayhustling 8y ago
State of the art on most NLP tasks is now provided by deep learning, thanks to a host of techniques for dealing with both categorical and structurally complex data, such as LSTMs. Here's a great resource for keeping up with development in that area: https://github.com/sebastianruder/NLP-progress https://github.com/sebastianruder/NLP-progress
yorwba 8y ago
That argument seems to be based on the idea that because written language uses a discrete encoding, it's impossible to use continuous functions. But as soon as you replace deterministic discrete values by continuous probabilities of those values, you're back in the space of continuous functions. Even non-NN approaches to NLP use that representation, because optimizing continuous parameters is something we can do very well. Discrete optimization is much harder.
- YeGoblynQueenne 8y ago
  And how well do all those techniques work when it comes to representing meaning, rather than structure?
  - yorwba 8y ago
    I think meaning isn't really separable from structure. But assuming you're talking about semantics vs. syntax, then continuous representations can certainly do a lot more than just syntax. They can model how words change in meaning over time [1], they can identify words that mean the same in different languages [2] and use that to translate whole sentences [3] without ever seeing a direct translation. Of course none of those tasks are solved to perfection yet, but some kind of meaning is definitely being captured. [1] https://arxiv.org/abs/1703.00607 https://arxiv.org/abs/1703.00607 [2] https://arxiv.org/abs/1710.04087 https://arxiv.org/abs/1710.04087 [3] https://arxiv.org/abs/1710.11041 https://arxiv.org/abs/1710.11041
    - YeGoblynQueenne 8y ago
      I disagree. If I tell you that argoubr + vliugo - viyt = plqtv, do you now understand what any of those words mean? If I say that "gye" must always precede "turo", do you know what "gye turo" means, or why it doesn't make sense to say "turo" without "gye"? What kind of meaning is captured by the relations between these words that these rules I just described, define? Would it help if I gave you all the possible ways that every word in the language those words are drawn from are to be articulated together, without telling you what any of those wors mean? And how would I explain those words' meaning without recourse to words whose meaning you already know? Also, that's what I mean by "structure" - the context in which tokens are found in text. I don't want to say "syntax" or "semantics" because it seems to me those are loaded terms that have very specific, er, meaning and I am not a linguist to be able to unpick it.
      - yorwba 8y ago
        > Would it help if I gave you all the possible ways that every word in the language those words are drawn from are to be articulated together, without telling you what any of those wors mean? And how would I explain those words' meaning without recourse to words whose meaning you already know? That's why I linked the two papers on unsupervised machine translation. Given enough text in an unknown language, it is already possible to turn those unknown words into words which humans understand.
        YeGoblynQueenne 8y ago
        But that's humans. Humans already understand language. I don't disagree that humans can extract meaning from language models. But that only speaks to the ability of humans to extract meaning from even very badly formed language. Not to the ability of those models to represent meaning. Edit: OK, I don't think I understand what you, er, mean. I think you're saying that language models essentially have a kind of understanding of language, and not just the ability to reproduce it without understanding it. Is that what you are saying?

evrydayhustling 8y ago

State of the art on most NLP tasks is now provided by deep learning, thanks to a host of techniques for dealing with both categorical and structurally complex data, such as LSTMs. Here's a great resource for keeping up with development in that area: https://github.com/sebastianruder/NLP-progress https://github.com/sebastianruder/NLP-progress

yorwba 8y ago

That argument seems to be based on the idea that because written language uses a discrete encoding, it's impossible to use continuous functions. But as soon as you replace deterministic discrete values by continuous probabilities of those values, you're back in the space of continuous functions. Even non-NN approaches to NLP use that representation, because optimizing continuous parameters is something we can do very well. Discrete optimization is much harder.

YeGoblynQueenne 8y ago
And how well do all those techniques work when it comes to representing meaning, rather than structure?
- yorwba 8y ago
  I think meaning isn't really separable from structure. But assuming you're talking about semantics vs. syntax, then continuous representations can certainly do a lot more than just syntax. They can model how words change in meaning over time [1], they can identify words that mean the same in different languages [2] and use that to translate whole sentences [3] without ever seeing a direct translation. Of course none of those tasks are solved to perfection yet, but some kind of meaning is definitely being captured. [1] https://arxiv.org/abs/1703.00607 https://arxiv.org/abs/1703.00607 [2] https://arxiv.org/abs/1710.04087 https://arxiv.org/abs/1710.04087 [3] https://arxiv.org/abs/1710.11041 https://arxiv.org/abs/1710.11041
  - YeGoblynQueenne 8y ago
    I disagree. If I tell you that argoubr + vliugo - viyt = plqtv, do you now understand what any of those words mean? If I say that "gye" must always precede "turo", do you know what "gye turo" means, or why it doesn't make sense to say "turo" without "gye"? What kind of meaning is captured by the relations between these words that these rules I just described, define? Would it help if I gave you all the possible ways that every word in the language those words are drawn from are to be articulated together, without telling you what any of those wors mean? And how would I explain those words' meaning without recourse to words whose meaning you already know? Also, that's what I mean by "structure" - the context in which tokens are found in text. I don't want to say "syntax" or "semantics" because it seems to me those are loaded terms that have very specific, er, meaning and I am not a linguist to be able to unpick it.
    - yorwba 8y ago
      > Would it help if I gave you all the possible ways that every word in the language those words are drawn from are to be articulated together, without telling you what any of those wors mean? And how would I explain those words' meaning without recourse to words whose meaning you already know? That's why I linked the two papers on unsupervised machine translation. Given enough text in an unknown language, it is already possible to turn those unknown words into words which humans understand.
      - YeGoblynQueenne 8y ago
        But that's humans. Humans already understand language. I don't disagree that humans can extract meaning from language models. But that only speaks to the ability of humans to extract meaning from even very badly formed language. Not to the ability of those models to represent meaning. Edit: OK, I don't think I understand what you, er, mean. I think you're saying that language models essentially have a kind of understanding of language, and not just the ability to reproduce it without understanding it. Is that what you are saying?