4 ms·

> I would never have considered Markov processes to be an adequate predictability model for natural language. Would you consider LSTM an adequate model?

TheOtherHobbes 10y ago
Personally, no. I think all of these models are essentially trivial and a long way from genuine NLP. That doesn't mean they're not useful in very narrow domains. But language is pretty much the definition of the ultimate wide domain, and trying to cover it with statistical correlations makes as much sense as word counting Shakespeare to try to generate some new plays.
- mark_l_watson 10y ago
  I think you might be pleasantly surprised by recent results using DL and LSTM for building models of natural language. The next advancement I would like to see is handling anaphora resolution (resolving pronouns to previous noun phrases in text, resolving words like 'there' to a place mentioned elsewhere in text, etc.) Progress has been so rapid that I bet I don't have to wait long.
  - nschucher 10y ago
    Have you seen the results from Dynamic Memory Networks? [0] The relevant example from the paper: I: Jane went to the hallway. I: Mary walked to the bathroom. I: Sandra went to the garden. I: Daniel went back to the garden. I: Sandra took the milk there. Q: Where is the milk? A: garden Obviously just a toy task, but as you said, progress is rapid! [0]: http://arxiv.org/abs/1506.07285 http://arxiv.org/abs/1506.07285
    - mark_l_watson 10y ago
      Thanks for the link!
  - crypto5 10y ago
    > I think you might be pleasantly surprised by recent results using DL and LSTM for building models of natural language. What exactly those models are modelling?
forgotpwtomain 10y ago
> Deep models are important because without the extra “dimension” of depth/abstraction, there is no way to construct “shortcuts” between random variables that are separated by large amounts of time with short-range interactions; 1D models will be doomed to exponential decay. From my non-professional perspective the above seems that it should have been very obvious - (and also that correlations between variables for natural languages would be better explained by multi-dimensional structure). That is if you told me that this were proved / formally supported as it is in this paper, my reaction would be a "that sounds like reasonable approximation" not - "that result sounds very surprising I must read the paper".

TheOtherHobbes 10y ago

Personally, no. I think all of these models are essentially trivial and a long way from genuine NLP. That doesn't mean they're not useful in very narrow domains. But language is pretty much the definition of the ultimate wide domain, and trying to cover it with statistical correlations makes as much sense as word counting Shakespeare to try to generate some new plays.

mark_l_watson 10y ago
I think you might be pleasantly surprised by recent results using DL and LSTM for building models of natural language. The next advancement I would like to see is handling anaphora resolution (resolving pronouns to previous noun phrases in text, resolving words like 'there' to a place mentioned elsewhere in text, etc.) Progress has been so rapid that I bet I don't have to wait long.
- nschucher 10y ago
  Have you seen the results from Dynamic Memory Networks? [0] The relevant example from the paper: I: Jane went to the hallway. I: Mary walked to the bathroom. I: Sandra went to the garden. I: Daniel went back to the garden. I: Sandra took the milk there. Q: Where is the milk? A: garden Obviously just a toy task, but as you said, progress is rapid! [0]: http://arxiv.org/abs/1506.07285 http://arxiv.org/abs/1506.07285
  - mark_l_watson 10y ago
    Thanks for the link!
- crypto5 10y ago
  > I think you might be pleasantly surprised by recent results using DL and LSTM for building models of natural language. What exactly those models are modelling?

forgotpwtomain 10y ago

> Deep models are important because without the extra “dimension” of depth/abstraction, there is no way to construct “shortcuts” between random variables that are separated by large amounts of time with short-range interactions; 1D models will be doomed to exponential decay. From my non-professional perspective the above seems that it should have been very obvious - (and also that correlations between variables for natural languages would be better explained by multi-dimensional structure). That is if you told me that this were proved / formally supported as it is in this paper, my reaction would be a "that sounds like reasonable approximation" not - "that result sounds very surprising I must read the paper".