4 ms·
This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, w
by mark_l_watson 10y ago
This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data.
BTW, I went to the North American Association of Computational Linguistics conference in April and it seemed like half the papers used LSTM.
Edit: the NAACL 2016 papers are here: http://aclweb.org/anthology/N/N16/ http://aclweb.org/anthology/N/N16/
- osipov 10y ago@mark_l_watson -- the paper also seems important in another respect -- in the conclusion the authors suggest abandoning loss functions as optimization objective functions in machine learning and replacing them with mutual information functions.
- forgotpwtomain 10y ago> This seems like a very important paper, basically showing that Markov models with exponential decay of influence of tokens by distance are often a poor model, where as deep neural networks with LSTM (long short term memory) has power law decay of influence decay, which performs better for a variety of sequential data. As someone outside of this field, it seems to me that this kind of result should have been very obviously foreseeable, hindsight bias and all that of course - but I would never have considered Markov processes to be an adequate predictability model for natural language. Though obviously the formalized results are important. Could someone with more knowledge comment on what the current working assumptions were prior to this paper and what the consequences would be?
- thomasahle 10y ago> I would never have considered Markov processes to be an adequate predictability model for natural language. Would you consider LSTM an adequate model?
- TheOtherHobbes 10y agoPersonally, no. I think all of these models are essentially trivial and a long way from genuine NLP. That doesn't mean they're not useful in very narrow domains. But language is pretty much the definition of the ultimate wide domain, and trying to cover it with statistical correlations makes as much sense as word counting Shakespeare to try to generate some new plays.
- mark_l_watson 10y agoI think you might be pleasantly surprised by recent results using DL and LSTM for building models of natural language. The next advancement I would like to see is handling anaphora resolution (resolving pronouns to previous noun phrases in text, resolving words like 'there' to a place mentioned elsewhere in text, etc.) Progress has been so rapid that I bet I don't have to wait long.
- nschucher 10y agoHave you seen the results from Dynamic Memory Networks? [0] The relevant example from the paper: I: Jane went to the hallway. I: Mary walked to the bathroom. I: Sandra went to the garden. I: Daniel went back to the garden. I: Sandra took the milk there. Q: Where is the milk? A: garden Obviously just a toy task, but as you said, progress is rapid! [0]: http://arxiv.org/abs/1506.07285 http://arxiv.org/abs/1506.07285
- mark_l_watson 10y agoThanks for the link!
- crypto5 10y ago> I think you might be pleasantly surprised by recent results using DL and LSTM for building models of natural language. What exactly those models are modelling?
- forgotpwtomain 10y ago
- jakub_h 10y ago> but I would never have considered Markov processes to be an adequate predictability model for natural language Isn't it the case that for very short distances (several elements), power decay and exponential decay are (or can be made, with proper constants) more similar? Thus, if predictive models were originally studied only for very short sequences in the past (limited computational resources!), it seems to make sense that this is a mistake that anyone could have made more easily back then.
- jakub_h 10y ago> This seems like a very important paper > mark_l_watson Hah, I'll take your word for it, then. :) Are there any recent comprehensive monographs you'd recommend for state-of-the-art NPL, for someone who has yet to enter the field?
- arnsholt 10y agoMonographs aren't really a thing in NLP, beyond theses. There are a couple of reference books most people lean on, though: * Foundations of Statistical Natural Language Processing by Manning and Schütze * Speech and Language Processing by Jurafsky and Martin (which is being revised for a third edition, which you can look at: https://web.stanford.edu/~jurafsky/slp3/ https://web.stanford.edu/~jurafsky/slp3/ ) Beyond that, you're basically stuck reading the research literature. On the up-side, most of that literature is freely available from the ACL anthology at http://aclweb.org/anthology/ http://aclweb.org/anthology/
- jakub_h 10y agoThank you very much. I'm aware of a large number of books that I could read, but if Sturgeon's law applies, or at least if a lot of the reading would be redundant anyway, I'll rather ask for pointers than to waste a lot of time by rediscovering the wheel. Basically, what I was asking for was the NLP's respected equivalent of Norvig and Russell, or similar books from other fields. Those two books seem to fit to bill.
- mark_l_watson 10y agoIf you have the time, I would start by taking Andrew Ng's machine learning class and then this NLP class when it is next offered in September 2016 https://www.coursera.org/learn/nlp https://www.coursera.org/learn/nlp
- jakub_h 10y ago> Andrew Ng's machine learning class Ah yes, the one I've heard about but still have to take. :) Well, I guess I should give Coursesa a chance. (Somehow I'm not fond of their "timelined" format, it seems redundant if you're communicating with a machine. I hope the future of online learning will avoid it like the plague.)
- stelfer 10y agoThe paper itself is a mess. It's very interesting but they aren't doing themselves any favors. They need to clean up the presentation and get rid of the distracting phrases like "fail epically".
- Gargoyle 10y agoGood ol HN comments never disappoint.
- slashcom 10y agoForgive me, but I'm less impressed by the paper. As far as I can tell, they've only really shown that (1) language is recursive, which we know already; (2) markov models cannot capture recursive languages, which we've known; and (3) RNNs can, which we've known. But so can PCFGs and many other formalisms from the past 25 years, which they ignore. I did not read it very closely though.
- evanb 10y agoFrom the article: > We can formalize the above considerations by giving rules for a toy language L over an alphabet A. In the parlance of theoretical linguistics, our language is generated by a stochastic or probabilistic context-free grammar (PCFG) [41–44]. We will discuss the relationship between our model and a generic PCFG in Section C.