5 ms·

Unless you are a researcher (in academia or a corporate research lab), you should think twice before spending your time with these papers. I have seen repeated

by osipov 6y ago

Unless you are a researcher (in academia or a corporate research lab), you should think twice before spending your time with these papers. I have seen repeated examples of information technology industry professionals who go off on a wild goose chase of trying to parse the papers and reproduce them. If you are a machine learning practitioner or a data scientist in the industry, it is highly likely that you are going to waste your time with these papers. Here's a concrete example from the list: "John Lafferty, Andrew McCallum, Fernando C.N. Pereira: Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data, ICML 2001." This used to be the defining paper in early 2000s. Today it is important only as a road marker in the NLP research history which turned out to lead down an unproductive route. Those who have not spent meaningful time in academia working on publishing their own research papers tend to fetishize them. The reality is that even the best papers in the field are a mess of ideas designed to please fickle reviewers and academic superiors. Most papers explore nooks and crannies of ideas that are irrelevant to an industry practitioner and are filled with assumptions that turn out to be impractical. Unfortunately reading research papers has become a self-reinforcing status symbol for practitioners to name-drop and generally show off their in-crowd status rather than to rely on the ideas in the papers for a source of useful and practical information.

dtjohnnyb 6y ago
As a counterpoint, "Marco Tulio Ribeiro et al.: Beyond Accuracy: Behavioral Testing of NLP Models with CheckList, ACL 2020" is a highly practical paper that should be incredibly useful in helping IT industry professionals to make their NLP/machine learning systems more robust. The more we can move towards software engineering practices of testing and monitoring, the better. Maybe it's the exception that proves the rule though, I do agree with your point in general!
ericd 6y ago
Do you have recommendations for getting the lay of the research land without reading the papers? Personally, I liked OpenAI's Spinning Up RL (which basically points you to useful papers) and Fast.ai's videos, but after those, it seems like reading papers are the main option.
- osipov 6y ago
  I find it more productive to follow the people of the academia rather than the papers.
aapppwe 6y ago
so what should nlp pratictioners and enthusiast read instead?
psyklic 6y ago
If your goal is learning and understanding, reading papers won't be a waste of time. If you need good results fast, they are less likely to be useful.
wenc 6y ago
Very true of applied research papers in general. Folks who've only either worked in academia, or in industry, but not both often don't appreciate the real distance between a journal publication and a corresponding practical application/implementation. A "hot" paper that is wrong but intriguing can sometimes trigger a flurry of derivative works, and unless someone tries to implement it in the real world (and deal with the constraints of systems as found, not as imagined), suddenly it ends up spawning an entire new field that works in theory but not in practice. The incentives of academia and industry are simply different. Exceptions exist however, like journal papers that describe products/algorithms/technology that already exist in the real-world, like FFTW [1] or IPOPT [2]. In such cases, the publication exists merely as a form of technical documentation that other academics can easily cite. Reality is a really good arbiter of how solid an idea is. [1] http://www.fftw.org/pldi99.pdf http://www.fftw.org/pldi99.pdf [2] https://github.com/coin-or/Ipopt https://github.com/coin-or/Ipopt
Der_Einzige 6y ago
CRFs are still state of the art on many domains, and in some places are only just now being beaten by (far more expensive) transformer models. What are you smoking with this claim that CRFs are not useful to read about? CRFs are used a lot within industry and are very effective and interpretable...
- JHonaker 6y ago
  Yea, CRFs and graphical models in general are extremely versatile! It’s a shame more people don’t think about them. The major problem with them is computation, but there are approximate methods like belief propagation, expectation propagation, and even sequential Monte Carlo that you can leverage depending on your inference goals.
whymauri 6y ago
Uh, I disagree? Some of the best scientific discoveries of the 2000s came from insights found in old papers (50s, 60s, 70s). For example, optogenetics. And now with Transformers, Hopfield learning and continuous Hebbian dynamics are making a small comeback. I mean, sure, don't implement the paper verbatim, but it's depressing to discard decades worth of work and insights only to rebuild it all again. Our disregard for past 'unsexy' work is one of the largest inefficiencies in science, hands down.
- melenaboija 6y ago
  The comment starts with "Unless you are a researcher...". If you are a practitioner you are just trying to use the result of some research that someone else has done before, mostly to not have to do that research again. Sure this result is possible thanks to revisiting old ideas, but using it does not mean you are discarding anything.
bratao 6y ago
I agree with but a nitpick. CRF is still very much used, even on Transformers architectures as the last layer in tasks such as NER. Many here in the leaderboard use it https://paperswithcode.com/sota/named-entity-recognition-ner-on-conll-2003 https://paperswithcode.com/sota/named-entity-recognition-ner...

by osipov 6y ago

dtjohnnyb 6y ago

As a counterpoint, "Marco Tulio Ribeiro et al.: Beyond Accuracy: Behavioral Testing of NLP Models with CheckList, ACL 2020" is a highly practical paper that should be incredibly useful in helping IT industry professionals to make their NLP/machine learning systems more robust. The more we can move towards software engineering practices of testing and monitoring, the better. Maybe it's the exception that proves the rule though, I do agree with your point in general!

ericd 6y ago

Do you have recommendations for getting the lay of the research land without reading the papers? Personally, I liked OpenAI's Spinning Up RL (which basically points you to useful papers) and Fast.ai's videos, but after those, it seems like reading papers are the main option.

osipov 6y ago
I find it more productive to follow the people of the academia rather than the papers.

aapppwe 6y ago

so what should nlp pratictioners and enthusiast read instead?

psyklic 6y ago

If your goal is learning and understanding, reading papers won't be a waste of time. If you need good results fast, they are less likely to be useful.

wenc 6y ago

Very true of applied research papers in general. Folks who've only either worked in academia, or in industry, but not both often don't appreciate the real distance between a journal publication and a corresponding practical application/implementation. A "hot" paper that is wrong but intriguing can sometimes trigger a flurry of derivative works, and unless someone tries to implement it in the real world (and deal with the constraints of systems as found, not as imagined), suddenly it ends up spawning an entire new field that works in theory but not in practice. The incentives of academia and industry are simply different. Exceptions exist however, like journal papers that describe products/algorithms/technology that already exist in the real-world, like FFTW [1] or IPOPT [2]. In such cases, the publication exists merely as a form of technical documentation that other academics can easily cite. Reality is a really good arbiter of how solid an idea is. [1] http://www.fftw.org/pldi99.pdf http://www.fftw.org/pldi99.pdf [2] https://github.com/coin-or/Ipopt https://github.com/coin-or/Ipopt

Der_Einzige 6y ago

CRFs are still state of the art on many domains, and in some places are only just now being beaten by (far more expensive) transformer models. What are you smoking with this claim that CRFs are not useful to read about? CRFs are used a lot within industry and are very effective and interpretable...

JHonaker 6y ago
Yea, CRFs and graphical models in general are extremely versatile! It’s a shame more people don’t think about them. The major problem with them is computation, but there are approximate methods like belief propagation, expectation propagation, and even sequential Monte Carlo that you can leverage depending on your inference goals.

whymauri 6y ago

Uh, I disagree? Some of the best scientific discoveries of the 2000s came from insights found in old papers (50s, 60s, 70s). For example, optogenetics. And now with Transformers, Hopfield learning and continuous Hebbian dynamics are making a small comeback. I mean, sure, don't implement the paper verbatim, but it's depressing to discard decades worth of work and insights only to rebuild it all again. Our disregard for past 'unsexy' work is one of the largest inefficiencies in science, hands down.

melenaboija 6y ago
The comment starts with "Unless you are a researcher...". If you are a practitioner you are just trying to use the result of some research that someone else has done before, mostly to not have to do that research again. Sure this result is possible thanks to revisiting old ideas, but using it does not mean you are discarding anything.

bratao 6y ago

I agree with but a nitpick. CRF is still very much used, even on Transformers architectures as the last layer in tasks such as NER. Many here in the leaderboard use it https://paperswithcode.com/sota/named-entity-recognition-ner-on-conll-2003 https://paperswithcode.com/sota/named-entity-recognition-ner...