7 ms·

Best Paper Awards at ACL 2020

36 points by polm23 6y ago

not_a_moth 6y ago
Thanks for posting. I was curious on "Climbing towards NLU: On Meaning, Form, and Understanding...". Interesting that a best paper award was given to a philosophical paper on distinction between meaning and form.
- mark_l_watson 6y ago
  I liked that article also. I went to NACL about 5 years ago, and was amazed that most papers even then were deep learning related. BTW, it is a small nuisance to not have article links on this web page. I had to copy titles/authors, do a web search, find and download. That said, between just skimming vs. careful reading, I probably spend an average of 20 minutes on a downloaded article so the overhead of finding and downloading is realy nothing.
- sqrt17 6y ago
  every couple decades, there is a new approach that promises to do solve more of the language understanding problems with less of humans actually thinking about and understanding the inner workings of language. Each time we get a phenomenon which is best described with the hype cycle analogy, namely consistent progress of what it can do, along with even faster progress of what people expect from the technique, until the actual progress cannot keep up with expectations and we settle at a new (much progressed but much disillusioned) level. BERT, GPT-2 and now GPT-3 have brought us to the tail end of such a hype cycle, with the most enthusiastic DNN enthusiasts still claiming that it's understanding meaning and general AI isn't far off, whereas the folks who give out ACL Best Paper awards (all long-time experts in their field) probably sigh and are thankful for a paper that breaks with the hype and tells the overenthusiasts that there's still stuff left to understand and discover. Do we need mediocre philosophical papers when we can have mediocre technical papers (or even fairly decent technical papers) instead? Hell no. But do we occasionally need a good and thoughtful philosophical paper to sober ourselves up? Admittedly yes.
teraflop 6y ago
I haven't read past the abstract, but "Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics" is an excellent title.
- nl 6y ago
  Great title and a great problem to work on.
nl 6y ago
GAIA: A Fine-grained Multimedia Knowledge Extraction System[1] looks pretty impressive. It's a huge amount of work they have done there - I guess DARPA funding buys a big team. The source is all available[2] but looks like a fair amount of work to get running. [1] https://www.aclweb.org/anthology/2020.acl-demos.11/ https://www.aclweb.org/anthology/2020.acl-demos.11/ [2] https://github.com/GAIA-AIDA https://github.com/GAIA-AIDA
lazyjeff 6y ago
If anyone likes checking out other lists of best paper awards, I've been collecting them (with the help of a few individuals and AllenAI) at https://jeffhuang.com/best_paper_awards/ https://jeffhuang.com/best_paper_awards/ There's 29 top CS conferences there with best papers since 1996. And NeurIPS and ICSA will be added soon...
- pabo 6y ago
  What an interesting collection, thanks! It's kind of surprising to me that Microsoft Research is the all-time leader, gathering more best paper awards than any top university. Wow. This in itself tells a story about Microsoft Research. Do you happen to have a time series on this? Would be interesting to see the trends in your data, e.g. to check how industrial research fares vs. academic research as a function of time. (There are some theories suggesting that industrial research is in a general decline [0]. Though the "best paper awards" metric is surely far from being perfect, it might still be viable as a proxy for corporate research performance.) [0] https://news.ycombinator.com/item?id=23246672 https://news.ycombinator.com/item?id=23246672
  - lazyjeff 6y ago
    Thanks for the comments. I don't have it in time-series format. You could scrape the page pretty easily to get it (even just a grep should get you most of what you want). I'm not sure if the N is big enough per year to give a reliable trend graph, but you're welcome to try.

36 points by polm23 6y ago

Thanks for posting. I was curious on "Climbing towards NLU: On Meaning, Form, and Understanding...". Interesting that a best paper award was given to a philosophical paper on distinction between meaning and form.

mark_l_watson 6y ago
I liked that article also. I went to NACL about 5 years ago, and was amazed that most papers even then were deep learning related. BTW, it is a small nuisance to not have article links on this web page. I had to copy titles/authors, do a web search, find and download. That said, between just skimming vs. careful reading, I probably spend an average of 20 minutes on a downloaded article so the overhead of finding and downloading is realy nothing.
sqrt17 6y ago
every couple decades, there is a new approach that promises to do solve more of the language understanding problems with less of humans actually thinking about and understanding the inner workings of language. Each time we get a phenomenon which is best described with the hype cycle analogy, namely consistent progress of what it can do, along with even faster progress of what people expect from the technique, until the actual progress cannot keep up with expectations and we settle at a new (much progressed but much disillusioned) level. BERT, GPT-2 and now GPT-3 have brought us to the tail end of such a hype cycle, with the most enthusiastic DNN enthusiasts still claiming that it's understanding meaning and general AI isn't far off, whereas the folks who give out ACL Best Paper awards (all long-time experts in their field) probably sigh and are thankful for a paper that breaks with the hype and tells the overenthusiasts that there's still stuff left to understand and discover. Do we need mediocre philosophical papers when we can have mediocre technical papers (or even fairly decent technical papers) instead? Hell no. But do we occasionally need a good and thoughtful philosophical paper to sober ourselves up? Admittedly yes.

teraflop 6y ago

I haven't read past the abstract, but "Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics" is an excellent title.

nl 6y ago
Great title and a great problem to work on.

nl 6y ago

GAIA: A Fine-grained Multimedia Knowledge Extraction System[1] looks pretty impressive. It's a huge amount of work they have done there - I guess DARPA funding buys a big team. The source is all available[2] but looks like a fair amount of work to get running. [1] https://www.aclweb.org/anthology/2020.acl-demos.11/ https://www.aclweb.org/anthology/2020.acl-demos.11/ [2] https://github.com/GAIA-AIDA https://github.com/GAIA-AIDA

lazyjeff 6y ago

If anyone likes checking out other lists of best paper awards, I've been collecting them (with the help of a few individuals and AllenAI) at https://jeffhuang.com/best_paper_awards/ https://jeffhuang.com/best_paper_awards/ There's 29 top CS conferences there with best papers since 1996. And NeurIPS and ICSA will be added soon...

pabo 6y ago
What an interesting collection, thanks! It's kind of surprising to me that Microsoft Research is the all-time leader, gathering more best paper awards than any top university. Wow. This in itself tells a story about Microsoft Research. Do you happen to have a time series on this? Would be interesting to see the trends in your data, e.g. to check how industrial research fares vs. academic research as a function of time. (There are some theories suggesting that industrial research is in a general decline [0]. Though the "best paper awards" metric is surely far from being perfect, it might still be viable as a proxy for corporate research performance.) [0] https://news.ycombinator.com/item?id=23246672 https://news.ycombinator.com/item?id=23246672
- lazyjeff 6y ago
  Thanks for the comments. I don't have it in time-series format. You could scrape the page pretty easily to get it (even just a grep should get you most of what you want). I'm not sure if the N is big enough per year to give a reliable trend graph, but you're welcome to try.