3 ms·

At which NLP tasks are they state the art? The only one where they are really competitive is dependency parsing. (from my own disjonction of cases) Also were t

At which NLP tasks are they state the art? The only one where they are really competitive is dependency parsing. (from my own disjonction of cases) Also were there any new real SOTA on any NLP tasks since last summer? I feel like accuracy progress has frozen.. What I would love would be to get a notification/mail when a task from which I subscribed got a new SOTA (from paperswithcode.com obviously).

tastroder 7y ago
> At which NLP tasks are they state the art? The only one where they are really competitive is dependency parsing. (from my own disjonction of cases) At tasks that actually involve graphs presumably. https://arxiv.org/pdf/1901.00596.pdf https://arxiv.org/pdf/1901.00596.pdf https://paperswithcode.com/task/graph-classification https://paperswithcode.com/task/graph-classification has a bunch of GNNs ranked #1 > Also were there any new real SOTA on any NLP tasks since last summer? I feel like accuracy progress has frozen.. That's pretty normal for winter/spring, it's not conference season.
- The_rationalist 7y ago
  it's not conference season Weird ^^ Imagine that I'm a scientist and that I made a big discovery X during winter. But for audience/visibility I only want to publish my results on conference Y during summer. Somebody, right before summer make the same discovery than mine. How do I protect the fact that I am the first one discoverer if I publish after the second?
  - p1esk 7y ago
    You post it on arxiv
    - The_rationalist 7y ago
      So the answer would be the SOTA results are on arxiv but are posted on paperswithcode.com leaderboards only at conference time? Sounds unlikely.
huac 7y ago
> Also were there any new real SOTA on any NLP tasks since last summer? I feel like accuracy progress has frozen.. Just in the last week there are two papers which claim SOTA on different tasks. Microsoft released Turing NLG (https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/ https://www.microsoft.com/en-us/research/blog/turing-nlg-a-1...) recently which claims SOTA on a couple tasks, it seems like the same transformer architecture, but with more layers and parameters, made feasible by training efficiency improvements. The biggest one seems to be partitioning how the model learns across different processes instead of replicating those states, which significantly improves communication, memory overhead, and training speed. Deepmind released the Compressive Transformer (https://deepmind.com/blog/article/A_new_model_and_dataset_for_long-range_memory https://deepmind.com/blog/article/A_new_model_and_dataset_fo...) which claims SOTA on two other "long-range" benchmarks. My understanding of the improvement here is that instead of discarding older states, in the traditional attention layer, the compressive transformer learns which states to keep, and which states to remove. I think these are two good examples of paper archetypes -- one where SOTA is achieved through more layers/training data/neurons (and the more interesting contribution is the improvements to model parallelism in training), and one where SOTA is achieved through a new/improved model architecture. I wonder, for most industrial practitioners, how much either paper is useful though. The Microsoft paper helps for training billion parameter models, but most won't train a model that deep; the Deepmind paper helps for training models over very long sequences, but most people aren't using book-length sequences.* * I remember reading somewhere that the attention mechanisms tend to only remember around 5 states (would love to see a source or study on this), which is pretty short, so would be interesting to try this model and see if the compressed transformer/attention mechanism can remember longer sequences.
- nl 7y ago
  Neither of these are Graph NNs though (although it was a weird question because Graph NNs solve a different problem)
  - huac 7y ago
    yes, true. there has been progress on NLP SOTA, just not from graph NN architectures. I have thought about using graph NN's where I am currently using RNN's, but they seem fairly immature / difficult to train over very large datasets so have not made much progress. I was following the work of https://www.octavian.ai/ https://www.octavian.ai/, but have not seen much recently. http://web.stanford.edu/class/cs224w/info.html http://web.stanford.edu/class/cs224w/info.html also looks interesting. looking briefly over the student projects, there appear to be a couple nlp projects (e.g. http://web.stanford.edu/class/cs224w/project/26418192.pdf http://web.stanford.edu/class/cs224w/project/26418192.pdf) but most leverage knowledge graph concepts.
nl 7y ago
Graph NNs aren't really used for SOTA NLP tasks. On the other hand, larger and larger transformer networks are constantly improving. Assuming you are in the Northern Hemisphere so "last summer" means Julyish, then Google's T5[1] and Microsoft Turing-NLG[2] come to mind. I find keeping an eye on HuggingFace's models list[3] is useful for this. [1] https://arxiv.org/abs/1910.10683 https://arxiv.org/abs/1910.10683 [2] https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/ https://www.microsoft.com/en-us/research/blog/turing-nlg-a-1... [3] https://github.com/huggingface/transformers#model-architectures https://github.com/huggingface/transformers#model-architectu...

tastroder 7y ago

> At which NLP tasks are they state the art? The only one where they are really competitive is dependency parsing. (from my own disjonction of cases) At tasks that actually involve graphs presumably. https://arxiv.org/pdf/1901.00596.pdf https://arxiv.org/pdf/1901.00596.pdf https://paperswithcode.com/task/graph-classification https://paperswithcode.com/task/graph-classification has a bunch of GNNs ranked #1 > Also were there any new real SOTA on any NLP tasks since last summer? I feel like accuracy progress has frozen.. That's pretty normal for winter/spring, it's not conference season.

The_rationalist 7y ago
it's not conference season Weird ^^ Imagine that I'm a scientist and that I made a big discovery X during winter. But for audience/visibility I only want to publish my results on conference Y during summer. Somebody, right before summer make the same discovery than mine. How do I protect the fact that I am the first one discoverer if I publish after the second?
- p1esk 7y ago
  You post it on arxiv
  - The_rationalist 7y ago
    So the answer would be the SOTA results are on arxiv but are posted on paperswithcode.com leaderboards only at conference time? Sounds unlikely.

huac 7y ago

> Also were there any new real SOTA on any NLP tasks since last summer? I feel like accuracy progress has frozen.. Just in the last week there are two papers which claim SOTA on different tasks. Microsoft released Turing NLG (https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/ https://www.microsoft.com/en-us/research/blog/turing-nlg-a-1...) recently which claims SOTA on a couple tasks, it seems like the same transformer architecture, but with more layers and parameters, made feasible by training efficiency improvements. The biggest one seems to be partitioning how the model learns across different processes instead of replicating those states, which significantly improves communication, memory overhead, and training speed. Deepmind released the Compressive Transformer (https://deepmind.com/blog/article/A_new_model_and_dataset_for_long-range_memory https://deepmind.com/blog/article/A_new_model_and_dataset_fo...) which claims SOTA on two other "long-range" benchmarks. My understanding of the improvement here is that instead of discarding older states, in the traditional attention layer, the compressive transformer learns which states to keep, and which states to remove. I think these are two good examples of paper archetypes -- one where SOTA is achieved through more layers/training data/neurons (and the more interesting contribution is the improvements to model parallelism in training), and one where SOTA is achieved through a new/improved model architecture. I wonder, for most industrial practitioners, how much either paper is useful though. The Microsoft paper helps for training billion parameter models, but most won't train a model that deep; the Deepmind paper helps for training models over very long sequences, but most people aren't using book-length sequences.* * I remember reading somewhere that the attention mechanisms tend to only remember around 5 states (would love to see a source or study on this), which is pretty short, so would be interesting to try this model and see if the compressed transformer/attention mechanism can remember longer sequences.

nl 7y ago
Neither of these are Graph NNs though (although it was a weird question because Graph NNs solve a different problem)
- huac 7y ago
  yes, true. there has been progress on NLP SOTA, just not from graph NN architectures. I have thought about using graph NN's where I am currently using RNN's, but they seem fairly immature / difficult to train over very large datasets so have not made much progress. I was following the work of https://www.octavian.ai/ https://www.octavian.ai/, but have not seen much recently. http://web.stanford.edu/class/cs224w/info.html http://web.stanford.edu/class/cs224w/info.html also looks interesting. looking briefly over the student projects, there appear to be a couple nlp projects (e.g. http://web.stanford.edu/class/cs224w/project/26418192.pdf http://web.stanford.edu/class/cs224w/project/26418192.pdf) but most leverage knowledge graph concepts.

nl 7y ago

Graph NNs aren't really used for SOTA NLP tasks. On the other hand, larger and larger transformer networks are constantly improving. Assuming you are in the Northern Hemisphere so "last summer" means Julyish, then Google's T5[1] and Microsoft Turing-NLG[2] come to mind. I find keeping an eye on HuggingFace's models list[3] is useful for this. [1] https://arxiv.org/abs/1910.10683 https://arxiv.org/abs/1910.10683 [2] https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/ https://www.microsoft.com/en-us/research/blog/turing-nlg-a-1... [3] https://github.com/huggingface/transformers#model-architectures https://github.com/huggingface/transformers#model-architectu...