4 ms·

Could you give an example of a major task that you think the state of the art could be trivially improved on with the xlnet approach?

by lacker 6y ago

sooheon 6y ago
Long (>2048 tokens) sequences. But GP is too focused on hyping XLNet for some reason. There are much more elegant attempts at improving the transformer architecture in just the past 8 months: Reformer, Performer, Macaron Net, and my current pet paper, Normalized Attention Pooling (https://arxiv.org/abs/2005.09561 https://arxiv.org/abs/2005.09561).
- The_rationalist 6y ago
  But GP is too focused on hyping XLNet for some reason. Yeah some reasons, it might be because of stars alignment and could variate inversely with the weather. Or it might be because it's the paper with the biggest number of first place at SOTA leaderboards? https://paperswithcode.com/paper/xlnet-generalized-autoregressive-pretraining https://paperswithcode.com/paper/xlnet-generalized-autoregre... I've queried most of your examples on the SOTA database that is paperswithcode.com and they have almost zero results. You illustrate the problem, if researchers like you don't even know the general SOTA, how can it be expected to be beaten? But beyond scientists ignorance there is also the problem of models not submitting their results to paperswithcode.com or not testing them extensively but only on niche benchmarks. This second behavior sentence such potentially promising models to remain unknown and therefore mostly irrelevant. It's always remarkable how one can be a smart researcher and yet not adjust its behavior to be rational regarding those two flaws (not seeking SOTA knowledge, and not promoting SOTA knowledge; READ, WRITE)
The_rationalist 6y ago
I asked the (at the time) SOTA author to replace its BERT implementation with XLnet. He accepted and won 0.5% of accuracy on constituency parsing https://github.com/sebastianruder/NLP-progress/blob/master/english/constituency_parsing.md https://github.com/sebastianruder/NLP-progress/blob/master/e... which is actually a huge, it means 5% less errors. Which tasks remain to be tried? Most actually but an obvious one would be coreference resolution https://github.com/sebastianruder/NLP-progress/blob/master/english/coreference_resolution.md https://github.com/sebastianruder/NLP-progress/blob/master/e...

Could you give an example of a major task that you think the state of the art could be trivially improved on with the xlnet approach?

by lacker 6y ago

Could you give an example of a major task that you think the state of the art could be trivially improved on with the xlnet approach?

sooheon 6y ago

Long (>2048 tokens) sequences. But GP is too focused on hyping XLNet for some reason. There are much more elegant attempts at improving the transformer architecture in just the past 8 months: Reformer, Performer, Macaron Net, and my current pet paper, Normalized Attention Pooling (https://arxiv.org/abs/2005.09561 https://arxiv.org/abs/2005.09561).

The_rationalist 6y ago
But GP is too focused on hyping XLNet for some reason. Yeah some reasons, it might be because of stars alignment and could variate inversely with the weather. Or it might be because it's the paper with the biggest number of first place at SOTA leaderboards? https://paperswithcode.com/paper/xlnet-generalized-autoregressive-pretraining https://paperswithcode.com/paper/xlnet-generalized-autoregre... I've queried most of your examples on the SOTA database that is paperswithcode.com and they have almost zero results. You illustrate the problem, if researchers like you don't even know the general SOTA, how can it be expected to be beaten? But beyond scientists ignorance there is also the problem of models not submitting their results to paperswithcode.com or not testing them extensively but only on niche benchmarks. This second behavior sentence such potentially promising models to remain unknown and therefore mostly irrelevant. It's always remarkable how one can be a smart researcher and yet not adjust its behavior to be rational regarding those two flaws (not seeking SOTA knowledge, and not promoting SOTA knowledge; READ, WRITE)

The_rationalist 6y ago

I asked the (at the time) SOTA author to replace its BERT implementation with XLnet. He accepted and won 0.5% of accuracy on constituency parsing https://github.com/sebastianruder/NLP-progress/blob/master/english/constituency_parsing.md https://github.com/sebastianruder/NLP-progress/blob/master/e... which is actually a huge, it means 5% less errors. Which tasks remain to be tried? Most actually but an obvious one would be coreference resolution https://github.com/sebastianruder/NLP-progress/blob/master/english/coreference_resolution.md https://github.com/sebastianruder/NLP-progress/blob/master/e...