5 ms·

I observe the state of the art on most Nlp tasks since many years: In 2018,2019 there was huge progress made each year on most tasks. 2020,except for a few tas

I observe the state of the art on most Nlp tasks since many years: In 2018,2019 there was huge progress made each year on most tasks. 2020,except for a few tasks have mostly stagnated... NLP accuracy is generally not production ready but the pace of progress was quick enough to have huge hopes. The root cause of the evil is: Nobody has build upon the state of the art pre trained language: XLnet while there are hundreds of declinaisons of BERTs. Just because of Google being behind it, if XLnet was owned by Google 2020 would have been different. I also believe that pre trained language have reached a plateau and we need new original ideas such as bringing variational autoencoder to Nlp and using metaoptimizers such as Ranger. The most pathetic one is that: Many major Nlp tasks have old SOTA in BERT just because nobody cared of using (not improving) XLnet on them which is absolute shame, I mean on many major tasks we could trivially win many percents of accuracy but nobody qualified bothered to do it,where goes the money then? To many NIH papers I guess. There's also not enough synergies, there are many interesting ideas that just needs to be combined and I think there's not enough funding for that, it's not exciting enough... I pray for 2021 to be a better year for AI, otherwise it will show evidence for a new AI progress winter

lacker 6y ago
Could you give an example of a major task that you think the state of the art could be trivially improved on with the xlnet approach?
- sooheon 6y ago
  Long (>2048 tokens) sequences. But GP is too focused on hyping XLNet for some reason. There are much more elegant attempts at improving the transformer architecture in just the past 8 months: Reformer, Performer, Macaron Net, and my current pet paper, Normalized Attention Pooling (https://arxiv.org/abs/2005.09561 https://arxiv.org/abs/2005.09561).
  - The_rationalist 6y ago
    But GP is too focused on hyping XLNet for some reason. Yeah some reasons, it might be because of stars alignment and could variate inversely with the weather. Or it might be because it's the paper with the biggest number of first place at SOTA leaderboards? https://paperswithcode.com/paper/xlnet-generalized-autoregressive-pretraining https://paperswithcode.com/paper/xlnet-generalized-autoregre... I've queried most of your examples on the SOTA database that is paperswithcode.com and they have almost zero results. You illustrate the problem, if researchers like you don't even know the general SOTA, how can it be expected to be beaten? But beyond scientists ignorance there is also the problem of models not submitting their results to paperswithcode.com or not testing them extensively but only on niche benchmarks. This second behavior sentence such potentially promising models to remain unknown and therefore mostly irrelevant. It's always remarkable how one can be a smart researcher and yet not adjust its behavior to be rational regarding those two flaws (not seeking SOTA knowledge, and not promoting SOTA knowledge; READ, WRITE)
- The_rationalist 6y ago
  I asked the (at the time) SOTA author to replace its BERT implementation with XLnet. He accepted and won 0.5% of accuracy on constituency parsing https://github.com/sebastianruder/NLP-progress/blob/master/english/constituency_parsing.md https://github.com/sebastianruder/NLP-progress/blob/master/e... which is actually a huge, it means 5% less errors. Which tasks remain to be tried? Most actually but an obvious one would be coreference resolution https://github.com/sebastianruder/NLP-progress/blob/master/english/coreference_resolution.md https://github.com/sebastianruder/NLP-progress/blob/master/e...
dpflan 6y ago
Thanks for the information. Do you know how the pandemic affected research output for 2020?
liviosoares 6y ago
Just to clarify one of your points regarding Google's involvement: XLnet, and the underlying TransformerXL technology, did have Google researchers involved: * https://ai.googleblog.com/2019/01/transformer-xl-unleashing-potential-of.html https://ai.googleblog.com/2019/01/transformer-xl-unleashing-... * https://arxiv.org/pdf/1901.02860.pdf https://arxiv.org/pdf/1901.02860.pdf * https://arxiv.org/pdf/1906.08237.pdf https://arxiv.org/pdf/1906.08237.pdf My understanding is that a CMU student interned at Google and developed most of the pieces of TransformerXL, which formed the basis of XLNet. The student and the Google researcher further collaborated with CMU researchers to finalize the work. (For the record, I think the remainder of your points do not match my understanding of NLP, which I do research in, but I just really wanted to clarify the XLNet story a bit).
bratao 6y ago
I do not agree with this. I work heavily with NLP models for production in the Legal domain (where my baseline is where a 8GB 1080 must predict more than 1000 words/sec). This year was when our team glued enough pieces of Deep Learning to outperform our previous statistic/old ML pipeline that was been optimized for years. Little things compound such as optimizers ( Ranger/Adahessian), better RNN ( IndRNN, Linear Transformers, Hopfield networks ) and techniques (cache everywhere, Torch script,gradient accumulation training)
- maxlamb 6y ago
  Interesting. What's the main goal(s) of your NLP models?
  - bratao 6y ago
    We work on multiple models, all related to legal proceedings and lawsuits, such as: - Structure Judicial Federal Register texts - Identify entities in Legal texts (citation to laws, other lawsuits) - Predict time to completion, risk and amount due of a lawsuit - Classifying judicial proceedings to non lawyers
    - grumple 6y ago
      How accurate is your prediction of time/risk/amount? How useful is identifying entities or classifying proceedings?
- danieldk 6y ago
  I do not agree with this. I work heavily with NLP models for production in the Legal domain (where my baseline is where a 8GB 1080 must predict more than 1000 words/sec). What kind of network are you using? I can do near-SoTA multi-task syntax annotation [1] with ~4000 tokens/s (~225 sentences/s) on a CPU with 4 threads using a transformer. Predicting 1000 words/second on a reasonably modern a GPU is easy, even with a relatively deep transformer network. [1] 8 tasks, including dependency parsing.
p1esk 6y ago
It'd be ironic if your comment was generated by GPT-3. But forget GPT-3. In 10 years, looking back at AI history, the year 2020 will probably be viewed as the point separating pre GPT-4 and post GPT-4 epochs. GPT-4 is the model I expect to make things interesting again, not just in NLP, but in AI.
- freyr 6y ago
  Are any of the recent NLP advancements due to improvements beyond throwing more data and horsepower at “dumb” models? Will GPT-4 be any different? It seems like the current approaches will always fall short of our loftier AI aspirations, but we’re reaching a level of mimicry where we can start to ask, “Does it matter for this task?”
  - disgruntledphd2 6y ago
    No, almost all the progress is driven by bigger GPUs and datasets. To be fair, things like CNN's and BERT were definitely massive improvements, but a lot of modern AI is just throwing compute at problems and seeing what sticks.
  - p1esk 6y ago
    Will GPT-4 be any different? That's the point - it does not need to be different. If it demonstrates similar improvement to what we saw with GPT-1 --> GPT-2 --> GPT-3, then it will be enough to actually start using it. It's like the progression MNIST --> CIFAR-10 --> ImageNet --> the point where object recognition is good enough for real world applications. But in addition to making it bigger, we can also make it better: smarter attention, external data queries, better word encoding, better data quality, more than one data type as input, etc. There's plenty of room for improvement.
- The_rationalist 6y ago
  It's not because something seems smart that it actually is. GPT - 3 is ridiculous in that it doesn't have semantic understanding. Text generation is the wrong task, it's cool to watch and actually fuel hype on the ML train but what you're really looking for is semantic parsing, which GPT or openAI has nothing to do with and is mostly underfunded.

lacker 6y ago

Could you give an example of a major task that you think the state of the art could be trivially improved on with the xlnet approach?

sooheon 6y ago
Long (>2048 tokens) sequences. But GP is too focused on hyping XLNet for some reason. There are much more elegant attempts at improving the transformer architecture in just the past 8 months: Reformer, Performer, Macaron Net, and my current pet paper, Normalized Attention Pooling (https://arxiv.org/abs/2005.09561 https://arxiv.org/abs/2005.09561).
- The_rationalist 6y ago
  But GP is too focused on hyping XLNet for some reason. Yeah some reasons, it might be because of stars alignment and could variate inversely with the weather. Or it might be because it's the paper with the biggest number of first place at SOTA leaderboards? https://paperswithcode.com/paper/xlnet-generalized-autoregressive-pretraining https://paperswithcode.com/paper/xlnet-generalized-autoregre... I've queried most of your examples on the SOTA database that is paperswithcode.com and they have almost zero results. You illustrate the problem, if researchers like you don't even know the general SOTA, how can it be expected to be beaten? But beyond scientists ignorance there is also the problem of models not submitting their results to paperswithcode.com or not testing them extensively but only on niche benchmarks. This second behavior sentence such potentially promising models to remain unknown and therefore mostly irrelevant. It's always remarkable how one can be a smart researcher and yet not adjust its behavior to be rational regarding those two flaws (not seeking SOTA knowledge, and not promoting SOTA knowledge; READ, WRITE)
The_rationalist 6y ago
I asked the (at the time) SOTA author to replace its BERT implementation with XLnet. He accepted and won 0.5% of accuracy on constituency parsing https://github.com/sebastianruder/NLP-progress/blob/master/english/constituency_parsing.md https://github.com/sebastianruder/NLP-progress/blob/master/e... which is actually a huge, it means 5% less errors. Which tasks remain to be tried? Most actually but an obvious one would be coreference resolution https://github.com/sebastianruder/NLP-progress/blob/master/english/coreference_resolution.md https://github.com/sebastianruder/NLP-progress/blob/master/e...

dpflan 6y ago

Thanks for the information. Do you know how the pandemic affected research output for 2020?

liviosoares 6y ago

Just to clarify one of your points regarding Google's involvement: XLnet, and the underlying TransformerXL technology, did have Google researchers involved: * https://ai.googleblog.com/2019/01/transformer-xl-unleashing-potential-of.html https://ai.googleblog.com/2019/01/transformer-xl-unleashing-... * https://arxiv.org/pdf/1901.02860.pdf https://arxiv.org/pdf/1901.02860.pdf * https://arxiv.org/pdf/1906.08237.pdf https://arxiv.org/pdf/1906.08237.pdf My understanding is that a CMU student interned at Google and developed most of the pieces of TransformerXL, which formed the basis of XLNet. The student and the Google researcher further collaborated with CMU researchers to finalize the work. (For the record, I think the remainder of your points do not match my understanding of NLP, which I do research in, but I just really wanted to clarify the XLNet story a bit).

bratao 6y ago

I do not agree with this. I work heavily with NLP models for production in the Legal domain (where my baseline is where a 8GB 1080 must predict more than 1000 words/sec). This year was when our team glued enough pieces of Deep Learning to outperform our previous statistic/old ML pipeline that was been optimized for years. Little things compound such as optimizers ( Ranger/Adahessian), better RNN ( IndRNN, Linear Transformers, Hopfield networks ) and techniques (cache everywhere, Torch script,gradient accumulation training)

maxlamb 6y ago
Interesting. What's the main goal(s) of your NLP models?
- bratao 6y ago
  We work on multiple models, all related to legal proceedings and lawsuits, such as: - Structure Judicial Federal Register texts - Identify entities in Legal texts (citation to laws, other lawsuits) - Predict time to completion, risk and amount due of a lawsuit - Classifying judicial proceedings to non lawyers
  - grumple 6y ago
    How accurate is your prediction of time/risk/amount? How useful is identifying entities or classifying proceedings?
danieldk 6y ago
I do not agree with this. I work heavily with NLP models for production in the Legal domain (where my baseline is where a 8GB 1080 must predict more than 1000 words/sec). What kind of network are you using? I can do near-SoTA multi-task syntax annotation [1] with ~4000 tokens/s (~225 sentences/s) on a CPU with 4 threads using a transformer. Predicting 1000 words/second on a reasonably modern a GPU is easy, even with a relatively deep transformer network. [1] 8 tasks, including dependency parsing.

p1esk 6y ago

It'd be ironic if your comment was generated by GPT-3. But forget GPT-3. In 10 years, looking back at AI history, the year 2020 will probably be viewed as the point separating pre GPT-4 and post GPT-4 epochs. GPT-4 is the model I expect to make things interesting again, not just in NLP, but in AI.

freyr 6y ago
Are any of the recent NLP advancements due to improvements beyond throwing more data and horsepower at “dumb” models? Will GPT-4 be any different? It seems like the current approaches will always fall short of our loftier AI aspirations, but we’re reaching a level of mimicry where we can start to ask, “Does it matter for this task?”
- disgruntledphd2 6y ago
  No, almost all the progress is driven by bigger GPUs and datasets. To be fair, things like CNN's and BERT were definitely massive improvements, but a lot of modern AI is just throwing compute at problems and seeing what sticks.
- p1esk 6y ago
  Will GPT-4 be any different? That's the point - it does not need to be different. If it demonstrates similar improvement to what we saw with GPT-1 --> GPT-2 --> GPT-3, then it will be enough to actually start using it. It's like the progression MNIST --> CIFAR-10 --> ImageNet --> the point where object recognition is good enough for real world applications. But in addition to making it bigger, we can also make it better: smarter attention, external data queries, better word encoding, better data quality, more than one data type as input, etc. There's plenty of room for improvement.
The_rationalist 6y ago
It's not because something seems smart that it actually is. GPT - 3 is ridiculous in that it doesn't have semantic understanding. Text generation is the wrong task, it's cool to watch and actually fuel hype on the ML train but what you're really looking for is semantic parsing, which GPT or openAI has nothing to do with and is mostly underfunded.