10 ms·

The paper is relevant to the original post because it points out that NLP models show state of the art results that are overfit to their respective datasets. Th

The paper is relevant to the original post because it points out that NLP models show state of the art results that are overfit to their respective datasets. This means that we may not be advancing as fast as we think. The issue here is also that few people have the monetary resources to provide a secondary analysis on these models. It's too expensive.

Eridrus 7y ago
I think this is way too broad an interpretation to draw from a paper that shows that a single dataset was bad. NLP datasets have had a good chunk of these issues, but people have been aware of this issue for a few years and newer datasets, e.g. SQuAD 2.0 explicitly try to address these sorts of issues.
- disgruntledphd2 7y ago
  Any chance you can provide some more recent results that justify your lack of concern? I'm an outsider to NLP, but have been very surprised by the recent improvements, so would be interested in anything that covers their limitations and newer datasets which help us to get over them.
  - Eridrus 7y ago
    The main breakthrough in the last 2 years is increasing success in unsupervised/self-supervised pre-training on very large datasets transfering to other tasks, e.g. ELMo, BERT, XLNet. Have a look at the BERT & XLNet papers, they have results on a wide range of datasets, including SQuAD 2.0. I think it's worth keeping in mind that all of the headlines about "surpases human performance" are 100% bullshit regardless of whether they are about vision, ASR or NLP. Performance on a dataset is not the same thing as real world usefulness. But this takes nothing away from the fact that ELMo/BERT/XLNet make it far easier to pull a model off the shelf and fine tune it for your task and get good performance. And the gains here are so broad and not task specific that issues with individual datasets are not really an issue, it is the massive gains in pretraining that are the real story, and if that even helps you overfit a test set, it shows that they now have enough built-in linguistic knowledge that we previously didn't have to even overfit a bad test set. This is obviously bad news for you if you thought that these tasks were basically solved, but it's completely irrelevant if you were building systems since these methods all result in relative gains on whatever language task they were pointed at. The bigger issue IMO with many recent NLP papers is not that they are expensive to train from scratch, which is a one time cost, but that they are expensive to even do inference with, which makes it hard to actually deploy them in real systems. But if your question is "is recent progress in NLP illusory" I think the answer is clearly no since the gains were across a broad range of tasks, and frankly I've used BERT and it makes my application better, the main issue I have is the runtime issue.

Eridrus 7y ago

I think this is way too broad an interpretation to draw from a paper that shows that a single dataset was bad. NLP datasets have had a good chunk of these issues, but people have been aware of this issue for a few years and newer datasets, e.g. SQuAD 2.0 explicitly try to address these sorts of issues.

disgruntledphd2 7y ago
Any chance you can provide some more recent results that justify your lack of concern? I'm an outsider to NLP, but have been very surprised by the recent improvements, so would be interested in anything that covers their limitations and newer datasets which help us to get over them.
- Eridrus 7y ago
  The main breakthrough in the last 2 years is increasing success in unsupervised/self-supervised pre-training on very large datasets transfering to other tasks, e.g. ELMo, BERT, XLNet. Have a look at the BERT & XLNet papers, they have results on a wide range of datasets, including SQuAD 2.0. I think it's worth keeping in mind that all of the headlines about "surpases human performance" are 100% bullshit regardless of whether they are about vision, ASR or NLP. Performance on a dataset is not the same thing as real world usefulness. But this takes nothing away from the fact that ELMo/BERT/XLNet make it far easier to pull a model off the shelf and fine tune it for your task and get good performance. And the gains here are so broad and not task specific that issues with individual datasets are not really an issue, it is the massive gains in pretraining that are the real story, and if that even helps you overfit a test set, it shows that they now have enough built-in linguistic knowledge that we previously didn't have to even overfit a bad test set. This is obviously bad news for you if you thought that these tasks were basically solved, but it's completely irrelevant if you were building systems since these methods all result in relative gains on whatever language task they were pointed at. The bigger issue IMO with many recent NLP papers is not that they are expensive to train from scratch, which is a one time cost, but that they are expensive to even do inference with, which makes it hard to actually deploy them in real systems. But if your question is "is recent progress in NLP illusory" I think the answer is clearly no since the gains were across a broad range of tasks, and frankly I've used BERT and it makes my application better, the main issue I have is the runtime issue.