4 ms·

I'm surprised that the comments are so critical here. Regarding the dataset criticisms: these are canonical datasets in ML research, I see no problem with using

by anjc 8y ago

I'm surprised that the comments are so critical here. Regarding the dataset criticisms: these are canonical datasets in ML research, I see no problem with using them for evaluations here. Using canonical datasets helps with reproducibility and should be encouraged. Secondly, there was no claim that polynomial regression is better than NN in all cases, or is more effective than SOTA boosting techniques. The claim seems to be that PR is more simple, scrutable, and effective, for certain datasets when compared to NN. This seems like a non-controversial finding to me, given that it's common knowledge. I appreciate seeing results like this.

p1esk 8y ago
I have a feeling that no one cares about these "canonical ML datasets" in 2018. If this paper was published back in 1998, it might have had generated more interest. Today we have a whole range of "canonical" DL datasets for every important application (image classification, object detection, speech recognition, NLP, games, etc). I might be wrong. How are their results relevant today?
- anjc 8y ago
  I don't understand your point? Of course there are a whole range of datasets, this paper chose some of them, and performed evaluations on them. If you want to reproduce their research, you can use the same algorithms on the same datasets. A quick search shows that hundreds of published papers have used these datasets just in 2018. Did you somehow already know the findings that they've made?
  - p1esk 8y ago
    My point is that the datasets they chose to test their ideas on are obscure, and/or small. From their results, it's not clear, even to someone familiar with the field, if they achieved anything impressive. For example, I've done a ton of work on image classification. Pretty much all papers on image classification published in the last 5 years have used one of the three "canonical" datasets: MNIST, CIFAR, or ImageNet. In fact, one can argue that MNIST has been a canonical image classification dataset since LeCun's famous paper in 1998. Also, anything published on the topic in 2018 better demonstrate good results on ImageNet to be taken seriously (unless it's from Hinton). I see "Letter Recognition Dataset" from UCI ML repository. What's the hell is that? Can you point to any paper published in a respectable conference (NIPS/ICML/CVPR) in the last 5 years that used it? They showed results from 9 irrelevant datasets, while a single one on ImageNet would instantly convinced everyone they are on to something.
    - anjc 8y ago
      > I see "Letter Recognition Dataset" from UCI ML repository. What's the hell is that? Can you point to any paper published in a respectable conference (NIPS/ICML/CVPR) in the last 5 years that used it? Why are you pro MNIST but anti Letter Recognition Dataset? A quick search of Scholar shows hundreds of papers that have used it in the last 4 years. Again, I'm still not sure what you want. They aren't doing a paper on NN vs PR for image classification. They aren't trying to get a higher score on a Kaggle classification challenge. Their hypothesis is that PR is a good alternative to NN, and their findings show that - for certain datasets - they are correct. Will more research arise from this? Will future results be mindblowing? Maybe, who knows. It's just research, and somebody has to do it.
      - p1esk 8y ago
        Their hypothesis is that PR is a good alternative to NN No, they are not correct, because they have not compared their model to state of the art NNs. That's exactly why I insist they use standard, canonical datasets, such as ImageNet: because it's easy to find well-known state of the art, and compare the models. Google Scholar, last 4 years: "Letter Recognition Dataset": 98 results, "MNIST": 10,000 results, "Imagenet": 18,000 results. They aren't doing a paper on NN vs PR for image classification If they include image classification results, and claim their model compares favorably to NNs, then yes, their paper is, at least in part, on NN vs PR for image classification. I only commented on image classification results because that's the field I'm familiar with. I suspect other results they presented would also be unconvincing to people working in those areas. Will more research arise from this? If they want more research to arise from that, then they should have made sure their results are convincing, or at least promising. As presented in the paper, they are not.
        nightski 8y ago
        Your obsession with state of the art on the specific subset of data that deep nn excels at blinds you to the entire point of this discussion.
claytonjy 8y ago
To your second point, the problem is their empirical results are entirely uninteresting. Who cares if PR beats poorly-specified NN's on datasets no one claims good NN performance on? They're winning at a game no one else is playing, or wants to play.

by anjc 8y ago

p1esk 8y ago

I have a feeling that no one cares about these "canonical ML datasets" in 2018. If this paper was published back in 1998, it might have had generated more interest. Today we have a whole range of "canonical" DL datasets for every important application (image classification, object detection, speech recognition, NLP, games, etc). I might be wrong. How are their results relevant today?

anjc 8y ago
I don't understand your point? Of course there are a whole range of datasets, this paper chose some of them, and performed evaluations on them. If you want to reproduce their research, you can use the same algorithms on the same datasets. A quick search shows that hundreds of published papers have used these datasets just in 2018. Did you somehow already know the findings that they've made?
- p1esk 8y ago
  My point is that the datasets they chose to test their ideas on are obscure, and/or small. From their results, it's not clear, even to someone familiar with the field, if they achieved anything impressive. For example, I've done a ton of work on image classification. Pretty much all papers on image classification published in the last 5 years have used one of the three "canonical" datasets: MNIST, CIFAR, or ImageNet. In fact, one can argue that MNIST has been a canonical image classification dataset since LeCun's famous paper in 1998. Also, anything published on the topic in 2018 better demonstrate good results on ImageNet to be taken seriously (unless it's from Hinton). I see "Letter Recognition Dataset" from UCI ML repository. What's the hell is that? Can you point to any paper published in a respectable conference (NIPS/ICML/CVPR) in the last 5 years that used it? They showed results from 9 irrelevant datasets, while a single one on ImageNet would instantly convinced everyone they are on to something.
  - anjc 8y ago
    > I see "Letter Recognition Dataset" from UCI ML repository. What's the hell is that? Can you point to any paper published in a respectable conference (NIPS/ICML/CVPR) in the last 5 years that used it? Why are you pro MNIST but anti Letter Recognition Dataset? A quick search of Scholar shows hundreds of papers that have used it in the last 4 years. Again, I'm still not sure what you want. They aren't doing a paper on NN vs PR for image classification. They aren't trying to get a higher score on a Kaggle classification challenge. Their hypothesis is that PR is a good alternative to NN, and their findings show that - for certain datasets - they are correct. Will more research arise from this? Will future results be mindblowing? Maybe, who knows. It's just research, and somebody has to do it.
    - p1esk 8y ago
      Their hypothesis is that PR is a good alternative to NN No, they are not correct, because they have not compared their model to state of the art NNs. That's exactly why I insist they use standard, canonical datasets, such as ImageNet: because it's easy to find well-known state of the art, and compare the models. Google Scholar, last 4 years: "Letter Recognition Dataset": 98 results, "MNIST": 10,000 results, "Imagenet": 18,000 results. They aren't doing a paper on NN vs PR for image classification If they include image classification results, and claim their model compares favorably to NNs, then yes, their paper is, at least in part, on NN vs PR for image classification. I only commented on image classification results because that's the field I'm familiar with. I suspect other results they presented would also be unconvincing to people working in those areas. Will more research arise from this? If they want more research to arise from that, then they should have made sure their results are convincing, or at least promising. As presented in the paper, they are not.
      - nightski 8y ago
        Your obsession with state of the art on the specific subset of data that deep nn excels at blinds you to the entire point of this discussion.

claytonjy 8y ago

To your second point, the problem is their empirical results are entirely uninteresting. Who cares if PR beats poorly-specified NN's on datasets no one claims good NN performance on? They're winning at a game no one else is playing, or wants to play.