4 ms·

I don't understand your point? Of course there are a whole range of datasets, this paper chose some of them, and performed evaluations on them. If you want to r

by anjc 8y ago

I don't understand your point? Of course there are a whole range of datasets, this paper chose some of them, and performed evaluations on them. If you want to reproduce their research, you can use the same algorithms on the same datasets. A quick search shows that hundreds of published papers have used these datasets just in 2018. Did you somehow already know the findings that they've made?

p1esk 8y ago
My point is that the datasets they chose to test their ideas on are obscure, and/or small. From their results, it's not clear, even to someone familiar with the field, if they achieved anything impressive. For example, I've done a ton of work on image classification. Pretty much all papers on image classification published in the last 5 years have used one of the three "canonical" datasets: MNIST, CIFAR, or ImageNet. In fact, one can argue that MNIST has been a canonical image classification dataset since LeCun's famous paper in 1998. Also, anything published on the topic in 2018 better demonstrate good results on ImageNet to be taken seriously (unless it's from Hinton). I see "Letter Recognition Dataset" from UCI ML repository. What's the hell is that? Can you point to any paper published in a respectable conference (NIPS/ICML/CVPR) in the last 5 years that used it? They showed results from 9 irrelevant datasets, while a single one on ImageNet would instantly convinced everyone they are on to something.
- anjc 8y ago
  > I see "Letter Recognition Dataset" from UCI ML repository. What's the hell is that? Can you point to any paper published in a respectable conference (NIPS/ICML/CVPR) in the last 5 years that used it? Why are you pro MNIST but anti Letter Recognition Dataset? A quick search of Scholar shows hundreds of papers that have used it in the last 4 years. Again, I'm still not sure what you want. They aren't doing a paper on NN vs PR for image classification. They aren't trying to get a higher score on a Kaggle classification challenge. Their hypothesis is that PR is a good alternative to NN, and their findings show that - for certain datasets - they are correct. Will more research arise from this? Will future results be mindblowing? Maybe, who knows. It's just research, and somebody has to do it.
  - p1esk 8y ago
    Their hypothesis is that PR is a good alternative to NN No, they are not correct, because they have not compared their model to state of the art NNs. That's exactly why I insist they use standard, canonical datasets, such as ImageNet: because it's easy to find well-known state of the art, and compare the models. Google Scholar, last 4 years: "Letter Recognition Dataset": 98 results, "MNIST": 10,000 results, "Imagenet": 18,000 results. They aren't doing a paper on NN vs PR for image classification If they include image classification results, and claim their model compares favorably to NNs, then yes, their paper is, at least in part, on NN vs PR for image classification. I only commented on image classification results because that's the field I'm familiar with. I suspect other results they presented would also be unconvincing to people working in those areas. Will more research arise from this? If they want more research to arise from that, then they should have made sure their results are convincing, or at least promising. As presented in the paper, they are not.
    - nightski 8y ago
      Your obsession with state of the art on the specific subset of data that deep nn excels at blinds you to the entire point of this discussion.

by anjc 8y ago

p1esk 8y ago

My point is that the datasets they chose to test their ideas on are obscure, and/or small. From their results, it's not clear, even to someone familiar with the field, if they achieved anything impressive. For example, I've done a ton of work on image classification. Pretty much all papers on image classification published in the last 5 years have used one of the three "canonical" datasets: MNIST, CIFAR, or ImageNet. In fact, one can argue that MNIST has been a canonical image classification dataset since LeCun's famous paper in 1998. Also, anything published on the topic in 2018 better demonstrate good results on ImageNet to be taken seriously (unless it's from Hinton). I see "Letter Recognition Dataset" from UCI ML repository. What's the hell is that? Can you point to any paper published in a respectable conference (NIPS/ICML/CVPR) in the last 5 years that used it? They showed results from 9 irrelevant datasets, while a single one on ImageNet would instantly convinced everyone they are on to something.

anjc 8y ago
> I see "Letter Recognition Dataset" from UCI ML repository. What's the hell is that? Can you point to any paper published in a respectable conference (NIPS/ICML/CVPR) in the last 5 years that used it? Why are you pro MNIST but anti Letter Recognition Dataset? A quick search of Scholar shows hundreds of papers that have used it in the last 4 years. Again, I'm still not sure what you want. They aren't doing a paper on NN vs PR for image classification. They aren't trying to get a higher score on a Kaggle classification challenge. Their hypothesis is that PR is a good alternative to NN, and their findings show that - for certain datasets - they are correct. Will more research arise from this? Will future results be mindblowing? Maybe, who knows. It's just research, and somebody has to do it.
- p1esk 8y ago
  Their hypothesis is that PR is a good alternative to NN No, they are not correct, because they have not compared their model to state of the art NNs. That's exactly why I insist they use standard, canonical datasets, such as ImageNet: because it's easy to find well-known state of the art, and compare the models. Google Scholar, last 4 years: "Letter Recognition Dataset": 98 results, "MNIST": 10,000 results, "Imagenet": 18,000 results. They aren't doing a paper on NN vs PR for image classification If they include image classification results, and claim their model compares favorably to NNs, then yes, their paper is, at least in part, on NN vs PR for image classification. I only commented on image classification results because that's the field I'm familiar with. I suspect other results they presented would also be unconvincing to people working in those areas. Will more research arise from this? If they want more research to arise from that, then they should have made sure their results are convincing, or at least promising. As presented in the paper, they are not.
  - nightski 8y ago
    Your obsession with state of the art on the specific subset of data that deep nn excels at blinds you to the entire point of this discussion.