4 ms·

I don't understand how papers like this end up on the homepage of HN. The abstract makes very broad claims, but the paper itself only runs toy experiments on ti

I don't understand how papers like this end up on the homepage of HN. The abstract makes very broad claims, but the paper itself only runs toy experiments on tiny datasets with tiny neural networks. No good ML practitioner believes that tiny, shallow, fully-connected neural networks are the best algorithm for every problem (just look at Kaggle results if you don't believe me). Especially for small, easy datasets like the ones analyzed in this paper, NNs are often not the best choice. However, for large scale image classification, density modeling, sequence modeling, etc. (none of which are tested in the paper), NNs are SOTA. Hilariously, the paper only compares polynomial regression to tiny neural networks. I bet if they had thrown in results from XGBoost or other classical ML techniques, polynomial regression would be blown out of the water. There's also some heuristics you can use to tell when a paper like this isn't necessarily representative of the field of ML: - Missing citations (e.g. "It is well-known that NNs are prone to overfitting [Chollet and Allaire(2018)], which has been the subject of much study, e.g. [?]."). - Inconsistent formatting (e.g. the tables on page 7) - Sentences like "Much more empirical work is needed to explore these issues." following something that sounds like an easy experiment to try.

luckyt 8y ago
You're absolutely right. A good way to tell if a paper is legit is search it on Google Scholar and see how many citations it has: this paper has none. If it's an arXiv preprint (not in a peer-reviewed journal) and also a low number of citations, definitely take it with a grain of salt.
- anjc 8y ago
  This is clearly a bad way to tell if a paper is legit, seeing as this particular paper seems to have been published a week ago and will therefore probably not have any citations for close to a year. Judge papers based on their methodology, evaluations and findings (at least). For some fields, you'll find the most cutting edge research on arXiv.
  - luckyt 8y ago
    Yes, ideally you would carefully read the paper and evaluate its methodology, evaluations, and results, but this requires you to be knowledgeable in the field. That's why peer review exists -- some experts in the field read it and determine if it passes the bar, so you don't have to. Citation count is useful for determining the relative importance of published papers. 10-100 citations is ok, 100+ citations means it made a significant contribution, 1000+ citations means it's a landmark paper in the field and probably worth reading. In the case of this paper, it's very new so citation count is not very meaningful, but it's a useful heuristic to evaluate most papers.
    - anjc 8y ago
      A lot of 'gaming the system' and backscratching goes on in Academia. Relying on citation counts, journal/conference names, and the oversight of 2 or 3 peers (who may be 1st year PhD students) is a veerrrrry dubious heuristic. Having said that, it'll certainly cut down the size of your reading list.
    - Someone 8y ago
      https://www.sciencedirect.com/science/article/pii/0022072889800063 https://www.sciencedirect.com/science/article/pii/0022072889... has been cited 603 times, so it is somewhere between “made a significant contribution” and “landmark paper in the field”?
hooloovoo_zoo 8y ago
If anyone is wondering, that missing reference appears to be https://www.researchgate.net/publication/3857463_Overfitting_and_neural_networks_Conjugate_gradient_and_backpropagation https://www.researchgate.net/publication/3857463_Overfitting....
cs702 8y ago
I came here to say pretty much the same things, but would add another heuristic: - Grandiose-sounding claims ("the NNAEPR principle") backed only by generic informal arguments and/or well-known facts ("...for more realistic activation functions, simply note that they themselves can usually be approximated by polynomials...").
mindcrime 8y ago
I don't understand how papers like this end up on the homepage of HN. I don't understand what it is you don't understand. Clearly HN is not a site dedicated strictly to academics and people focused on the ceremony of academia. So if a paper contains even an idea that might be interesting, and seems likely to foster some interesting discussion, it's probably going to get upvotes. It making the front page or not presumably depends on which other stories are also contending for a front-page spot at the same time. In any case, getting a lot of upvotes doesn't mean that the paper was "good", that its conclusions were justified, or anything else. It just means it was something that people thought was worth talking about. Missing citations (e.g. "It is well-known that NNs are prone to overfitting [Chollet and Allaire(2018)], which has been the subject of much study, e.g. [?]."). - Inconsistent formatting (e.g. the tables on page 7) It's a pre-print, and not a published paper, no? My assumption was that the authors intended to fix those in a subsequent revision.
- sdenton4 8y ago
  At least in my end of the world (math), typically the paper is sent for review and posted to arxiv at the same time; the paper should already pretty much meet the quality bar for publication. The ML community I think suffers from some (shall we say) poor sportsmanship, with people occasionally posting rather half-baked papers to establish priority on an idea, before it's even been fully fleshed out or explored. This paper feels like that, on both the 'science' and 'bothering to use reasonable LaTeX' fronts.
  - mindcrime 8y ago
    At least in my end of the world (math), typically the paper is sent for review and posted to arxiv at the same time; the paper should already pretty much meet the quality bar for publication. Ah, interesting. At first blush that seems like that kinda defeats the purpose of releasing a pre-print, but I can kinda see why someone might do it that way.

luckyt 8y ago

You're absolutely right. A good way to tell if a paper is legit is search it on Google Scholar and see how many citations it has: this paper has none. If it's an arXiv preprint (not in a peer-reviewed journal) and also a low number of citations, definitely take it with a grain of salt.

anjc 8y ago
This is clearly a bad way to tell if a paper is legit, seeing as this particular paper seems to have been published a week ago and will therefore probably not have any citations for close to a year. Judge papers based on their methodology, evaluations and findings (at least). For some fields, you'll find the most cutting edge research on arXiv.
- luckyt 8y ago
  Yes, ideally you would carefully read the paper and evaluate its methodology, evaluations, and results, but this requires you to be knowledgeable in the field. That's why peer review exists -- some experts in the field read it and determine if it passes the bar, so you don't have to. Citation count is useful for determining the relative importance of published papers. 10-100 citations is ok, 100+ citations means it made a significant contribution, 1000+ citations means it's a landmark paper in the field and probably worth reading. In the case of this paper, it's very new so citation count is not very meaningful, but it's a useful heuristic to evaluate most papers.
  - anjc 8y ago
    A lot of 'gaming the system' and backscratching goes on in Academia. Relying on citation counts, journal/conference names, and the oversight of 2 or 3 peers (who may be 1st year PhD students) is a veerrrrry dubious heuristic. Having said that, it'll certainly cut down the size of your reading list.
  - Someone 8y ago
    https://www.sciencedirect.com/science/article/pii/0022072889800063 https://www.sciencedirect.com/science/article/pii/0022072889... has been cited 603 times, so it is somewhere between “made a significant contribution” and “landmark paper in the field”?

hooloovoo_zoo 8y ago

If anyone is wondering, that missing reference appears to be https://www.researchgate.net/publication/3857463_Overfitting_and_neural_networks_Conjugate_gradient_and_backpropagation https://www.researchgate.net/publication/3857463_Overfitting....

cs702 8y ago

I came here to say pretty much the same things, but would add another heuristic: - Grandiose-sounding claims ("the NNAEPR principle") backed only by generic informal arguments and/or well-known facts ("...for more realistic activation functions, simply note that they themselves can usually be approximated by polynomials...").

mindcrime 8y ago

I don't understand how papers like this end up on the homepage of HN. I don't understand what it is you don't understand. Clearly HN is not a site dedicated strictly to academics and people focused on the ceremony of academia. So if a paper contains even an idea that might be interesting, and seems likely to foster some interesting discussion, it's probably going to get upvotes. It making the front page or not presumably depends on which other stories are also contending for a front-page spot at the same time. In any case, getting a lot of upvotes doesn't mean that the paper was "good", that its conclusions were justified, or anything else. It just means it was something that people thought was worth talking about. Missing citations (e.g. "It is well-known that NNs are prone to overfitting [Chollet and Allaire(2018)], which has been the subject of much study, e.g. [?]."). - Inconsistent formatting (e.g. the tables on page 7) It's a pre-print, and not a published paper, no? My assumption was that the authors intended to fix those in a subsequent revision.

sdenton4 8y ago
At least in my end of the world (math), typically the paper is sent for review and posted to arxiv at the same time; the paper should already pretty much meet the quality bar for publication. The ML community I think suffers from some (shall we say) poor sportsmanship, with people occasionally posting rather half-baked papers to establish priority on an idea, before it's even been fully fleshed out or explored. This paper feels like that, on both the 'science' and 'bothering to use reasonable LaTeX' fronts.
- mindcrime 8y ago
  At least in my end of the world (math), typically the paper is sent for review and posted to arxiv at the same time; the paper should already pretty much meet the quality bar for publication. Ah, interesting. At first blush that seems like that kinda defeats the purpose of releasing a pre-print, but I can kinda see why someone might do it that way.