6 ms·

Summarizing, they tried replicating the results of 18 recent papers, and determined that because of lack of available code or data, they were only able to attem

by nkurz 7y ago

Summarizing, they tried replicating the results of 18 recent papers, and determined that because of lack of available code or data, they were only able to attempt replications of 7 of them. Of these, they found that the "improvements" in 6 of them could also be achieved with a simpler non-machine learning approach (such as nearest neighbor). For the 7th (Mult-VAE), they were able to almost match the improvement with better tuning of one of the baselines used in that paper. Does this mean that the published results are garbage? No! But what does it mean? First, don't miss the main claim: less than half the published papers have sufficient code and data available for replication. While one might argue exactly what the standards for availability should be, if journals and reviewers were to demand it, this number can definitely be improved. Greater availability of data and code makes for better science. Second, my guess is that it means that paper authors are using excess "puffery" in an attempt to improve the chances that their papers are accepted for publication. They've decided that using a weak baseline and claiming a 50% improvement is more likely to result in publication than using a realistic baseline and claiming a 0-5% improvement. Unfortunately, they are probably right. The useful change (in my opinion) would be to change the standards by which papers are judged to be publishable. Reviewers should push back against weak baselines, but be more accepting of results that make modest claims. You can still have a useful paper even if the degree of improvement is small --- or even nonexistent. Maybe it will inspire someone else, maybe it's useful in an ensemble, or maybe it applies to a case where the baseline wouldn't. It's the puffery that's the problem, not the publication. Creating a system that rewards authors who more honestly evaluate their work would be an overall win.

goostavos 7y ago
Is there some background reason that academics generally don't provide code in their papers? Hell, in a lot of papers I've tried looking at recently[0], there will be paragraphs and paragraphs of english text describing an algorithm rather than even just plunking down some pseudo code. It makes pulling out the actual content of the papers such a slog. [0] exhibit A: https://ieeexplore.ieee.org/document/1017616 https://ieeexplore.ieee.org/document/1017616
- mayank 7y ago
  Research code is generally an abomination, scraped together by MS/PhD students who may never have had any industry exposure to best practices, norms, testing, etc. There may also be the fear of releasing buggy code, which if/when discovered, will discredit the paper. Far safer to just not release the code, since it isn't mandated (which should absolutely 100% change) Source: am former producer/publisher of abomination research code.
  - dv_dt 7y ago
    I wonder if a journal could be established that emphasizes code integrated papers, something that might accept submissions as jupyter notebooks for example. Edit: Perhaps one way to encourage this is via topic "reuse" papers, which present further detail on a previously published papers topic, but this time with code and more detailed discussion. It would at least help the situations in that the author gets some reuse of effort, but in a way that still shows new information and advances the field.
    - scribu 7y ago
      Formatting articles as jupyter notebooks might help with reproducibility, but it wouldn't necessarily help with understanding. distill.pub has a journal that strongly encourages interactive articles: https://distill.pub/journal/ https://distill.pub/journal/ It's a lot of work, tho: > One good test for whether your article is a fit for Distill is whether your collaborators and you are willing to put in whatever time is necessary to write and illustrate an outstanding article. In our experience, this often takes 100+ hours.
      - dv_dt 7y ago
        Reproducibility is one of the core problems of the article. I'm not sure any format can assure understanding by itself - that's more the purview of the reviewers I would think. Applying the criticism to my own suggestion: even the reproducibility issue is only tangentially helped by a Jupyter format, it may only bring reviewers to the point where one might execute the notebook (though there are still issues there if its an exceeding long computation or needs special hardware setup, or the notebook executes w/o demonstrating actual computation).
thefourthchime 7y ago
Veritasium did a great piece on this. https://www.youtube.com/watch?v=42QuXLucH3Q https://www.youtube.com/watch?v=42QuXLucH3Q
- igravious 7y ago
  Superb link. The end is humbling, “What gets me is the thought that even trying our best to figure out what's true, using our most sophisticated and rigorous mathematical tools: peer review, and the standards of practice, we still get it wrong so often; so how frequently do we delude ourselves when we're not using the scientific method? As flawed as our science may be, it is far away more reliable than any other way of knowing that we have.”
YeGoblynQueenne 7y ago
>> Second, my guess is that it means that paper authors are using excess "puffery" in an attempt to improve the chances that their papers are accepted for publication. They've decided that using a weak baseline and claiming a 50% improvement is more likely to result in publication than using a realistic baseline and claiming a 0-5% improvement. Unfortunately, they are probably right. I fear this contradicts your assertion that published results are not garbage. If researchers are claiming their systems are strong based on the fact that they beat weak baselines, then their claims are rubbish and their scholarship shoddy. If, on top of that, their work is difficult to reproduce, or replicate (by the distinction in the paper) then the published results are beyond garbage- they are meaningles garbage, garbage that fails to prove even its weak and insignificant claim to beat a weak baseline. Basically, this is exactly the kind of shoddy scholarship that Backus accused social scientists of, in his infamous speech about "Cargo-Cult Science". In that speech, he points out that basing new research on previous weak research can only lead in the proliferation of weak results (my interpretation of his words, from memory). This seems very much to be the case in machine learning research today. This article only provides further evidence that there is something rotten to the core in machine learning research. >> Reviewers should push back against weak baselines, but be more accepting of results that make modest claims. Agreed about the second part, however, the paper in the OP says that the way things are, it is very difficult to know which baselines are strong. My interpretation: because they're all trying to beat weak baselines that became established by beating earlier weak baselines. It's weak baselines all the way down.
- SubiculumCode 7y ago
  Weak baselines are not necessarily inappropriate. If using a new approach the point is to show that it is a viable direction for further exploration and refinement, not always whether it can beat finely tuned, well explored approaches.
  - prashp 7y ago
    I disagree with you. Weak baselines are inappropriate, especially when claiming 'state-of-the-art' results, which is very common in ML literature these days. If weak baselines are used as proof of superiority for a particular method, there are infinite other methods/minor adjustments which can be shown to be superior using the same logic.
krick 7y ago
Lately I was entertaining an idea of how this whole thing may be improved, but I don't really see how can I possibly influence it, so let me splash it out here. What do we have now: 1. We have a common platform, which is a de-facto standard for sharing (somewhat math-related) papers: arxiv.org. Basically, anyone can upload anything to arxiv.org, there's not much of quality review required. Which, all in all, I think is great. I mean, it's orders of magnitude better than other fields, where both publishing and accessing a published paper is expensive, painstakingly slow and inconvenient process which is obviously great for the journals that have a monopoly over it, but less so for everyone else. All being justified by the need of peer-review (which is not even sure to be working alright, because (1) we all know the quality of the vast number of studies in some social field is shit, and (2) you can even publish a completely bullshit paper in a very reputable journal if you try hard enough). 2. Much of the research is done in the academia (because, well, that's supposedly the reason it exists), but, paradoxically, the main moving force of the progress isn't really interested in progress. It is a competitive environment, so everybody is interested in publishing a paper which will be recognized as an "important one" later on, so the. They are not interested in anyone learning anything and especially not interested in making their rivals (and that is what any other researcher is, a rival) able to make a follow-up contribution easily. 3. Hence, many important studies by reputable scientists lack quality, which everyone knows, including the scientists. Papers are written in a passive-agressive tone, in particularly confusing terms an inconvenient format, they often lack the source code and the source data, a minor one-sentence contribution to the current state-of-the-art is spread around 20 pages which you have to read fully to understand what exactly the contribution is. 4. To summarize: everyone is interested in publicity of their work, nobody is interested in improving their work because it provides them the high ground, but everyone (including even those outside of the academia) is interested in improving others work, since it helps them to learn. Improvements on the others work, obviously, are not shared unless it can be framed as a follow-up study. How (I think) it might be improved. Assume a platform everyone uses to share their work, let's call it arxiv2. arxiv2 has a ranking system, which pushes the papers it "thinks" are of the better quality to the top. "Quality" is (maybe not exhaustively, but that's the main idea) a set of domain-specific flags which are considered to be beneficial for a study to be conclusive in this field, like "has a source code", "has a source data", "has a benchmark/accuracy/specificity/precision metrics", "summary/abstract succinctly captures the main result". Depending on the field, features may be more complicated in order to address some common problems like p-hacking or non-representative sampling in the study. But all the metrics have to be concrete. Nobody is required to include anything, but if they don't, and if others mark the work as lacking some features, it has less chance to get to the top. Everyone can vote for the paper, some minimal moderation may take place. Disputes may be solved similarly to how they are solved on wikipedia/stackoverflow (which are, admittedly, awful and frustrating, but order of magnitudes better than the status-quo in the research).
otakucode 7y ago
>Greater availability of data and code makes for better science. Lack of availability of data and code should be an immediate disqualifier from publication for research. In any field. There is no substantial difference between withholding source code or data and withholding experimental procedures or list of materials tested. I can only think that the people who withhold such things are either actively malicious or are completely ignorant of the stupendous danger of cutting corners in science. Given the history we have to work with, it's pretty clear that the distance from cutting a corner in a way that seems harmless to the deaths of millions of people is very short. This is not something people should trifle with.
scottlocklin 7y ago
The dirty secret of ML research is almost all of it is shit (much worse than Sturgeon's law), and something like KNN with minor tweaks will almost always win. I'm out in the trenches trying to turn ding dong papers into tools that does something useful, and the "muh mape 50% better" papers are effectively always something that should be ignored. Actually better algorithms and implementations (something like gradient boost machines/xgboost) are rare enough attempting to find them in the literature is futile.

by nkurz 7y ago

goostavos 7y ago

Is there some background reason that academics generally don't provide code in their papers? Hell, in a lot of papers I've tried looking at recently[0], there will be paragraphs and paragraphs of english text describing an algorithm rather than even just plunking down some pseudo code. It makes pulling out the actual content of the papers such a slog. [0] exhibit A: https://ieeexplore.ieee.org/document/1017616 https://ieeexplore.ieee.org/document/1017616

mayank 7y ago
Research code is generally an abomination, scraped together by MS/PhD students who may never have had any industry exposure to best practices, norms, testing, etc. There may also be the fear of releasing buggy code, which if/when discovered, will discredit the paper. Far safer to just not release the code, since it isn't mandated (which should absolutely 100% change) Source: am former producer/publisher of abomination research code.
- dv_dt 7y ago
  I wonder if a journal could be established that emphasizes code integrated papers, something that might accept submissions as jupyter notebooks for example. Edit: Perhaps one way to encourage this is via topic "reuse" papers, which present further detail on a previously published papers topic, but this time with code and more detailed discussion. It would at least help the situations in that the author gets some reuse of effort, but in a way that still shows new information and advances the field.
  - scribu 7y ago
    Formatting articles as jupyter notebooks might help with reproducibility, but it wouldn't necessarily help with understanding. distill.pub has a journal that strongly encourages interactive articles: https://distill.pub/journal/ https://distill.pub/journal/ It's a lot of work, tho: > One good test for whether your article is a fit for Distill is whether your collaborators and you are willing to put in whatever time is necessary to write and illustrate an outstanding article. In our experience, this often takes 100+ hours.
    - dv_dt 7y ago
      Reproducibility is one of the core problems of the article. I'm not sure any format can assure understanding by itself - that's more the purview of the reviewers I would think. Applying the criticism to my own suggestion: even the reproducibility issue is only tangentially helped by a Jupyter format, it may only bring reviewers to the point where one might execute the notebook (though there are still issues there if its an exceeding long computation or needs special hardware setup, or the notebook executes w/o demonstrating actual computation).

thefourthchime 7y ago

Veritasium did a great piece on this. https://www.youtube.com/watch?v=42QuXLucH3Q https://www.youtube.com/watch?v=42QuXLucH3Q

igravious 7y ago
Superb link. The end is humbling, “What gets me is the thought that even trying our best to figure out what's true, using our most sophisticated and rigorous mathematical tools: peer review, and the standards of practice, we still get it wrong so often; so how frequently do we delude ourselves when we're not using the scientific method? As flawed as our science may be, it is far away more reliable than any other way of knowing that we have.”

YeGoblynQueenne 7y ago

>> Second, my guess is that it means that paper authors are using excess "puffery" in an attempt to improve the chances that their papers are accepted for publication. They've decided that using a weak baseline and claiming a 50% improvement is more likely to result in publication than using a realistic baseline and claiming a 0-5% improvement. Unfortunately, they are probably right. I fear this contradicts your assertion that published results are not garbage. If researchers are claiming their systems are strong based on the fact that they beat weak baselines, then their claims are rubbish and their scholarship shoddy. If, on top of that, their work is difficult to reproduce, or replicate (by the distinction in the paper) then the published results are beyond garbage- they are meaningles garbage, garbage that fails to prove even its weak and insignificant claim to beat a weak baseline. Basically, this is exactly the kind of shoddy scholarship that Backus accused social scientists of, in his infamous speech about "Cargo-Cult Science". In that speech, he points out that basing new research on previous weak research can only lead in the proliferation of weak results (my interpretation of his words, from memory). This seems very much to be the case in machine learning research today. This article only provides further evidence that there is something rotten to the core in machine learning research. >> Reviewers should push back against weak baselines, but be more accepting of results that make modest claims. Agreed about the second part, however, the paper in the OP says that the way things are, it is very difficult to know which baselines are strong. My interpretation: because they're all trying to beat weak baselines that became established by beating earlier weak baselines. It's weak baselines all the way down.

SubiculumCode 7y ago
Weak baselines are not necessarily inappropriate. If using a new approach the point is to show that it is a viable direction for further exploration and refinement, not always whether it can beat finely tuned, well explored approaches.
- prashp 7y ago
  I disagree with you. Weak baselines are inappropriate, especially when claiming 'state-of-the-art' results, which is very common in ML literature these days. If weak baselines are used as proof of superiority for a particular method, there are infinite other methods/minor adjustments which can be shown to be superior using the same logic.

krick 7y ago

Lately I was entertaining an idea of how this whole thing may be improved, but I don't really see how can I possibly influence it, so let me splash it out here. What do we have now: 1. We have a common platform, which is a de-facto standard for sharing (somewhat math-related) papers: arxiv.org. Basically, anyone can upload anything to arxiv.org, there's not much of quality review required. Which, all in all, I think is great. I mean, it's orders of magnitude better than other fields, where both publishing and accessing a published paper is expensive, painstakingly slow and inconvenient process which is obviously great for the journals that have a monopoly over it, but less so for everyone else. All being justified by the need of peer-review (which is not even sure to be working alright, because (1) we all know the quality of the vast number of studies in some social field is shit, and (2) you can even publish a completely bullshit paper in a very reputable journal if you try hard enough). 2. Much of the research is done in the academia (because, well, that's supposedly the reason it exists), but, paradoxically, the main moving force of the progress isn't really interested in progress. It is a competitive environment, so everybody is interested in publishing a paper which will be recognized as an "important one" later on, so the. They are not interested in anyone learning anything and especially not interested in making their rivals (and that is what any other researcher is, a rival) able to make a follow-up contribution easily. 3. Hence, many important studies by reputable scientists lack quality, which everyone knows, including the scientists. Papers are written in a passive-agressive tone, in particularly confusing terms an inconvenient format, they often lack the source code and the source data, a minor one-sentence contribution to the current state-of-the-art is spread around 20 pages which you have to read fully to understand what exactly the contribution is. 4. To summarize: everyone is interested in publicity of their work, nobody is interested in improving their work because it provides them the high ground, but everyone (including even those outside of the academia) is interested in improving others work, since it helps them to learn. Improvements on the others work, obviously, are not shared unless it can be framed as a follow-up study. How (I think) it might be improved. Assume a platform everyone uses to share their work, let's call it arxiv2. arxiv2 has a ranking system, which pushes the papers it "thinks" are of the better quality to the top. "Quality" is (maybe not exhaustively, but that's the main idea) a set of domain-specific flags which are considered to be beneficial for a study to be conclusive in this field, like "has a source code", "has a source data", "has a benchmark/accuracy/specificity/precision metrics", "summary/abstract succinctly captures the main result". Depending on the field, features may be more complicated in order to address some common problems like p-hacking or non-representative sampling in the study. But all the metrics have to be concrete. Nobody is required to include anything, but if they don't, and if others mark the work as lacking some features, it has less chance to get to the top. Everyone can vote for the paper, some minimal moderation may take place. Disputes may be solved similarly to how they are solved on wikipedia/stackoverflow (which are, admittedly, awful and frustrating, but order of magnitudes better than the status-quo in the research).

otakucode 7y ago

>Greater availability of data and code makes for better science. Lack of availability of data and code should be an immediate disqualifier from publication for research. In any field. There is no substantial difference between withholding source code or data and withholding experimental procedures or list of materials tested. I can only think that the people who withhold such things are either actively malicious or are completely ignorant of the stupendous danger of cutting corners in science. Given the history we have to work with, it's pretty clear that the distance from cutting a corner in a way that seems harmless to the deaths of millions of people is very short. This is not something people should trifle with.

scottlocklin 7y ago

The dirty secret of ML research is almost all of it is shit (much worse than Sturgeon's law), and something like KNN with minor tweaks will almost always win. I'm out in the trenches trying to turn ding dong papers into tools that does something useful, and the "muh mape 50% better" papers are effectively always something that should be ignored. Actually better algorithms and implementations (something like gradient boost machines/xgboost) are rare enough attempting to find them in the literature is futile.