4 ms·

As an outsider to Academia, this looks like a long array of excuses and saddens me. A huge number of hours are wasted every year by undergrads fighting with inc

As an outsider to Academia, this looks like a long array of excuses and saddens me. A huge number of hours are wasted every year by undergrads fighting with incomplete papers when we could do better. We could enforce higher standards. Everything in a paper should be explainable and reproducible. Have a look at efforts by PapersWithCode, Arxiv Sanity, colah's blog, or 3blue1brown in either curating content or explaining concepts. I couldn't find a single excuse in this blog post for which we can't come up with a solution, if we have the consensus to enforce it.

vlovich123 6y ago
Publishing the code isn’t a panacea. Someone still has to find the bug. It hopefully would make it easier to come to a consensus faster when there is a dispute. In practice I think ultimately there is little glamour and fruitfulness for exploration. What might be impactful is setting up prizes for papers to be disproven or corrected. Ultimately though we can produce, as a species, far more information than we could ever hope to verify.
- lucfranken 6y ago
  I think there are quite some software engineers willing to take a look at the code to test/validate/fix interesting research proposals. When the right amount of info is shared a lot of cooperation is possible and challenging for both I suppose. The added value for both is learning the other skill. Gaining a broader knowledge to engage further in career and life.
andi999 6y ago
I think there is one good reason: eventually the student (or post doc) is supposed to come up with a new process (trying to implement a theoretical idea/possibility), and this is when he needs all the experience he gained when reestablishing a 'known' process (like riding a bike with side wheels). If she doesnt have the experience then it is too much. A metapher would be: if you make summit trails too easy, none of these mountaineers will be able to successfully challenge a new path. But I agree that the article is not good.
- pas 6y ago
  But... but... there is ample room to literally experiment, change variables, optimize methods, eliminate smaller problems, and more importantly to test other hypotheses. it should not depend on mindless/random trial-and-error guesswork and faith-based replication. yes, it's hard to write everything down well. look at the recent DARPA paralell replication study. but it gives very high quality research/science for a little overhead.
zhdc1 6y ago
I agree to a small extent, but this is very field dependent. Social sciences, where you're taking a bunch of data and running a Python, R, or Julia script against it? Sure. Require the PI to archive the data (or anonymized version where PII is an issue) along with the script and a quick description of the method. When you cross over to more STEM oriented fields, tacit knowledge (e.g., the 'experience' or stuff you pick up from your colleagues) becomes much more important. Reproducibility is still important, but you can only expect the PI to provide so much knowledge before it becomes untenable.
- disgruntledphd2 6y ago
  People do run experiments in social science too. There's just as much, if not more, tacit knowledge required to reproduce those experiments correctly (even after accounting for differential participant populations).
yiyus 6y ago
Enforcing higher standards is way more difficult than you think, even impractical most of the time. Having consensus and enforcing it is not trivial. Requiring from every PhD student the level of quality of 3blue1brown is extremely naive. Your comment is the equivalent of saying there should be no armies and instead we should be peaceful and love each other. Great idea. And those excuses of why I cannot connect A to B? Let's just reach consensus to use the same connector everywhere! I'd like to live in that world, but unfortunately it is not the one we got.
- altvali 6y ago
  You're making a strawman here. What I want is that prestigious journals/conferences not accept papers unless they come with the code, dataset and weights required to replicate the result, and every formula come with explanations of the terms. If the terms are not widely used, a more thorough explanation should be requested by reviewers. It's not that difficult. There's already a Distill Prize for Clarity in Machine Learning. A great spark would be if a company like DeepMind or OpenAI would enforce standards like these internally and host a conference that rewards papers at that standard. It would be a great PR move, at great benefit for humanity.
  - yiyus 6y ago
    There are many good papers that would not fulfill your requirements. I agree that ideally they would go through several revisions until every detail is fixed, but rejection levels would skyrocket. We would not have many great papers. I am not too familiar with the AI field, but I know that at least in my field the quality is very often too poor. We have to try to improve it. But if I rejected every paper I review because every detail to replicate it is not there, I would have rejected 99% of them. And some of the ones that eventually got accepted were very valid and very useful papers, and replication not so important after all.

vlovich123 6y ago

Publishing the code isn’t a panacea. Someone still has to find the bug. It hopefully would make it easier to come to a consensus faster when there is a dispute. In practice I think ultimately there is little glamour and fruitfulness for exploration. What might be impactful is setting up prizes for papers to be disproven or corrected. Ultimately though we can produce, as a species, far more information than we could ever hope to verify.

lucfranken 6y ago
I think there are quite some software engineers willing to take a look at the code to test/validate/fix interesting research proposals. When the right amount of info is shared a lot of cooperation is possible and challenging for both I suppose. The added value for both is learning the other skill. Gaining a broader knowledge to engage further in career and life.

andi999 6y ago

I think there is one good reason: eventually the student (or post doc) is supposed to come up with a new process (trying to implement a theoretical idea/possibility), and this is when he needs all the experience he gained when reestablishing a 'known' process (like riding a bike with side wheels). If she doesnt have the experience then it is too much. A metapher would be: if you make summit trails too easy, none of these mountaineers will be able to successfully challenge a new path. But I agree that the article is not good.

pas 6y ago
But... but... there is ample room to literally experiment, change variables, optimize methods, eliminate smaller problems, and more importantly to test other hypotheses. it should not depend on mindless/random trial-and-error guesswork and faith-based replication. yes, it's hard to write everything down well. look at the recent DARPA paralell replication study. but it gives very high quality research/science for a little overhead.

zhdc1 6y ago

I agree to a small extent, but this is very field dependent. Social sciences, where you're taking a bunch of data and running a Python, R, or Julia script against it? Sure. Require the PI to archive the data (or anonymized version where PII is an issue) along with the script and a quick description of the method. When you cross over to more STEM oriented fields, tacit knowledge (e.g., the 'experience' or stuff you pick up from your colleagues) becomes much more important. Reproducibility is still important, but you can only expect the PI to provide so much knowledge before it becomes untenable.

disgruntledphd2 6y ago
People do run experiments in social science too. There's just as much, if not more, tacit knowledge required to reproduce those experiments correctly (even after accounting for differential participant populations).

yiyus 6y ago

Enforcing higher standards is way more difficult than you think, even impractical most of the time. Having consensus and enforcing it is not trivial. Requiring from every PhD student the level of quality of 3blue1brown is extremely naive. Your comment is the equivalent of saying there should be no armies and instead we should be peaceful and love each other. Great idea. And those excuses of why I cannot connect A to B? Let's just reach consensus to use the same connector everywhere! I'd like to live in that world, but unfortunately it is not the one we got.

altvali 6y ago
You're making a strawman here. What I want is that prestigious journals/conferences not accept papers unless they come with the code, dataset and weights required to replicate the result, and every formula come with explanations of the terms. If the terms are not widely used, a more thorough explanation should be requested by reviewers. It's not that difficult. There's already a Distill Prize for Clarity in Machine Learning. A great spark would be if a company like DeepMind or OpenAI would enforce standards like these internally and host a conference that rewards papers at that standard. It would be a great PR move, at great benefit for humanity.
- yiyus 6y ago
  There are many good papers that would not fulfill your requirements. I agree that ideally they would go through several revisions until every detail is fixed, but rejection levels would skyrocket. We would not have many great papers. I am not too familiar with the AI field, but I know that at least in my field the quality is very often too poor. We have to try to improve it. But if I rejected every paper I review because every detail to replicate it is not there, I would have rejected 99% of them. And some of the ones that eventually got accepted were very valid and very useful papers, and replication not so important after all.