3 ms·

> I'll often ask authors of a paper whether it really works. Asking this often leads to the authors revealing holes in their papers, bits where the results aren

> I'll often ask authors of a paper whether it really works. Asking this often leads to the authors revealing holes in their papers, bits where the results aren't quite as good as depicted or things they tried that didn't work. When I was a student we also had to write papers in order to learn how to do research and document the results. At the end of each paper there had to be a section that discussed weaknesses and shortcomings of the research done. If there were factors left out from the paper for some reason or issues/problems that could not be dealt with in order to finish the paper (there was limited time as we had one semester maximum, this should not be an issue for Phds though) we had to mention it there. I considered this normal as it provides that transparency readers might need for their own research.

machinelearning 5y ago
Who actually cares about real research these days? Everyone participating seems to turn a blind eye to the issues and act like its someone else's problem. Students & Professors: "Publish or perish", if everyone else is doping, there's no other way to compete. Read the other comment from the newly minted PhD student below. Industrial labs: Our research is so compute intensive that it doesn't matter what the results are, no one is going to be able to compete or reproduce the work anyway. Plus, let's not release any code because it was built on our proprietary closed source infrastructure. Gathering mindshare and acquiring talent is primary, science is secondary. Conference organizers: We would like to have a higher bar for reproducibility but we don't have the resources to setup a standardized evaluation benchmark and we can't force authors to provide code. The truth is that we live in a disingenuous culture where people no longer have the moral strength to do what is right. They would much rather prefer to engage in virtue signaling than fix any of the real problems for fear of repercussion from the community and missing their KPIs. Everyone knows this has been a huge problem for the past several years and we need a strong group of people to organize and do something about this.

machinelearning 5y ago

Who actually cares about real research these days? Everyone participating seems to turn a blind eye to the issues and act like its someone else's problem. Students & Professors: "Publish or perish", if everyone else is doping, there's no other way to compete. Read the other comment from the newly minted PhD student below. Industrial labs: Our research is so compute intensive that it doesn't matter what the results are, no one is going to be able to compete or reproduce the work anyway. Plus, let's not release any code because it was built on our proprietary closed source infrastructure. Gathering mindshare and acquiring talent is primary, science is secondary. Conference organizers: We would like to have a higher bar for reproducibility but we don't have the resources to setup a standardized evaluation benchmark and we can't force authors to provide code. The truth is that we live in a disingenuous culture where people no longer have the moral strength to do what is right. They would much rather prefer to engage in virtue signaling than fix any of the real problems for fear of repercussion from the community and missing their KPIs. Everyone knows this has been a huge problem for the past several years and we need a strong group of people to organize and do something about this.