3 ms·
To be fair, in my field (deep learning, computer vision), the papers often do not contain enough information to reproduce the results. To take a recent example,
by m0zg 6y ago
To be fair, in my field (deep learning, computer vision), the papers often do not contain enough information to reproduce the results. To take a recent example, Google's EfficientDet paper did not contain enough detail to be able to implement BiFPN, so nobody could replicate their results until official implementation was released. And even then, to the best of my knowledge, nobody has been able to train the models to the same accuracy in PyTorch - the results matching Google's merely port the TensorFlow weights.
Much of the recent "efficient" DL work is like that. Efficient models are notoriously difficult to train, and all manner of secret sauce is simply not mentioned, and without it you won't get the same result. At higher levels of precision, a single percentage point of a metric can mean 10% increase in error rate, so this is not negligible.
To the authors' credit though, a lot of this work does get released in full source code form, so even if you can't achieve the same result on your own hardware, you can at least test the results using the provided weights, and see that they _are_ in fact achievable.
- StandardFuture 6y agoThis is why science needs to get rid of "papers" and make everything a public repository of information. GitHub, etc. could be used to file "Issues" (like your anecdote) to ensure a recorded history of what the paper might be missing or how it could be better expressed, etc. In fact, I really do not understand why in 2020 we are writing papers/PDFs anymore. We have more than enough software and software tooling to have superior formats for sharing and discussing information. Is our biggest hurdle to this that the most experienced scientists/professors/researchers in our day are not as technically competent as they would need to be? Otherwise, why is this not already the norm? We do not need ArXiv or journals, we need GitXiv.
- m0zg 6y agoThat papers are unnecessary is a common misconception shared by those not in the field. Papers are _extremely_ useful. I read a couple carefully every week and skim many others. But I will concede that a paper is not sufficient in most cases, at least in my field. And researchers have only very limited amount of time which they can devote to engaging in "discussions". Arguably their time would be much better spent creating the next great thing rather than fixing bugs in their latest paper, as frustrating as that is for everyone else.
- justinmeiners 6y agoI think the author of that comment was saying a paper alone is typically not sufficient. A researcher should summarize their work in a written paper AND provide resources, code, media, etc.
- m0zg 6y agoYes, but don't hold your breath for: 1. Researchers ditching papers as the primary means of exchanging ideas. 2. Researchers going back and fixing bugs, or even reviewing PRs. None of this is going to happen.
- justinmeiners 6y agoI don't think that's what's being requested. It's ok if resources are dumped and left "as is". We just need more than we are getting, and it's not like they don't have that, it's just not as expected for them to publish it.