3 ms·

> While it's not a dichotomy, who will be seen as "better"? Someone who spent tons of time to write reproducible clean code with tests, tracked down all minor d

> While it's not a dichotomy, who will be seen as "better"? Someone who spent tons of time to write reproducible clean code with tests, tracked down all minor details and pondered about things that may change the results by 0.1 percent, or someone who was a bit sloppy but got 3 publications in the meantime and got to know famous professors at conferences? This is a common narrative, but I'm not certain if the example you give is typical. I've written about this before: https://news.ycombinator.com/item?id=18743531 https://news.ycombinator.com/item?id=18743531 Since 2018 I have tried to publish at least one paper debunking something every year. So far, if anything, being more careful has led me to publish more, not less. Admittedly, I don't think everyone should be as careful as I am as people would quickly run into diminishing returns, but the idea that being careful necessarily means publishing less has not been true in my experience. Also, I don't think the typical case is something changing the results by 0.1%. In my experience when I catch an error it's usually larger than that. The first debunking paper I published was about something that was one or two orders of magnitude off the correct value in the typical case, but still received 300 citations... Ultimately I think it would be best if an academic field adopts uniform standards for publication. That way, a sloppy researcher can't pump out 3 bad papers. Many academic communities have nominal standards that are not enforced. Some examples are discussed here: https://www.osti.gov/biblio/1141709 https://www.osti.gov/biblio/1141709

bonoboTP 7y ago
It often feels like a quixotic fight. Strong claims sell papers, but are hard to stand behind for a researcher with high standards, who is very critical of overall practices in a field. Their "best" recourse in the idealistic sense would be to publish an overarching meta analysis of the field, and pointing out problematic practices, gaining enemies and being labeled as a crusader. In my field, computer vision, a major problem is uncomparable setups, far reaching claims etc. A major pattern is introducing some fancy model, showing that it gets a bit better results that could well be due to noise or some tuning on the test set (not even secretly, just trying many different things and putting the bold formatting on your best number and claiming it as SOTA). Whether it was really due to your fancy model or not is hard to prove. But my experience is that mundane, pedestrian changes in a model introduce way bigger performance changes than whether or not you use the fancy module or architectural tweak that is proposed in mediocre published works. Which also means you can absolutely not compare models created by different people, especially when the performance gap is small. Tiny details can screw up the whole ordering, so trying to interpret the order and draw conclusions from it is equivalent to reading tea leaves. Now surely this applies mostly to mediocre research. But that is the majority. Top of the line research is more solid, but if you're a small unknown researcher, your best strategy is to start getting out there, publish and network.

bonoboTP 7y ago

It often feels like a quixotic fight. Strong claims sell papers, but are hard to stand behind for a researcher with high standards, who is very critical of overall practices in a field. Their "best" recourse in the idealistic sense would be to publish an overarching meta analysis of the field, and pointing out problematic practices, gaining enemies and being labeled as a crusader. In my field, computer vision, a major problem is uncomparable setups, far reaching claims etc. A major pattern is introducing some fancy model, showing that it gets a bit better results that could well be due to noise or some tuning on the test set (not even secretly, just trying many different things and putting the bold formatting on your best number and claiming it as SOTA). Whether it was really due to your fancy model or not is hard to prove. But my experience is that mundane, pedestrian changes in a model introduce way bigger performance changes than whether or not you use the fancy module or architectural tweak that is proposed in mediocre published works. Which also means you can absolutely not compare models created by different people, especially when the performance gap is small. Tiny details can screw up the whole ordering, so trying to interpret the order and draw conclusions from it is equivalent to reading tea leaves. Now surely this applies mostly to mediocre research. But that is the majority. Top of the line research is more solid, but if you're a small unknown researcher, your best strategy is to start getting out there, publish and network.