3 ms·

> I often wonder why we spend so much time trying to justify certain research avenues over others and don't let people just research what they find is interesti

> I often wonder why we spend so much time trying to justify certain research avenues over others and don't let people just research what they find is interesting. I honestly feel this is mostly because of a (possibly well-justified) fear that, were we to not do that, almost all the money would quickly get captured by grifters whose ideas are fake and of no research value, but who were best at convincing they're worth funding. We can't read peoples' hearts, so we need to force out some output we can discriminate on.

godelski 2y ago
I believe you are correct, but at the same time, are we not already giving a lot of money to grifters? For most research funding, I think the incentives are misaligned for the type of grifters we're trying to prevent, i.e. people just pocketing the money. The money is too low and the barriers are too high. The grifting that does happen is more about metric hacking and is about the status one achieves in academia. Yeah, there's high profile people that make good money on this, but at the same time many in the respective fields are not shocked to find that their papers were faked. There were usually existing accusations. But the structures in place are not well aligned to oust or investigate them. The publish or perish paradigm gives little time for replication experiments and you should come with strong evidence before making accusations. We're also getting better at this, but more awareness and more discovery has not led to these people losing their positions our status. (We even see universities protect them at times) Take this recent plagiarism case[0]. Most authors have quite good citation records and metrics, but how does a work that's so egregious not cause other works to be investigated, it warrants suspicion. We have network graphs of researchers, with their collaborators, but why do we not build these graphs for abusers? Most of the time the abuse is purged behind closed doors and so it even makes it harder to build the networks. CS is especially egregious for this. I lost a lot of respect for a bunch of my ML researcher peers when they promoted Rabbit. The demo was so egregiously faked and none of the claims made sense given the state of research. But we throw tons of money at these types of problems. We thrive on hype bubbles and don't purge the conmen. Hell, we often reward bad behavior. Like how that intern that got fired from Bytedance for manipulating code won Best Paper and NeurIPS[1]. The HumanEval paper is ridiculously naive (to think you can write "leetcode style problems" and that because you wrote them "by hand" that they won't be spoiled by training on GitHub. You can find identical code to most of the canonical solutions trivially!). There's tons of memes in ML around plagiarism and as best as I can tell all these authors are still doing just fine. How an egregious case doesn't result in a 1 year probation for a conference is beyond me. We suspend students for this academic misconduct but bury it for professors and grad students? The incentives are all wrong and we need to take a serious look at ourselves. Because we are in fact protecting the grifters. It is just lucky that most people are uninterested in grifting in this domain, though as said, there isn't much incentive to (obviously changing with the prominence in ML). [0] https://openreview.net/forum?id=cIKQp84vqN https://openreview.net/forum?id=cIKQp84vqN [1] https://www.wired.com/story/bytedance-intern-best-paper-neurips/ https://www.wired.com/story/bytedance-intern-best-paper-neur...

godelski 2y ago

I believe you are correct, but at the same time, are we not already giving a lot of money to grifters? For most research funding, I think the incentives are misaligned for the type of grifters we're trying to prevent, i.e. people just pocketing the money. The money is too low and the barriers are too high. The grifting that does happen is more about metric hacking and is about the status one achieves in academia. Yeah, there's high profile people that make good money on this, but at the same time many in the respective fields are not shocked to find that their papers were faked. There were usually existing accusations. But the structures in place are not well aligned to oust or investigate them. The publish or perish paradigm gives little time for replication experiments and you should come with strong evidence before making accusations. We're also getting better at this, but more awareness and more discovery has not led to these people losing their positions our status. (We even see universities protect them at times) Take this recent plagiarism case[0]. Most authors have quite good citation records and metrics, but how does a work that's so egregious not cause other works to be investigated, it warrants suspicion. We have network graphs of researchers, with their collaborators, but why do we not build these graphs for abusers? Most of the time the abuse is purged behind closed doors and so it even makes it harder to build the networks. CS is especially egregious for this. I lost a lot of respect for a bunch of my ML researcher peers when they promoted Rabbit. The demo was so egregiously faked and none of the claims made sense given the state of research. But we throw tons of money at these types of problems. We thrive on hype bubbles and don't purge the conmen. Hell, we often reward bad behavior. Like how that intern that got fired from Bytedance for manipulating code won Best Paper and NeurIPS[1]. The HumanEval paper is ridiculously naive (to think you can write "leetcode style problems" and that because you wrote them "by hand" that they won't be spoiled by training on GitHub. You can find identical code to most of the canonical solutions trivially!). There's tons of memes in ML around plagiarism and as best as I can tell all these authors are still doing just fine. How an egregious case doesn't result in a 1 year probation for a conference is beyond me. We suspend students for this academic misconduct but bury it for professors and grad students? The incentives are all wrong and we need to take a serious look at ourselves. Because we are in fact protecting the grifters. It is just lucky that most people are uninterested in grifting in this domain, though as said, there isn't much incentive to (obviously changing with the prominence in ML). [0] https://openreview.net/forum?id=cIKQp84vqN https://openreview.net/forum?id=cIKQp84vqN [1] https://www.wired.com/story/bytedance-intern-best-paper-neurips/ https://www.wired.com/story/bytedance-intern-best-paper-neur...