3 ms·

Sounds to me like they want to replace the current grant process with something that resembles Google's peer bonus system. Popular sorts will get the lion's sh

Sounds to me like they want to replace the current grant process with something that resembles Google's peer bonus system. Popular sorts will get the lion's share, renegades will get peanuts. And this would compound existing problems in the tenure system wherein the decision to grant you tenure is in part determined by your competitors at other universities. Imagine, for example, if your promotion at Google were in part decided by Jeff Bezos. In my own post-doc experience, I caught a National Academy of Sciences member at Johns Hopkins faking data 20 years ago this year, and he decided to threaten me over it. Since no one stood up for me at the time, I left for tech a month or so later. And it's a shame because the method that exposed him was an early form of adversarial search that's now all the rage with Deep Learning. I don't see how a grant system based on a Mean Girls(tm) model of funding allocation would have prevented this.

coherentpony 10y ago
> In my own post-doc experience, I caught a National Academy of Sciences member at Johns Hopkins faking data 20 years ago this year, and he decided to threaten me over it. That's a bold claim. Do you have evidence to support this claim? Or a name? Could you point us to the data set they faked? If others aren't able to reproduce their data (under the same circumstances), that's fraud.
- oneshot908 10y ago
  There's no benefit to me in outing the professor at this point so I'm not going to do so. If you feel that undermines my credibility, I'm OK with that. I don't need a lawsuit. But the reason my own research was never published was that my post-doc advisor demanded my work achieve the same Nobel-prize worthy result that this professor was claiming to have attained. To put that into perspective, imagine if the requirement to submit a machine learning paper to arxiv was building SkyNet. And this was a status quo requirement for publication so researchers tended to make bold claims about doing so, all of which eventually got debunked, and there still hasn't been a Nobel prize for the achievement yet 20+ years later. There has been significant progress though. The professor went on to complete a long and successful career but the work in question was exposed as bogus shortly thereafter by a competition that was the equivalent of ImageNet for the field. It was yet another example of predicting the training set in an age when people didn't really understand yet why that would be a problem. And it wouldn't have amounted to data fakery except that said professor denied there was any issue whatsoever when I presented counterexamples to the model located by adversarial search and then proceeded to threaten me professionally, paraphrasing: "Your data could take me down, and if that happens, I'm taking you with me." It's mostly behind me now, but I bring it up in this thread because I don't see how this new model of funding would have prevented this or a myriad of other issues with science funding.
  - xaa 10y ago
    Ugh, shiver. I've seen a variety of "problematic" data analysis behavior by very senior people, so I have no trouble believing this story. Nor do I blame you for not elaborating to random internet people. Although it is sad that his intimidation was not only rewarded, but apparently intimidated you right out of the field. But IME, they usually try to cover their bases by (fake?) incompetence, rather than this kind of thing. That is, they claim to be unable to see severe problems even when the evidence is placed right in front of them. Plausible deniability, I guess. In your example, it would be like "I don't see why we can't build Skynet. And if we claim to have built Skynet and have overstated our case, we can just fix it in future work. Anyway, who cares what we claim as long as we made a good-faith effort and get that next grant funded." Personally though, I doubt whether most fabricated data is fabricated by PIs themselves, but by postdocs etc put exactly under the kind of impossible pressure you're describing. Then, of course, they can take credit for the successes and pretend to be shocked and outraged if the fabrication ever comes to light. The VW scandal probably happens somewhere in science daily. I agree with your conclusion. Also, the distribution of "credit" or "popularity" is highly exponential in any field, so this proposal would just exacerbate the already rich-and-senior get more rich-and-senior phenomenon.
  - yeukhon 10y ago
    Whistleblower. Announce it annonymously, or just do it. Seriously, we can't get bullied like this. If we let this go, we will repeat the fraud research Harvard scientists did after getting paid by the sugar industry and hundreds of other scientists willingly to write paper with subjective evidences and conclusions because they were paid to write in certain way. We need to stand up. I am willing to support you if you have concrete evidences.
    - xaa 10y ago
      You're really underestimating how much this happens. One incident is a drop in the bucket, and there won't be a whistleblower every time. IMO the only way to fix it, especially in CS type disciplines, is to make it a requirement for publication that all the code is available and easily reproducible. For other disciplines, I don't know the answer. At its core, this is a problem of very high competition and of failures being unpublishable. Really, it's not a great thing to out highly respected scientists who fabricate -- even though they richly deserve it -- because it erodes public confidence in science. It would be better if we could find a way to prevent these things before they happen.

coherentpony 10y ago

> In my own post-doc experience, I caught a National Academy of Sciences member at Johns Hopkins faking data 20 years ago this year, and he decided to threaten me over it. That's a bold claim. Do you have evidence to support this claim? Or a name? Could you point us to the data set they faked? If others aren't able to reproduce their data (under the same circumstances), that's fraud.

oneshot908 10y ago
There's no benefit to me in outing the professor at this point so I'm not going to do so. If you feel that undermines my credibility, I'm OK with that. I don't need a lawsuit. But the reason my own research was never published was that my post-doc advisor demanded my work achieve the same Nobel-prize worthy result that this professor was claiming to have attained. To put that into perspective, imagine if the requirement to submit a machine learning paper to arxiv was building SkyNet. And this was a status quo requirement for publication so researchers tended to make bold claims about doing so, all of which eventually got debunked, and there still hasn't been a Nobel prize for the achievement yet 20+ years later. There has been significant progress though. The professor went on to complete a long and successful career but the work in question was exposed as bogus shortly thereafter by a competition that was the equivalent of ImageNet for the field. It was yet another example of predicting the training set in an age when people didn't really understand yet why that would be a problem. And it wouldn't have amounted to data fakery except that said professor denied there was any issue whatsoever when I presented counterexamples to the model located by adversarial search and then proceeded to threaten me professionally, paraphrasing: "Your data could take me down, and if that happens, I'm taking you with me." It's mostly behind me now, but I bring it up in this thread because I don't see how this new model of funding would have prevented this or a myriad of other issues with science funding.
- xaa 10y ago
  Ugh, shiver. I've seen a variety of "problematic" data analysis behavior by very senior people, so I have no trouble believing this story. Nor do I blame you for not elaborating to random internet people. Although it is sad that his intimidation was not only rewarded, but apparently intimidated you right out of the field. But IME, they usually try to cover their bases by (fake?) incompetence, rather than this kind of thing. That is, they claim to be unable to see severe problems even when the evidence is placed right in front of them. Plausible deniability, I guess. In your example, it would be like "I don't see why we can't build Skynet. And if we claim to have built Skynet and have overstated our case, we can just fix it in future work. Anyway, who cares what we claim as long as we made a good-faith effort and get that next grant funded." Personally though, I doubt whether most fabricated data is fabricated by PIs themselves, but by postdocs etc put exactly under the kind of impossible pressure you're describing. Then, of course, they can take credit for the successes and pretend to be shocked and outraged if the fabrication ever comes to light. The VW scandal probably happens somewhere in science daily. I agree with your conclusion. Also, the distribution of "credit" or "popularity" is highly exponential in any field, so this proposal would just exacerbate the already rich-and-senior get more rich-and-senior phenomenon.
- yeukhon 10y ago
  Whistleblower. Announce it annonymously, or just do it. Seriously, we can't get bullied like this. If we let this go, we will repeat the fraud research Harvard scientists did after getting paid by the sugar industry and hundreds of other scientists willingly to write paper with subjective evidences and conclusions because they were paid to write in certain way. We need to stand up. I am willing to support you if you have concrete evidences.
  - xaa 10y ago
    You're really underestimating how much this happens. One incident is a drop in the bucket, and there won't be a whistleblower every time. IMO the only way to fix it, especially in CS type disciplines, is to make it a requirement for publication that all the code is available and easily reproducible. For other disciplines, I don't know the answer. At its core, this is a problem of very high competition and of failures being unpublishable. Really, it's not a great thing to out highly respected scientists who fabricate -- even though they richly deserve it -- because it erodes public confidence in science. It would be better if we could find a way to prevent these things before they happen.