4 ms·

The incentive system in science is immensely screwed up. Researchers are rewarded not for high quality work flows, reproducible results, and solid methodology

by Fede_V 11y ago

The incentive system in science is immensely screwed up. Researchers are rewarded not for high quality work flows, reproducible results, and solid methodology but for neat, linear stories with a simple high impact result. Imagine that a scientist designs a very intelligent experiment to test a plausible hypothesis - she has an excellent plan to analyze the results (pre-registered, to avoid p-value hacking) - and after doing the experiment, she finds no evidence of the effect that she was looking for. Why should the result (which is the only thing the scientist is not in control of) determine the reward given to the scientist? Funding should be given to investigate a particular phenomena, not to find high impact results.

jacquesm 11y ago
The reason funding goes to those that find 'good' results is the same reason why funding goes to those that start successful start-ups. Even though both parties probably have less control over the outcome than they think the perception is that they do and that betting on them again is a better bet than betting on someone that did not succeed to score. That's how we're wired, we simply like success stories much more than that we like stories of failure.
- digi_owl 11y ago
  Just world?
  - verbin217 11y ago
    False inherencies are the staple of those that wish to understand the world such that they might exploit it. If they weren't prepared to demonstrate that the pursuit of a solution was necessarily a fool's game then they'd have to give an actual shit.
- pbhjpbhj 11y ago
  The analogy doesn't completely work. If the data shows your hypothesis is false, or doesn't support its truth then it doesn't matter if you're the best experimenter in your field nor if you worked most competently the world is what it is (barring poor experiment design!). An hypothesis isn't a product, falsifying your hypothesis is success - you've increased the pool of human knowledge (or increased the support for an element of it). In business good and bad products can both lead to profit and can both lead to losses. Profit and loss can both be considered success and failure. On your last sentence - I don't think it's as self-evident as it seems. We tend to like the successes that profit us but in general I'm not sure we humans are that congratulatory of other's success. Schadenfreude seems very popular.
- dba7dba 11y ago
  Startup and science experiments are different. Startups fail, although nobody involved likes it. Science experiments are supposed to FAIL, many times. That's what science is.
Lawtonfogle 11y ago
In many cases the funding is politically motivated. In turn this corrupts the results by a type of 'survival of the fittest' where fitness is determined by one's biases aligning with one's funding sources.
TimPC 11y ago
While it's noisy and high variance, the approach is similar to start-ups in that we want experiments to be designed in ways that they are likely to confirm underlying theories. This makes positive results more valuable than negative results and something the scientist has some influence over, but not complete control of. Successful experiments increase the chance of future experiments being successful in much the same way that having previously had a successful start-up increases the chances of future start-up success. Particularly when committees cover huge areas, it's a lot easier to fund based on track records than to accurately determine experiment quality of an experiment well outside your expertise.
- Fede_V 11y ago
  I hope I'm not reading what you wrote uncharitably, but, by rewarding people with positive results, we are inevitably encouraging people to either design their experiments in ways that confirm their hypothesis, or at worse, to falsify their data. Thankfully, I've seen very little of the latter, but I've seen quite a bit of the former. Even unconsciously, people are afraid to design experiments that would give a black and white answer and falsify a cherish theory. People with lots of positive results are not necessarily better scientists. They are just as often less scrupulous and better salesmen - Peter Thiel noticed that as well.
- learnstats2 11y ago
  Yes, and that causes so many problems. As you indicate, all of science is easily biased by what has been done and what has been published in the past. Negative results are essential for a complete understanding of science and when they are repressed, complete access to the globally available data set is not available. If I'm an ethically-dubious pharma or lobby group, I love this situation. I keep funding testing until someone comes up with a positive result by chance or p-value hacking - then fanfare that result all the way to the media and to the bank. (Negative results don't need to be seen) If I'm a cynical scientist, I take the money and p-value hack until I get the results my funding body wants, so that I have access to more money in their next round of funding. Now I'm far more successful than a completely honest scientist. In fact, the honest scientist will either seek to replicate what the cynical scientist did, or leave the profession.
sukilot 11y ago
If she waits or gets someone else to misprove the positive result first, she can build a career as a debunker...

by Fede_V 11y ago

jacquesm 11y ago

The reason funding goes to those that find 'good' results is the same reason why funding goes to those that start successful start-ups. Even though both parties probably have less control over the outcome than they think the perception is that they do and that betting on them again is a better bet than betting on someone that did not succeed to score. That's how we're wired, we simply like success stories much more than that we like stories of failure.

digi_owl 11y ago
Just world?
- verbin217 11y ago
  False inherencies are the staple of those that wish to understand the world such that they might exploit it. If they weren't prepared to demonstrate that the pursuit of a solution was necessarily a fool's game then they'd have to give an actual shit.
pbhjpbhj 11y ago
The analogy doesn't completely work. If the data shows your hypothesis is false, or doesn't support its truth then it doesn't matter if you're the best experimenter in your field nor if you worked most competently the world is what it is (barring poor experiment design!). An hypothesis isn't a product, falsifying your hypothesis is success - you've increased the pool of human knowledge (or increased the support for an element of it). In business good and bad products can both lead to profit and can both lead to losses. Profit and loss can both be considered success and failure. On your last sentence - I don't think it's as self-evident as it seems. We tend to like the successes that profit us but in general I'm not sure we humans are that congratulatory of other's success. Schadenfreude seems very popular.
dba7dba 11y ago
Startup and science experiments are different. Startups fail, although nobody involved likes it. Science experiments are supposed to FAIL, many times. That's what science is.

Lawtonfogle 11y ago

In many cases the funding is politically motivated. In turn this corrupts the results by a type of 'survival of the fittest' where fitness is determined by one's biases aligning with one's funding sources.

TimPC 11y ago

While it's noisy and high variance, the approach is similar to start-ups in that we want experiments to be designed in ways that they are likely to confirm underlying theories. This makes positive results more valuable than negative results and something the scientist has some influence over, but not complete control of. Successful experiments increase the chance of future experiments being successful in much the same way that having previously had a successful start-up increases the chances of future start-up success. Particularly when committees cover huge areas, it's a lot easier to fund based on track records than to accurately determine experiment quality of an experiment well outside your expertise.

Fede_V 11y ago
I hope I'm not reading what you wrote uncharitably, but, by rewarding people with positive results, we are inevitably encouraging people to either design their experiments in ways that confirm their hypothesis, or at worse, to falsify their data. Thankfully, I've seen very little of the latter, but I've seen quite a bit of the former. Even unconsciously, people are afraid to design experiments that would give a black and white answer and falsify a cherish theory. People with lots of positive results are not necessarily better scientists. They are just as often less scrupulous and better salesmen - Peter Thiel noticed that as well.
learnstats2 11y ago
Yes, and that causes so many problems. As you indicate, all of science is easily biased by what has been done and what has been published in the past. Negative results are essential for a complete understanding of science and when they are repressed, complete access to the globally available data set is not available. If I'm an ethically-dubious pharma or lobby group, I love this situation. I keep funding testing until someone comes up with a positive result by chance or p-value hacking - then fanfare that result all the way to the media and to the bank. (Negative results don't need to be seen) If I'm a cynical scientist, I take the money and p-value hack until I get the results my funding body wants, so that I have access to more money in their next round of funding. Now I'm far more successful than a completely honest scientist. In fact, the honest scientist will either seek to replicate what the cynical scientist did, or leave the profession.

sukilot 11y ago

If she waits or gets someone else to misprove the positive result first, she can build a career as a debunker...