3 ms·

Equally broadly, there's huge pressure to report positive results. If the incentive structure were even tilted a bit to make reporting negative or ambiguous re

Equally broadly, there's huge pressure to report positive results. If the incentive structure were even tilted a bit to make reporting negative or ambiguous results less career-damaging, we would see much less of this. I don't think most scientists want to fudge the truth, but in many cases they feel that they have no choice. The pressure in most of these sciences is to consistently get a paper for every expensive experiment that is done, but if you're doing that, and published work has to be a positive result, you're not really doing science. You could just publish your hypothesis and be done with it if you already know the result of the experiment will be ginned up in some way to be positive.

chongli 3y ago
If the incentive structure were even tilted a bit to make reporting negative or ambiguous results less career-damaging, we would see much less of this I don't see how we do this. Ultimately, the buck stops with the public. Governments fund the sciences because the public demands it. If scientists spend more time reporting negative or ambiguous results then the public begin to question what they're paying for. People want breakthroughs and scientific revolutions, not "business as usual." And this bias is not limited to the public, scientists themselves want to see new science.
- mistermann 3y ago
  > Governments fund the sciences because the public demands it. Is this a scientific fact? Secondly, is it an epistemically flawless fact (which is similar, but also different)? In what way exactly does "the public" "demand" that which science does?
  - chongli 3y ago
    Is this a scientific fact? Secondly, is it an epistemically flawless fact (which is similar, but also different)? Why don’t you check on Google Scholar and report back to me? In what way exactly does "the public" "demand" that which science does? In the economic way that we say most things are demanded. I’m not saying that people are going to be out in the streets protesting if scientists start publishing more negative results. I’m saying that the general public attitude towards science (the one measured in polls) is affected by the public’s perception that scientists are producing important and meaningful results.
    - mistermann 3y ago
      > Why don’t you check on Google Scholar and report back to me? Standard conventions for burden of proof I suppose. Surely you don't actually have no evidence to back up your claim, do you? > In the economic way that we say most things are demanded. Are you discerning support for science from economic data of some sort? >I’m saying that the general public attitude towards science (the one measured in polls) is affected by the public’s perception that scientists are producing important and meaningful results. And might it be possible that the public's perception of science could be affected by all of the free marketing it's gotten in the last five or so years?
- godelski 3y ago
  I don't think this is entirely a good model. People do want advancements and "breakthroughs" but the truth is that they don't understand any of those things. So if good science is happening, even in a small proportion, then your argument just suggests that reporting is more important. With respect to the public, I generally agree. But with this in mind, it would be probably more effective to just increase the total funding and randomly hand it out. You'd get bad apples like we do now and similar stories of ineffective spending -- where the public only understands "big number" -- but you'd also have more "successes" which are however you want to define them. That's because the truth of it is, for science progress is "business as usual." As for "breakthroughs" and "revolutions" well, those are exceedingly rare, and I'm willing to bet that several of the ones you are thinking of (not all) are less leaps and bounds than you perceive and more shoulders on "giants," where the "giants" are just several people in a trench coat. So I don't get how this supports the current framework.
godelski 3y ago
> there's huge pressure to report positive results. This is basically just noting Goodhart's Law, or as I like to say "Warning." Evaluation is incredibly difficult and every metric can be hacked. Making "meritocracy" a fool's errand since a metric alone is not enough (it needs the associated uncertainty value, which is non-zero). The problem here is that we're searching through an incredibly high dimensional non-convex solution space and assuming that a simple optimization function is a global optimizer. Instead I'd just love to see academics pursuing their ideas and discussing them in the open. I don't think you'd have a significant problem getting people to generally align in a single direction, as currently happens, but I think it also allows for others to challenge those directions as well. Essentially research is performing some sampling in that search space but we're turning the learning rate way down. So we get plenty of works where novelty and incrementalism is indeterminable, due to the speed in which we must produce. Instead let the academics determine their own parameters and let time sort it out. Ensembling is a very useful technique, especially when uncertainty is high.

chongli 3y ago

If the incentive structure were even tilted a bit to make reporting negative or ambiguous results less career-damaging, we would see much less of this I don't see how we do this. Ultimately, the buck stops with the public. Governments fund the sciences because the public demands it. If scientists spend more time reporting negative or ambiguous results then the public begin to question what they're paying for. People want breakthroughs and scientific revolutions, not "business as usual." And this bias is not limited to the public, scientists themselves want to see new science.

mistermann 3y ago
> Governments fund the sciences because the public demands it. Is this a scientific fact? Secondly, is it an epistemically flawless fact (which is similar, but also different)? In what way exactly does "the public" "demand" that which science does?
- chongli 3y ago
  Is this a scientific fact? Secondly, is it an epistemically flawless fact (which is similar, but also different)? Why don’t you check on Google Scholar and report back to me? In what way exactly does "the public" "demand" that which science does? In the economic way that we say most things are demanded. I’m not saying that people are going to be out in the streets protesting if scientists start publishing more negative results. I’m saying that the general public attitude towards science (the one measured in polls) is affected by the public’s perception that scientists are producing important and meaningful results.
  - mistermann 3y ago
    > Why don’t you check on Google Scholar and report back to me? Standard conventions for burden of proof I suppose. Surely you don't actually have no evidence to back up your claim, do you? > In the economic way that we say most things are demanded. Are you discerning support for science from economic data of some sort? >I’m saying that the general public attitude towards science (the one measured in polls) is affected by the public’s perception that scientists are producing important and meaningful results. And might it be possible that the public's perception of science could be affected by all of the free marketing it's gotten in the last five or so years?
godelski 3y ago
I don't think this is entirely a good model. People do want advancements and "breakthroughs" but the truth is that they don't understand any of those things. So if good science is happening, even in a small proportion, then your argument just suggests that reporting is more important. With respect to the public, I generally agree. But with this in mind, it would be probably more effective to just increase the total funding and randomly hand it out. You'd get bad apples like we do now and similar stories of ineffective spending -- where the public only understands "big number" -- but you'd also have more "successes" which are however you want to define them. That's because the truth of it is, for science progress is "business as usual." As for "breakthroughs" and "revolutions" well, those are exceedingly rare, and I'm willing to bet that several of the ones you are thinking of (not all) are less leaps and bounds than you perceive and more shoulders on "giants," where the "giants" are just several people in a trench coat. So I don't get how this supports the current framework.

godelski 3y ago

> there's huge pressure to report positive results. This is basically just noting Goodhart's Law, or as I like to say "Warning." Evaluation is incredibly difficult and every metric can be hacked. Making "meritocracy" a fool's errand since a metric alone is not enough (it needs the associated uncertainty value, which is non-zero). The problem here is that we're searching through an incredibly high dimensional non-convex solution space and assuming that a simple optimization function is a global optimizer. Instead I'd just love to see academics pursuing their ideas and discussing them in the open. I don't think you'd have a significant problem getting people to generally align in a single direction, as currently happens, but I think it also allows for others to challenge those directions as well. Essentially research is performing some sampling in that search space but we're turning the learning rate way down. So we get plenty of works where novelty and incrementalism is indeterminable, due to the speed in which we must produce. Instead let the academics determine their own parameters and let time sort it out. Ensembling is a very useful technique, especially when uncertainty is high.