4 ms·

The deeper you get into the research industry, the crazier things get. We are taught growing up that scientists value truth above everything and scientists are

by socrates1998 8y ago

The deeper you get into the research industry, the crazier things get. We are taught growing up that scientists value truth above everything and scientists are sort of seen as these demigods of the universe. Yet, sadly, we know that they are just people who want to be right more than they want the truth too often. I mean, if you spent 3-5 years (sometimes much more) of your prime working on something only to come to the conclusion that your wrong would you just accept it? OR you "re-adjust" the data and somehow make your p-value significant, then BOOM, your research is valuable! This is why so many studies are not reproducible.

dalore 8y ago
It's not that bad faith. When their initial hypothesis is wrong, instead of considering the null hypothesis (which is the case of accepting you're wrong), you create new conclusions from the data to say why you're still correct. It's the same thing as you said, but it's not done with malintent but done with the just wanting to prove yourself right.
nonbel 8y ago
>"I mean, if you spent 3-5 years (sometimes much more) of your prime working on something only to come to the conclusion that your wrong would you just accept it? OR you "re-adjust" the data and somehow make your p-value significant, then BOOM, your research is valuable!" From what I've seen I don't think there is any "OR". That is standard behavior and people don't even believe there is anything wrong with it. There really is always some legitimate reason to adjust the data... the problem is that the method of drawing conclusions (checking for significant p-values) is incompatible with this. So one or the other practice needs to go.
- 3pt14159 8y ago
  Scientists / scientists in training are starting to understand it. I think almost every last field taught in university would benefit from more statistics courses, with at least one of them from a bayesian perspective. The problem is that statistics is boring. Most things in university are causal. Statistics is kinda fuzzy and unintuitive.
  - nonbel 8y ago
    I think you have the blame reversed. These problems were caused by statisticians who failed to understand there is always a reason to "adjust" the data. They taught people to apply methods that were inapplicable to the real-life research conditions they would face.
    - ethbro 8y ago
      Agreed. Statistics can be counter-intuitive, but it's not rocket science. Playing the "keeper of the true flame" and hiding the internals in intro stats courses does everyone a disservice. I get there are a huge amount of non-mathematical scientists who go through these courses, but they should at least be equipped to get a funny feeling if they see something odd (and therefore go ask for advice). Rather than just believe "that's the number that came out." Mathematically rigorous and vetting meta-studies seem like they would help here too... validating methods and that studies reproduce within a range.
snarf21 8y ago
As mentioned in other threads, the role of self-preservation is hard to dismiss. Most universities drop the task of getting external funding to the researcher. In order to maintain their position they must bring in X dollars. Of course, the results of all fundings have to be made meaningful to the funder. I get it, they need to eat too. It is a shame that the Harvards of the world don't have huge research departments where the goal is always the public good funded through their monstrous endowments. I know RWJ does this kind of research but they only run so many and we have 1000s of researchers at countless universities all trying to just stay alive who will always pick up trash studies to pay the bills.
watwut 8y ago
I increasingly think that we should stop to teach kids to blindly trust to this or that group of people (army, scientists, president, past historical characters) and teach them to deal with complexity and truth instead.
- croon 8y ago
  One of those groups are massively more likely to deal in thruth though. Otherwise I agree; people are usually too keen on easy answers instead of complex correct answers.
  - 81318536 8y ago
    If you're insinuating that the 'scientists' group in that list is more truthy, I'd completely disagree due to the bias involved. If you're going to pick one group in that list for realism in understanding, it's obvious you'd choose the army and not purely academic modern scholars who consistently struggle with solving the most basic perverse incentives such as unavailability of open access peer review, Pagerank-as-tenure-prerequisites, and pervasive social norms of qualifying career success as 'results' driven rather than clarifying the scope of human knowledge through the overwhelmingly important outcomes of 'failure.' Sure, you're wouldn't trust a public figure who only has a short term length to contain a problematic national policy dogma informed by 'studies.' See the outcome of Lysenkoism in the Soviet era as an example of posturing dubious scientism as political party ideology. And past historical characters aren't really sources of truth due to the distorting lens of the incumbent narrative writers who need to legitimize policy either through hacking the populace's susceptibility to voice merging rhetoric or indoctrination of a national origin myth that gives meaning to most citizens who are actively seeking authoritative external confirmation of a double-truth to ease the cognitive-dissonance everyone feels when reality doesn't match the cherry-picked politically-harmonious model. But to trust the army, that's somewhat different. Similar to many imperial armies of antiquity, the US army's sole strength is logistical in nature and not tactical. We can't fight based on risky 'war genius' maneuvering and would completely flounder in a scenario of being overwhelmed in asymmetrical/low-tech/low-intensity/decentralized conflict scenarios, so we instead gain supremacy in being able to deploy massive amounts of resources to any location on Earth faster than anyone else and at a sustained rate. If reality didn't match predictions of our models in forecasting things such as cost-effective mitigation of nutritional demands for soldiers / displaced populations / war engine back home, long-term regional and global climatic patterns and short-term weather conditions, estimates of extractable fuels and necessary technologies to overcome resource shortages, etc, then you probably have much more to worry about on a very small timescale than pondering over truth in general and especially whether sugar is 'good' or 'bad' for you. Note that the army's view of certain scientific knowledge and distilled policies can be vastly different than what emerges from the mouths of the faces we see on TV be it consumer ads with white-coat actors or government ads with black-coat actors.
Sileni 8y ago
It's strange to me that the gate is based on success, and not just a panel of your peers/mentors judging the way you performed the study. After all, negative results are just as valuable as positive results, unless we decide we're going to ignore them and let someone else run the same experiment... which is what we're doing. At least when considering someone for a PhD or Masters, you'd think the point would be to learn good practice, not to actually push the field forward. There's just not as much frontier as when the rules were set.
- acchow 8y ago
  > After all, negative results are just as valuable as positive results This is not true. It's incredibly easy (and you can be infinitely lazy) to come up with any bad hypothesis that has almost no chance of turning out positive results. Good hypothesis: Existence of Higgs boson at 125 GeV. Bad hypothesis: Eating chocolate in the morning decreases the risk of developing lung cancer. We value positive results more because negative results are incredibly easy to make just by coming up with bad hypotheses.
  - dragonwriter 8y ago
    Negative results on “good” hypotheses (by your implicit standard, which seems to be “prior probability of positive results”), though, are still as valuable as positive results (and negative results on hypotheses that seem obvious intuitively are arguably more valuable than positive results.)
    - acchow 8y ago
      agree, but in order to incentivize people to come up with "good hypotheses" we need to devalue negative results. Otherwise everyone can just suck up all the funding for total garbage hypotheses that will almost definitely end up with negative results anyway. "A mutation in the P53AIP1 has a protective effect against alzheimers" Yeah, I totally picked a random gene off wikipedia. This will probably have a negative result. So helpful.
  - Sileni 8y ago

dalore 8y ago

It's not that bad faith. When their initial hypothesis is wrong, instead of considering the null hypothesis (which is the case of accepting you're wrong), you create new conclusions from the data to say why you're still correct. It's the same thing as you said, but it's not done with malintent but done with the just wanting to prove yourself right.

nonbel 8y ago

>"I mean, if you spent 3-5 years (sometimes much more) of your prime working on something only to come to the conclusion that your wrong would you just accept it? OR you "re-adjust" the data and somehow make your p-value significant, then BOOM, your research is valuable!" From what I've seen I don't think there is any "OR". That is standard behavior and people don't even believe there is anything wrong with it. There really is always some legitimate reason to adjust the data... the problem is that the method of drawing conclusions (checking for significant p-values) is incompatible with this. So one or the other practice needs to go.

3pt14159 8y ago
Scientists / scientists in training are starting to understand it. I think almost every last field taught in university would benefit from more statistics courses, with at least one of them from a bayesian perspective. The problem is that statistics is boring. Most things in university are causal. Statistics is kinda fuzzy and unintuitive.
- nonbel 8y ago
  I think you have the blame reversed. These problems were caused by statisticians who failed to understand there is always a reason to "adjust" the data. They taught people to apply methods that were inapplicable to the real-life research conditions they would face.
  - ethbro 8y ago
    Agreed. Statistics can be counter-intuitive, but it's not rocket science. Playing the "keeper of the true flame" and hiding the internals in intro stats courses does everyone a disservice. I get there are a huge amount of non-mathematical scientists who go through these courses, but they should at least be equipped to get a funny feeling if they see something odd (and therefore go ask for advice). Rather than just believe "that's the number that came out." Mathematically rigorous and vetting meta-studies seem like they would help here too... validating methods and that studies reproduce within a range.

snarf21 8y ago

As mentioned in other threads, the role of self-preservation is hard to dismiss. Most universities drop the task of getting external funding to the researcher. In order to maintain their position they must bring in X dollars. Of course, the results of all fundings have to be made meaningful to the funder. I get it, they need to eat too. It is a shame that the Harvards of the world don't have huge research departments where the goal is always the public good funded through their monstrous endowments. I know RWJ does this kind of research but they only run so many and we have 1000s of researchers at countless universities all trying to just stay alive who will always pick up trash studies to pay the bills.

watwut 8y ago

I increasingly think that we should stop to teach kids to blindly trust to this or that group of people (army, scientists, president, past historical characters) and teach them to deal with complexity and truth instead.

croon 8y ago
One of those groups are massively more likely to deal in thruth though. Otherwise I agree; people are usually too keen on easy answers instead of complex correct answers.
- 81318536 8y ago
  If you're insinuating that the 'scientists' group in that list is more truthy, I'd completely disagree due to the bias involved. If you're going to pick one group in that list for realism in understanding, it's obvious you'd choose the army and not purely academic modern scholars who consistently struggle with solving the most basic perverse incentives such as unavailability of open access peer review, Pagerank-as-tenure-prerequisites, and pervasive social norms of qualifying career success as 'results' driven rather than clarifying the scope of human knowledge through the overwhelmingly important outcomes of 'failure.' Sure, you're wouldn't trust a public figure who only has a short term length to contain a problematic national policy dogma informed by 'studies.' See the outcome of Lysenkoism in the Soviet era as an example of posturing dubious scientism as political party ideology. And past historical characters aren't really sources of truth due to the distorting lens of the incumbent narrative writers who need to legitimize policy either through hacking the populace's susceptibility to voice merging rhetoric or indoctrination of a national origin myth that gives meaning to most citizens who are actively seeking authoritative external confirmation of a double-truth to ease the cognitive-dissonance everyone feels when reality doesn't match the cherry-picked politically-harmonious model. But to trust the army, that's somewhat different. Similar to many imperial armies of antiquity, the US army's sole strength is logistical in nature and not tactical. We can't fight based on risky 'war genius' maneuvering and would completely flounder in a scenario of being overwhelmed in asymmetrical/low-tech/low-intensity/decentralized conflict scenarios, so we instead gain supremacy in being able to deploy massive amounts of resources to any location on Earth faster than anyone else and at a sustained rate. If reality didn't match predictions of our models in forecasting things such as cost-effective mitigation of nutritional demands for soldiers / displaced populations / war engine back home, long-term regional and global climatic patterns and short-term weather conditions, estimates of extractable fuels and necessary technologies to overcome resource shortages, etc, then you probably have much more to worry about on a very small timescale than pondering over truth in general and especially whether sugar is 'good' or 'bad' for you. Note that the army's view of certain scientific knowledge and distilled policies can be vastly different than what emerges from the mouths of the faces we see on TV be it consumer ads with white-coat actors or government ads with black-coat actors.

Sileni 8y ago

It's strange to me that the gate is based on success, and not just a panel of your peers/mentors judging the way you performed the study. After all, negative results are just as valuable as positive results, unless we decide we're going to ignore them and let someone else run the same experiment... which is what we're doing. At least when considering someone for a PhD or Masters, you'd think the point would be to learn good practice, not to actually push the field forward. There's just not as much frontier as when the rules were set.

acchow 8y ago
> After all, negative results are just as valuable as positive results This is not true. It's incredibly easy (and you can be infinitely lazy) to come up with any bad hypothesis that has almost no chance of turning out positive results. Good hypothesis: Existence of Higgs boson at 125 GeV. Bad hypothesis: Eating chocolate in the morning decreases the risk of developing lung cancer. We value positive results more because negative results are incredibly easy to make just by coming up with bad hypotheses.
- dragonwriter 8y ago
  Negative results on “good” hypotheses (by your implicit standard, which seems to be “prior probability of positive results”), though, are still as valuable as positive results (and negative results on hypotheses that seem obvious intuitively are arguably more valuable than positive results.)
  - acchow 8y ago
    agree, but in order to incentivize people to come up with "good hypotheses" we need to devalue negative results. Otherwise everyone can just suck up all the funding for total garbage hypotheses that will almost definitely end up with negative results anyway. "A mutation in the P53AIP1 has a protective effect against alzheimers" Yeah, I totally picked a random gene off wikipedia. This will probably have a negative result. So helpful.
- Sileni 8y ago