6 ms·

I'm not a scientist, nor am I familiar with statistics. I could imagine this being troubling for a scientist or someone familiar with the field. But for a non-s

I'm not a scientist, nor am I familiar with statistics. I could imagine this being troubling for a scientist or someone familiar with the field. But for a non-scientist it's fairly devastating if you consider the implications. What it means is that if I trust a study - any study (from my perspective) - I'm essentially flipping a coin. If non-scientist citizens can't rely on what is coming out of the field, then it seems like a massive problem that needs solving before any other.

sixstringbudha 8y ago
> that needs solving before any other. Sadly, I think there is huge incentive to maintain status quo. It is essentially a loophole to manipulate the scientific consensus without the actual scientists being dishonest. Some wise once said "When a measure becomes a target, it ceases to be a good measure.". I think that what we call as "science" today, has thus that fallen prey to this.
j7ake 8y ago
You should be considering not just one study but how independent studies fit together into a grander picture.
- rayiner 8y ago
  That's not a useful thing to say to the general public--they're not capable of that. They count on scientists to police themselves.
  - tenaciousDaniel 8y ago
    I get his point, though in the end I agree with you. The interface between the scientific community and the public at large should be reliable. At this point in time it doesn't seem to be.
  - j7ake 8y ago
    Scientific papers aren’t written for the lay audience, it’s a conversation amongst scientists.
the8472 8y ago
You shouldn't trust any single study. Researchers are not perfect, reviewers are not perfect, the systems they observe are complex. Even in the "hard" sciences even if the experimental data itself is solid there can be confounders that were overlooked and will only be uncovered by follow-up studies, depending on importance of the result it might take years. That's why have meta-analysis and systematic reviews and even those aren't exactly bullet-proof.
- deleted 8y ago
  [deleted]
- User23 8y ago
  Meta analysis is just the weighted average of expert opinion. Which makes it worthless when the experts don’t actually know anything. https://jamanetwork.com/journals/jama/fullarticle/2698337 https://jamanetwork.com/journals/jama/fullarticle/2698337 You should be skeptical of any so-called science that cannot actually rigorously apply the scientific method, be it for practical, ethical, or any other reasons.
- btrettel 8y ago
  > Even in the "hard" sciences even if the experimental data itself is solid there can be confounders that were overlooked and will only be uncovered by follow-up studies, depending on importance of the result it might take years. I work in fluid dynamics. In one subfield I work in there is at least one confounder in most studies (Weber number and Reynolds number are confounded, in particular) and an important variable is often omitted (turbulence intensity). Many people don't seem aware that turbulence intensity matters, either, and most who do think it matters don't want to measure or estimate it, so it tends to be ignored. Just because it's hard to measure does not mean it's not important! This is supposed to be "hard" science, but sometimes I feel it's not much better than social science. How long has this been a problem? I'd say over 50 years, easily! At a recent conference, one of my papers addressed these issues briefly (I tried to avoid them in a data analysis), but I think it'll take more than that to correct these problems. So I'm planning a new article specifically addressing these concerns. I still expect progress to be slow, but any movement in the right direction would be appreciated...
cm2187 8y ago
The first step is to not call science what isn’t science. Social sciences is a heavily politicised field which is more akin to journalism than anything else. It doesn’t mean it is not interesting but it does not carry the authority of hard sciences. Medecine is more problematic. Many studies are at the border of statistical significance but it is impossible to conduct large scale experiments with humans, so it’s not like if we have a better alternative.
- weberc2 8y ago
  > Social sciences is a heavily politicised field which is more akin to journalism than anything else. You're being downvoted, but my understanding is that lots of social science researchers openly refer to themselves and their discipline as activism. Am I mistaken? Is this observation controversial? EDIT: Now that people are downvoting me, perhaps someone could take a moment to explain where I'm mistaken?
  - Fomite 8y ago
    I haven't downvoted you, but "Science" and "Activism" are not mutually exclusive.
    - weberc2 8y ago
      Fair point, but isn’t it cause for skepticism? It seems that science-minded folks are skeptical of the research done by various politically-oriented think-tanks; isn’t a politically oriented field or department similar?
      - Fomite 8y ago
        The question is what direction that's in. Activism that is, in essence, urging the adoption of ones findings, if obtained through valid experiments, isn't really biased. Politically-oriented think-tanks are looking to build research that supports a particular political outcome. That's a pretty strong contrast to say "We have found that X is harmful, and will advocate for people having reduced exposure to X." Consider, for example, the number of infectious disease epidemiologists, vaccine developers and doctors who have essentially been forced to engage in "activism" thanks to anti-vaxx movements.

sixstringbudha 8y ago

> that needs solving before any other. Sadly, I think there is huge incentive to maintain status quo. It is essentially a loophole to manipulate the scientific consensus without the actual scientists being dishonest. Some wise once said "When a measure becomes a target, it ceases to be a good measure.". I think that what we call as "science" today, has thus that fallen prey to this.

j7ake 8y ago

You should be considering not just one study but how independent studies fit together into a grander picture.

rayiner 8y ago
That's not a useful thing to say to the general public--they're not capable of that. They count on scientists to police themselves.
- tenaciousDaniel 8y ago
  I get his point, though in the end I agree with you. The interface between the scientific community and the public at large should be reliable. At this point in time it doesn't seem to be.
- j7ake 8y ago
  Scientific papers aren’t written for the lay audience, it’s a conversation amongst scientists.

the8472 8y ago

You shouldn't trust any single study. Researchers are not perfect, reviewers are not perfect, the systems they observe are complex. Even in the "hard" sciences even if the experimental data itself is solid there can be confounders that were overlooked and will only be uncovered by follow-up studies, depending on importance of the result it might take years. That's why have meta-analysis and systematic reviews and even those aren't exactly bullet-proof.

deleted 8y ago
[deleted]
User23 8y ago
Meta analysis is just the weighted average of expert opinion. Which makes it worthless when the experts don’t actually know anything. https://jamanetwork.com/journals/jama/fullarticle/2698337 https://jamanetwork.com/journals/jama/fullarticle/2698337 You should be skeptical of any so-called science that cannot actually rigorously apply the scientific method, be it for practical, ethical, or any other reasons.
btrettel 8y ago
> Even in the "hard" sciences even if the experimental data itself is solid there can be confounders that were overlooked and will only be uncovered by follow-up studies, depending on importance of the result it might take years. I work in fluid dynamics. In one subfield I work in there is at least one confounder in most studies (Weber number and Reynolds number are confounded, in particular) and an important variable is often omitted (turbulence intensity). Many people don't seem aware that turbulence intensity matters, either, and most who do think it matters don't want to measure or estimate it, so it tends to be ignored. Just because it's hard to measure does not mean it's not important! This is supposed to be "hard" science, but sometimes I feel it's not much better than social science. How long has this been a problem? I'd say over 50 years, easily! At a recent conference, one of my papers addressed these issues briefly (I tried to avoid them in a data analysis), but I think it'll take more than that to correct these problems. So I'm planning a new article specifically addressing these concerns. I still expect progress to be slow, but any movement in the right direction would be appreciated...

cm2187 8y ago

The first step is to not call science what isn’t science. Social sciences is a heavily politicised field which is more akin to journalism than anything else. It doesn’t mean it is not interesting but it does not carry the authority of hard sciences. Medecine is more problematic. Many studies are at the border of statistical significance but it is impossible to conduct large scale experiments with humans, so it’s not like if we have a better alternative.

weberc2 8y ago
> Social sciences is a heavily politicised field which is more akin to journalism than anything else. You're being downvoted, but my understanding is that lots of social science researchers openly refer to themselves and their discipline as activism. Am I mistaken? Is this observation controversial? EDIT: Now that people are downvoting me, perhaps someone could take a moment to explain where I'm mistaken?
- Fomite 8y ago
  I haven't downvoted you, but "Science" and "Activism" are not mutually exclusive.
  - weberc2 8y ago
    Fair point, but isn’t it cause for skepticism? It seems that science-minded folks are skeptical of the research done by various politically-oriented think-tanks; isn’t a politically oriented field or department similar?
    - Fomite 8y ago
      The question is what direction that's in. Activism that is, in essence, urging the adoption of ones findings, if obtained through valid experiments, isn't really biased. Politically-oriented think-tanks are looking to build research that supports a particular political outcome. That's a pretty strong contrast to say "We have found that X is harmful, and will advocate for people having reduced exposure to X." Consider, for example, the number of infectious disease epidemiologists, vaccine developers and doctors who have essentially been forced to engage in "activism" thanks to anti-vaxx movements.