7 ms·

Let's not forget though that a great deal of "science" is in fact trash[1]. The problem isn't really people being anti-science or pro-science. The problem is sc

by darawk 4y ago

Let's not forget though that a great deal of "science" is in fact trash[1]. The problem isn't really people being anti-science or pro-science. The problem is science being done poorly, whether by scientists in the credentialed sense, or amateurs. There is no pat "trust science more" or "trust amateurs less" answer here. The actual answer is that if you want to understand research, you need to actually understand mathematical statistics and the philosophy of statistics fairly deeply. There just isn't any way around it. 1. https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo...

specialist 4y ago
In other words, emphasize the process more than the outcomes. If the scientific process used proves sound, then I have more confidence in the outcomes. Alas, that's a hard sell to laypersons thru the mediums of soundbites and tweets.
quanto 4y ago
Completely agreed. While I strongly believe in citizen science and people's right (and perhaps obligation) to critique established science, there is just so much poor analyses done by people to criticize some scientific findings they do not approve, motivated emotionally or otherwise. This phenomenon does not bring us closer to finding scientific facts or resolving the replication crisis. People should learn some basic analytic tools first.
- j3th9n 4y ago
  Scientific facts do not exist, scientific observations do.
- bigfudge 4y ago
  We should really just do (and certainly publish) less research.
  - bigfudge 4y ago
    I can see why this has been downvoted, but I didn’t mean to sound like a luddite. I really just think there is a lot of low quality research, especially in the social sciences, done primarily to keep an publishing schedule up for pressured academics. We’d often be better to spend more time thinking about and planning fewer better studies.
russdill 4y ago
I think there's two extremes here. One is the issue covered well above. There is a great deal of junk science that gets published. That is a problem and it does erode trust. But in some ways it's also how the sausage gets made, there's going to be room for things to get published they later gets refuted. People rightly so have distrust for results coming out in fields they don't have good knowledge in. Without becoming an expert yourself it's very difficult to know who or what to trust. On the other end there's distrust of broad scientific consensus across different professions, countries, etc. It's the distrust at these levels that is the increasing problem we are facing today.
- tsimionescu 4y ago
  > People rightly so have distrust for results coming out in fields they don't have good knowledge in. Without becoming an expert yourself it's very difficult to know who or what to trust. A problem here is that there are fields of science that are almost certainly bogus in themselves. One very likely candidate is nutrition, which seems to be fumbling in the dark and has a long history of producing worse recommendations than doing nothing (e.g. replace fat with sugar). More controversially, the entire field of economics is seen by some to be very suspect from a basic foundations view.
  - FrenchDevRemote 4y ago
    >A problem here is that there are fields of science that are almost certainly bogus in themselves. One very likely candidate is nutrition, which seems to be fumbling in the dark and has a long history of producing worse recommendations than doing nothing (e.g. replace fat with sugar). It's not the field that is bogus in that case. People were quite literally bribed to push this. This could happen anywhere, anytime, in any field.
    - tsimionescu 4y ago
      That was just an example, but we can go into more detail - at every point, nutrition studies are highly questionable. Sample sizes are usually minuscule (you can find important studies with 5-10 subjects, and even the largest studies rarely have more than a few hundred), they are often biased samples (only overweight people, only people with heart disease or diabetes etc.), they often don't account for likely confounding factors (using weight without accounting for muscle mass, no accounting for stress, time in the sun etc.) and on and on. And all of this in a subject where we basically don't have any clear idea about how much metabolism differs from one person to the next, based on what factors (e.g. gut microbiome has only been recently identified as a major component of digestion that can differ significantly between people; psychological effects of diet are even less well understood, even though your food choices are obviously not coming from some pure realm of reason). Basically the digestive system is far too complex for us to understand from first principles at this time. Your diet has a very complex and very slow effect on your body, with some exceptions. Numerous diseases and environmental factors impact how this plays out exactly. So, to do real research in nutrition, the only chance right now would be to conduct massive studies over long periods of time with rigorous controls on subjects' nutrition and activities - which is basically impossible, or at least prohibitively expensive. Instead, we get conclusions drawn from studies of a few dozen people over a few months or at best a year (in "long-term" studies). Or, we get conclusions drawn from comparing diet across huge populations ("the Mediterranean diet", "the Japanese diet", "the American diet" etc) with no possibility to control for obvious differences in nutrients, environmental factors, lifestyle differences, access to healthcare etc. Both of these are worthless conclusions, they don't tell you anything at all. The only successes of nutrition science have been identifying the most basic nutrients we need to survive at a basic level (protein, fat, carbohydrates, and the various vitamins and minerals). Basically nothing beyond that should be trusted. As a fun historical note, after the discovery of the macro-nutrients there was a budding field of nutrition scientists confidently recommending optimal diets using scientific methods. Unfortunately, they had no idea about the existence of micro-nutrients, so actually following some of their diets you could actually end up getting scurvy or other serious malnutrition diseases. The current slew is not that bad, but I wouldn't be surprised if in the future we will look back similarly at some common diet advice of today.
avereveard 4y ago
part of the issue is that producing shit research cost very little, and debunking shit research cost a lot. just pick up one of the many "x reasons why earth is not a sphere" videos, some are pretty easy to debunk, but other require to understand i.e. potential fields (if earth is round why don't train engineer take into account curvature when laying tracks and variations thereof)
- cgriswald 4y ago
  Shit research can sometimes be debunked just by running the numbers given by the shit researchers. Flat earth theorists don’t often offer up actual research (and the one group I’ve seen try got negative results and concluded they messed up, not that they were wrong). A theory ought to be able to answer questions like “Why don’t train engineers take the curvature of the Earth into account?” The problem is when someone comes along and thinks an unanswered question (or even just someone not knowing the answer off the top of their head) proves a theory is completely false (or worse, proves their favorite theory correct). (And to even believe the Earth is flat is to be a conspiracy theorist so in this particular case no prove will ever suffice anyway.)
sandgiant 4y ago
I agree, but I don't think this is limited to science. I think a great deal of everything is in fact trash. This is why we need education and good faith discussion.
- gadders 4y ago
  https://en.wikipedia.org/wiki/Sturgeon%27s_law https://en.wikipedia.org/wiki/Sturgeon%27s_law
  - Kye 4y ago
    Would Sturgeon's Law fall under the 10% or the 90% of popular to cite laws?
gadders 4y ago
Science itself is the best method we have for exploring and making sense of the world around us. The method is rock solid. In between us (gen pop) and The Method are scientists, and scientists are just as fallible as any other group of people - lawyers, politicians, coders, shop assistants.
starfallg 4y ago
>The problem isn't really people being anti-science or pro-science. The problem is science being done poorly, whether by scientists in the credentialed sense, or amateurs. That's a very simplistic take on it. Bad science is a necessary part of the process and dealt with accordingly by the scientific method. The problem is that science that is bad or incomplete is being reported as fact or truth, or arguably even worse, as entertainment in order to gain an audience. This is what actually eroded the trust in science, as people kept repeating things that reinforced and further misshaped their biases. Human nature tends to distrust stuff we don't understand. Hence, our trust in science, which in many fields is often beyond the understanding of laymen, have to have constant reinforcement. However, the goal of media, and especially social media is to increase eyeballs for their content, and the truth sells a lot less well than sensationalised content pandering to the audience. Simply put, there is not much profit in reporting science truthfully, and every incentive to sensationalise it.
rkangel 4y ago
One way that journals could actually add value (a concept to which they seem resistant) would be to review the statistical analysis. Statistics is hard and easy to get subtley wrong, and is often an independent skill to the underlying science. If journals had statistics experts to critique the analysis techniques prior to publishing it would be a great improvement in the confidence in which we could read papers.
- Hendrikto 4y ago
  If only journals actually wanted to contribute… They just want to skim.
- bigfudge 4y ago
  I think this is a great idea, but in my experience there are surprisingly few such experts available. In practice, most statisticians not doing active methods research (I'm thinking of 'trial statisticians' mostly here, in CTU's) just cargo cult whatever procedures previous trials used. I guess they would pick up issues around sample size, but without also integrating that with some substantive knowledge about plausible effect sizes I'm not sure what value they would have. Plus, it would reduce the number of publishable papers quite substantially including from high profile authors/groups, so I don't think they want the fight. We should also remember that most journal editors are also involved in publishing this research — they often have no real incentives to make things awkward.

by darawk 4y ago

specialist 4y ago

In other words, emphasize the process more than the outcomes. If the scientific process used proves sound, then I have more confidence in the outcomes. Alas, that's a hard sell to laypersons thru the mediums of soundbites and tweets.

quanto 4y ago

Completely agreed. While I strongly believe in citizen science and people's right (and perhaps obligation) to critique established science, there is just so much poor analyses done by people to criticize some scientific findings they do not approve, motivated emotionally or otherwise. This phenomenon does not bring us closer to finding scientific facts or resolving the replication crisis. People should learn some basic analytic tools first.

j3th9n 4y ago
Scientific facts do not exist, scientific observations do.
bigfudge 4y ago
We should really just do (and certainly publish) less research.
- bigfudge 4y ago
  I can see why this has been downvoted, but I didn’t mean to sound like a luddite. I really just think there is a lot of low quality research, especially in the social sciences, done primarily to keep an publishing schedule up for pressured academics. We’d often be better to spend more time thinking about and planning fewer better studies.

russdill 4y ago

I think there's two extremes here. One is the issue covered well above. There is a great deal of junk science that gets published. That is a problem and it does erode trust. But in some ways it's also how the sausage gets made, there's going to be room for things to get published they later gets refuted. People rightly so have distrust for results coming out in fields they don't have good knowledge in. Without becoming an expert yourself it's very difficult to know who or what to trust. On the other end there's distrust of broad scientific consensus across different professions, countries, etc. It's the distrust at these levels that is the increasing problem we are facing today.

tsimionescu 4y ago
> People rightly so have distrust for results coming out in fields they don't have good knowledge in. Without becoming an expert yourself it's very difficult to know who or what to trust. A problem here is that there are fields of science that are almost certainly bogus in themselves. One very likely candidate is nutrition, which seems to be fumbling in the dark and has a long history of producing worse recommendations than doing nothing (e.g. replace fat with sugar). More controversially, the entire field of economics is seen by some to be very suspect from a basic foundations view.
- FrenchDevRemote 4y ago
  >A problem here is that there are fields of science that are almost certainly bogus in themselves. One very likely candidate is nutrition, which seems to be fumbling in the dark and has a long history of producing worse recommendations than doing nothing (e.g. replace fat with sugar). It's not the field that is bogus in that case. People were quite literally bribed to push this. This could happen anywhere, anytime, in any field.
  - tsimionescu 4y ago
    That was just an example, but we can go into more detail - at every point, nutrition studies are highly questionable. Sample sizes are usually minuscule (you can find important studies with 5-10 subjects, and even the largest studies rarely have more than a few hundred), they are often biased samples (only overweight people, only people with heart disease or diabetes etc.), they often don't account for likely confounding factors (using weight without accounting for muscle mass, no accounting for stress, time in the sun etc.) and on and on. And all of this in a subject where we basically don't have any clear idea about how much metabolism differs from one person to the next, based on what factors (e.g. gut microbiome has only been recently identified as a major component of digestion that can differ significantly between people; psychological effects of diet are even less well understood, even though your food choices are obviously not coming from some pure realm of reason). Basically the digestive system is far too complex for us to understand from first principles at this time. Your diet has a very complex and very slow effect on your body, with some exceptions. Numerous diseases and environmental factors impact how this plays out exactly. So, to do real research in nutrition, the only chance right now would be to conduct massive studies over long periods of time with rigorous controls on subjects' nutrition and activities - which is basically impossible, or at least prohibitively expensive. Instead, we get conclusions drawn from studies of a few dozen people over a few months or at best a year (in "long-term" studies). Or, we get conclusions drawn from comparing diet across huge populations ("the Mediterranean diet", "the Japanese diet", "the American diet" etc) with no possibility to control for obvious differences in nutrients, environmental factors, lifestyle differences, access to healthcare etc. Both of these are worthless conclusions, they don't tell you anything at all. The only successes of nutrition science have been identifying the most basic nutrients we need to survive at a basic level (protein, fat, carbohydrates, and the various vitamins and minerals). Basically nothing beyond that should be trusted. As a fun historical note, after the discovery of the macro-nutrients there was a budding field of nutrition scientists confidently recommending optimal diets using scientific methods. Unfortunately, they had no idea about the existence of micro-nutrients, so actually following some of their diets you could actually end up getting scurvy or other serious malnutrition diseases. The current slew is not that bad, but I wouldn't be surprised if in the future we will look back similarly at some common diet advice of today.

avereveard 4y ago

part of the issue is that producing shit research cost very little, and debunking shit research cost a lot. just pick up one of the many "x reasons why earth is not a sphere" videos, some are pretty easy to debunk, but other require to understand i.e. potential fields (if earth is round why don't train engineer take into account curvature when laying tracks and variations thereof)

cgriswald 4y ago
Shit research can sometimes be debunked just by running the numbers given by the shit researchers. Flat earth theorists don’t often offer up actual research (and the one group I’ve seen try got negative results and concluded they messed up, not that they were wrong). A theory ought to be able to answer questions like “Why don’t train engineers take the curvature of the Earth into account?” The problem is when someone comes along and thinks an unanswered question (or even just someone not knowing the answer off the top of their head) proves a theory is completely false (or worse, proves their favorite theory correct). (And to even believe the Earth is flat is to be a conspiracy theorist so in this particular case no prove will ever suffice anyway.)

sandgiant 4y ago

I agree, but I don't think this is limited to science. I think a great deal of everything is in fact trash. This is why we need education and good faith discussion.

gadders 4y ago
https://en.wikipedia.org/wiki/Sturgeon%27s_law https://en.wikipedia.org/wiki/Sturgeon%27s_law
- Kye 4y ago
  Would Sturgeon's Law fall under the 10% or the 90% of popular to cite laws?

gadders 4y ago

Science itself is the best method we have for exploring and making sense of the world around us. The method is rock solid. In between us (gen pop) and The Method are scientists, and scientists are just as fallible as any other group of people - lawyers, politicians, coders, shop assistants.

starfallg 4y ago

>The problem isn't really people being anti-science or pro-science. The problem is science being done poorly, whether by scientists in the credentialed sense, or amateurs. That's a very simplistic take on it. Bad science is a necessary part of the process and dealt with accordingly by the scientific method. The problem is that science that is bad or incomplete is being reported as fact or truth, or arguably even worse, as entertainment in order to gain an audience. This is what actually eroded the trust in science, as people kept repeating things that reinforced and further misshaped their biases. Human nature tends to distrust stuff we don't understand. Hence, our trust in science, which in many fields is often beyond the understanding of laymen, have to have constant reinforcement. However, the goal of media, and especially social media is to increase eyeballs for their content, and the truth sells a lot less well than sensationalised content pandering to the audience. Simply put, there is not much profit in reporting science truthfully, and every incentive to sensationalise it.

rkangel 4y ago

One way that journals could actually add value (a concept to which they seem resistant) would be to review the statistical analysis. Statistics is hard and easy to get subtley wrong, and is often an independent skill to the underlying science. If journals had statistics experts to critique the analysis techniques prior to publishing it would be a great improvement in the confidence in which we could read papers.

Hendrikto 4y ago
If only journals actually wanted to contribute… They just want to skim.
bigfudge 4y ago
I think this is a great idea, but in my experience there are surprisingly few such experts available. In practice, most statisticians not doing active methods research (I'm thinking of 'trial statisticians' mostly here, in CTU's) just cargo cult whatever procedures previous trials used. I guess they would pick up issues around sample size, but without also integrating that with some substantive knowledge about plausible effect sizes I'm not sure what value they would have. Plus, it would reduce the number of publishable papers quite substantially including from high profile authors/groups, so I don't think they want the fight. We should also remember that most journal editors are also involved in publishing this research — they often have no real incentives to make things awkward.