6 ms·

Once, years ago, I got into a debate with somebody online who just kept dropping links to studies after making a statement as if it proved them correct. One da

Once, years ago, I got into a debate with somebody online who just kept dropping links to studies after making a statement as if it proved them correct. One day, I got bored and decided to actually read every single study. The studies said nothing close to what the debater suggested and seriously opened my eyes to just how terrible some studies can be in terms of quality. Now, I just assume that anybody who starts link bombing in a conversation has no idea what they're talking about and can't engage on a logical discussion.

hospitalJail 3y ago
If you ever want to reject a study, just read it. You will find something to nit pick.
- ben_w 3y ago
  I'm not saying you're wrong — I've got my name on exactly one paper so what would I know — but that being true would suggest that peer review is fundamentally broken.
  - jvanderbot 3y ago
    No, it would suggest that every paper has something in it that can be considered a fatal flaw by a person biased against it. Peer review is precisely supposed to admit this and minimize flaws in conclusions given inevitable flaws in methodology. Sort of a research paper equivalent of: "If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him." by Cardinal Richelieu
  - matusp 3y ago
    Peer review is not a certificate of truthfulness, peer review just says that a grad student reading the paper did not found anything too suspicious there and it looks like a typical paper in that field.
    - some_random 3y ago
      A huge part of the problem is that the layperson is taught otherwise.
      - hospitalJail 3y ago
        2nd grade we learned the scientific method. I don't remember peer reviewed being a thing. Replication was critical. Also, I've seen peer reviewed papers with atrocious stuff in them, unless I know the peer, it doesn't mean much to me. Give me 100 independent groups coming to the same conclusions. That has far more weight than a few of your friends/professors reading your paper.
    - svnt 3y ago
      Peer review, at its core, is a social consensus process. It can work, but it is structurally inclined to propagate agreement. I wonder if it serves to reinforce this problem, as those more motivated to reinforce certain literatures or perspectives will surface those repeatedly in reviews.
      - all2 3y ago
        There's a popular work that comes to this conclusion in its beginning chapters [0]. The basic premise is that the act of science and knowledge acquisition cyclically devolves into ideology and is then disrupted. Typically those who initially disrupt a scientific dogma are not treated well. Eventually the old guard literally dies off and the new ideas can begin to take hold. [0] The Structure of Scientific Revolutions, T. Kuhn
        consilient 3y ago
        > The basic premise is that the act of science and knowledge acquisition cyclically devolves into ideology and is then disrupted. This can happen of course, but it's not really what Kuhn is talking about. Kuhnian paradigm shifts are to science what massive refactorings are to a codebase: the problem isn't that the old code was wrong (though it might also be wrong), it's that it can't be extended to meet new requirements. And while some of us are better at writing extensible code than others, no one gets it right every single time.
  - hospitalJail 3y ago
    Not exactly. Control for that in a future experiment? To be fair, as you start to leave chemistry, true knowledge(the goal of science) is impossible to find. The best we can get is little glimpses of the truth. To be fair, I have seen studies that were pretty bullet proof. It just seems that most are lazy or simply impossible to prevent variables.
  - caddemon 3y ago
    I mean I think peer review kind of is fundamentally broken, but at the same time this phenomenon isn't necessarily a sign of it. Even within technical papers, let alone pop writing or internet discussions, many of the claims citations are provided for are broad e.g. "X can increase people's anxiety". A very rigorous study might exist showing that when X happens a specific way it does increase some form of anxiety in some specific subset of the population, but using it as a citation for the broad statement without further context can be misleading at times. It's in fact possible that by considering different subsets of the same statement you'd get an opposite directional effect. That can certainly be used to confuse people, especially in a situation where it is unlikely most readers will dive deeply into the cited works. As far as nit picking - the same general principle applies. There will always be some tradeoff between scope, rigor, and available resources to do the study. If we waited for papers to be perfect in both correctness and interestingness/utility there would be almost nothing to ever publish. So I think the systemic problem here is moreso an undervaluing of review article and text book type resources (both reading them and writing them) in favor of vomiting out random individual paper citations for whatever claim. Science needs more heterogeneity in the roles different PIs fill for the system. Improving peer review process would be great (and might indirectly help), but I don't feel it's the root.
rahimnathwani 3y ago
When reading the first draft of the California Math Framework, I followed many of the citations. My experience was similar to yours. In particular: - some of the cited studies did not claim what the citation said they did - some of the cited studies did claim what the citation said they did, but the experimental results were far too weak to support these claims - some of the cited studies had such weak experimental design, that it would be hard to conclude anything about the subject at hand
- arp242 3y ago
  This is one reason Wikipedia discourages using primary sources (i.e. individual studies) as references. Not that using secondary sources is perfect either, but you can "prove" almost anything by linking to studies as there are just so many of them. Do 1,000 studies on homeopathy and some of them will show a positive effect (even when done well, ignoring there are also many bad studies on these kind of topics).
indymike 3y ago
> One day, I got bored and decided to actually read every single study. Nothing is more entertaining than the meeting after a business presentation where someone fact checks the reference after the power point. When we have vendor presentations, we started doing this because... so many claims, so little truth.
DontchaKnowit 3y ago
Yep. This is why it makes me nearly physically ill when I hear people use terms like "science denier" or say things like "<insert political party here> don't believe in science" or "our laws should be based on science". It is invariably some of the most scientifically illiterate, ideologically entrenched, and intellectually lukewarm people who spout this garbage as a retort to any sort of argument with which they do not wish to engage.
- bdowling 3y ago
  My favorite term like that was “scientific consensus”. I used to point out that up until the 17th century the scientific consensus was that the Sun revolved around the Earth. I would get nothing but blank stares in response. It turns out that people who talk about “scientific consensus” typically don’t know whether the Earth revolves around the Sun, or vice-versa.
  - DontchaKnowit 3y ago
    It is especially frustrating when scientific ideas are applied to moral dilemmas. I see this from the far left wing frequently. The irony is that this same mindset is what led to the proliferation of eugenics apologists in the 1800s. The scientific consensus had absolutely no objections to eugenics on scientific grounds. In fact, "science" would seem to support it. Yet it is widely agreed that this practice is morally reprehensible nowadays, because humans were able to put aside their hubris and apply their moral reasoning.
    - rnk 3y ago
      Plenty of people realized that eugenics was unacceptable who were in the scientific world then, just like plenty of people understood that slavery was wrong. Which doesn't take away from the fact that plenty of terrible policies came out because leaders did have those views. I don't know what left wing ideas you've seen that are wrong, I'm sure that I've seen some left wing ideas that are wrong. I'm more worried about what do the leaders and the effective speakers in groups promote. Saying that I'm making you more free by taking away your choice of library books, or your choice about medical care, that's not an extreme view that only 1% of people have. It's pretty much the consensus leadership view of Republicans.
svnt 3y ago
Maybe assess the argument by the first two studies. If either of them do not support the point (or as has frequently happened for me, actually oppose the point) then you can reply for posterity and no one else needs to replicate your debunking effort. “Your first two studies not only don’t support your argument but invalidate it, and I stopped reading after that.” ends most disagreements. You will have probably done more work than the poster who often has just discovered keyword searches and related articles in google scholar or science direct, but everyone will learn something.
beebmam 3y ago
What are some examples of the claims people made vs what the research actually demonstrated?
- troupe 3y ago
  mRNA vaccines are completely safe - Obviously false just like it is false for any vaccine since there are always going to be rare side effects in some members of the population. They very well may be just as safe as any other vaccine, but the truth is there are some things we won't know until a lot more time goes by. You can't say what the long term side effects are of something until it has been around for a long time. Wind power is better for the environment - Similar to above, we don't really have data on the full lifecycle environmental costs of large scale wind farms. This is especially true because a good portion of the impact depends on how long the turbines last, what type of repair/recycle technologies are developed, and what type of policies get applied to recycling them. We see some things that look promising but others that are concerning.
  - Delk 3y ago
    Saying "mRNA vaccines are completely safe" is obviously not pedantically true. The same could be said of anything, not just mRNA vaccines. Nothing is ever absolutely 100 % safe. It would, at least in theory, be better to realistically say what we knew and didn't know about the risks. But public communication about those kinds of risks and tradeoffs is hard. If you say there might be any kind of a risk at all, some people are going to get hung up on that or grossly overestimate those risks compared to the benefits (or to the risks of not getting the vaccine -- also not something everyone would be affected by but clearly a non-zero risk). Taking that into account, "completely safe" might be a better approximation than most other ones you could make.
    - troupe 3y ago
      While I understand what you are saying, I would disagree. Saying that something is completely safe when it isn't (as you say nothing is completely safe) makes it appear that people are being lied to when someone does have some type of reaction. I think we are better of thinking of risk in terms of every day risk management. Comparing risks to things like driving 10 miles, getting hit by a meteor while sleeping, etc. can help create a better understanding of the risks.
Salgat 3y ago
I'll often read through the studies in an argument but it is very unfeasible to keep up with these people because they can just keep throwing crap at you, and if you disagree with the study, they'll throw another at you until you give up. I remember reading through one 80 page paper on how a public option for healthcare would bankrupt us, but if you actually read through the thing, it never accounts for all the money saved by not being spent on private healthcare (which would fund the entire thing). My local power company had a similar situation where they posted a huge study explaining why solar was costing them a bunch of money, but if you actually read the study it explains that it considers lost income from solar as costing them money (in the same way that moving to a more efficient AC system would "cost" the power company money). It's all bullshit, and it's too easy to abuse studies to defend your position.
njarboe 3y ago
It is sort of like how when some people want to prove something is true they quote the Bible. I find the behavior especially strange when they are quoting the bible to prove the bible is true.
sn9 3y ago
If someone replies within a few minutes with a link to a study, you can usually just assume it doesn't even support their point unless they're actually a researcher in the field. It takes way longer even for experts to read and digest a paper thoroughly to evaluate the methodology and results. Googling for 5 minutes and skimming the abstract for keywords that might be related to whatever you were arguing about is usually the norm online. It takes years of training and feedback to learn to properly evaluate a paper in your own field, and that's after years of undergraduate education at least. Most people online skip even the basic textbook level background and think they understand what they're reading.

hospitalJail 3y ago

If you ever want to reject a study, just read it. You will find something to nit pick.

ben_w 3y ago
I'm not saying you're wrong — I've got my name on exactly one paper so what would I know — but that being true would suggest that peer review is fundamentally broken.
- jvanderbot 3y ago
  No, it would suggest that every paper has something in it that can be considered a fatal flaw by a person biased against it. Peer review is precisely supposed to admit this and minimize flaws in conclusions given inevitable flaws in methodology. Sort of a research paper equivalent of: "If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him." by Cardinal Richelieu
- matusp 3y ago
  Peer review is not a certificate of truthfulness, peer review just says that a grad student reading the paper did not found anything too suspicious there and it looks like a typical paper in that field.
  - some_random 3y ago
    A huge part of the problem is that the layperson is taught otherwise.
    - hospitalJail 3y ago
      2nd grade we learned the scientific method. I don't remember peer reviewed being a thing. Replication was critical. Also, I've seen peer reviewed papers with atrocious stuff in them, unless I know the peer, it doesn't mean much to me. Give me 100 independent groups coming to the same conclusions. That has far more weight than a few of your friends/professors reading your paper.
  - svnt 3y ago
    Peer review, at its core, is a social consensus process. It can work, but it is structurally inclined to propagate agreement. I wonder if it serves to reinforce this problem, as those more motivated to reinforce certain literatures or perspectives will surface those repeatedly in reviews.
    - all2 3y ago
      There's a popular work that comes to this conclusion in its beginning chapters [0]. The basic premise is that the act of science and knowledge acquisition cyclically devolves into ideology and is then disrupted. Typically those who initially disrupt a scientific dogma are not treated well. Eventually the old guard literally dies off and the new ideas can begin to take hold. [0] The Structure of Scientific Revolutions, T. Kuhn
      - consilient 3y ago
        > The basic premise is that the act of science and knowledge acquisition cyclically devolves into ideology and is then disrupted. This can happen of course, but it's not really what Kuhn is talking about. Kuhnian paradigm shifts are to science what massive refactorings are to a codebase: the problem isn't that the old code was wrong (though it might also be wrong), it's that it can't be extended to meet new requirements. And while some of us are better at writing extensible code than others, no one gets it right every single time.
- hospitalJail 3y ago
  Not exactly. Control for that in a future experiment? To be fair, as you start to leave chemistry, true knowledge(the goal of science) is impossible to find. The best we can get is little glimpses of the truth. To be fair, I have seen studies that were pretty bullet proof. It just seems that most are lazy or simply impossible to prevent variables.
- caddemon 3y ago
  I mean I think peer review kind of is fundamentally broken, but at the same time this phenomenon isn't necessarily a sign of it. Even within technical papers, let alone pop writing or internet discussions, many of the claims citations are provided for are broad e.g. "X can increase people's anxiety". A very rigorous study might exist showing that when X happens a specific way it does increase some form of anxiety in some specific subset of the population, but using it as a citation for the broad statement without further context can be misleading at times. It's in fact possible that by considering different subsets of the same statement you'd get an opposite directional effect. That can certainly be used to confuse people, especially in a situation where it is unlikely most readers will dive deeply into the cited works. As far as nit picking - the same general principle applies. There will always be some tradeoff between scope, rigor, and available resources to do the study. If we waited for papers to be perfect in both correctness and interestingness/utility there would be almost nothing to ever publish. So I think the systemic problem here is moreso an undervaluing of review article and text book type resources (both reading them and writing them) in favor of vomiting out random individual paper citations for whatever claim. Science needs more heterogeneity in the roles different PIs fill for the system. Improving peer review process would be great (and might indirectly help), but I don't feel it's the root.

rahimnathwani 3y ago

When reading the first draft of the California Math Framework, I followed many of the citations. My experience was similar to yours. In particular: - some of the cited studies did not claim what the citation said they did - some of the cited studies did claim what the citation said they did, but the experimental results were far too weak to support these claims - some of the cited studies had such weak experimental design, that it would be hard to conclude anything about the subject at hand

arp242 3y ago
This is one reason Wikipedia discourages using primary sources (i.e. individual studies) as references. Not that using secondary sources is perfect either, but you can "prove" almost anything by linking to studies as there are just so many of them. Do 1,000 studies on homeopathy and some of them will show a positive effect (even when done well, ignoring there are also many bad studies on these kind of topics).

indymike 3y ago

> One day, I got bored and decided to actually read every single study. Nothing is more entertaining than the meeting after a business presentation where someone fact checks the reference after the power point. When we have vendor presentations, we started doing this because... so many claims, so little truth.

DontchaKnowit 3y ago

Yep. This is why it makes me nearly physically ill when I hear people use terms like "science denier" or say things like "<insert political party here> don't believe in science" or "our laws should be based on science". It is invariably some of the most scientifically illiterate, ideologically entrenched, and intellectually lukewarm people who spout this garbage as a retort to any sort of argument with which they do not wish to engage.

bdowling 3y ago
My favorite term like that was “scientific consensus”. I used to point out that up until the 17th century the scientific consensus was that the Sun revolved around the Earth. I would get nothing but blank stares in response. It turns out that people who talk about “scientific consensus” typically don’t know whether the Earth revolves around the Sun, or vice-versa.
- DontchaKnowit 3y ago
  It is especially frustrating when scientific ideas are applied to moral dilemmas. I see this from the far left wing frequently. The irony is that this same mindset is what led to the proliferation of eugenics apologists in the 1800s. The scientific consensus had absolutely no objections to eugenics on scientific grounds. In fact, "science" would seem to support it. Yet it is widely agreed that this practice is morally reprehensible nowadays, because humans were able to put aside their hubris and apply their moral reasoning.
  - rnk 3y ago
    Plenty of people realized that eugenics was unacceptable who were in the scientific world then, just like plenty of people understood that slavery was wrong. Which doesn't take away from the fact that plenty of terrible policies came out because leaders did have those views. I don't know what left wing ideas you've seen that are wrong, I'm sure that I've seen some left wing ideas that are wrong. I'm more worried about what do the leaders and the effective speakers in groups promote. Saying that I'm making you more free by taking away your choice of library books, or your choice about medical care, that's not an extreme view that only 1% of people have. It's pretty much the consensus leadership view of Republicans.

svnt 3y ago

Maybe assess the argument by the first two studies. If either of them do not support the point (or as has frequently happened for me, actually oppose the point) then you can reply for posterity and no one else needs to replicate your debunking effort. “Your first two studies not only don’t support your argument but invalidate it, and I stopped reading after that.” ends most disagreements. You will have probably done more work than the poster who often has just discovered keyword searches and related articles in google scholar or science direct, but everyone will learn something.

beebmam 3y ago

What are some examples of the claims people made vs what the research actually demonstrated?

troupe 3y ago
mRNA vaccines are completely safe - Obviously false just like it is false for any vaccine since there are always going to be rare side effects in some members of the population. They very well may be just as safe as any other vaccine, but the truth is there are some things we won't know until a lot more time goes by. You can't say what the long term side effects are of something until it has been around for a long time. Wind power is better for the environment - Similar to above, we don't really have data on the full lifecycle environmental costs of large scale wind farms. This is especially true because a good portion of the impact depends on how long the turbines last, what type of repair/recycle technologies are developed, and what type of policies get applied to recycling them. We see some things that look promising but others that are concerning.
- Delk 3y ago
  Saying "mRNA vaccines are completely safe" is obviously not pedantically true. The same could be said of anything, not just mRNA vaccines. Nothing is ever absolutely 100 % safe. It would, at least in theory, be better to realistically say what we knew and didn't know about the risks. But public communication about those kinds of risks and tradeoffs is hard. If you say there might be any kind of a risk at all, some people are going to get hung up on that or grossly overestimate those risks compared to the benefits (or to the risks of not getting the vaccine -- also not something everyone would be affected by but clearly a non-zero risk). Taking that into account, "completely safe" might be a better approximation than most other ones you could make.
  - troupe 3y ago
    While I understand what you are saying, I would disagree. Saying that something is completely safe when it isn't (as you say nothing is completely safe) makes it appear that people are being lied to when someone does have some type of reaction. I think we are better of thinking of risk in terms of every day risk management. Comparing risks to things like driving 10 miles, getting hit by a meteor while sleeping, etc. can help create a better understanding of the risks.

Salgat 3y ago

I'll often read through the studies in an argument but it is very unfeasible to keep up with these people because they can just keep throwing crap at you, and if you disagree with the study, they'll throw another at you until you give up. I remember reading through one 80 page paper on how a public option for healthcare would bankrupt us, but if you actually read through the thing, it never accounts for all the money saved by not being spent on private healthcare (which would fund the entire thing). My local power company had a similar situation where they posted a huge study explaining why solar was costing them a bunch of money, but if you actually read the study it explains that it considers lost income from solar as costing them money (in the same way that moving to a more efficient AC system would "cost" the power company money). It's all bullshit, and it's too easy to abuse studies to defend your position.

njarboe 3y ago

It is sort of like how when some people want to prove something is true they quote the Bible. I find the behavior especially strange when they are quoting the bible to prove the bible is true.

sn9 3y ago

If someone replies within a few minutes with a link to a study, you can usually just assume it doesn't even support their point unless they're actually a researcher in the field. It takes way longer even for experts to read and digest a paper thoroughly to evaluate the methodology and results. Googling for 5 minutes and skimming the abstract for keywords that might be related to whatever you were arguing about is usually the norm online. It takes years of training and feedback to learn to properly evaluate a paper in your own field, and that's after years of undergraduate education at least. Most people online skip even the basic textbook level background and think they understand what they're reading.