6 ms·

Do we have good data to the contrary, though? Is there evidence that most science is good? The replicability crisis has certainly given some evidence that much

by bumby 2y ago

Do we have good data to the contrary, though? Is there evidence that most science is good? The replicability crisis has certainly given some evidence that much of published research could be classified as 'not good' in some domains. (And maybe there is an important distinction to make between 'bad' science and 'fraudulent' science). If there isn't good evidence to show that most science is good, the best we can probably claim is that we don't know.

matthewdgreen 2y ago
I don’t think “there is an important distinction between ‘bad’ science and ‘fraudulent’ science” should have the word “maybe” attached to it. Of course there is a difference, just as there is a huge distinction between premeditated murder and accidental deaths. The correct level of both should be zero, but one is vastly more damaging and also more rare than the other. As an aside: the problem I have with the OP is that it isn’t arguing about scientific quality, it’s arguing that science is rife with actual fraud. Typically in the midst of such conversations the conversation switches from actual accusations of fraud to the replication crisis in small-effect-size social science experimental work, which seems to be more an issue of people being careless than malicious —- and importantly, is an issue that the scientific community is addressing through improved standards. With respect to your question: “is there evidence that most science is good”, I’m not even sure how you would answer that question. All I know is that if you compare the world of my grandmother to the world of today, the two are so different as to be unrecognizable, and scientific advances are responsible for most of the difference. That progress continues every single day, and we’re both alive because of science was at least good enough to make our continued existence possible.
- bumby 2y ago
  It reads like most of your issue is with my phrasing, which was deliberate to soften the comment so it doesn’t come across as accusatory. >I’m not even sure how you would answer the question. It should probably be answered in the same way we’d answer the converse. Something like “here is the data and the tests that show the data quality is high and the results are replicable.” My point was, if you can’t show that, then the claims that the work is good may not be scientifically valid. To put a finer point on it, we know there is replicability crisis in many corners of science. But we don’t likely know how much of that is due to bad science or outright fraud unless we do the science-y work to check. And unfortunately many of the incentives are aligned against that very thing.
  - matthewdgreen 2y ago
    I have not claimed that all scientific work is good or valid. The post I responded to said: “given the shameless plague of fraudulent papers, and the gaming of citations, my faith in experts and scientists has been damaged.” I picked a bone with that given the total lack of evidence for the claim. More generally I have a problem with a broader class of people who make claims like this, since I find these claims unlikely and unjustified. You came back to me with a different question: “how do we know that all science is good?” I genuinely don’t know how to answer that question. And bluntly, I think it would be incredibly hard to answer it: the question is up there with “how many grains of sand are there on the world’s beaches.” I do think that clearly science, in whatever state it is in, has historically worked amazingly well. The world we live in is a testament to that. Maybe the answer is that it doesn’t matter, that ecience has always been mediocre and yet we go to space and cure diseases with even low-quality science. A more general answer is that everyone doing science is working incredibly hard to find the answers to hard questions. We are always doing so imperfectly, with imperfect methods and imperfect tools and imperfect brains and yet we soldier on. If you want to join this effort, either to further it or bring evidence to criticize and improve it, we are all excited to have you. But simply posing impossible-to-answer questions is not necessarily doing either thing.
    - bumby 2y ago
      >more general answer is that everyone doing science is working incredibly hard to find the answers to hard questions. I think this is too romantic of a view. Yes, some (or even many) are working hard to answer hard questions. Others are seeking money, status and/or prestige as their top priority, just like practically any other field. It’s no different than saying financiers are working incredibly hard to create a rising tide that lifts all boats. It makes for a lofty narrative, but it’s also ignoring much of human nature. And if you think that question is impossible to answer, it should at the very least give you pause to think about the confidence you have in your conclusion. Otherwise it reads as “I can’t prove it, but I know it’s true.” That’s dogma, not science.
      - matthewdgreen 2y ago

by bumby 2y ago

matthewdgreen 2y ago

I don’t think “there is an important distinction between ‘bad’ science and ‘fraudulent’ science” should have the word “maybe” attached to it. Of course there is a difference, just as there is a huge distinction between premeditated murder and accidental deaths. The correct level of both should be zero, but one is vastly more damaging and also more rare than the other. As an aside: the problem I have with the OP is that it isn’t arguing about scientific quality, it’s arguing that science is rife with actual fraud. Typically in the midst of such conversations the conversation switches from actual accusations of fraud to the replication crisis in small-effect-size social science experimental work, which seems to be more an issue of people being careless than malicious —- and importantly, is an issue that the scientific community is addressing through improved standards. With respect to your question: “is there evidence that most science is good”, I’m not even sure how you would answer that question. All I know is that if you compare the world of my grandmother to the world of today, the two are so different as to be unrecognizable, and scientific advances are responsible for most of the difference. That progress continues every single day, and we’re both alive because of science was at least good enough to make our continued existence possible.

bumby 2y ago
It reads like most of your issue is with my phrasing, which was deliberate to soften the comment so it doesn’t come across as accusatory. >I’m not even sure how you would answer the question. It should probably be answered in the same way we’d answer the converse. Something like “here is the data and the tests that show the data quality is high and the results are replicable.” My point was, if you can’t show that, then the claims that the work is good may not be scientifically valid. To put a finer point on it, we know there is replicability crisis in many corners of science. But we don’t likely know how much of that is due to bad science or outright fraud unless we do the science-y work to check. And unfortunately many of the incentives are aligned against that very thing.
- matthewdgreen 2y ago
  I have not claimed that all scientific work is good or valid. The post I responded to said: “given the shameless plague of fraudulent papers, and the gaming of citations, my faith in experts and scientists has been damaged.” I picked a bone with that given the total lack of evidence for the claim. More generally I have a problem with a broader class of people who make claims like this, since I find these claims unlikely and unjustified. You came back to me with a different question: “how do we know that all science is good?” I genuinely don’t know how to answer that question. And bluntly, I think it would be incredibly hard to answer it: the question is up there with “how many grains of sand are there on the world’s beaches.” I do think that clearly science, in whatever state it is in, has historically worked amazingly well. The world we live in is a testament to that. Maybe the answer is that it doesn’t matter, that ecience has always been mediocre and yet we go to space and cure diseases with even low-quality science. A more general answer is that everyone doing science is working incredibly hard to find the answers to hard questions. We are always doing so imperfectly, with imperfect methods and imperfect tools and imperfect brains and yet we soldier on. If you want to join this effort, either to further it or bring evidence to criticize and improve it, we are all excited to have you. But simply posing impossible-to-answer questions is not necessarily doing either thing.
  - bumby 2y ago
    >more general answer is that everyone doing science is working incredibly hard to find the answers to hard questions. I think this is too romantic of a view. Yes, some (or even many) are working hard to answer hard questions. Others are seeking money, status and/or prestige as their top priority, just like practically any other field. It’s no different than saying financiers are working incredibly hard to create a rising tide that lifts all boats. It makes for a lofty narrative, but it’s also ignoring much of human nature. And if you think that question is impossible to answer, it should at the very least give you pause to think about the confidence you have in your conclusion. Otherwise it reads as “I can’t prove it, but I know it’s true.” That’s dogma, not science.
    - matthewdgreen 2y ago