5 ms·

Independent replication is the cornerstone of Karl Popper's formulation of the scientific method. If we were really holding to philosophical basis for how scien

Independent replication is the cornerstone of Karl Popper's formulation of the scientific method. If we were really holding to philosophical basis for how science is supposed to work, then we'd be careful to consider results that have only been demonstrated by one laboratory to be tentative at best, and be cautious about making policy decisions based on it until there has been truly independent replication. Physicists seem to be really good about this, and many other aspects of implementing the scientific method too. I wish the other sciences would get on board. It would eliminate almost all the chronic problems that plague biological and social sciences: falsification, p-hacking, failing to notice honest methodological mistakes, outright fraud, etc. I fear that the problem is, we can't get there from here for social reasons. The people at the top of these fields - the ones who drive culture in academic institutions, set publication standards for journals, influence where grant money is allocated, etc. - all got there by using sloppy methods and getting lucky. I think that, on some level, many of them know it, and know that fixing the rotten core of their field inevitably involves subjecting their own work - and, by extension, reputations - to a level of scrutiny that it is unlikely to survive.

stonethrowaway 2y ago
> I wish the other sciences would get on board It’s not about science it’s about politics and corruption and power plays. At the expense of being accused of posting lazily, you really have to follow the money. 2 more weeks to flatten the curve while the oligarchs pillage the economy some more.
- fastball 2y ago
  It's definitely not just politics. Popper was not just a champion of reproducibility but also a champion of falsifiability. This is something many a scientist also loses sight of. People will formulate their hypothesis and then try to prove it, rather than trying to disprove it, which is often the much more effective strategy for avoiding bias and other issues. You see this most often with the "soft" sciences, but it crops up in the harder ones as well. Focusing on reproducibility and falsifiability would go a long way to improving the current state of science, regardless of the political games happening at the same time.
  - ToucanLoucan 2y ago
    To expand on that person's thought though, that's the problem: reproducibility and falsifiability don't make money, and in our system, what doesn't make money, costs money. There is absolutely zero funding out there available for studies to reproduce the results of other studies. And, in addition, any source of funding that is offered for regular non-reproducing studies also comes with expectations of the scientist in charge of the study. You can say "it shouldn't be this way" all you like but the fact remains that when a fossil-fuel company funds a study into if humans affect global climate change, they obviously have an answer in mind that they're wanting to hear, and not only that, probably sought out a scientist who has an existing reputation that implies they will deliver that answer, just as, say, a governmental probe into climate change might do the exact opposite. Every funding organization chooses the studies to fund run by the scientists they also choose to fund with an outcome in mind. Hopefully, this is less true with Government funded research? But it's still absolutely true, and also worth noting that Government-backed research has been in a steady decline since the 1980's. And I mean, that's just money. There is also zero career progression or public impact on reproducibility. One cannot reasonably build a sustainable career in science simply by reproducing studies. Nobody gives a shit about those. Those don't get you jobs at better, more prestigious institutions. Those don't get you interviewed on CNN. Those don't get you, hell, funding for more research you might actually want to do. The scientific community itself is a community of people, it is not immune from the corrosive and negative aspects of any community created by humans, it inherits all our faults and foibles just the same as any other, it's affected by similar biases, it's paralyzed into the same inaction, and it caves to the same influences. Truly neutral, unbiased thinking and therefore action is, IMHO, impossible. Anyone who says otherwise has either deluded themselves into thinking their agenda is somehow magically no agenda, or is well aware they have an agenda and wish that fact to remain unacknowledged. No one is truly neutral. Every thought you have, every action you consider, every idea that comes to mind is a totality of every previous thought, action and idea you've had and the fact that so many people claim to be unbiased or neutral doesn't change this. You're not objective, you're alive. That doesn't mean there's no truth to be had or found, and indeed, broadly, the scientific method as outlined in our school curricula is the closest thing humanity has thus far created to a truth producing methodology. But it remains built of humans and so inherits their limitations, and we should remain cognizant of that.
deleted 2y ago
[deleted]
matthewdgreen 2y ago
>Physicists seem to be really good about this Consider the possibility that what you're observing is not so much different cultures, but different levels of resources being provided. Experimental physics receives vast amounts of funding from scientific agencies. Much of this goes to support expensive equipment like particle accelerators and neutrino detectors. However the funding of this infrastructure doesn't occur in a vacuum: the operating budget for these systems also requires the resources to perform careful data collection and analysis. You cannot justify running a quick low-budget experiment with the LHC, simply because it costs too much. It's worth pointing out that most of the boogeyman results people hold up in the "replication crisis" literature are in the fields of the social sciences, where funding is vastly less available. Here you get weaker techniques, fewer resources for replication, and many people who are struggling just to fund any small studies at all. What you immediately diagnose as malicious behavior is much more adequately explained by lack of resources. It is very human to think that the problem is bad humans. But in the real world, the biggest factor is almost always resources. Everything else is downstream of that.
- jackcosgrove 2y ago
  The social sciences also have a much more limited ability to perform controlled experiments because of ethical reasons, which cannot be resolved with more money. So much of the studies published in these fields are observational rather than controlled experiments. Physics (and chemistry, to a lesser extent biology, and medicine lesser still) has an advantage in that it's exploring the parts of the universe where controlled experiments are possible. I don't think it's all about resources, and I think the social sciences are less resourced in part because grant-making bodies know that these fields suffer from these intrinsic limitations.
  - bunderbunder 2y ago
    It kind of depends on where you are in physics. It might be easy for a materials scientist to do a controlled experiment, but it's approximately impossible for an astrophysicist. You can't exactly randomly assign celestial bodies to treatment and non-treatment groups; all you can do is observe.
cauch 2y ago
But "not giving beta blockers" is also a decision. In that context, there is no "we can freeze time for the rest of the world while scientists add new studies", you have to choose. Imagine a parallel universe where beta blockers are a good solution and where Poldermans' study was fraudulent and said that beta blockers are bad. According to you, doctors should have not trusted Poldermans' study and, therefore, should have continued to give beta blockers. So, in this parallel universe, your definition of "not trusting the study" = "doing exactly the opposite of what not trusting the study means in the first universe". Then there are also parallel universes where Poldermans' study was not fraudulent. What about there? Is adopting the opposite of the conclusion the correct things to do while waiting for new studies? Or are we rather saying "well, I know there is not much there, but the probability that the study is crap is lower than the probability that it is not, so in the meanwhile, let's follow its conclusions, it's the best bet in the meanwhile"
- mikeiz404 2y ago
  You make good points. I think in order to make a good decision in all these possible scenarios (since you don't know which one a specific study might be in) is to make sure you have a good understanding of what the likelihoods are any single study is in one of these scenarios (ex: fraudulent). Right now it would appear the perception and the reality of the likelihood a study is trustworthy is in disagreement. Hopefully new incentives coupled with public statistics can help fix this.
  - cauch 2y ago
    I'm not sure you can say if the trust is overestimated or not based on the fact that "yes/no" decision are close to 100% in favor of trusting each study. Let's imagine you distrust a lot: you believe that a study has 45% chance to be fraudulent. You have 100 studies S1, S2, S3, ..., each saying that a process is better than another. You take study S1. It concludes you should do G1 and avoid B1. You have 55% chance doing G1 will save life and doing B1 will kill. Conclusion: let's do G1, doing B1 would be stupid: 55% chance of killing instead of 45%. You take study S2. It concludes you should do G2 and avoid B2. You have 55% chance doing G2 will save life and doing B2 will kill. Conclusion: let's do G2, doing B2 would be stupid: 55% chance of killing instead of 45%. ... So, at the end, you will follow the conclusion of 100% of the studies, even if you only trust 55% of them. You end up making 45% of errors, but 55% of good choice. Now, let's say you want to follow a ratio of conclusions that is similar to your level of trust. You want it to be following 55% of the conclusions. Which one do you follow? If you pick 55% at random, let's say studies 2, 3, 6, 7, 8, ... What is the probability that those are the correct studies? If you do a quick simulation, you will see that you end up making 49% of errors, and 51% of good choice. So, still accepting that you will follow a bad study is still better than randomly discard a study "just so I remove some studies that may be bad". You may say that you will discard the studies that look bad, not take them randomly, but the trick is that you cannot tell which study look bad. Poldermans study looked convincing at first, and it's few years later that problems were found, and they were found by people having the means to properly investigate (they had access to insider info that you will not have access to). Possibly there were more good studies that looked worst than Poldermans'. edit: also, a large fraction of fraudulent studies is done to hide "non conclusive" results, which means that a fraudulent study that concludes that G is better than B does not mean that B is better than G, it just means they have no idea which one is the best. So, you cannot also pretend that a fishy study implies that the opposite conclusion is proven.
Der_Einzige 2y ago
That's why you reject Karl Popper's epistemology and embrace the the epistemological anarchism of Feyerabend. - https://en.wikipedia.org/wiki/Against_Method https://en.wikipedia.org/wiki/Against_Method
Ferret7446 2y ago
> Physicists seem to be really good about this, and many other aspects of implementing the scientific method too. That's because physics is relatively immune from politics. Other fields are on the other side of the "practice vs theory" spectrum and thus attract a lot of politics/interests which influence the science. https://xkcd.com/435/ https://xkcd.com/435/

stonethrowaway 2y ago

> I wish the other sciences would get on board It’s not about science it’s about politics and corruption and power plays. At the expense of being accused of posting lazily, you really have to follow the money. 2 more weeks to flatten the curve while the oligarchs pillage the economy some more.

fastball 2y ago
It's definitely not just politics. Popper was not just a champion of reproducibility but also a champion of falsifiability. This is something many a scientist also loses sight of. People will formulate their hypothesis and then try to prove it, rather than trying to disprove it, which is often the much more effective strategy for avoiding bias and other issues. You see this most often with the "soft" sciences, but it crops up in the harder ones as well. Focusing on reproducibility and falsifiability would go a long way to improving the current state of science, regardless of the political games happening at the same time.
- ToucanLoucan 2y ago
  To expand on that person's thought though, that's the problem: reproducibility and falsifiability don't make money, and in our system, what doesn't make money, costs money. There is absolutely zero funding out there available for studies to reproduce the results of other studies. And, in addition, any source of funding that is offered for regular non-reproducing studies also comes with expectations of the scientist in charge of the study. You can say "it shouldn't be this way" all you like but the fact remains that when a fossil-fuel company funds a study into if humans affect global climate change, they obviously have an answer in mind that they're wanting to hear, and not only that, probably sought out a scientist who has an existing reputation that implies they will deliver that answer, just as, say, a governmental probe into climate change might do the exact opposite. Every funding organization chooses the studies to fund run by the scientists they also choose to fund with an outcome in mind. Hopefully, this is less true with Government funded research? But it's still absolutely true, and also worth noting that Government-backed research has been in a steady decline since the 1980's. And I mean, that's just money. There is also zero career progression or public impact on reproducibility. One cannot reasonably build a sustainable career in science simply by reproducing studies. Nobody gives a shit about those. Those don't get you jobs at better, more prestigious institutions. Those don't get you interviewed on CNN. Those don't get you, hell, funding for more research you might actually want to do. The scientific community itself is a community of people, it is not immune from the corrosive and negative aspects of any community created by humans, it inherits all our faults and foibles just the same as any other, it's affected by similar biases, it's paralyzed into the same inaction, and it caves to the same influences. Truly neutral, unbiased thinking and therefore action is, IMHO, impossible. Anyone who says otherwise has either deluded themselves into thinking their agenda is somehow magically no agenda, or is well aware they have an agenda and wish that fact to remain unacknowledged. No one is truly neutral. Every thought you have, every action you consider, every idea that comes to mind is a totality of every previous thought, action and idea you've had and the fact that so many people claim to be unbiased or neutral doesn't change this. You're not objective, you're alive. That doesn't mean there's no truth to be had or found, and indeed, broadly, the scientific method as outlined in our school curricula is the closest thing humanity has thus far created to a truth producing methodology. But it remains built of humans and so inherits their limitations, and we should remain cognizant of that.

deleted 2y ago

[deleted]

matthewdgreen 2y ago

>Physicists seem to be really good about this Consider the possibility that what you're observing is not so much different cultures, but different levels of resources being provided. Experimental physics receives vast amounts of funding from scientific agencies. Much of this goes to support expensive equipment like particle accelerators and neutrino detectors. However the funding of this infrastructure doesn't occur in a vacuum: the operating budget for these systems also requires the resources to perform careful data collection and analysis. You cannot justify running a quick low-budget experiment with the LHC, simply because it costs too much. It's worth pointing out that most of the boogeyman results people hold up in the "replication crisis" literature are in the fields of the social sciences, where funding is vastly less available. Here you get weaker techniques, fewer resources for replication, and many people who are struggling just to fund any small studies at all. What you immediately diagnose as malicious behavior is much more adequately explained by lack of resources. It is very human to think that the problem is bad humans. But in the real world, the biggest factor is almost always resources. Everything else is downstream of that.

jackcosgrove 2y ago
The social sciences also have a much more limited ability to perform controlled experiments because of ethical reasons, which cannot be resolved with more money. So much of the studies published in these fields are observational rather than controlled experiments. Physics (and chemistry, to a lesser extent biology, and medicine lesser still) has an advantage in that it's exploring the parts of the universe where controlled experiments are possible. I don't think it's all about resources, and I think the social sciences are less resourced in part because grant-making bodies know that these fields suffer from these intrinsic limitations.
- bunderbunder 2y ago
  It kind of depends on where you are in physics. It might be easy for a materials scientist to do a controlled experiment, but it's approximately impossible for an astrophysicist. You can't exactly randomly assign celestial bodies to treatment and non-treatment groups; all you can do is observe.

cauch 2y ago

But "not giving beta blockers" is also a decision. In that context, there is no "we can freeze time for the rest of the world while scientists add new studies", you have to choose. Imagine a parallel universe where beta blockers are a good solution and where Poldermans' study was fraudulent and said that beta blockers are bad. According to you, doctors should have not trusted Poldermans' study and, therefore, should have continued to give beta blockers. So, in this parallel universe, your definition of "not trusting the study" = "doing exactly the opposite of what not trusting the study means in the first universe". Then there are also parallel universes where Poldermans' study was not fraudulent. What about there? Is adopting the opposite of the conclusion the correct things to do while waiting for new studies? Or are we rather saying "well, I know there is not much there, but the probability that the study is crap is lower than the probability that it is not, so in the meanwhile, let's follow its conclusions, it's the best bet in the meanwhile"

mikeiz404 2y ago
You make good points. I think in order to make a good decision in all these possible scenarios (since you don't know which one a specific study might be in) is to make sure you have a good understanding of what the likelihoods are any single study is in one of these scenarios (ex: fraudulent). Right now it would appear the perception and the reality of the likelihood a study is trustworthy is in disagreement. Hopefully new incentives coupled with public statistics can help fix this.
- cauch 2y ago
  I'm not sure you can say if the trust is overestimated or not based on the fact that "yes/no" decision are close to 100% in favor of trusting each study. Let's imagine you distrust a lot: you believe that a study has 45% chance to be fraudulent. You have 100 studies S1, S2, S3, ..., each saying that a process is better than another. You take study S1. It concludes you should do G1 and avoid B1. You have 55% chance doing G1 will save life and doing B1 will kill. Conclusion: let's do G1, doing B1 would be stupid: 55% chance of killing instead of 45%. You take study S2. It concludes you should do G2 and avoid B2. You have 55% chance doing G2 will save life and doing B2 will kill. Conclusion: let's do G2, doing B2 would be stupid: 55% chance of killing instead of 45%. ... So, at the end, you will follow the conclusion of 100% of the studies, even if you only trust 55% of them. You end up making 45% of errors, but 55% of good choice. Now, let's say you want to follow a ratio of conclusions that is similar to your level of trust. You want it to be following 55% of the conclusions. Which one do you follow? If you pick 55% at random, let's say studies 2, 3, 6, 7, 8, ... What is the probability that those are the correct studies? If you do a quick simulation, you will see that you end up making 49% of errors, and 51% of good choice. So, still accepting that you will follow a bad study is still better than randomly discard a study "just so I remove some studies that may be bad". You may say that you will discard the studies that look bad, not take them randomly, but the trick is that you cannot tell which study look bad. Poldermans study looked convincing at first, and it's few years later that problems were found, and they were found by people having the means to properly investigate (they had access to insider info that you will not have access to). Possibly there were more good studies that looked worst than Poldermans'. edit: also, a large fraction of fraudulent studies is done to hide "non conclusive" results, which means that a fraudulent study that concludes that G is better than B does not mean that B is better than G, it just means they have no idea which one is the best. So, you cannot also pretend that a fishy study implies that the opposite conclusion is proven.

Der_Einzige 2y ago

That's why you reject Karl Popper's epistemology and embrace the the epistemological anarchism of Feyerabend. - https://en.wikipedia.org/wiki/Against_Method https://en.wikipedia.org/wiki/Against_Method

Ferret7446 2y ago

> Physicists seem to be really good about this, and many other aspects of implementing the scientific method too. That's because physics is relatively immune from politics. Other fields are on the other side of the "practice vs theory" spectrum and thus attract a lot of politics/interests which influence the science. https://xkcd.com/435/ https://xkcd.com/435/