5 ms·

I think you need to draw a line between "not replicable" and fraudulent. This is about the latter. This is the kind of corruption that spreads. It's a very bad

I think you need to draw a line between "not replicable" and fraudulent. This is about the latter. This is the kind of corruption that spreads. It's a very bad thing. It is related to the former problem of "not replicable," which is about application of the "scientific method." On this, my worldview has evolved a lot over the last decade. I think we've been applying overly Popperian methods, and formalisations of the scientific method where they don't work well. Say you are studying the effects of a breakfast program, 2 hr recess or push-up regiment on high schoolers. The study is well constructed, faithfully conducted, etc. You find interesting, potentially actionable ways of reducing obesity, improving psych health or something. Do/Should we even expect such a study to replicate in 1990s Tokyo, 2000s Miami, School A, School B, etc. Does that mean this study should not be conducted? Does it mean we can't generate knowledge and understanding of things this way? Is there any daylight between "anecdote" and a result that qualifies for publication in Physics? A lot of articles published in many fields feel like they are being hammered into a Popperian mold they don't fit into. Everyone knows it, but no one is willing to say "this method doesn't work here. we're actually doing a different thing." Instead, they sort of eye roll the article into the frame and let readers unscramble it for themselves.

rwbhn 5y ago
If the results are not replicable, how can they be actionable?
- ethanbond 5y ago
  Think about the replication crisis as actually a generalization crisis. The study said what it said for the exact parameters given. The question is what were all the parameters and which ones are the final result actually sensitive to? It’s actionable by trying it in a new circumstance (set of parameters) and observing whether it fails - over and over. Failure doesn’t mean the claim is wrong/untrue, it means you’ve changed a parameter that happens to be important: now find it!
  - jtbayly 5y ago
    Or it means there was fraud, which is why they are ultimately not able to be separated.
  - DanielBMarkham 5y ago
    Yes. Find it. And when you do, you'll have something reproducible. The purpose of science is to advance the species. Given this, when that happens, this other thing(s) occur (or occur with a certain frequency) I agree that this is a generalization problem. I just don't believe that this particular way of describing it is helpful. If you'll allow me an apocryphal example, when Newton saw the apple fall, if he had generalized to "all things from apple trees fall at the same rate", he would have been correct for a while, then ludicrously incorrect once the leaves started blowing off in Autumn. If at that point he had tried to figure out exactly what kind of tree produces this gravity thing, or which parts of trees, we wouldn't be any further along than we were before. Generalizations work along multiple axes. With some degree of imprecision, every experiment can be a success (or failure). This is why we have the scientific method in the first place. Now, perhaps you're making the argument that this is somehow, for lack of a better term, half-science. It's not right, but it's not wrong either. It's truthy. The logical follow-up would be: ok, where is the second half? In this and many other cases there is no second half. Papers were published, people received awards, wrote books, became famous, and lots of other folks used their work. So even if somehow this is better than nothing, in practice I'm not seeing it.
    - ethanbond 5y ago
      To be clear, in this case there appears to have been fraud ie actual falsification of data. That’s very different from the broader replication crisis, which is more of the shape of unknown parameters. The two get blurry in some places due to eg publication incentives pushing people’s interpretation of legitimate data into semi-fraudulence. But there are still two ends (at least) on the spectrum of non-replicable science. I think framing it in this lens is helpful because it points toward a solution: capturing more of the parameters more explicitly and varying them more deliberately. If we just say, “oh x y z studies aren’t replicable, field [x y z] is bullshit,” we are ending our inquiry prematurely. We are ending it at the falsification of Newton’s initial theory and saying that because leaves fall at a different rate, that gravity doesn’t exist.
- dalbasal 5y ago
  Let me continue the example... We found that students participating in the "2hr active recess program" have meaningfully better health outcomes, psychological health outcomes, and better grades. It was tested faithfully in 25 schools over 10 years. It expands to other places. Different countries. Different decades. Different people run it in different ways. The kids are different. In some cases, the kids are already so active that the extra activity is detrimental. In some places, it's really hot. Kids tend to read, or do more organized indoor activities. Basically, the world is not a laboratory. You're not predicting the trajectory of a projectile, where unknowables can be quantified as a function. It's all 3 body problems all the time. Yet, we as humans do manage to make sense of the world. We run schools, companies. We learn more about how to do those things. Formal studies are an important part of learning about how to do those things. Scientific mindset and methodologies are useful to this. It's a lot better than just philosophizing the whole thing. The type of knowledge we gain, OTOH, isn't like F=MA. It never will be. The value of the above study does relate/imply to it containing knowledge. It's a partial knowledge of a general problem. It is not complete knowledge of a particular problem, like you would get in a lab. Neither lab experiments or wild studies tend to replicate in one another's world. No matter how good the study, how many times it replicated, or how well it worked throughout the 80s and 90s... there's no guarantee that your program is working well or a guaranteed explanation for why it isn't.
  - kortilla 5y ago
    > Different countries. Different decades. Different people run it in different ways. The kids are different. In some cases, the kids are already so active that the extra activity is detrimental. In some places, it's really hot. Kids tend to read, or do more organized indoor activities. Elucidating confounding variables and controlling for them is the entire fucking job of a scientist. To your example, if they publish a paper that explicitly highlights that it worked well for sedentary kids with typical diets of X, Y, and Z in region Q, then it failing to be a found in other environments isn’t a problem. It’s a problem when these clowns do a test on one specific group of 20 people and then generalize to, “doing X leads to Y in humans”. The answer is absolutely not loosening up the constraints. It’s to keep lambasting people who generalize before anything has been reproduced widely. Popper’s science is fine. Morons who don’t realize what they measured are the problem.
nyolfen 5y ago
the replication crisis came to a head a few years ago when it was revealed that two thirds of social psychology findings failed to replicate -- this is not just empiricism taken too far, this is the mass production of false knowledge due to poor training at best or incentive misalignment at worst
- dalbasal 5y ago
  True, and a lot of it is fraud or not far. However, a big chunk of this statistic is not actually fraud. It's just false empiricism. If you look at a lot of the actual papers, I think you'll find that a lot of them just aren't trying to do empiricism... at least not strict empiricism. They're forced into that structure by a convention. Either (1) "Not Science" becomes OK and extends to economics, some areas of ecology, and such. (2) The Scientific Method gets renamed "Method A," and the concept of scientific pursuit is more flexible, depending on what you are studying or (3) We force these poor bastards doing economics, psychology and such to pretend that they're doing a method that they're not... or publish in the prestigious "Unscientific Economics & Psychology" journal.
- tpoacher 5y ago
  It is not even about malice. It is a straightforward consequence of how hypothesis testing is conducted. Greenland et al have a nice section on this: > Despite its shortcomings for interpreting current data, power can be useful for designing studies and for understanding why replication of ‘‘statistical significance’’ will often fail even under ideal conditions. Studies are often designed or claimed to have 80 % power against a key alternative when using a 0.05 significance level, although in execution often have less power due to unanticipated problems such as low subject recruitment. Thus, if the alternative is correct and the actual power of two studies is 80 %, the chance that the studies will both show P ≤ 0.05 will at best be only 0.80(0.80) = 64 %; furthermore, the chance that one study shows P ≤ 0.05 and the other does not (and thus will be misinterpreted as showing conflicting results) is 2(0.80)0.20 = 32 % or about 1 chance in 3. Similar calculations taking account of typical problems suggest that one could anticipate a ‘‘replication crisis’’ even if there were no publication or reporting bias, simply because current design and testing conventions treat individual study results as dichotomous outputs of ‘‘significant’’/‘‘nonsignificant’’ or ‘‘reject’’/‘‘accept.’’
rossdavidh 5y ago
I think the current situation calls for a more rigorous application of Popperian standards, not less. Nothing about "this study applies to this specific situation" is counter to Popper. However, if it fails to replicate in other situations, it tends to suggest at the very least "might be too specific to this situation to have practical application". This happens in software as well. If my system works on my specific laptop but anyone with a slightly different version of the programming language, imported libraries, or operating system will be unlikely to get it to work, then it's just not good software. The solution is not (usually) to put something in the README that says "will only work with python 3.6.7 on Ubuntu 18.04", the solution is to change the software so it is more generally applicable. And it's on me the programmer, to make that happen. If the scientific phenomenon is specific to 1990's Tokyo but not 2000s Miami, then some major variable is being left out, and the science requires finding out what that is. I agree with your basic point that there is a big difference between non-replicable and fraud, though.
- dalbasal 5y ago
  >> Nothing about "this study applies to this specific situation" is counter to Popper. However, if it fails to replicate in other situations, it tends to suggest at the very least "might be too specific to this situation to have practical application". Maybe not technically, but it does imply that "this specific situation" should be defined specifically, hence controls. I agree that broad applicability, and replicability are strong indicators of truth value. That said, there are pursuits of rational knowledge outside of Popperian ideals. That's not represented in the way Science, as an institutions works.

rwbhn 5y ago

If the results are not replicable, how can they be actionable?

ethanbond 5y ago
Think about the replication crisis as actually a generalization crisis. The study said what it said for the exact parameters given. The question is what were all the parameters and which ones are the final result actually sensitive to? It’s actionable by trying it in a new circumstance (set of parameters) and observing whether it fails - over and over. Failure doesn’t mean the claim is wrong/untrue, it means you’ve changed a parameter that happens to be important: now find it!
- jtbayly 5y ago
  Or it means there was fraud, which is why they are ultimately not able to be separated.
- DanielBMarkham 5y ago
  Yes. Find it. And when you do, you'll have something reproducible. The purpose of science is to advance the species. Given this, when that happens, this other thing(s) occur (or occur with a certain frequency) I agree that this is a generalization problem. I just don't believe that this particular way of describing it is helpful. If you'll allow me an apocryphal example, when Newton saw the apple fall, if he had generalized to "all things from apple trees fall at the same rate", he would have been correct for a while, then ludicrously incorrect once the leaves started blowing off in Autumn. If at that point he had tried to figure out exactly what kind of tree produces this gravity thing, or which parts of trees, we wouldn't be any further along than we were before. Generalizations work along multiple axes. With some degree of imprecision, every experiment can be a success (or failure). This is why we have the scientific method in the first place. Now, perhaps you're making the argument that this is somehow, for lack of a better term, half-science. It's not right, but it's not wrong either. It's truthy. The logical follow-up would be: ok, where is the second half? In this and many other cases there is no second half. Papers were published, people received awards, wrote books, became famous, and lots of other folks used their work. So even if somehow this is better than nothing, in practice I'm not seeing it.
  - ethanbond 5y ago
    To be clear, in this case there appears to have been fraud ie actual falsification of data. That’s very different from the broader replication crisis, which is more of the shape of unknown parameters. The two get blurry in some places due to eg publication incentives pushing people’s interpretation of legitimate data into semi-fraudulence. But there are still two ends (at least) on the spectrum of non-replicable science. I think framing it in this lens is helpful because it points toward a solution: capturing more of the parameters more explicitly and varying them more deliberately. If we just say, “oh x y z studies aren’t replicable, field [x y z] is bullshit,” we are ending our inquiry prematurely. We are ending it at the falsification of Newton’s initial theory and saying that because leaves fall at a different rate, that gravity doesn’t exist.
dalbasal 5y ago
Let me continue the example... We found that students participating in the "2hr active recess program" have meaningfully better health outcomes, psychological health outcomes, and better grades. It was tested faithfully in 25 schools over 10 years. It expands to other places. Different countries. Different decades. Different people run it in different ways. The kids are different. In some cases, the kids are already so active that the extra activity is detrimental. In some places, it's really hot. Kids tend to read, or do more organized indoor activities. Basically, the world is not a laboratory. You're not predicting the trajectory of a projectile, where unknowables can be quantified as a function. It's all 3 body problems all the time. Yet, we as humans do manage to make sense of the world. We run schools, companies. We learn more about how to do those things. Formal studies are an important part of learning about how to do those things. Scientific mindset and methodologies are useful to this. It's a lot better than just philosophizing the whole thing. The type of knowledge we gain, OTOH, isn't like F=MA. It never will be. The value of the above study does relate/imply to it containing knowledge. It's a partial knowledge of a general problem. It is not complete knowledge of a particular problem, like you would get in a lab. Neither lab experiments or wild studies tend to replicate in one another's world. No matter how good the study, how many times it replicated, or how well it worked throughout the 80s and 90s... there's no guarantee that your program is working well or a guaranteed explanation for why it isn't.
- kortilla 5y ago
  > Different countries. Different decades. Different people run it in different ways. The kids are different. In some cases, the kids are already so active that the extra activity is detrimental. In some places, it's really hot. Kids tend to read, or do more organized indoor activities. Elucidating confounding variables and controlling for them is the entire fucking job of a scientist. To your example, if they publish a paper that explicitly highlights that it worked well for sedentary kids with typical diets of X, Y, and Z in region Q, then it failing to be a found in other environments isn’t a problem. It’s a problem when these clowns do a test on one specific group of 20 people and then generalize to, “doing X leads to Y in humans”. The answer is absolutely not loosening up the constraints. It’s to keep lambasting people who generalize before anything has been reproduced widely. Popper’s science is fine. Morons who don’t realize what they measured are the problem.

nyolfen 5y ago

the replication crisis came to a head a few years ago when it was revealed that two thirds of social psychology findings failed to replicate -- this is not just empiricism taken too far, this is the mass production of false knowledge due to poor training at best or incentive misalignment at worst

dalbasal 5y ago
True, and a lot of it is fraud or not far. However, a big chunk of this statistic is not actually fraud. It's just false empiricism. If you look at a lot of the actual papers, I think you'll find that a lot of them just aren't trying to do empiricism... at least not strict empiricism. They're forced into that structure by a convention. Either (1) "Not Science" becomes OK and extends to economics, some areas of ecology, and such. (2) The Scientific Method gets renamed "Method A," and the concept of scientific pursuit is more flexible, depending on what you are studying or (3) We force these poor bastards doing economics, psychology and such to pretend that they're doing a method that they're not... or publish in the prestigious "Unscientific Economics & Psychology" journal.
tpoacher 5y ago
It is not even about malice. It is a straightforward consequence of how hypothesis testing is conducted. Greenland et al have a nice section on this: > Despite its shortcomings for interpreting current data, power can be useful for designing studies and for understanding why replication of ‘‘statistical significance’’ will often fail even under ideal conditions. Studies are often designed or claimed to have 80 % power against a key alternative when using a 0.05 significance level, although in execution often have less power due to unanticipated problems such as low subject recruitment. Thus, if the alternative is correct and the actual power of two studies is 80 %, the chance that the studies will both show P ≤ 0.05 will at best be only 0.80(0.80) = 64 %; furthermore, the chance that one study shows P ≤ 0.05 and the other does not (and thus will be misinterpreted as showing conflicting results) is 2(0.80)0.20 = 32 % or about 1 chance in 3. Similar calculations taking account of typical problems suggest that one could anticipate a ‘‘replication crisis’’ even if there were no publication or reporting bias, simply because current design and testing conventions treat individual study results as dichotomous outputs of ‘‘significant’’/‘‘nonsignificant’’ or ‘‘reject’’/‘‘accept.’’

rossdavidh 5y ago

I think the current situation calls for a more rigorous application of Popperian standards, not less. Nothing about "this study applies to this specific situation" is counter to Popper. However, if it fails to replicate in other situations, it tends to suggest at the very least "might be too specific to this situation to have practical application". This happens in software as well. If my system works on my specific laptop but anyone with a slightly different version of the programming language, imported libraries, or operating system will be unlikely to get it to work, then it's just not good software. The solution is not (usually) to put something in the README that says "will only work with python 3.6.7 on Ubuntu 18.04", the solution is to change the software so it is more generally applicable. And it's on me the programmer, to make that happen. If the scientific phenomenon is specific to 1990's Tokyo but not 2000s Miami, then some major variable is being left out, and the science requires finding out what that is. I agree with your basic point that there is a big difference between non-replicable and fraud, though.

dalbasal 5y ago
>> Nothing about "this study applies to this specific situation" is counter to Popper. However, if it fails to replicate in other situations, it tends to suggest at the very least "might be too specific to this situation to have practical application". Maybe not technically, but it does imply that "this specific situation" should be defined specifically, hence controls. I agree that broad applicability, and replicability are strong indicators of truth value. That said, there are pursuits of rational knowledge outside of Popperian ideals. That's not represented in the way Science, as an institutions works.