5 ms·

Think about the replication crisis as actually a generalization crisis. The study said what it said for the exact parameters given. The question is what were al

Think about the replication crisis as actually a generalization crisis. The study said what it said for the exact parameters given. The question is what were all the parameters and which ones are the final result actually sensitive to? It’s actionable by trying it in a new circumstance (set of parameters) and observing whether it fails - over and over. Failure doesn’t mean the claim is wrong/untrue, it means you’ve changed a parameter that happens to be important: now find it!

jtbayly 5y ago
Or it means there was fraud, which is why they are ultimately not able to be separated.
DanielBMarkham 5y ago
Yes. Find it. And when you do, you'll have something reproducible. The purpose of science is to advance the species. Given this, when that happens, this other thing(s) occur (or occur with a certain frequency) I agree that this is a generalization problem. I just don't believe that this particular way of describing it is helpful. If you'll allow me an apocryphal example, when Newton saw the apple fall, if he had generalized to "all things from apple trees fall at the same rate", he would have been correct for a while, then ludicrously incorrect once the leaves started blowing off in Autumn. If at that point he had tried to figure out exactly what kind of tree produces this gravity thing, or which parts of trees, we wouldn't be any further along than we were before. Generalizations work along multiple axes. With some degree of imprecision, every experiment can be a success (or failure). This is why we have the scientific method in the first place. Now, perhaps you're making the argument that this is somehow, for lack of a better term, half-science. It's not right, but it's not wrong either. It's truthy. The logical follow-up would be: ok, where is the second half? In this and many other cases there is no second half. Papers were published, people received awards, wrote books, became famous, and lots of other folks used their work. So even if somehow this is better than nothing, in practice I'm not seeing it.
- ethanbond 5y ago
  To be clear, in this case there appears to have been fraud ie actual falsification of data. That’s very different from the broader replication crisis, which is more of the shape of unknown parameters. The two get blurry in some places due to eg publication incentives pushing people’s interpretation of legitimate data into semi-fraudulence. But there are still two ends (at least) on the spectrum of non-replicable science. I think framing it in this lens is helpful because it points toward a solution: capturing more of the parameters more explicitly and varying them more deliberately. If we just say, “oh x y z studies aren’t replicable, field [x y z] is bullshit,” we are ending our inquiry prematurely. We are ending it at the falsification of Newton’s initial theory and saying that because leaves fall at a different rate, that gravity doesn’t exist.
dalbasal 5y ago
I disagree. You're not going to find the parameter which makes a high school nutrition program deterministic. It shouldn't be the goal.
- beagle3 5y ago
  That’s a strawman. Reproducibility does not require determinism. If you can’t find the parameters that make it reproducible, what value does it have in the first place?
- goatlover 5y ago
  Which means you can’t generalize to other high schools. If the scientific result doesn’t generalize, is it still science? How can we use the science to improve HS nutrition?
jfengel 5y ago
Generalizing further, it touches on the Demarcation Problem. What is the difference between scientific hypothesis revision and pseudoscientific goalpost moving? Surely there is one, but it's maddeningly hard to pin down. Especially when the experiments are hard to perform and control properly, which applies both to psychology and astronomy.
- dalbasal 5y ago
  This kind of demarcation usually hints that you are dealing with a subject that is not as concrete as you had supposed. "Science" doesn't really have a concrete definition, a pseudoscience just means stuff that is pretending to be science. Remember that scientists existed before philosophy of science, including formalisations of The Scientific Method. Darwin wasn't following the scientific method, the method followed him.
moonchrome 5y ago
> Think about the replication crisis as actually a generalization crisis. The study said what it said for the exact parameters given. The question is what were all the parameters and which ones are the final result actually sensitive to? If you can't control parameters so that the results you are claiming are reproducible you are just publishing noise ?
- epistasis 5y ago
  How would you know if parameters are controlled until other people try the same thing with other parameters? Should a journal paper never be published until the field has established all the fundamental properties? How could it discover those fundamental properties without the publications?

jtbayly 5y ago

Or it means there was fraud, which is why they are ultimately not able to be separated.

DanielBMarkham 5y ago

Yes. Find it. And when you do, you'll have something reproducible. The purpose of science is to advance the species. Given this, when that happens, this other thing(s) occur (or occur with a certain frequency) I agree that this is a generalization problem. I just don't believe that this particular way of describing it is helpful. If you'll allow me an apocryphal example, when Newton saw the apple fall, if he had generalized to "all things from apple trees fall at the same rate", he would have been correct for a while, then ludicrously incorrect once the leaves started blowing off in Autumn. If at that point he had tried to figure out exactly what kind of tree produces this gravity thing, or which parts of trees, we wouldn't be any further along than we were before. Generalizations work along multiple axes. With some degree of imprecision, every experiment can be a success (or failure). This is why we have the scientific method in the first place. Now, perhaps you're making the argument that this is somehow, for lack of a better term, half-science. It's not right, but it's not wrong either. It's truthy. The logical follow-up would be: ok, where is the second half? In this and many other cases there is no second half. Papers were published, people received awards, wrote books, became famous, and lots of other folks used their work. So even if somehow this is better than nothing, in practice I'm not seeing it.

ethanbond 5y ago
To be clear, in this case there appears to have been fraud ie actual falsification of data. That’s very different from the broader replication crisis, which is more of the shape of unknown parameters. The two get blurry in some places due to eg publication incentives pushing people’s interpretation of legitimate data into semi-fraudulence. But there are still two ends (at least) on the spectrum of non-replicable science. I think framing it in this lens is helpful because it points toward a solution: capturing more of the parameters more explicitly and varying them more deliberately. If we just say, “oh x y z studies aren’t replicable, field [x y z] is bullshit,” we are ending our inquiry prematurely. We are ending it at the falsification of Newton’s initial theory and saying that because leaves fall at a different rate, that gravity doesn’t exist.

dalbasal 5y ago

I disagree. You're not going to find the parameter which makes a high school nutrition program deterministic. It shouldn't be the goal.

beagle3 5y ago
That’s a strawman. Reproducibility does not require determinism. If you can’t find the parameters that make it reproducible, what value does it have in the first place?
goatlover 5y ago
Which means you can’t generalize to other high schools. If the scientific result doesn’t generalize, is it still science? How can we use the science to improve HS nutrition?

jfengel 5y ago

Generalizing further, it touches on the Demarcation Problem. What is the difference between scientific hypothesis revision and pseudoscientific goalpost moving? Surely there is one, but it's maddeningly hard to pin down. Especially when the experiments are hard to perform and control properly, which applies both to psychology and astronomy.

dalbasal 5y ago
This kind of demarcation usually hints that you are dealing with a subject that is not as concrete as you had supposed. "Science" doesn't really have a concrete definition, a pseudoscience just means stuff that is pretending to be science. Remember that scientists existed before philosophy of science, including formalisations of The Scientific Method. Darwin wasn't following the scientific method, the method followed him.

moonchrome 5y ago

> Think about the replication crisis as actually a generalization crisis. The study said what it said for the exact parameters given. The question is what were all the parameters and which ones are the final result actually sensitive to? If you can't control parameters so that the results you are claiming are reproducible you are just publishing noise ?

epistasis 5y ago
How would you know if parameters are controlled until other people try the same thing with other parameters? Should a journal paper never be published until the field has established all the fundamental properties? How could it discover those fundamental properties without the publications?