5 ms·
This is a nice summary of fivethirtyeight's science crisis work, but (as with most treatments of the subject) it leaves me fundamentally unsatisfied. Apparentl
by pc2g4d 9y ago
This is a nice summary of fivethirtyeight's science crisis work, but (as with most treatments of the subject) it leaves me fundamentally unsatisfied.
Apparently we've identified the problem enough to say "there's a problem". We can say "look at all the ways you can manipulate this analysis to achieve the desired outcome". We can say "gosh, science is harder than we thought". But it seems we're still far from a convincing solution.
The fact that statistical analysis is so liable to manipulation seems to call the entire thing into question. In the article they take comfort from the fact that many of the labs analyzing the red card/race data arrived at similar conclusions. One would assume this is because they made similar choices in the analysis. But what says those were the right sorts of choices? Could it not simply be that the labs shared the same biases and errors, making that outcome more common? Is a proper analysis really determined by (essentially) democratic vote? If that's what we've arrived at, it gives me less rather than more confidence in the robustness of the scientific process.
It feels like something fundamental has to be reimagined. It's difficult to prove things about the world---but maybe it's actually near-impossible? Or maybe we need to get real about the cost of actually demonstrating anything reliably. Instead of individual labs running one-off experiments it becomes researchers collaborating openly to build the perfect experiments, which are then run by many different labs, then analyzed collectively in the open for strengths and weaknesses, then reformulated, sent out again, and so on iteratively until at the end of years of research one little bit of probably-truth drips out the bottom of the system.
But that bit would be something we could build on.
- tpetricek 9y agoMaybe the thing that needs to be reimagined is the naive image of science as an infallible way of always getting true answers. Science works, just not in as simple way as one might naively think. Feyerabend's "Against Method" is my favourite place to start :-).
- averagewall 9y agoI like your idea about collaborating on bigger experiments. One big experiment can be both more convincing and more accessible than 1000 little ones all measuring different incomparable things that take 1000s of man-hours just to identify if they're what you want or not. The trouble is everyone wants to be the special snowflake discovering the special effect in their own special experiment, so now we have too much messy unreliable and inaccessible data to be of much use in the real world.
- sungx105 9y agoAnalysis being determined by a(n essentially) democratic vote is actually just replication of results. While the system is highly flawed now, I would argue that collaborating to build perfect experiments carried out by multiple labs would not be a good idea. You point out if people made the same sorts of choices then perhaps everyone just has the same biases and errors. But if all the people on one experiment collaborate, then they'll definitely all have the same biases. Instead, independent verification in which the nitty-gritty details are abstracted away for a more general methods section would allow for more alternate takes at approaching a problem.
- analog31 9y agoIt's difficult to prove things about the world---but maybe it's actually near-impossible? Scientists always hesitate to talk about "proof," since we hold our beliefs on a tentative basis. However, some scientific theories have been corroborated to such a degree that they can be regarded as proven for all practical purposes, within broad domains. These include classical mechanics, relativity, gravitation, quantum mechanics, thermodynamics, electromagnetism, and Darwinian evolution. This level of "proof" is trusted on a daily basis in regular lab work, e.g., if an electrical circuit violates Ohm's Law, you don't throw out Ohm's Law. You look for a bad solder joint. Granted, science is presently dominated by the life sciences. I'm sure the physical sciences have their own problems -- we shouldn't be overconfident -- but they may be qualitatively different problems. In my view, a journalist shouldn't generalize about "science" without showing that they've considered whether the generalization is justified. Disclosure: My background is in physics, and my day job is in scientific instrumentation. It's a business where people are obsessive about replication. When an instrument goes out the door, it's been put through a stringent set of tests that could be described as replications, and the results of these tests are compared across different designs, models, batches of components, and even competing brands, all around the world.