5 ms·
Worth pointing out is that, while the studies are almost certainly wrong, they develop into less wrong that builds on the old more wrong. Take the Bohr atom, h
by danteembermage 19y ago
Worth pointing out is that, while the studies are almost certainly wrong, they develop into less wrong that builds on the old more wrong.
Take the Bohr atom, horribly wrong, but immensely valuable in it's time (and still pedagogically). Then you add sub-orbitals, orbital hybridization theory, and the quantum mechanical model, all almost certainly wrong.
Even in the social sciences you have results like "if we make charitable gifts tax deductible, revenue will go down, but gifts will go up more than the lost revenue" later studies came along using people as their own control by studying their behavior through time instead of just over one year, and the opposite is true, tax revenue goes down more than giving goes up. So the original study was "wrong" but to get the right answer you have to start with the simple and build up.
There are certainly areas that are so entangled with inter-relations and data difficulties that make empirical studies worthless until someone comes up with decent models to untangle it all, but I think those are the exception rather than the rule.
- mechanical_fish 19y agoI think I agree with your main point, but it's important to distinguish between various kinds of "wrongness". The Bohr atom is "wrong" in the way that a cartoon is wrong. Newtonian mechanics (and non-relativistic quantum mechanics, for that matter) are "wrong" because they are approximations: at certain length scales or time scales they work perfectly, but at other scales they break down and need to be replaced by something more precise. The kind of "wrong" that this article is discussing is much different: biased experimental design and lousy statistics. Unlike the other two kinds of "wrong", this is actually a problem. I've seen thousands of hours wasted - some of them by me - due to lack of knowledge of the principles of statistical significance, or because someone found a way to fool himself into cherry-picking the most interesting data points. Public service announcement: If you're still in school as you read this, take a stats course!
- apathy 19y ago> If you're still in school as you read this, take a stats course! If you're not, you can consult with a decent statistician at most institutions. Might save your career, in fact...
- npk 19y agoYou raise a great point, but consider a different viewpoint. epidemiological studies are based on complex systems, systems with many variables and non-linearities. Atoms are a lot simpler (ha!) Here's what I think is going on: a hot paper comes out, presenting an interesting theory. People either like or don't like the theory based on preconceived notions, look at their data, plot every permutation of Xi versus Xj and publish the "juiciest plot." Ok, it sounds like I'm saying that the "scientific method" has broken down. I'm not, the preconceived notion is subtle, difficult, and hard to eradicate! So, people aren't starting with simple and building up. Not because they don't want to, but because they can't.