8 ms·
About 40% of economics experiments fail replication survey (2016)
- rossdavidh 8y agoSo, their take is that 60% replication is good news. Yikes.
- lsh123 8y agoBetter than 50-50 I guess
- thaumasiotes 8y agoOK, but why 50-50? There's no reason to expect that a priori. It's also better then 55-45 and 20-80, but none of those is good.
- justtopost 8y agoI think he was implying that better than 50% implies its at least not mostly bullshit, or that we could perhaps draw usful inferrences if we are right at least most of the time. But agreed, none are particularly enthusing.
- forgingahead 8y agoI think he was being sarcastic and highlighting the absurdity of the situation.
- hannob 8y agoI hate to break it to you, but that's the given standard of science. There's pretty good evidence that a mixture of publication bias and p-hacking very likely leads to the majority of studies being false: https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo... Given that in most fields publication bias and p-hacking is standard and basically written into the way science is published (i.e. you only publish your results after you've done your study, so noone knows what you intended to study when you started) it's normal to assume that most things published are simply wrong.
- air7 8y agoWith the current Publish or Perish zeitgeist in science this number can be seen as high. Also, even if assuming legitimate significant results, it's hard to control for every possible confounding variable: Maybe part of the effect is due to some sociological parameters of of the test group, maybe it's the heat/cold, maybe it only happens on Tuesdays. Even the same stimulus can change its meaning over time (like the Williams video). Humans make for finicky test subjects. IMO a single scientific paper shouldn't be considered much more than a small indicator that an effect might exist and it will require further work. I think popular media does Science a disservice by ignoring this and headlining single papers as "fact". So given that, 60% is not that bad...
- rossdavidh 8y agoTrue, but also not that good. All of the heat/cold/only on Tuesdays stuff should have been baked into the original experiment, really. Not that they should be able to take into account everything, but 60% replication does not indicate a very high standard. I mean it's not much better than if you just took and educated guess, with no data, since you could probably do better than 50/50. The whole point of the study is that it is supposed to be better than an educated guess.
- HillaryBriss 8y agoThe study's abstract ( http://science.sciencemag.org/content/351/6280/1433 http://science.sciencemag.org/content/351/6280/1433 ) says We found a significant effect in the same direction as in the original study for 11 replications (61%); on average, the replicated effect size is 66% of the original. On average, the replicated effect size is only 66% of the original effect size. What explains this?
- thaumasiotes 8y agoAssume you measure an effect and get two values, the estimated effect size in standard deviations and a p-value representing the probability that, if the effect size were zero in reality, you would have estimated an effect size at least as large as the one you did get. In phase two, assume you publish a paper if the p-value is less than 0.05, regardless of the effect size. If p >= 0.05, you don't publish anything. We are now done with the assumptions. p-values get smaller as the effect size increases, and they get smaller as the sample size increases. There is a concept called statistical power which measures the smallest effect size it is possible for a study to detect at a given p-value, given the (fixed) sample size that that study has to work with. Larger sample sizes mean more statistical power and mean that it's possible to estimate a smaller effect size for the same p-value. Adding this all up, we can see that: - if the true effect size is small; - AND if the sample size is "small", defined relative to the true effect size; - AND if we filter studies by whether they meet a p-value threshold ("reach statistical significance"); - THEN the only studies that can be published will find effect sizes that are too large. They do not have the power to detect the true effect size; the only possible results are "no effect" and "unrealistically large effect". The quick summary is that a reduced effect size on replication is a strong indicator that the original finding was spurious. As replications continue they will trend toward the higher of the true effect size or the floor set by the available statistical power.
- overeater 8y agoThere's a simpler mathematical explanation. It's called Regression Towards the Mean: https://en.wikipedia.org/wiki/Regression_toward_the_mean https://en.wikipedia.org/wiki/Regression_toward_the_mean
- deleted 8y ago[deleted]
- archgoon 8y agoThis seems to be on par with psychology and medicine. https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis
- forapurpose 8y agoI'm not sure it's a crisis. Nobody who understands research expects every study to replicate, so what is a reasonable standard? 40% having weaker effects isn't a surprise to me, a complete amateur, for whatever that's worth. The point of research is to push the into novel territories of knowledge; like new tech at startups, that's necessarily going to include many failures. If it's a critique of the scientific method(s), I'm all for very serious work on improving it. But I think the question is, how? Does someone know something that works better that they are holding back from us? Has any method in the history of humanity produced better results (or made more sense)? Should all science stop until it's perfect? What should we do going forward? EDIT: I meant to add: Maybe social sciences are just really hard. They are a bit less deterministic than many aspects of physics, for example, where gravity is so predictable that a small perturbation thousands of light years away can tell us what's happening there.
- captain_perl 8y ago> Maybe social sciences are just really hard. More likely is that they're not actually science.
- misabon 8y agoSounds snarky, but perhaps this is true in that Economics cannot be science in the Popper sense since our experiments are usually too noisy to lead to reasonably falsify any hypothesis. Still, I don’t think we’ve come upon a better method of knowledge acquisition in this realm.
- Emma_Goldman 8y agoI think there are profound limits to what can be understood from the application of natural scientific methods to the social world, because it's an open and non-linear system in which the units - people, groups - reflect on and learn from their experience. Also, while in the natural world you can easily create operational codings of variables because they follow deductively from proven theories (e.g. temperature), you cannot do that for the social world. There is no single and authoritative concept of 'power', for instance. All that said, there is no one scientific model. Evolutionary biology has never pretended to by anything like physics. It's an open and non-linear system. It cannot make predictions, and nearly every generalisation it makes is either relatively trivial, or has a great many exceptions. For the most part, it's a historical science.
- mmwelt 8y agoThe article is dated Mar. 3, 2016. It could be informative to add a (2016) to the title.
- dang 8y agoSure. Done.
- budadre75 8y agoIn machine learning's POV, this means the models these researchers proposed underfit the real-world.
- krob 8y agoIsn't that how you know something is based in science? commonality of people given the procedures to replicate, they come out with the same results consistently? so then this would mean non-scientific results. I guess they're not scientifically sound studies.
- cjohansson 8y agoIt's better than nothing though, but it's probably wise to only used reproduced science to back up arguments based on science.
- chewz 8y agoI am still hoping that some day economy will find it's place where it belongs - next to astrology, numerology and alchemy. This is with tongue in cheek but as a mathematician (financial math) I find economic equations and theorems laughable and it's scientific method just a thin veil over ideology.
- candiodari 8y agoThis is a problem math majors have with pretty much every other science out there. You can rank them if you like, and there are parts that are better and worse (in economics you have game theory, which I'm sure you're not going to find too ideological, and you have experimental economics ... which is little better than social sciences). https://xkcd.com/435/ https://xkcd.com/435/
- Ntrails 8y agoMy experience of economics teaching is something like this: - Here is a set of assumptions which are obviously wrong - Given that set of assumptions, we can make this model, which has been shown not to hold true in reality due to the above - Repeat ad nauseum
- candiodari 8y agoTrue, but consider utilitarianism. Have you ever tried modeling the economy using a more reasonable set of assumptions ? Not so easy. But yeah I get it. Obviously the basic economic assumptions ignore: 1) government in general, but especially tax law. It's WAY too complex to model, unfortunately (and I doubt that's a coincidence), nor is it even constant. 2) it ignores that the vast majority of markets are supply- or demand- constricted. In reality demand graphs flatten off at some point. Let's take bread. There is only demand for so much bread, and lowering the prices past a point no longer increases demand at all. So you can see it exists. Supply bound is similar. We are just not going to produce more land than there is. That market is absolutely supply bound. The weird thing economics glosses over is that nearly every market is either supply constricted or demand constricted, and the common equations just don't work there. 3) information. Economics assumes perfect information, when that obviously doesn't exist. Information spreads though, so over very long time frames perhaps there's a case to be made that perfect information does exist as long as the information is old enough. The thing is, whilst I might agree that the first one is at least theoretically solvable, it's just too much work. The second and third are impossible to model. And I bet you could come up with 10 more.
- gbacon 8y agoThis is a well-known problem. Hypotheses must be continually verified anew by experience. In an experiment they can generally be subjected to a particular method of examination. Various hypotheses are linked together into a system, and everything is deduced that must logically follow from them. Then experiments are performed again and again to verify the hypotheses in question. One tests whether new experience conforms to the expectations required by the hypotheses. Two assumptions are necessary for these methods of verification: the possibility of controlling the conditions of the experiment, and the existence of experimentally discoverable constant relations whose magnitudes admit of numerical determination. If we wish to call a proposition of empirical science true (with whatever degree of certainty or probability an empirically derived proposition can have) when a change of the relevant conditions in all observed cases leads to the results we have been led to expect, then we may say that we possess the means of testing the truth of such propositions. With regard to historical experience, however, we find ourselves in an entirely different situation. Here we lack the possibility not only of performing a controlled experiment in order to observe the individual determinants of a change, but also of discovering numerical constants. We can observe and experience historical change only as the result of the combined action of a countless number of individual causes that we are unable to distinguish according to their magnitudes. We never find fixed relationships that are open to numerical calculation. The long cherished assumption that a proportional relationship, which could be expressed in an equation, exists between prices and the quantity of money has proved fallacious; and as a result the doctrine that knowledge of human action can be formulated in quantitative terms has lost its only support. See Epistemological Problems of Economics by Ludwig von Mises. https://mises.org/library/epistemological-problems-economics https://mises.org/library/epistemological-problems-economics Looking backward to past events involving conscious, acting people is history, not economics. Because human beings are not falling stones, the methods of physics are a poor fit.