3 ms·

“A recent study by economists at the Federal Reserve found that less than half of the published papers they examined could be replicated, even when given help f

by oglop 3y ago

“A recent study by economists at the Federal Reserve found that less than half of the published papers they examined could be replicated, even when given help from the original authors. Mr Levitt’s counterintuitive results have fallen out of fashion and economists in general have become more sceptical.” Didn’t realize replication was so bad in Econ too.

jdougan 3y ago
Depends on which part of econ. Micro and econometrics fare pretty well (there was even an econ pseudo-Nobel awarded to Vernon Smith for figuring out how to do reproducible experiments in micro), but macro is the darkest of swamps.
immibis 3y ago
If the purpose of half of economics study was to justify political decisions, rather than to learn facts about the world, that replication rate would be what you'd expect.
- FridgeSeal 3y ago
  > If the purpose of half of economics study was to justify political decisions I mean, let’s be honest, is it _not_ actually this? Maybe economics should just give up the charade and accept its role as a political tool and not something with a scientific base so comically bad, even the softest of other sciences look positively mathematical in comparison.
  - anamax 3y ago
    There are lots of good reasons why you can't admit that it's a con, most of which are some variation on "it's good to lie."
  - Alpha3031 3y ago
    Which specific social sciences do you believe are doing better?
    - 7thaccount 3y ago
      None of them I would guess. Math is math and the hard sciences like physics, chemistry, biology,and engineering are all good assuming somebody isn't mining p-values or something like that. The social sciences require a ton of data and simulating something like the economy with trillions of variables and bounded rationality just isn't possible. If you wanted to simulate an agent like myself you'd likely need millions of lines of probabilistic code and then you'd need millions of other agents running on a supercomputer the size of Rhode Island.
      - Alpha3031 3y ago
        Biomed is already too complex to simulate fully, which is why simulating is just, not really a thing that is done to find new knowledge. Heck, some quantum interactions are already (so far) computationally intractable, which is why quantum chemistry exists as a field at all, it's far from trivial. That doesn't mean the best practices developed for medicine won't help in the social sciences as well. The issue with degrees of freedom can be mostly pinned down with preregistration, among other measures. It won't solve the problem of having more type I errors at any specific error rate the more science you do, but it does give assurances the target type I error rate is accurate.
  - immibis 3y ago
    If you want to use propaganda but you tell people your propaganda is propaganda, it won't work.
tomrod 3y ago
It's bad. But the field has moved towards causal inference methods anyhow, and away from the "cutenomics" Freakonomics championed.
deleted 3y ago
[deleted]
nothercastle 3y ago
I mean half went be bad but like 10-20% would be the expected if it matched the other sciences
P_I_Staker 3y ago
Econ is a notorious "nothing and everything topic" (not being fair here, but economists also talk about this). It doesn't really fit anywhere. It's not very practical to study scientifically. I don't want to insult Sociologists here, but economics can be very much like sociology for money issues. It's intuitive that replication would be terrible in economics (not taking a shot at you for not "realizing"; we don't think about every subject in the world). What you really want to do is take one society, run it for 5-200 years. Then go back in time, take the same society, and try again with one small change. In practice, you might need to repeat many times because you would expect chance to change such a complicated system (butterfly effect). "In practice"... listen to me, I'm being absurd. This is clearly impossible. Of course there's ways to try anyway, but they're all known to be "bad science".

by oglop 3y ago

jdougan 3y ago

Depends on which part of econ. Micro and econometrics fare pretty well (there was even an econ pseudo-Nobel awarded to Vernon Smith for figuring out how to do reproducible experiments in micro), but macro is the darkest of swamps.

immibis 3y ago

If the purpose of half of economics study was to justify political decisions, rather than to learn facts about the world, that replication rate would be what you'd expect.

FridgeSeal 3y ago
> If the purpose of half of economics study was to justify political decisions I mean, let’s be honest, is it _not_ actually this? Maybe economics should just give up the charade and accept its role as a political tool and not something with a scientific base so comically bad, even the softest of other sciences look positively mathematical in comparison.
- anamax 3y ago
  There are lots of good reasons why you can't admit that it's a con, most of which are some variation on "it's good to lie."
- Alpha3031 3y ago
  Which specific social sciences do you believe are doing better?
  - 7thaccount 3y ago
    None of them I would guess. Math is math and the hard sciences like physics, chemistry, biology,and engineering are all good assuming somebody isn't mining p-values or something like that. The social sciences require a ton of data and simulating something like the economy with trillions of variables and bounded rationality just isn't possible. If you wanted to simulate an agent like myself you'd likely need millions of lines of probabilistic code and then you'd need millions of other agents running on a supercomputer the size of Rhode Island.
    - Alpha3031 3y ago
      Biomed is already too complex to simulate fully, which is why simulating is just, not really a thing that is done to find new knowledge. Heck, some quantum interactions are already (so far) computationally intractable, which is why quantum chemistry exists as a field at all, it's far from trivial. That doesn't mean the best practices developed for medicine won't help in the social sciences as well. The issue with degrees of freedom can be mostly pinned down with preregistration, among other measures. It won't solve the problem of having more type I errors at any specific error rate the more science you do, but it does give assurances the target type I error rate is accurate.
- immibis 3y ago
  If you want to use propaganda but you tell people your propaganda is propaganda, it won't work.

tomrod 3y ago

It's bad. But the field has moved towards causal inference methods anyhow, and away from the "cutenomics" Freakonomics championed.

deleted 3y ago

[deleted]

nothercastle 3y ago

I mean half went be bad but like 10-20% would be the expected if it matched the other sciences

P_I_Staker 3y ago

Econ is a notorious "nothing and everything topic" (not being fair here, but economists also talk about this). It doesn't really fit anywhere. It's not very practical to study scientifically. I don't want to insult Sociologists here, but economics can be very much like sociology for money issues. It's intuitive that replication would be terrible in economics (not taking a shot at you for not "realizing"; we don't think about every subject in the world). What you really want to do is take one society, run it for 5-200 years. Then go back in time, take the same society, and try again with one small change. In practice, you might need to repeat many times because you would expect chance to change such a complicated system (butterfly effect). "In practice"... listen to me, I'm being absurd. This is clearly impossible. Of course there's ways to try anyway, but they're all known to be "bad science".