9 ms·
Grad Student Who Took Down Reinhart and Rogoff Explains Why They're Wrong
- wpietri 13y agoSeems like a great argument for more transparency. This guy wasted a lot of time trying to guess what they had done. Given that publishing all the supporting materials is approximately free, perhaps journals could start requiring a git link that contains data, code, and paper drafts.
- niggler 13y ago"journals could start requiring a git link that contains data, code, and paper drafts." But when the goal is to create a result rather than report it, transparency is the enemy
- carbocation 13y ago> But when the goal is to create a result rather than report it, providing data, transparency is the enemy I strongly believe that errors are a much more pervasive problem in science and related fields than malice is.
- niggler 13y ago"science and related fields " But we are talking about economics, and republican groups like fox news eat it up. My favorite example is the 2011 chart distorting the display of the unemployment rate: http://mediamatters.org/blog/2011/12/12/today-in-dishonest-fox-news-charts/185162 http://mediamatters.org/blog/2011/12/12/today-in-dishonest-f...
- carbocation 13y agoFox News is not in the business of science. I used the term "related fields" as a euphemism for Econ and Psych because I didn't want to get drawn into a demarcation debate. Wherever the line is drawn, the dishonesty of some random Fox News chart is not related to the honesty or dishonesty of actual scientific research. The much bigger problem in science is error.
- niggler 13y ago" didn't want to get drawn into a demarcation debate." Econ and Psych really do lie in a gray area, and by avoid the demarcation debate you completely miss the relevance of the issue at hand. If policy makers weren't using this study to justify more austerity, then we probably wouldn't have such a prolonged discussion.
- jerf 13y agoDistorting the truth about the state of the economy is not something confined to one partisan side, and the need for data transparency extends beyond economics regardless.
- deleted 13y ago[deleted]
- wpietri 13y agoThere's also a bunch of stuff between error and malice. As we saw yesterday here, it's very obvious in pharmaceutical research, where the stuff getting funded and published is often heavily biased despite the best intentions of everybody (or almost everybody). I suspect there are plenty of similar issues in economics. People with money are much more likely to support researchers whose work benefits or protects people with money. E.g., I happened to read a paper from a U of Chicago prof arguing that insider trading is actually beneficial. Boy, I wonder who the big donors are there. Probably not Mother Jones Magazine.
- kevinpet 13y agoThe economic benefit of insider trading is a well established economic possibility. It's not a pure transfer to rich people.
- dragonwriter 13y ago> I strongly believe that errors are a much more pervasive problem in science and related fields than malice is. In science generally, probably. In areas tightly connected to perennial areas of sharp ideological policy divides, like macroeconomics, I'm less convinced.
- tempaccount9473 13y ago> I strongly believe that errors are a much more pervasive problem in science and related fields than malice is. The problem is that economics is not a field related to science. The vast majority of public policy economics starts with a conclusion, and then creates facts to support that conclusion. This is not science, it is religion. (insert disclamier about this being an overgeneralization)
- deleted 13y ago[deleted]
- eli 13y agoMany respectable journals already do this. From Science: All data necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science. All computer codes involved in the creation or analysis of data must also be available to any reader of Science. After publication, all reasonable requests for data and materials must be fulfilled. [...] http://www.sciencemag.org/site/feature/contribinfo/prep/gen_info.xhtml#dataavail http://www.sciencemag.org/site/feature/contribinfo/prep/gen_...
- jessriedel 13y agoPaper drafts? That's a sure fire way to make sure people prepare their papers in secret. The Presidental Records Act didn't suddenly increase the transparency of the presidency, it ensured the president doesn't use email and prompted the Bush administration to move to a secret email server: http://en.wikipedia.org/wiki/Presidential_Records_Act http://en.wikipedia.org/wiki/Presidential_Records_Act http://en.wikipedia.org/wiki/Bush_White_House_e-mail_controversy http://en.wikipedia.org/wiki/Bush_White_House_e-mail_controv... By all means, force researchers to publish all the tools necessary to reproduce their results. But you can't expect to set up surveillance in their head.
- wpietri 13y agoYou might have a point, but I think that's a terrible analogy. It could go as you suggest. Or it could be like a locker room: if everybody is naked, then nobody cares. What made me add drafts to the lists is Daniel Dennett's energetic description in Consciousness Explained of how he repeatedly circulates drafts of papers to colleagues for comment. At least in philosophy, that's an important part of the process. Having to show interim steps would make fraud much harder, and it's a zero-overhead thing if people are already backing up their work.
- jessriedel 13y agoYes, I often circulate drafts of my papers to colleagues too. I don't post them for the world to see into perpetuity until they reach a certain level of quality. Your shower analogy doesn't work because there's no way to force people to post drafts. We already have the option posting of drafts. It's called personal websites and/or the arXiv.
- wpietri 13y agoWell, the mechanism I suggested for forcing was journals requiring it for publication. People would be obliged to keep a version history of some sort. Careful writers do already, and it's easily automated, so I don't think enforcement would be hard.
- twog 13y agoAt my startup, http://banyan.co http://banyan.co, we are aiming to tackle transparency in academia using git. Our products barely a few months old, but were shipping new features & improvements daily.
- freyr 13y ago> This guy wasted a lot of time trying to guess what they had done. According to the article, many people wasted a lot of time attempting to recreate their results. Further, the paper is highly-cited and it probably shaped, directly or indirectly, opinions, further research directions, and possibly even policy. We are focusing on the Excel error, which was likely a mistake. I'm really struggling though to justify their choice of weighting/averaging. It's confounding that two highly regarded and experienced academics would choose something so particularly bad. It absolutely should have been noted in the text.
- codeulike 13y agoIts fascinating that part of the problem in the Reinhart and Rogoff paper comes down to an Excel formula error http://nymag.com/daily/intelligencer/2013/04/grad-student-who-shook-global-austerity-movement.html http://nymag.com/daily/intelligencer/2013/04/grad-student-wh...
- niggler 13y agoHN discussion: https://news.ycombinator.com/item?id=5575631 https://news.ycombinator.com/item?id=5575631
- ScottBurson 13y agoOne columnist I read said something like "it will be interesting to see what historians make of the fact that a global economic policy was driven in part by an error in an Excel spreadsheet".
- pseut 13y agoIt's rare that we get such a clean natural experiment like this in economics; maybe future economists will thank R&R for cleverly introducing this negative exogenous policy shock (sarcasm).
- danielweber 13y agoAusterity was a big thing in Europe, and they mostly came to that decision before R+R's paper came out in 2010.
- jgrahamc 13y agoI have argued in the past that data and code need to be submitted with academic papers: http://blog.jgc.org/2013/04/the-importance-of-open-code.html http://blog.jgc.org/2013/04/the-importance-of-open-code.html This is just the latest example of why, IMHO, code needs to be open.
- carbocation 13y agoMore specifically for people not following jgrahamc's full story, he and co-authors wrote a paper making the argument that data and code need to be submitted with academic papers, and the pre-eminent scientific journal, Nature, saw fit to publish their paper. So the idea is getting attention; hopefully it will also gain traction, too.
- jgrahamc 13y agoInformation about the Nature paper: http://blog.jgc.org/2012/02/case-for-open-computer-programs.html http://blog.jgc.org/2012/02/case-for-open-computer-programs.... My co-authors and I chose Nature as the target for the paper because of its reputation and because I believe in going to 11 (http://en.wikipedia.org/wiki/Up_to_eleven http://en.wikipedia.org/wiki/Up_to_eleven).
- latj 13y agoI didnt realize this was something that was still up for debate. In my experience source code has been a requirement since I entered the game- especially for high impact journals.
- carbocation 13y agoAs late as at least 2010, source code wasn't required for Nature or NEJM. People could request it from you, but it wasn't required to actually be submitted up front.
- jurassic 13y agoYes! We need to see more material under Matt Might's CRAPL license. I feel a strong urge to write off any result that doesn't have accompanying code with it. http://matt.might.net/articles/crapl/ http://matt.might.net/articles/crapl/
- tokenadult 13y agoFrom Jelte Wicherts writing in Frontiers of Computational Neuroscience (an open-access journal) comes a set of general suggestions Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389/fncom.2012.00020 http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00020/full http://www.frontiersin.org/Computational_Neuroscience/10.338... on how to make the peer-review process in scientific publishing more reliable. Wicherts does a lot of research on this issue to try to reduce the number of dubious publications in his main discipline, the psychology of human intelligence. It appears that the discipline of economics research needs help with data openness too. "With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed peer review entails a community-wide review system in which reviews are published online and rated by peers. This ensures accountability of reviewers, thereby increasing academic quality of reviews. Second, reviewers who write many highly regarded reviews may move to higher editorial positions. Third, online publication of data ensures the possibility of independent verification of inferential claims in published papers. This counters statistical errors and overly positive reporting of statistical results. We illustrate the benefits of these strategies by discussing an example in which the classical publication system has gone awry, namely controversial IQ research. We argue that this case would have likely been avoided using more transparent publication practices. We argue that the proposed system leads to better reviews, meritocratic editorial hierarchies, and a higher degree of replicability of statistical analyses."
- MaysonL 13y agoNote that the Reinhart-Rogoff article was not peer-reviewed.
- harshpotatoes 13y agoErSo, there are a few things I don't quite understand. In the original study, Renhart had erroneously averaged 7 numbers together to get a slightly negative growth. The 'correct' result is obtained from averaging eight numbers together, resulting in slightly positive growth. What is unclear to me, is how are either of these numbers considered to be very significant? With so few samples in the average, I would draw the conclusion that both measurements agree with one another and the error bars for their measured result are quite a bit larger than they seem to be suggesting. Is this common in economics?
- jessriedel 13y agoMacroecon has the simultaneous hindrances of (1) not being able to run a controlled experiment and (2) a dearth of data. It doesn't excuse people for being far too confident in the conclusions they draw, but it does explain why such large differences of opinions can persist (especially on such a politically charged subject). People can't be pulled away from their priors.
- dragonwriter 13y ago> Macroecon has the simultaneous hindrances of (1) not being able to run a controlled experiment and (2) a dearth of data. Well, except #1 should be "not being able to run experiments in laboratory conditions". Statistical controls in an experiment are still controls, and it is still possible though (because of the point made briefly by #2, to run statistically-controlled experiments) sometimes of limited utility; the dearth of data (specifically, the small number of data points compared to the number of independent variables that need to be controlled for) is what limits the utility of many experiments with statistical controls in the field.
- jessriedel 13y agoAgreed.
- pseut 13y agoPolitical constraints and welfare concerns effectively rule out "controlled experiment" at the present time.
- davesims 13y agoFor me this was the 'tl;dr' quote: "It would be absurd to think that governments never have to worry about their level of indebtedness. The aim of our paper was much more narrowly focused. We show that, contrary to R&R, there is no definitive threshold for the public debt/GDP ratio, beyond which countries will invariably suffer a major decline in GDP growth."
- SilasX 13y agoR&R weren't trying to claim that, were they?
- bhb916 13y agoI think they were. Speaking as someone who is generally pro-austerity, I think R&R concluded that after 90% debt/GDP there was major drop-off in GDP growth. The corrections all but destroy this assertion. That's not to say that debt and GDP growth aren't inversely proportional/correlated -- just that there is no magic level that changes the relationship substantially. Honestly, there are so many variables and dimensions in this system and so little data I'm not sure we should ever be drawing sweeping conclusions like this.
- SilasX 13y ago>I think they were. Speaking as someone who is generally pro-austerity, I think R&R concluded that after 90% debt/GDP there was major drop-off in GDP growth. That doesn't mean they were asserting what the grad students claim, that the falloff in growth is unavoidable, which sounds like an extreme strawman.
- yyqux 13y agoR&R's defense strategy seems currently to pretend that they weren't claiming anything in particular by the paper originally: "hey man, I was just putting it out there. I didn't, like, mean anything by it." But their own public pronouncements, and the way they responded to other's interpretations of the results, pretty much indicated that they though they'd found firm evidence that there was some big non-linearity at 90%.
- beefman 13y agoMore debate over 20 data points with no explanatory power one way or the other (as Yglesias pointed out). This is not uncommon in macro. Another recent case also centered on work by R&R http://johnbtaylorsblog.blogspot.com/2012/10/simple-proof-that-strong-growth-has.html http://johnbtaylorsblog.blogspot.com/2012/10/simple-proof-th... (one of a host of commentary pieces on both sides... but essentially none in the middle)
- driverdan 13y agoThis was an interesting and well written followup but I'm disappointed in Herndon's decision to publish it on Business Insider. BI is generally full of blogspam and has interstitial ads.
- niggler 13y agoNonzero chance Herndon gets a percentage of the ad revenue.
- pseut 13y agoTrue, but BI is actually really good for econ news.
- cjensen 13y agoAgreed. BI is edited by Henry Blodget who was charged with securities fraud as an analyst which lead to him paying millions in settlement and being permanently banned from the securities industry. Really not where you want to publish anything meant to be taken seriously. http://en.wikipedia.org/wiki/Henry_Blodget http://en.wikipedia.org/wiki/Henry_Blodget
- ttrreeww 13y agoThat grad student won't be getting the big job offers in industry, great way to destroy your career by biting the hand that feeds you. (Sarcastic, but it's real life)
- nazgulnarsil 13y agoDoesn't matter, he'll now have a lucrative career being a USG debt apologist.
- 205guy 13y agoI think this method of averaging should hereby be called the Reinhart-Rogoff Average, RRA for short. I should program it into my business intelligence software, it's gotta be useful for something--like for cooking the books.
- zaroth 13y agoIt's not related to the article, but rather the site hosting it. I noticed more than one fishy looking 3rd party domain loading while the page downloaded, so went into Inspector to see what was up. There are one or more resources loaded from each of the following domains, in many cases including javascript... 2mdn.net, scorecardresearch.com, bizographics.com, tynt.com, optimizely.com, google.com, sail-horizon.com, facebook.com, 247realmedia.com,akamaihd.net, vizu.com, pubmatic.com, imrworldwide.com, advertising.com, googlesyndication.com, doubleclick.net, chartbeat.com, sharethrough.com, fbcdn.net, skimresources.com, gstatic.com, stumbleupon.com, tynt.com, adadvisor.net, youtube.com, shareth.ru, agkn.com, yimg.com 'shareth.ru' seemed particularly suspect, until I realized it was probably sharethrough.com trying to be cute. There are so many domains being trusted here the drive-bys could have drive-bys.
- justin_vanw 13y agoJournals shouldn't publish papers without all the supporting data and computer code necessary to reproduce the result. This seems obvious, but established academics have a vested interest in not sharing. Sharing data and code opens their work up to impeachment (as is seen here), and it gives others a jumping off point to extend their work. Both of these side effects of sharing code and data are bad for the careers of successful academics, but good for literally everyone else in the world (including, ironically, these academics). It should be a no brainer, but then again, who referees these papers? The very people who have the most to lose by such a change. It will take a lot of clamoring from the outside to bring about a world where scientific work is considered not legitimate without full documentation of the experiment performed.
- YokoZar 13y agoI recall hearing somewhere that the original paper wasn't actually published in a peer-reviewed journal, so requirements for data and code wouldn't really have applied anyway. Perhaps the greater lesson is that we should trust proper journals for a reason.
- ikhare 13y agoThis whole situation makes me think: is there a static analysis tool for excel? Seems like missing a few rows in an excel calculation could be flagable.
- tonyarkles 13y agoThere sort of is one built-in. It usually warns you if you're doing things on a group of cells and you're missing one of them.