12 ms·
Undisclosed tinkering in Excel behind economics paper
- grey-area 3y agoThe title is a bit odd. The article details a sloppy use of substitute data from adjacent rows or even columns to fill gaps in data, excel has little to do with it. The worst example is using United Kingdom data to fill gaps in USA data because they were adjacent in the list.
- smt88 3y agoExcel has everything to do with it. From the article: > In email correspondence seen by Retraction Watch and a follow-up Zoom call, Heshmati told the student he had used Excel’s autofill function to mend the data. The most charitable interpretation is that the professor has no idea what autofill is or isn't capable of doing, so he misused it.
- grey-area 3y agoThat’s not even the worst case though, worse by far than interpolating (whatever method is used) is simply taking data from countries with a similar name to fill gaps.
- InSteady 3y ago"What was your rational for using data from the Netherlands to fill in the missing values in New Zealand?" "Um, both countries start with the letter N"
- jakeinspace 3y agoShould’ve used Denmark rather than Netherlands, as it’s the Old Zealand.
- kgwgk 3y agoDo _you_ have an idea of what autofill is and what is or isn’t capable of doing? When he dragged down a selection he _saw_ the values being copied or a trend being extrapolated.
- LeifCarrotson 3y agoFlash Fill and Series Fill are great at recognizing simple patterns like 1,2,3,4,5 or 2,4,6,8,10 and can sometimes (unreliably!) recognize more complicated multiplications, repetitions or formulaic string modifications (like house numbers and street names). I use both frequently. But neither these tools nor an LLM has real understanding, they're not going out into the real world and collecting data. The results need to be verified. Automatically filling an index column with 1,2,3... is one thing, automatically filling a data column with guesses from pattern matching is different. This problem is only going to get worse as LLMs proliferate and can more reliably but still imperfectly and still opaquely fill in more complicated data.
- smt88 3y agoI don't use autofill precisely because I have no insight into the algorithm behind it. I suspect it's very "stupid," but I don't know exactly how stupid it is. It sounds like he didn't realize or notice that autofill was using adjacent cells (from other countries, in this instance). That behavior would shock me as well. I thought autofill only took columns of data into account.
- kgwgk 3y ago> It sounds like he didn't realize or notice that autofill was using adjacent cells (from other countries, in this instance). No, it doesn’t sound like that: “in several instances where there were no observations to use for the autofill operation, the professor had taken the values from an adjacent country in the spreadsheet.” It was not Excel’s fault.
- secondcoming 3y agoRight, but the mistake wasn't an issue with Excel, it was with the professor misusing it.
- kurthr 3y agoMight as well blame ChatGPT for writing your term paper. If they'd been using R/Matlab/Python written by a copilot it would have been just as much gibberish.
- smt88 3y agoYes, I understand that and agree. The headline isn't blaming Excel. "Undisclosed tinkering in Excel" is not far from the accurate headline: "Misuse of commonly-used Excel tool..." The interesting thing is that a supposedly sophisticated person was using an unsophisticated tool and then relying on the data it produced. Failing to mention "Excel" in this case is doing a disservice to the thrust of the problem.
- grey-area 3y agoThe accurate headline would be incomplete data falsified. The tools have nothing to do with it, and the same could have been done with paper and pencil. Autocomplete is the problem, it just repeats a single number or a series, and when using it you see exactly what the results were. This was not done by mistake or in ignorance or because the tool encouraged them, it was done to knowingly falsify incomplete data so that they could finish their paper and publish.
- scotty79 3y agoAlso Netherlands to New Zealand. Hilarious.
- riedel 3y agoSuch things are encouraged by the excel UI because it fails to make relationships and manual corrections easily visible. The human computer interface matters beyond UX.
- martynr 3y agoSurely this can’t be justified as a ux driven “oversight” - I am not an economics professor but I would be feeling totally sketchy if I manipulated a data source in such an apparently arbitrary manner. In combination with the (reportedly)blatant lack of any reference to the methodology in his paper this feels like wilful gross misconduct in professor-land. IMO this sort of thing rightly undermines confidence and credibility of research in a much broader context and should be a sackable matter if shown to be true as asserted.
- csydas 3y agoI get what you're saying, but I would disagree that the focus should be on Excel and its UI/UX. As annoying as Excel is when you're working with _specific_ data that suffers from Excel's attempts to be helpful, I'd wager for most use cases Excel is doing exactly what most users expect it to, or the change is benign at best for most use cases. That isn't to diminish when Excel's help has disastrous consequences, but rather to say it shouldn't be unexpected. If your data requires precision, there are better tools than Excel for working with your data and there should be steps taken by more experienced members of the team to ensure the team doesn't get bit by Excel's features. Excel's behaviors are well known and I'd have to imagine at least one member of any research team has been bitten by Excel's assistance features before during research. I would be surprised if any serious educational institution that teaches about ethics and common pitfalls in research *didn't* caution about how Excel might corrupt research [0]. 0 - Okay, maybe not _that_ surprised, but I would be disappointed and think less of the institution/instructor.
- croes 3y agoExcel makes such things convenient
- bjornsing 3y agoA related anecdote: Jönköping University is not allowed to call itself a university in Swedish (“universitet”), since that word is, by law, reserved for institutions that meet certain requirements. To get around this it uses the English name “Jönköping University” in Swedish as well. But formally it’s a so called “högskola”, more often translated as ”college”.
- zdimension 3y agoIn France, "Licence" and "Master" (Bachelor's and Master's degree respectively) are protected diplomas that can only be delivered by universities (or by schools that depend on universities). To circumvent this, private non-state-certified schools deliver things called "Bachelor", "MSc", "MBA" or "Mastère" that walk and quack like regular diplomas but since they're using those specific names, they're not actually state-recognized diplomas. Which means that if a company is hiring "people with a Licence" they can reasonably deny someone with a "Bachelor" because even though it's also a 3-year degree, there is no guarantee whatsoever that it's actually worth anything. Anybody can create a company and hand out sheets of paper with "MSc in Psychoceramics" written on it. This is a big problem for high schoolers who usually don't know the difference and end up paying thousands of euros for private schools that deliver close-to-worthless degrees (not all are worthless, but their worth is mostly correlated to how well-known the school is, whereas if you get a real state-certified degree from a random university anywhere in France it'll be recognized by any company).
- wodenokoto 3y agoWouldn’t that also make the professor _not_ a professor?
- dash2 3y ago"Journal of Cleaner Production" is not "highly ranked" in economics. Economists are pretty hierarchical about their journals and know which are the top 5, which are top field journals etc. That one is a non-entity, I have literally never heard of it.
- nsajko 3y ago"Highly ranked" probably refers to a formal metric like the impact factor. See https://en.wikipedia.org/wiki/Journal_ranking https://en.wikipedia.org/wiki/Journal_ranking
- kratom_sandwich 3y agoIt appears not to be an economics journal at all, at least it is not featured in RePEc's rankings: https://ideas.repec.org/top/top.journals.simple.html https://ideas.repec.org/top/top.journals.simple.html
- boppo1 3y agoWhat are the top 5?
- Shrezzing 3y agoDepends a bit on the branch/focus of economics you're looking at, and which nation/region you want to read about, but a broadly uncontroversial list would be: The Quarterly Journal of Economics, Oxford University Press (colloquial known as "the QUGE") Econometrica, Econometric Society Journal of Economic Literature, American Economic Association Journal of Economic Growth, Springer Journal of Financial Economics, Elsevier Maybe swap in the Journal of Political Economy if you're looking for more political leaning research. Swap in the American Economic Review if you're interested in more American-centric research.
- othello 3y agoThat's incorrect, the Top 5 in econ are (in no particular order): - Econometrica - American Economic Review - Quarterly Journal of Economics - Journal of Political Economy - Review of Economic Studies
- scotty79 3y agoI'm not surprised that results of this study were not surprising.
- bouncing 3y ago> If I had other intensions [sic] and did not believe in my imputation approach, I would not share the data with him. Admittedly this isn’t my field. But that sounds a lot like “I only show my work to people who will validate it,” does it not?
- timmb 3y agoI disagree, I think it means “my actions have demonstrated that I’m not trying to cover anything up”
- munch117 3y agoThat's my interpretation as well. It doesn't put him in a much better light, though. It's like he's so incompetent that he didn't even realise that there was something that needed covering up.
- anjc 3y agoThere wasn't something that needed covering up. If he'd reported the methodology in the paper there would have been no issues. It's likely that this was omitted from the paper as an oversight; such imputation likely wouldn't have affected the outcome of peer review.
- thimp 3y agoOh this is nothing. Go look at the social sciences. They get data that compromises their entire profession, tell the university it’s inconclusive and wait until the research grant runs out and fail to publish. Then do the same thing again. I have seen this with my own eyes. A bit of excel drama is usually behind their published papers. Also seen that with my own eyes. The field has trouble collecting data too because it’s expensive and very subjective. The worst one I saw was somehow turning 7 data points via imputation and interpolation into a hundred or so. The principal researcher on that I actually had to teach BIDMAS to and her calculations were completely whacked out (excel again)
- screenoridesaga 3y agoA lot of people are upset when their grant doesn’t get funded and it’s easy to suppose there’s some sort of sinister motive behind it. In truth it’s probably that the science is very difficult and only a tiny fraction of people get anywhere in social sciences. Try marine biology, they are always looking for decent papers.
- thimp 3y ago[flagged]
- concordDance 3y agoWhat profession have you seen compromised?
- thimp 3y ago[flagged]
- FrustratedMonky 3y agoBecause there are bad apples, doesn't mean you should throw out the scientific method. Over 50% of social science studies do replicate. Sorry that isn't high enough. We should definitely just give up on understanding ourselves. Or, maybe fix the financial incentives. Any industry across all fields, if the incentives are miss-matched, will produce crap.
- lkdfjlkdfjlg 3y agoYou post on HN, people see it as a UI problem. Everything is tech, everything is tooling. But if you knew anything the history of science, you'd know that a sloppy scientist with good tools gets you sloppy science, and a diligent scientist with bad tools might still get you as-good-as-possible science.
- LeifCarrotson 3y agoPeople aren't uniform, but they're all human. Improvements in tooling raise the baseline universally, making the "sloppy" scientists malfeasance obvious and helping the "diligent" scientist avoid inadvertent mistakes. Moralizing about humans being humans and wishing we were better gets us nowhere, but technology carries us into the future.
- lkdfjlkdfjlg 3y agoI'm not moralizing. There's no moral anywhere in what I wrote. Being sloppy isn't a moral judgement.
- moron4hire 3y agoThis is called "econometrics".
- nequo 3y agoThis is not econometrics though. Actual econometrics is upfront that you have missing data, performs partial identification rather than pretending that you have point identification, and gives you upper and lower bounds on what the true parameter value might be. Only if you did that with this paper, you’d have bounds so wide that no useful conclusion could be drawn, as indicated by the author’s statement that without imputation such data is useless.
- readthenotes1 3y agoThis reminds me of an error in the Excel sheet used by the EU to justify austerity programs back in 2008 the led to trouble in the PIGS countries. Someone double checked that and found that there was an error in the equation is correction show that austerity programs were mildly worse for recovering from economic disasters. -- On an editorial note, it seems to me that economics is just highfalutin astrology and this does nothing to convince me otherwise
- codeulike 3y agoReinhart, Rogoff... and Herndon: The student who caught out the profs (2013) https://www.bbc.co.uk/news/magazine-22223190 https://www.bbc.co.uk/news/magazine-22223190
- nequo 3y agoIf you see conclusions drawn from country level data, you’ve gotta be careful. Unfortunately, there is a field of economics that uses such data extensively. Other fields have a, shall we say, weaker taste for it.
- immibis 3y agoMany excuses have been used to justify austerity. If this study didn't work, they'd use another. If 99 studies said austerity was bad and 1 study said it was good, I suspect that one would be more likely to get published; in either case it would get used to justify austerity. It seems to me like austerity is the pre-determined conclusion, and then they go looking for reasons.
- 745frm 3y ago[flagged]
- blackbear_ 3y agoDid you really just make an account to post this comment? That's cowardly, and sad.
- 745frm 3y ago[flagged]
- raid2000 3y agoI expected to read a paper about some obscure Excel trick to manipulate stats output. Instead, this is just old-fashioned manipulation by hand or "imputation" as the paper describes it. > In email correspondence seen by Retraction Watch and a follow-up Zoom call, Heshmati told the student he had used Excel’s autofill function to mend the data. He had marked anywhere from two to four observations before or after the missing values and dragged the selected cells down or up, depending on the case. The program then filled in the blanks. If the new numbers turned negative, Heshmati replaced them with the last positive value Excel had spit out.
- MostlyStable 3y agoThe crazy thing about it is that the author doesn't seem to understand why it's bad. He doesn't appear to be hiding it. He just says "yeah that's what I did, whoops, I forgot to say it in the paper". He's either decided that acting like a complete moron is better than being thought of as an an intentional fraud, or else he really does think it was totally above board.
- richbell 3y agoI had a similar thought. My interpretation is that he genuinely thinks what he did was ok, because Excel has computer magic. This quote seems unintentionally telling: > "If we do not use imputation, such data is almost useless,” Heshmati said. He added that the description of the data in the paper as “balanced” referred to “the final data” – that is, the mended dataset.
- FrustratedMonky 3y ago""Is the evidence for austerity based on an Excel spreadsheet error?" Lets not forget that people/politicians also will ignore errors if they support their point of view. This can lead to policy decisions that impact the global economy. The US 'Right' believe strongly in biblical/moral individual responsibility, so any study that supports that gets a pass. This lead to 'austerity' measures after 2008. Because "everyone should just be more responsible in their spending". ""They actually found, using a different method, that the economic growth would be around 2.2 percent. Reinhart and Rogoff admitted that their spreadsheet was accidentally omitting 5 rows in an Average formula, and claimed that the corrected result would be a positive growth but of just 0.2 percent."" https://www.powerusersoftwares.com/post/2016/08/11/the-excel-formula-error-that-initiated-austerity-policies-after-the-crisis#:~:text=They%20actually%20found%2C%20using%20a,but%20of%20just%200.2%20percent https://www.powerusersoftwares.com/post/2016/08/11/the-excel.... https://www.theguardian.com/politics/2013/apr/18/uncovered-error-george-osborne-austerity https://www.theguardian.com/politics/2013/apr/18/uncovered-e... https://www.washingtonpost.com/news/wonk/wp/2013/04/16/is-the-best-evidence-for-austerity-based-on-an-excel-spreadsheet-error/ https://www.washingtonpost.com/news/wonk/wp/2013/04/16/is-th...
- gjvc 3y agosee also https://eusprig.org/ https://eusprig.org/
- hermitcrab 3y agoThere is plenty of software that can do imputation for you. Video of how to do it in our Easy Data Transform software at: https://www.youtube.com/watch?v=WXAGhtqI5xw https://www.youtube.com/watch?v=WXAGhtqI5xw Is is jaw-dropping that a professor would use Excel auto-fill.
- SebFender 3y ago“The reason it’s cheating isn’t that he’s done it, but that he hasn’t written it down,” Sorry but no. Every day my team takes decisions that have significant risk management impacts and we clearly state the subjective implications. Even when we have numbers to back things up, we always state the nature of our decisions.