15 ms·

What happens when scientists admit error

221 points by lelf 6y ago

jzer0cool 6y ago
Is there a convenient place to input web url to get at the article text? I saw someone post a few days ago ...
- notriddle 6y ago
  https://sci-hub.tw https://sci-hub.tw?
  - jzer0cool 6y ago
    I couldn't get it to work with authors post, but first time I am hearing of this site. Thank you for sharing!
- justinclift 6y ago
  Guessing you're thinking of https://archive.today https://archive.today? That gives you a web page where you can give the full URL to the paywalled one, and it'll give back an "https://archive.xx/address" https://archive.xx/address" URL where you can see the full thing. eg: https://archive.vn/0ikco https://archive.vn/0ikco in this instance
- SynasterBeiter 6y ago
  I use DuckDuckGo's iOS browser. If you use it's Clear Data functionality, most sites will display the article (because they think you're a first-time visitor).
- DonCopal 6y ago
  You're thinking of Outline.com
  - jzer0cool 6y ago
    Yes! This is what I was looking for.
saystupidthings 6y ago
A good read! It helped me a lot too -- I've had issues where I would find things constantly wrong in my code and find out my results were not right. It never got to the point of publication, but it was embarrassing to share with my advisor and co-authors. However, with some work we were able to continue to improve the algorithm and get good results down the line! Because of this, though, I am always paranoid of mistakes in my coding, but this article helped that fear a bit.
chaps 6y ago
Since when does medium require sign-in to read? Anyone have a different link to the post?
- renewiltord 6y ago
  Clear your medium cookies. It's a "lots of visits" thing.
- jtvjan 6y ago
  Disable JavaScript or use a text browser like Lynx. I've put the article in this paste: https://pastebin.com/raw/pe6dfUtJ https://pastebin.com/raw/pe6dfUtJ
gfrff 6y ago
She said this never happens. This happened all the time. It's just that she was the first scientist to take the honest route and do a retraction rather than letting it slip. I'm impressed with her integrity, but surprised by her naivete.
- SubiculumCode 6y ago
  She is not the first scientist to fess to a mistake. Heck, one of my colleagues went through the same thing last year, and he also did the right thing, and other scientists also appreciated his open honesty about the matter, even though his mistake meant the retraction of a high impact article.
throwaway285524 6y ago
It's great that she did the right thing and had the paper retracted, but this is still terrible on so many levels. Maybe with a strong effect for every single subject a little more skepticism would have been warranted in the first place? Some manual spot checking if possible, or using a minimal independent implementation of the analysis code? Who knows if she'd gotten her grant, her assistant professorship without the publication of this incorrect finding. Who knows who didn't get any of that because they were a bit too careful in their work.
- Thorrez 6y ago
  > Who knows if she'd gotten her grant, her assistant professorship without the publication of this incorrect finding. Who knows who didn't get any of that because they were a bit too careful in their work. On the other hand, if she hadn't wasted her time on this useless study she might have done more useful studies and her career would be better than it is right now. She might have gotten even better grants if she hadn't made this error, and maybe fewer other people would have gotten grants. Maybe those other people being more careful helped them rather than hurt them. I don't see how speculating like this is very useful.
  - darkerside 6y ago
    I think that underplays the difficulty of the work being done here. Or maybe it just underestimates the amount of luck involved. The downside is clearly greater than the upside. Why would you even imply the opposite?
    - Thorrez 6y ago
      I don't fully understand the relation between the difficulty of the work, the luck involved, and upside vs downside. I don't even know what you mean by the "downside", is that the downside of making the mistake, or the downside of not making the mistake? I'm saying the downside of making the mistake (for her) might be larger than the upside of making the mistake. I don't think that's obviously wrong. But my real main point is that speculation of this kind (in either direction) isn't very productive.
pdar4123 6y ago
Pro tip - When “every single one of the 96 participants show[s] the effect“ u have a technical artifact. This is scientific common sense.
- danieltillett 6y ago
  Not really. If you are counting fingers and toes you would not be surprised if all 96 participants had 10 of each. It is all about context.
  - pdar4123 6y ago
    But you would be surprised if they all had 11, 10 is your null hypothesis. It’s not context, it’s proper controls and the scientific method. This was very very careless
    - jhanschoo 6y ago
      Your rebuttal stands but not your point, for the comment you were replying to didn't properly contextualize their argument in a scientific experiment. It's more like serve participants canned vegetable soup, but in the non-control group warm it up before serving. Here you'd reasonably expect the non-control to be universally more favorable. They did have a null hypothesis and a control group. The problem was that the non-control was different from the control in a way they were careless with, that the scientific methodology does not catch (that peer review and replication ought to have caught). e.g. if the favorable results were better explained by subjects seeing soup taken out of the pot vs. from the can rather than the temperature itself.
  - matheusmoreira 6y ago
    > If you are counting fingers and toes you would not be surprised if all 96 participants had 10 of each. It is absolutely possible to find people with an abnormal number of fingers or toes. https://en.wikipedia.org/wiki/Oligodactyly https://en.wikipedia.org/wiki/Oligodactyly https://en.wikipedia.org/wiki/Polydactyly https://en.wikipedia.org/wiki/Polydactyly If 100% of the studied humans have 10 fingers and 10 toes, something's wrong with the study. In this case, the problem is the extremely small sample size: it simply doesn't represent the general human population. In the article author's case, a programming error caused these results. 0% and 100% are always suspect no matter the context they appear in.
jes5199 6y ago
we still trust software too much. I suspect that most software-controlled experiments are going to have errors like this! we should require that every experiment has at least two clean-room implementations of its logic, and a battery of smoke tests for common mistakes
- robertlagrant 6y ago
  We certainly trust scientists' software too much :-)
danieltillett 6y ago
While I applaud Julia for retracting the paper, I do wonder if she handed back the NIH grant she was given off the back of this false result. Some researcher just missed out on a NIH grant that would have been funded if not for this error.
- jhauris 6y ago
  Every indication is this was an honest mistake that was discovered due to the grant. The costs associated with the experiment where still there. Taking back grants in this situation would disincentivize scientific honesty.
  - danieltillett 6y ago
    I wasn’t suggesting the grant be taken back, but that she voluntarily hand it back.
    - Ididntdothis 6y ago
      Unless she is independently wealthy there would be a strong incentive for her to not admit the error if there was an expectation to hand the money back. Where is it supposed to come from?
      - danieltillett 6y ago
        I am not suggesting she hand back the money already spent, just that she might think about handing what is left. I would hand back a grant that was given due to a false result that occurred because of my error. Julia should be praised for doing the right thing with the paper, I just think she should think about going further.
        Ididntdothis 6y ago
        I don't think so. I am not returning my salary when I screw up at work and I have made bad decisions that cost 100s of thousands. As long it comes from an honest effort mistakes are part of the game.
lexpar 6y ago
What a nightmare. I bet this raises the hair on the back of the necks of a lot of other researchers. So much time and momentum can be invested into a research track like this.
- Pfhreak 6y ago
  Is this a nightmare? I read a story of someone who had a very reasonable, strong emotional response to the error, but ultimately got credit for coming clean and republished their results with new data. (And a different conclusion.) This is exactly how I'd expect something like this to work -- the author isn't a bad person because they made an error. The co-authors aren't bad people because they failed to catch it. Software and science are hard, mistakes are going to happen. If anything, I think the researchers learned valuable lessons, and are better researchers as a result. They have an anecdote they can share with more junior researchers about this frightening thing that happened to them, and use that to grow more people. We should celebrate people who take the time to handle their mistakes properly and share the lessons openly.
  - lexpar 6y ago
    I don't think she's a bad person, and I appreciate her response to the situation. But building a year (years?) around a project, attending multiple conferences for it, bringing up a couple students on a research idea, then finding it's all a software bug? That's a nightmare.
neonate 6y ago
https://archive.md/0ikco https://archive.md/0ikco
tehlike 6y ago
I think this can be considered a scientific version of blameless postmortem. There is no shame in making mistakes, and being honest about it should take us further as a society.
peterlk 6y ago
Remember when scientists in Italy were sent to prison for not predicting an earthquake correctly [0]? I know this isn't really what the article is about, but scientists are allowed to be wrong unless and until politics is involved. [0] https://www.scientificamerican.com/article/italian-scientists-get/ https://www.scientificamerican.com/article/italian-scientist...
- rs23296008n1 6y ago
  I wonder to what extent this has effected their public policy around healthcare? Were doctors afraid of warning about severity for fears if they got it wrong and their advice was used to close schools/businesses.
- Fragoel2 6y ago
  I live in the city struck by the earthquake and I can tell you that what the article states is not correct. The scientists went under trial not for a wrong prediction but for downplaying the possibile risk after thousands of smaller earthquakes were recorded in the area.
- im3w1l 6y ago
  While they may be scientists by training, predicting earthquakes is an application and not science itself. Like don't be wrong about that. Similar to how you should not be wrong about planes falling out of the sky, bridges collapsing, or medicines killing all the patients.
sn41 6y ago
The greatest and in some sense heartbreaking admission of error I know of, is Frege's letter to Bertrand Russell: https://sites.tufts.edu/histmath/files/2015/11/Frege-Letter-to-Russell.pdf https://sites.tufts.edu/histmath/files/2015/11/Frege-Letter-... Russell himself says "As I think about acts of integrity and grace, there is nothing to compare with Frege's dedication for truth."
- Daub 6y ago
  This is beautiful and humbling. An example of a scientist who loved being wrong was Fred Hoyle. Quote: ‘it is better to be interesting and wrong than boring and right’.
  - amelius 6y ago
    > ‘it is better to be interesting and wrong than boring and right’ This is why fake news is such a hit ;)
    - OrgNet 6y ago
      sometimes I think Trump makes these dumb quotes just for free publicity... hopefully he doesn't kill too many people in the process...
      - asdkjh345fd 6y ago
        Who is it that really bears the blame for these unsubstantiated bleach drinking incidents? Trump, who did not say anything like "drink bleach", or the media who openly lie and run headlines like "Trump tells everyone to drink bleach!"? Or even non-media hucksters trying to get publicity by tweeting things like "I can’t believe I have to say this, but please don’t drink bleach"?
        neltnerb 6y ago
        You can't be a huckster if you aren't selling. They aren't making a profit on anyone's gullibility, they're just horrified.
rixed 6y ago
> I’ve never heard of a comparable situation. Yet it is hard to believe scientists would be the ones who only ship flawless software. And easier to imagine that most often, when one realises a previous publication's result was indeed impacted by an error and faces the difficult decision to retract or pretend they haven't noticed, they opt for the other route. I've found scientists in general to have less ego then average, as expected from people trained to care only for the facts, but they still operate in the larger framework of individualistic modern society. It probably takes a scientist and a woman, whose education generally encourage lesser ego to begin with, to admit such an error. Or a software author, but there there is no other alternative to guilt admittance than total ridicule, since we are drowning everyday in a world of errors.
- searchableguy 6y ago
  > Yet it is hard to believe scientists would be the ones who only ship flawless software. I mean, most of them don't even pin down their dependencies. I don't want to touch a lot of python code in the open written by scientists. Thinking about software reproducibility is the last thing I have noticed on repositories submitted with a paper on arxiv and other places of publishing. Not even a requirements.txt or mentioning what they used.
  - dgoldstein0 6y ago
    I've spent years learning and practicing good software engineering practices. Whenever I read about quality issues in scientific code, I wonder: how are they expected to learn these things?
    - searchableguy 6y ago
      I don't think you need to worry about more than a few principles - 1. Reproducibility This is very important for scientists so I expect them to know about package management and containers. You don't need to know everything about it. Virtual environments, docker and using a good package manager like poetry (which is just an abstraction over virtualenv) shouldn't take more than a few weeks. Once you are done with the basics, you can learn more as you build. 2. Git and documentation Add another few weeks. They don't need to write good code. They just need to document it which they should already be good at. Optimization, scalability, distribution and other things can be handled by engineers but without documentation, it makes the work a lot harder.
Nalta 6y ago
I really hope those grad students that she mentioned didn't get a couple years added on to their degrees as a result of this. I mean good on her for finding the error, but I can't imagine what it would be like to be told "cancel your thesis, I made a mistake and your work is now invalid."
- bpodgursky 6y ago
  Luckily, they were masters' students, so the damage will be limited to a semester or so at most (in the US, generally a master's thesis isn't the most important part of a degree). If it was a PhD thesis... I don't want to think about that.
  - irrational 6y ago
    I don’t know. It took me a few semesters to write mine and it was the main graduation requirement. This kind of situation would’ve set me back at least a year and cost me a ton of tuition money. It would’ve been economically and emotionally devastating for me.
  - OkayPhysicist 6y ago
    Our professor loves to tell the story of how one of his friends in grad school based his thesis on a series of experiments to be run by a mars probe. One morning my professor runs into him and he's just completely panicked. My professor asked what was wrong, and the guy replied "My thesis is a debris field on Mars". So, it happens
- irrational 6y ago
  Yeah, I was disheartened that she didn’t mention what the fallout for them was. The one that was about to defend was probably the most affected out of anyone in this story. The author ended up keeping her grant and got tenure. What happened to the students?
rs23296008n1 6y ago
The other side of this is that actual science happened. Admitting error is science being done right. The search for the truth is fundamental and detecting when it wasn't found is contributing to the greater truth. Questions around methodology abound but at its core this is science walking tall. Not hobbling along loaded down with sugar-coated lies about to collapse into a coma. That this is seen as an exception or extraordinary is quite illuminating in itself.
dschuetz 6y ago
Kudos for writing an article about that. Most scientists just don't care. Reproducible results are still considered overrated.
squarefoot 6y ago
Now if only politicians learned a thing or two from this guy. Not possible however: scientists talk to brains, politicians talk to bellies and appeal to instincts, which require them to appear strong by showing self confidence and never admit any errors.
z3t4 6y ago
This could possible be gamed/abused. Make great discovery - get grants and tenure - only to retract the finding - and keep the money.
nednar 6y ago
I'm surprised if updating papers to correct mistakes is not the common path. Especially with software being a crucial part of it. I mean, how many software products do we know of that shipped flawlessly in v1.0?
vinay_ys 6y ago
If anyone is curious like I was to read the actual paper and look at the source code, you can find them here: https://osf.io/b94yx/ https://osf.io/b94yx/ The paper authors made a mistake, fine. But the scientific process and peer review process should have caught it. It didn't. The author caught it accidentally and then luckily decided to come forward (bravo!). This begs the question of robustness of the whole scientific publishing process. I hope they adopt the practice of doing a blameless RCA and improve the scientific and academic peer review process.
- asdkjh345fd 6y ago
  >This begs the question of robustness of the whole scientific publishing process It raises the question. Begging the question is an unrelated logical fallacy. Unfortunately, there have been a ton of examples of the peer-review process being essentially useless. Things like people deliberately putting things in to test if the paper is even being read and none of the reviewers notice it.
  - vinay_ys 6y ago
    "This begs the question" is same as "This raises the question" – What is the logical fallacy in how it is phrased?
    - asdkjh345fd 6y ago
      >"This begs the question" is same as "This raises the question" No it isn't, that is a common error in modern English. Raising the question means bringing a question into focus. Begging the question means to assume the conclusion is correct in the premise. https://en.wikipedia.org/wiki/Begging_the_question https://en.wikipedia.org/wiki/Begging_the_question
      - vinay_ys 6y ago
        Thanks for pointing out that wiki link. It lead me to another link https://www.merriam-webster.com/words-at-play/beg-the-question https://www.merriam-webster.com/words-at-play/beg-the-questi... which seems to indicate my usage is quite acceptable in modern English.
unexaminedlife 6y ago
For me this story illuminates pretty clearly the fact that free-market ideals and academia don't mesh that well.
kortilla 6y ago
The most shocking part of this article was the revelation that the original article was not retracted. The entire conclusion and data were entirely wrong, yet the paper still stands. That’s really damning for the journal at least and likely speaks to a pretty bad culture in the wider field. Kudos to the author for doing the right thing, but the fact that there seems to be no way to remove a paper that is blatantly false because retractions are reserved for deliberate misconduct is horrifying. Not only does this setup long term fucked up incentives (no downside to fraud if you paint it as a whoopsie), but it also harms all work that had cited that work and anyone doing literature reviews not realizing the ground other papers were standing on has dissolved away.
- dwighttk 6y ago
  >The editor and publisher were understanding and ultimately opted not to retract the paper but to instead publish a revised version of the article, linked to from the original paper, with the results section updated to reflect the true (opposite) results. I don't see what the problem is.
  - kortilla 6y ago
    The problem is that the old copy is still floating around and by replacing it instead of retracting it, it’s not clear which papers are referencing the bullshit one and which ones are referencing the real results. Not retracting hid the turd from bibliometric tools that could have easily notifies you of poisoned papers.
rarescruceat 6y ago
Scientists are also humans and humans are always making errors. The only thing is that the probability of a scientist making an error is 10x lower than another human, only because they made 1000 test errors before they talk.
jbj 6y ago
Well done by the author and thanks for sharing, especilly because of the immense mental pressure. I think it is great that the journal was cooporative for this. It would be interesting to see a single journal implement a very easy undo-button for peer reviewed research and see how that reflects over a series of years compared to the current model. Although very ridgid, the scientiffic ecosystem is very robust, and we now see friction being removed with efficient pre-print servers. I remember some early covid papers were redrawn from bioRxiv by authors request.
seemslegit 6y ago
> The data set was gorgeous — every single one of the 96 participants showed the effect. Ideally, that would have been an uh-oh moment.
Gatsky 6y ago
So this error was in the code that actually ran the experiment, not the analysis. The experiment was effectively doomed from the beginning. The mistake amounts to failing to calibrate and test your experimental apparatus, which is a little hard to forgive, since it would have taken hardly any time or effort. Perhaps I’m being harsh but scientific enquiry is too important to be satisfied with conducting unrigorous experiments then apologising, which of course one could do ad infinitum.
- GiveOver 6y ago
  I agree. I write code all the time and I've never had a single bug. She was careless

221 points by lelf 6y ago

jzer0cool 6y ago

Is there a convenient place to input web url to get at the article text? I saw someone post a few days ago ...

notriddle 6y ago
https://sci-hub.tw https://sci-hub.tw?
- jzer0cool 6y ago
  I couldn't get it to work with authors post, but first time I am hearing of this site. Thank you for sharing!
justinclift 6y ago
Guessing you're thinking of https://archive.today https://archive.today? That gives you a web page where you can give the full URL to the paywalled one, and it'll give back an "https://archive.xx/address" https://archive.xx/address" URL where you can see the full thing. eg: https://archive.vn/0ikco https://archive.vn/0ikco in this instance
SynasterBeiter 6y ago
I use DuckDuckGo's iOS browser. If you use it's Clear Data functionality, most sites will display the article (because they think you're a first-time visitor).
DonCopal 6y ago
You're thinking of Outline.com
- jzer0cool 6y ago
  Yes! This is what I was looking for.

saystupidthings 6y ago

A good read! It helped me a lot too -- I've had issues where I would find things constantly wrong in my code and find out my results were not right. It never got to the point of publication, but it was embarrassing to share with my advisor and co-authors. However, with some work we were able to continue to improve the algorithm and get good results down the line! Because of this, though, I am always paranoid of mistakes in my coding, but this article helped that fear a bit.

chaps 6y ago

Since when does medium require sign-in to read? Anyone have a different link to the post?

renewiltord 6y ago
Clear your medium cookies. It's a "lots of visits" thing.
jtvjan 6y ago
Disable JavaScript or use a text browser like Lynx. I've put the article in this paste: https://pastebin.com/raw/pe6dfUtJ https://pastebin.com/raw/pe6dfUtJ

gfrff 6y ago

She said this never happens. This happened all the time. It's just that she was the first scientist to take the honest route and do a retraction rather than letting it slip. I'm impressed with her integrity, but surprised by her naivete.

SubiculumCode 6y ago
She is not the first scientist to fess to a mistake. Heck, one of my colleagues went through the same thing last year, and he also did the right thing, and other scientists also appreciated his open honesty about the matter, even though his mistake meant the retraction of a high impact article.

throwaway285524 6y ago

It's great that she did the right thing and had the paper retracted, but this is still terrible on so many levels. Maybe with a strong effect for every single subject a little more skepticism would have been warranted in the first place? Some manual spot checking if possible, or using a minimal independent implementation of the analysis code? Who knows if she'd gotten her grant, her assistant professorship without the publication of this incorrect finding. Who knows who didn't get any of that because they were a bit too careful in their work.

Thorrez 6y ago
> Who knows if she'd gotten her grant, her assistant professorship without the publication of this incorrect finding. Who knows who didn't get any of that because they were a bit too careful in their work. On the other hand, if she hadn't wasted her time on this useless study she might have done more useful studies and her career would be better than it is right now. She might have gotten even better grants if she hadn't made this error, and maybe fewer other people would have gotten grants. Maybe those other people being more careful helped them rather than hurt them. I don't see how speculating like this is very useful.
- darkerside 6y ago
  I think that underplays the difficulty of the work being done here. Or maybe it just underestimates the amount of luck involved. The downside is clearly greater than the upside. Why would you even imply the opposite?
  - Thorrez 6y ago
    I don't fully understand the relation between the difficulty of the work, the luck involved, and upside vs downside. I don't even know what you mean by the "downside", is that the downside of making the mistake, or the downside of not making the mistake? I'm saying the downside of making the mistake (for her) might be larger than the upside of making the mistake. I don't think that's obviously wrong. But my real main point is that speculation of this kind (in either direction) isn't very productive.

pdar4123 6y ago

Pro tip - When “every single one of the 96 participants show[s] the effect“ u have a technical artifact. This is scientific common sense.

danieltillett 6y ago
Not really. If you are counting fingers and toes you would not be surprised if all 96 participants had 10 of each. It is all about context.
- pdar4123 6y ago
  But you would be surprised if they all had 11, 10 is your null hypothesis. It’s not context, it’s proper controls and the scientific method. This was very very careless
  - jhanschoo 6y ago
    Your rebuttal stands but not your point, for the comment you were replying to didn't properly contextualize their argument in a scientific experiment. It's more like serve participants canned vegetable soup, but in the non-control group warm it up before serving. Here you'd reasonably expect the non-control to be universally more favorable. They did have a null hypothesis and a control group. The problem was that the non-control was different from the control in a way they were careless with, that the scientific methodology does not catch (that peer review and replication ought to have caught). e.g. if the favorable results were better explained by subjects seeing soup taken out of the pot vs. from the can rather than the temperature itself.
- matheusmoreira 6y ago
  > If you are counting fingers and toes you would not be surprised if all 96 participants had 10 of each. It is absolutely possible to find people with an abnormal number of fingers or toes. https://en.wikipedia.org/wiki/Oligodactyly https://en.wikipedia.org/wiki/Oligodactyly https://en.wikipedia.org/wiki/Polydactyly https://en.wikipedia.org/wiki/Polydactyly If 100% of the studied humans have 10 fingers and 10 toes, something's wrong with the study. In this case, the problem is the extremely small sample size: it simply doesn't represent the general human population. In the article author's case, a programming error caused these results. 0% and 100% are always suspect no matter the context they appear in.

jes5199 6y ago

we still trust software too much. I suspect that most software-controlled experiments are going to have errors like this! we should require that every experiment has at least two clean-room implementations of its logic, and a battery of smoke tests for common mistakes

robertlagrant 6y ago
We certainly trust scientists' software too much :-)

danieltillett 6y ago

While I applaud Julia for retracting the paper, I do wonder if she handed back the NIH grant she was given off the back of this false result. Some researcher just missed out on a NIH grant that would have been funded if not for this error.

jhauris 6y ago
Every indication is this was an honest mistake that was discovered due to the grant. The costs associated with the experiment where still there. Taking back grants in this situation would disincentivize scientific honesty.
- danieltillett 6y ago
  I wasn’t suggesting the grant be taken back, but that she voluntarily hand it back.
  - Ididntdothis 6y ago
    Unless she is independently wealthy there would be a strong incentive for her to not admit the error if there was an expectation to hand the money back. Where is it supposed to come from?
    - danieltillett 6y ago
      I am not suggesting she hand back the money already spent, just that she might think about handing what is left. I would hand back a grant that was given due to a false result that occurred because of my error. Julia should be praised for doing the right thing with the paper, I just think she should think about going further.
      - Ididntdothis 6y ago
        I don't think so. I am not returning my salary when I screw up at work and I have made bad decisions that cost 100s of thousands. As long it comes from an honest effort mistakes are part of the game.

lexpar 6y ago

What a nightmare. I bet this raises the hair on the back of the necks of a lot of other researchers. So much time and momentum can be invested into a research track like this.

Pfhreak 6y ago
Is this a nightmare? I read a story of someone who had a very reasonable, strong emotional response to the error, but ultimately got credit for coming clean and republished their results with new data. (And a different conclusion.) This is exactly how I'd expect something like this to work -- the author isn't a bad person because they made an error. The co-authors aren't bad people because they failed to catch it. Software and science are hard, mistakes are going to happen. If anything, I think the researchers learned valuable lessons, and are better researchers as a result. They have an anecdote they can share with more junior researchers about this frightening thing that happened to them, and use that to grow more people. We should celebrate people who take the time to handle their mistakes properly and share the lessons openly.
- lexpar 6y ago
  I don't think she's a bad person, and I appreciate her response to the situation. But building a year (years?) around a project, attending multiple conferences for it, bringing up a couple students on a research idea, then finding it's all a software bug? That's a nightmare.

neonate 6y ago

https://archive.md/0ikco https://archive.md/0ikco

tehlike 6y ago

I think this can be considered a scientific version of blameless postmortem. There is no shame in making mistakes, and being honest about it should take us further as a society.

peterlk 6y ago

Remember when scientists in Italy were sent to prison for not predicting an earthquake correctly [0]? I know this isn't really what the article is about, but scientists are allowed to be wrong unless and until politics is involved. [0] https://www.scientificamerican.com/article/italian-scientists-get/ https://www.scientificamerican.com/article/italian-scientist...

rs23296008n1 6y ago
I wonder to what extent this has effected their public policy around healthcare? Were doctors afraid of warning about severity for fears if they got it wrong and their advice was used to close schools/businesses.
Fragoel2 6y ago
I live in the city struck by the earthquake and I can tell you that what the article states is not correct. The scientists went under trial not for a wrong prediction but for downplaying the possibile risk after thousands of smaller earthquakes were recorded in the area.
im3w1l 6y ago
While they may be scientists by training, predicting earthquakes is an application and not science itself. Like don't be wrong about that. Similar to how you should not be wrong about planes falling out of the sky, bridges collapsing, or medicines killing all the patients.

sn41 6y ago

The greatest and in some sense heartbreaking admission of error I know of, is Frege's letter to Bertrand Russell: https://sites.tufts.edu/histmath/files/2015/11/Frege-Letter-to-Russell.pdf https://sites.tufts.edu/histmath/files/2015/11/Frege-Letter-... Russell himself says "As I think about acts of integrity and grace, there is nothing to compare with Frege's dedication for truth."

Daub 6y ago
This is beautiful and humbling. An example of a scientist who loved being wrong was Fred Hoyle. Quote: ‘it is better to be interesting and wrong than boring and right’.
- amelius 6y ago
  > ‘it is better to be interesting and wrong than boring and right’ This is why fake news is such a hit ;)
  - OrgNet 6y ago
    sometimes I think Trump makes these dumb quotes just for free publicity... hopefully he doesn't kill too many people in the process...
    - asdkjh345fd 6y ago
      Who is it that really bears the blame for these unsubstantiated bleach drinking incidents? Trump, who did not say anything like "drink bleach", or the media who openly lie and run headlines like "Trump tells everyone to drink bleach!"? Or even non-media hucksters trying to get publicity by tweeting things like "I can’t believe I have to say this, but please don’t drink bleach"?
      - neltnerb 6y ago
        You can't be a huckster if you aren't selling. They aren't making a profit on anyone's gullibility, they're just horrified.

rixed 6y ago

> I’ve never heard of a comparable situation. Yet it is hard to believe scientists would be the ones who only ship flawless software. And easier to imagine that most often, when one realises a previous publication's result was indeed impacted by an error and faces the difficult decision to retract or pretend they haven't noticed, they opt for the other route. I've found scientists in general to have less ego then average, as expected from people trained to care only for the facts, but they still operate in the larger framework of individualistic modern society. It probably takes a scientist and a woman, whose education generally encourage lesser ego to begin with, to admit such an error. Or a software author, but there there is no other alternative to guilt admittance than total ridicule, since we are drowning everyday in a world of errors.

searchableguy 6y ago
> Yet it is hard to believe scientists would be the ones who only ship flawless software. I mean, most of them don't even pin down their dependencies. I don't want to touch a lot of python code in the open written by scientists. Thinking about software reproducibility is the last thing I have noticed on repositories submitted with a paper on arxiv and other places of publishing. Not even a requirements.txt or mentioning what they used.
- dgoldstein0 6y ago
  I've spent years learning and practicing good software engineering practices. Whenever I read about quality issues in scientific code, I wonder: how are they expected to learn these things?
  - searchableguy 6y ago
    I don't think you need to worry about more than a few principles - 1. Reproducibility This is very important for scientists so I expect them to know about package management and containers. You don't need to know everything about it. Virtual environments, docker and using a good package manager like poetry (which is just an abstraction over virtualenv) shouldn't take more than a few weeks. Once you are done with the basics, you can learn more as you build. 2. Git and documentation Add another few weeks. They don't need to write good code. They just need to document it which they should already be good at. Optimization, scalability, distribution and other things can be handled by engineers but without documentation, it makes the work a lot harder.

Nalta 6y ago

I really hope those grad students that she mentioned didn't get a couple years added on to their degrees as a result of this. I mean good on her for finding the error, but I can't imagine what it would be like to be told "cancel your thesis, I made a mistake and your work is now invalid."

bpodgursky 6y ago
Luckily, they were masters' students, so the damage will be limited to a semester or so at most (in the US, generally a master's thesis isn't the most important part of a degree). If it was a PhD thesis... I don't want to think about that.
- irrational 6y ago
  I don’t know. It took me a few semesters to write mine and it was the main graduation requirement. This kind of situation would’ve set me back at least a year and cost me a ton of tuition money. It would’ve been economically and emotionally devastating for me.
- OkayPhysicist 6y ago
  Our professor loves to tell the story of how one of his friends in grad school based his thesis on a series of experiments to be run by a mars probe. One morning my professor runs into him and he's just completely panicked. My professor asked what was wrong, and the guy replied "My thesis is a debris field on Mars". So, it happens
irrational 6y ago
Yeah, I was disheartened that she didn’t mention what the fallout for them was. The one that was about to defend was probably the most affected out of anyone in this story. The author ended up keeping her grant and got tenure. What happened to the students?

rs23296008n1 6y ago

The other side of this is that actual science happened. Admitting error is science being done right. The search for the truth is fundamental and detecting when it wasn't found is contributing to the greater truth. Questions around methodology abound but at its core this is science walking tall. Not hobbling along loaded down with sugar-coated lies about to collapse into a coma. That this is seen as an exception or extraordinary is quite illuminating in itself.

dschuetz 6y ago

Kudos for writing an article about that. Most scientists just don't care. Reproducible results are still considered overrated.

squarefoot 6y ago

Now if only politicians learned a thing or two from this guy. Not possible however: scientists talk to brains, politicians talk to bellies and appeal to instincts, which require them to appear strong by showing self confidence and never admit any errors.

z3t4 6y ago

This could possible be gamed/abused. Make great discovery - get grants and tenure - only to retract the finding - and keep the money.

nednar 6y ago

I'm surprised if updating papers to correct mistakes is not the common path. Especially with software being a crucial part of it. I mean, how many software products do we know of that shipped flawlessly in v1.0?

vinay_ys 6y ago

If anyone is curious like I was to read the actual paper and look at the source code, you can find them here: https://osf.io/b94yx/ https://osf.io/b94yx/ The paper authors made a mistake, fine. But the scientific process and peer review process should have caught it. It didn't. The author caught it accidentally and then luckily decided to come forward (bravo!). This begs the question of robustness of the whole scientific publishing process. I hope they adopt the practice of doing a blameless RCA and improve the scientific and academic peer review process.

asdkjh345fd 6y ago
>This begs the question of robustness of the whole scientific publishing process It raises the question. Begging the question is an unrelated logical fallacy. Unfortunately, there have been a ton of examples of the peer-review process being essentially useless. Things like people deliberately putting things in to test if the paper is even being read and none of the reviewers notice it.
- vinay_ys 6y ago
  "This begs the question" is same as "This raises the question" – What is the logical fallacy in how it is phrased?
  - asdkjh345fd 6y ago
    >"This begs the question" is same as "This raises the question" No it isn't, that is a common error in modern English. Raising the question means bringing a question into focus. Begging the question means to assume the conclusion is correct in the premise. https://en.wikipedia.org/wiki/Begging_the_question https://en.wikipedia.org/wiki/Begging_the_question
    - vinay_ys 6y ago
      Thanks for pointing out that wiki link. It lead me to another link https://www.merriam-webster.com/words-at-play/beg-the-question https://www.merriam-webster.com/words-at-play/beg-the-questi... which seems to indicate my usage is quite acceptable in modern English.

unexaminedlife 6y ago

For me this story illuminates pretty clearly the fact that free-market ideals and academia don't mesh that well.

kortilla 6y ago

The most shocking part of this article was the revelation that the original article was not retracted. The entire conclusion and data were entirely wrong, yet the paper still stands. That’s really damning for the journal at least and likely speaks to a pretty bad culture in the wider field. Kudos to the author for doing the right thing, but the fact that there seems to be no way to remove a paper that is blatantly false because retractions are reserved for deliberate misconduct is horrifying. Not only does this setup long term fucked up incentives (no downside to fraud if you paint it as a whoopsie), but it also harms all work that had cited that work and anyone doing literature reviews not realizing the ground other papers were standing on has dissolved away.

dwighttk 6y ago
>The editor and publisher were understanding and ultimately opted not to retract the paper but to instead publish a revised version of the article, linked to from the original paper, with the results section updated to reflect the true (opposite) results. I don't see what the problem is.
- kortilla 6y ago
  The problem is that the old copy is still floating around and by replacing it instead of retracting it, it’s not clear which papers are referencing the bullshit one and which ones are referencing the real results. Not retracting hid the turd from bibliometric tools that could have easily notifies you of poisoned papers.

rarescruceat 6y ago

Scientists are also humans and humans are always making errors. The only thing is that the probability of a scientist making an error is 10x lower than another human, only because they made 1000 test errors before they talk.

jbj 6y ago

Well done by the author and thanks for sharing, especilly because of the immense mental pressure. I think it is great that the journal was cooporative for this. It would be interesting to see a single journal implement a very easy undo-button for peer reviewed research and see how that reflects over a series of years compared to the current model. Although very ridgid, the scientiffic ecosystem is very robust, and we now see friction being removed with efficient pre-print servers. I remember some early covid papers were redrawn from bioRxiv by authors request.

seemslegit 6y ago

> The data set was gorgeous — every single one of the 96 participants showed the effect. Ideally, that would have been an uh-oh moment.

Gatsky 6y ago

So this error was in the code that actually ran the experiment, not the analysis. The experiment was effectively doomed from the beginning. The mistake amounts to failing to calibrate and test your experimental apparatus, which is a little hard to forgive, since it would have taken hardly any time or effort. Perhaps I’m being harsh but scientific enquiry is too important to be satisfied with conducting unrigorous experiments then apologising, which of course one could do ad infinitum.

GiveOver 6y ago
I agree. I write code all the time and I've never had a single bug. She was careless