3 ms·

I'm a former astronomer and I've talked to some of my friends who are still in astronomy about this. One way they've put it is that there are some papers that

I'm a former astronomer and I've talked to some of my friends who are still in astronomy about this. One way they've put it is that there are some papers that end up being wrong for some subtle, non-obvious reason which gets worked out over the course of a series of publications. This is fine, this is how science progresses. Then there is work where the authors made basic, sloppy mistakes that other specialists in the field could have immediately caught. This work falls in the latter category, which is bad enough, but then it managed to get accepted to Nature (which tends to select for hype over rigor) and received breathless media coverage. I don't think anyone in astronomy is so concerned about this mistake not getting corrected. People in the field were immediately skeptical of the results and after some digging pretty conclusively showed that they must be wrong. The problem is more that extensive media coverage of sloppy work propagates a perception among the general public that scientists can't be trusted. The media does a poor job at communicating what results are speculative or disputed, and what results are solidly accepted by a large amount of evidence. If scientists are changing their mind about this and whether chocolate and wine are good for you, how can they be trusted on matters like climate change or even whether the Earth is round? Why give them billions of dollars to build telescopes if they're changing their mind every other day and can't seem to figure anything out? So I think most scientists tend to be very conservative about their results and only to try advertise results that they have a lot of confidence in so as to uphold this sort of social contract with the public. When they see other scientists go on a media blitz with some shoddy work, they view it as a violation of that social contract.

deleted 6y ago
[deleted]
gfodor 6y ago
I guess the challenge is that the recognition that the work was sloppy can have a number of explanations: - the group was collectively honest but incompetent, including reviewers. So, no way to self identify the problem. - the group was deliberately leaving errors unfixed to get press and attention. If so, they should be shunned but this seems unlikely in this case. - the group worked honestly, in good faith, and was competent but made honest errors, which are easier to see once the effort is exposed from a few people to the whole community. Many eyes make all bugs shallow and so on. In that case, it’s important to unpack how it’s possible to say with certainty certain mistakes were “sloppy” and it not being just the kind of thing one says once the mistakes are identified. In general my point is that criticisms like the one you mention often seem valid but leave me unsatisfied by not offering a fully explanatory theory, or having one but not voicing it.
- antognini 6y ago
  In this case my understanding is that the group was basically new to the particular topic and data analysis technique that they were studying. At some level their errors were understandable, it would have been easy for anyone new to it to make similar errors. But when you start on a new topic as a researcher it's really incumbent on you to do your background reading and make sure you understand the field you're contributing to. In addition to that, Nature clearly did not assign reviewers who were knowledgeable about that topic. So I would say that the first situation you laid out most accurately described what happened here. Obviously there can always be some Dunning-Kruger effect going on and it's hard to know what you don't know, so I don't think there's any easy solution. The main things that help when you're entering a new field are: 1. Doing a lot of background reading. A lot. 2. Showing a preprint to others in the field for their feedback. Of course if you're sitting on what you believe to be big results, it's hard to do these things because they will slow you down.

deleted 6y ago

[deleted]

gfodor 6y ago

I guess the challenge is that the recognition that the work was sloppy can have a number of explanations: - the group was collectively honest but incompetent, including reviewers. So, no way to self identify the problem. - the group was deliberately leaving errors unfixed to get press and attention. If so, they should be shunned but this seems unlikely in this case. - the group worked honestly, in good faith, and was competent but made honest errors, which are easier to see once the effort is exposed from a few people to the whole community. Many eyes make all bugs shallow and so on. In that case, it’s important to unpack how it’s possible to say with certainty certain mistakes were “sloppy” and it not being just the kind of thing one says once the mistakes are identified. In general my point is that criticisms like the one you mention often seem valid but leave me unsatisfied by not offering a fully explanatory theory, or having one but not voicing it.

antognini 6y ago
In this case my understanding is that the group was basically new to the particular topic and data analysis technique that they were studying. At some level their errors were understandable, it would have been easy for anyone new to it to make similar errors. But when you start on a new topic as a researcher it's really incumbent on you to do your background reading and make sure you understand the field you're contributing to. In addition to that, Nature clearly did not assign reviewers who were knowledgeable about that topic. So I would say that the first situation you laid out most accurately described what happened here. Obviously there can always be some Dunning-Kruger effect going on and it's hard to know what you don't know, so I don't think there's any easy solution. The main things that help when you're entering a new field are: 1. Doing a lot of background reading. A lot. 2. Showing a preprint to others in the field for their feedback. Of course if you're sitting on what you believe to be big results, it's hard to do these things because they will slow you down.