5 ms·

Wouldn't it be better for scientists to actually verify their finds before releasing breathless statements to the media? It feels like almost every exciting di

Wouldn't it be better for scientists to actually verify their finds before releasing breathless statements to the media? It feels like almost every exciting discovery is being quietly walked back a month or two later.

fastball 6y ago
Generally I've found that the problem is not breathless scientists, it is breathless journos. They're looking for the "next big scoop" and so mostly misinterpret findings in order to come up with the most sensational headline possible. Don't get me wrong, science has it's own (major) problems, such as losing sight of reproducibility, falsifiability, etc, but usually when it comes to "I read about a discovery that sounded fabulous but didn't pan out", the heightened expectations are usually coming from the media, not the other way around.
- Hyp3rion 6y ago
  This is correct. To add to your point, it's also the respective institute's media departments. In a university for example, there's a strong incentive to hype up your research to attract young, impressionable undergraduates to your school. Not just a journalist looking for the "next big scoop", but often a journalist looking for that at an institute that wants to also have "the next big thing".
mcphage 6y ago
This is how scientists verify their finds. They observe something, make some tests to see if their hypothesis holds up. If it does, they publish their findings, and let other scientists evaluate their claims. This is exactly what is supposed to happen.
- antognini 6y ago
  I'm a former astronomer and I've talked to some of my friends who are still in astronomy about this. One way they've put it is that there are some papers that end up being wrong for some subtle, non-obvious reason which gets worked out over the course of a series of publications. This is fine, this is how science progresses. Then there is work where the authors made basic, sloppy mistakes that other specialists in the field could have immediately caught. This work falls in the latter category, which is bad enough, but then it managed to get accepted to Nature (which tends to select for hype over rigor) and received breathless media coverage. I don't think anyone in astronomy is so concerned about this mistake not getting corrected. People in the field were immediately skeptical of the results and after some digging pretty conclusively showed that they must be wrong. The problem is more that extensive media coverage of sloppy work propagates a perception among the general public that scientists can't be trusted. The media does a poor job at communicating what results are speculative or disputed, and what results are solidly accepted by a large amount of evidence. If scientists are changing their mind about this and whether chocolate and wine are good for you, how can they be trusted on matters like climate change or even whether the Earth is round? Why give them billions of dollars to build telescopes if they're changing their mind every other day and can't seem to figure anything out? So I think most scientists tend to be very conservative about their results and only to try advertise results that they have a lot of confidence in so as to uphold this sort of social contract with the public. When they see other scientists go on a media blitz with some shoddy work, they view it as a violation of that social contract.
  - deleted 6y ago
    [deleted]
  - gfodor 6y ago
    I guess the challenge is that the recognition that the work was sloppy can have a number of explanations: - the group was collectively honest but incompetent, including reviewers. So, no way to self identify the problem. - the group was deliberately leaving errors unfixed to get press and attention. If so, they should be shunned but this seems unlikely in this case. - the group worked honestly, in good faith, and was competent but made honest errors, which are easier to see once the effort is exposed from a few people to the whole community. Many eyes make all bugs shallow and so on. In that case, it’s important to unpack how it’s possible to say with certainty certain mistakes were “sloppy” and it not being just the kind of thing one says once the mistakes are identified. In general my point is that criticisms like the one you mention often seem valid but leave me unsatisfied by not offering a fully explanatory theory, or having one but not voicing it.
    - antognini 6y ago
      In this case my understanding is that the group was basically new to the particular topic and data analysis technique that they were studying. At some level their errors were understandable, it would have been easy for anyone new to it to make similar errors. But when you start on a new topic as a researcher it's really incumbent on you to do your background reading and make sure you understand the field you're contributing to. In addition to that, Nature clearly did not assign reviewers who were knowledgeable about that topic. So I would say that the first situation you laid out most accurately described what happened here. Obviously there can always be some Dunning-Kruger effect going on and it's hard to know what you don't know, so I don't think there's any easy solution. The main things that help when you're entering a new field are: 1. Doing a lot of background reading. A lot. 2. Showing a preprint to others in the field for their feedback. Of course if you're sitting on what you believe to be big results, it's hard to do these things because they will slow you down.
- deleted 6y ago
  [deleted]
bpodgursky 6y ago
There's not a global secret cabal of scientists who can meet in the absence of media attention. If a paper is presented in a conference to other scientists, journalists can (and will) write breathtaking headlines about it.
- avs733 6y ago
  If only... Anyone who talks about scientific conspiracy theories really astonishes me because it is so obvious they do not know any actual scientists all that well.
  - Blikkentrekker 6y ago
    And no one here spoke of any scientific conspiracy theories. What was talked about was making statements too quickly to the media in excitement; the rebuttal to that was somehow that no scientific conspiracies run amok. I must say that I find that that in the vast majority of cases that a “conspiracy” is rebutted, no conspiracy was alleged at all. Someone, this morning, was also about a website that promoted the idea of the corona virus being a government conspiracy; all I could find on that website was the belief that mandatory curfews and closing ships went too far and curbed the citizen's freedoms too much. I found no alleged government conspiracy on it, and he could not point me to where to find this.
inshadows 6y ago
It's better to put it out and let others take a glance, isn't it? I don't think scientists actively seek media attention. I think it's other way around, media looking for hot stuff in science.
c3534l 6y ago
They thought they did. I heard they did a really good job of it, too. But it doesn't matter how much work you put into it, you can still be wrong.
crispycrafter2 6y ago
Publishing results that, to the best of your knowledge, are correct is a crucial part of the scientific endeavor. The fact this their results have been drawn into question is therefore a good thing. A hypothesis that has been proven to be false is a net positive to humanity as a whole. This reminds me of a excerpt from the opening line of my undergraduate thermodynamics textbook: "The theories presented in this work are true due to the absence of contrary evidence"

fastball 6y ago

Generally I've found that the problem is not breathless scientists, it is breathless journos. They're looking for the "next big scoop" and so mostly misinterpret findings in order to come up with the most sensational headline possible. Don't get me wrong, science has it's own (major) problems, such as losing sight of reproducibility, falsifiability, etc, but usually when it comes to "I read about a discovery that sounded fabulous but didn't pan out", the heightened expectations are usually coming from the media, not the other way around.

Hyp3rion 6y ago
This is correct. To add to your point, it's also the respective institute's media departments. In a university for example, there's a strong incentive to hype up your research to attract young, impressionable undergraduates to your school. Not just a journalist looking for the "next big scoop", but often a journalist looking for that at an institute that wants to also have "the next big thing".

mcphage 6y ago

This is how scientists verify their finds. They observe something, make some tests to see if their hypothesis holds up. If it does, they publish their findings, and let other scientists evaluate their claims. This is exactly what is supposed to happen.

antognini 6y ago
I'm a former astronomer and I've talked to some of my friends who are still in astronomy about this. One way they've put it is that there are some papers that end up being wrong for some subtle, non-obvious reason which gets worked out over the course of a series of publications. This is fine, this is how science progresses. Then there is work where the authors made basic, sloppy mistakes that other specialists in the field could have immediately caught. This work falls in the latter category, which is bad enough, but then it managed to get accepted to Nature (which tends to select for hype over rigor) and received breathless media coverage. I don't think anyone in astronomy is so concerned about this mistake not getting corrected. People in the field were immediately skeptical of the results and after some digging pretty conclusively showed that they must be wrong. The problem is more that extensive media coverage of sloppy work propagates a perception among the general public that scientists can't be trusted. The media does a poor job at communicating what results are speculative or disputed, and what results are solidly accepted by a large amount of evidence. If scientists are changing their mind about this and whether chocolate and wine are good for you, how can they be trusted on matters like climate change or even whether the Earth is round? Why give them billions of dollars to build telescopes if they're changing their mind every other day and can't seem to figure anything out? So I think most scientists tend to be very conservative about their results and only to try advertise results that they have a lot of confidence in so as to uphold this sort of social contract with the public. When they see other scientists go on a media blitz with some shoddy work, they view it as a violation of that social contract.
- deleted 6y ago
  [deleted]
- gfodor 6y ago
  I guess the challenge is that the recognition that the work was sloppy can have a number of explanations: - the group was collectively honest but incompetent, including reviewers. So, no way to self identify the problem. - the group was deliberately leaving errors unfixed to get press and attention. If so, they should be shunned but this seems unlikely in this case. - the group worked honestly, in good faith, and was competent but made honest errors, which are easier to see once the effort is exposed from a few people to the whole community. Many eyes make all bugs shallow and so on. In that case, it’s important to unpack how it’s possible to say with certainty certain mistakes were “sloppy” and it not being just the kind of thing one says once the mistakes are identified. In general my point is that criticisms like the one you mention often seem valid but leave me unsatisfied by not offering a fully explanatory theory, or having one but not voicing it.
  - antognini 6y ago
    In this case my understanding is that the group was basically new to the particular topic and data analysis technique that they were studying. At some level their errors were understandable, it would have been easy for anyone new to it to make similar errors. But when you start on a new topic as a researcher it's really incumbent on you to do your background reading and make sure you understand the field you're contributing to. In addition to that, Nature clearly did not assign reviewers who were knowledgeable about that topic. So I would say that the first situation you laid out most accurately described what happened here. Obviously there can always be some Dunning-Kruger effect going on and it's hard to know what you don't know, so I don't think there's any easy solution. The main things that help when you're entering a new field are: 1. Doing a lot of background reading. A lot. 2. Showing a preprint to others in the field for their feedback. Of course if you're sitting on what you believe to be big results, it's hard to do these things because they will slow you down.
deleted 6y ago
[deleted]

bpodgursky 6y ago

There's not a global secret cabal of scientists who can meet in the absence of media attention. If a paper is presented in a conference to other scientists, journalists can (and will) write breathtaking headlines about it.

avs733 6y ago
If only... Anyone who talks about scientific conspiracy theories really astonishes me because it is so obvious they do not know any actual scientists all that well.
- Blikkentrekker 6y ago
  And no one here spoke of any scientific conspiracy theories. What was talked about was making statements too quickly to the media in excitement; the rebuttal to that was somehow that no scientific conspiracies run amok. I must say that I find that that in the vast majority of cases that a “conspiracy” is rebutted, no conspiracy was alleged at all. Someone, this morning, was also about a website that promoted the idea of the corona virus being a government conspiracy; all I could find on that website was the belief that mandatory curfews and closing ships went too far and curbed the citizen's freedoms too much. I found no alleged government conspiracy on it, and he could not point me to where to find this.

inshadows 6y ago

It's better to put it out and let others take a glance, isn't it? I don't think scientists actively seek media attention. I think it's other way around, media looking for hot stuff in science.

c3534l 6y ago

They thought they did. I heard they did a really good job of it, too. But it doesn't matter how much work you put into it, you can still be wrong.

crispycrafter2 6y ago

Publishing results that, to the best of your knowledge, are correct is a crucial part of the scientific endeavor. The fact this their results have been drawn into question is therefore a good thing. A hypothesis that has been proven to be false is a net positive to humanity as a whole. This reminds me of a excerpt from the opening line of my undergraduate thermodynamics textbook: "The theories presented in this work are true due to the absence of contrary evidence"