5 ms·

They don't have to shoot for perfection. The bar has sunk so low even very basic techniques can yield huge improvements in trust: 1. Report the fact that disag

They don't have to shoot for perfection. The bar has sunk so low even very basic techniques can yield huge improvements in trust: 1. Report the fact that disagreement exists. Phrased differently, stop taking academics at their word when they claim there's a consensus. Do some basic web searches to find people who disagree. Get quotes from them. Stress on the word people; not just other academics but literally anyone disagreeing on scientific grounds eliminates the claim of consensus. Bloggers are fine. 2. READ the papers. Journalists never do this. I cannot express how frequently you can spot scientific fraud by just reading the underlying papers, even as a layman. If you lack expertise maybe you'll miss 90% of the tricks but catching 10% of them is still sufficient to notice something is wrong, and often you don't need any special training. Here are some of my own investigations of bad papers - it's often obvious and most of it doesn't require expertise to spot. https://blog.plan99.net/did-russian-bots-impact-brexit-ad66f08c014a https://blog.plan99.net/did-russian-bots-impact-brexit-ad66f... https://blog.plan99.net/fake-science-part-ii-bots-that-are-not-c66129e5e3f5 https://blog.plan99.net/fake-science-part-ii-bots-that-are-n... 3. Hold sources to account when it's proven that they were misleading you. Report on bad behavior to discourage it next time. 4. Be willing to report stories dug up by other people, even when they make Team University look bad. Note how the reporting Silver refers to hasn't been covered by legacy media outlets even though you don't need to be a scientist to understand what they're saying and how damning it is. In reality this stuff is easy. Nobody is asking for the NYT to engage in professional peer review of newly published papers. Just not assuming anything a professor says is gospel truth would be a good start, but there seems little chance of that happening :( Journalists depend so heavily on academics for rent-a-quote services and a constant flow of stories that getting tough would be biting the hand that feeds them.

thorncorona 3y ago
> 1. Report the fact that disagreement exists. Phrased differently, stop taking academics at their word when they claim there's a consensus. Do some basic web searches to find people who disagree. Get quotes from them. Stress on the word people; not just other academics but literally anyone disagreeing on scientific grounds eliminates the claim of consensus. Bloggers are fine. This is how you get journalists to report disagreements that don't exist in reality. For example, whether the earth is flat, whether climate change is real, etc. > 2. READ the papers. Literally nobody reads the papers. A huge amount of news isn't even investigated. A large number of news these days is regurgitated from other sources.
- mike_hearn 3y ago
  Journalists love reporting on flat Earthers even though their beliefs have no impact on anything in the real world: https://news.google.com/search?q=%22flat%20earth%22&hl=en-US&gl=US&ceid=US%3Aen https://news.google.com/search?q=%22flat%20earth%22&hl=en-US... Realistically, journalists like to report on fringe or weird beliefs so they can laugh at the people holding them, and dislike reporting on serious disagreement with things they want to be true. Agree that almost nobody is reading the papers, outside of random tweeters and bloggers. Journalists might as well start, though. Reading obscure documents is a part of the job, classically at least.
- SoftTalker 3y ago
  You can still look at credentials. If there is disagreement among people with appropriate background, experience, and education, it can be reported as a legitimate disagreement. The journalist should be evaluating the credentials of sources, but not what they say, and not on the basis of whether the journalist personally agrees with them.
- hackyhacky 3y ago
  > This is how you get journalists to report disagreements that don't exist in reality. For example, whether the earth is flat, whether climate change is real, etc. Correct. Part of the problem is that there is disagreement about whether a disagreement exists. Cranks believe that there is a vigorous debate about flat-earth/evolution/climate change, and scientists don't. Do how does one (journalist) objectively determine whether an issue is settled or not?
timr 3y ago
Great comment. One thing I'll add: if you don't have specific training in the field in question, just ignore any use of "scientific consensus" to justify an argument. The "scientific consensus" trope is just dressed-up appeal to authority, and even if there is such a "consensus", it's almost never broad enough to be applicable to whatever pop-science journalism thing you're reading. Also, even within the hallowed halls of academic science, most scientists are just repeating things they've heard other people say. Unless the "consensus" is amongst scientists who have spent their entire career studying the specific question (and by "specific", I mean...hyper specific, not just "in the same field", and certainly not something meaningless like "epidemiology"), this kind of thing just devolves into a popularity contest. You'd be shocked by how many PhDs just confidently repeat whatever silly thing they saw that morning in the New York Times. Also, since I'm already seeing the meme appear...people are waaaaay too worried about "amplifying fringe voices" these days. News flash: if you don't know what you're talking about, then you can't possibly know what you should or should not be "amplifying". Stick to what you know, be modest about what you don't know (which is most things), and let the facts sort themselves out over time. Science only works if contrarians get a voice.
- giantrobot 3y ago
  > Science only works if contrarians get a voice. This is more of the rather tired "contrarians are always right" meme that seems to crop up constantly on HN. Sometimes a contrarian is right and the accepted consensus is wrong. But that doesn't happen only because the contrarian position is contrarian, it's because the contrarians brought receipts. They applied proper scientific rigor and came up with a falsifiable theory that fits empirical observations and is sufficiently predictive. They also set out to disprove their hypothesis. Not all contrarians need a "voice". It's not worth anyone's time to rebut yet another unfounded and stupid perpetual motion theorem or electric universe bullshit. It's far easier to spam stupid contrarian ideas than to produce real rigorous scientific output.
  - runesofdoom 3y ago
    As Carl Sagan said, "They laughed at Columbus, they laughed at Fulton, they laughed at the Wright brothers. But they also laughed at Bozo the Clown."
ke88y 3y ago
I agree with your prescriptions, but worry the point might've slipped by. > READ the papers The point of my original comment is that most journalists don't have the educational background required to do this. And everything else flows downstream of that problem.
- mike_hearn 3y ago
  The issue is not that they can't do it, they just don't want to. Journalists with no science background are happy to write quite technical fact checks of articles - even articles written by scientists - when those articles are contradicting something the journalists are already invested in. And often the problems don't need specialist knowledge to spot. The before/after images purporting to be of surgery in this article can be detected as fraud by anyone: https://blog.plan99.net/fake-science-part-i-7e9764571422 https://blog.plan99.net/fake-science-part-i-7e9764571422 No expertise needed. Blatant stuff like that is more obvious than we'd hope.
  - mcpackieh 3y ago
    The issue is that journalists could do better if they wanted to, but they don't want to and there's no system of incentives in place that would make them want to. There's no such thing as a journalism license they could lose, and publishing nonsensical science articles doesn't hurt their careers in any way because nobody expects better of them. After all, they're just journalists with no STEM education... Even if you somehow forced science journalists to all get STEM dual majors, it still wouldn't make them care. They'd still take the path of least resistance and pump out slop. The only way to make them care is to put them under editors that enforce standards. But how do you make a publication care enough to hire editors that care? Even state funded university press departments notoriously sloppy. If they can't uphold standards, I doubt any organization can.
    - _djo_ 3y ago
      I don't think you have any idea of the kind of time pressures journalists are under, and how little time they have to devote hours or days of deep research to each and every story they work on. At best you can get that for a large story if you're an investigative journalist, but very few have that privilege. All of the journalists I know personally would love to have more time to dig into stories, to do more background research, to spend more time following them up, and to get to understand a subject more deeply. None are given that opportunity in today's high pressure, short-staffed, and cut-to-the-bones newsrooms.
hooverd 3y ago
wrt 1., there are an unfortunate number of cranks for every discipline. It would be funny to see journos publish the emails that every faculty member who releases their email in the university phonebook gets as Serious Disagreement* though.
dekhn 3y ago
it's rare for "layman" to find true errors in papers, and you weren't a layman when you did your investigation, as you say in the article. That said, 90% of all papers contain at least one important error that brings the conclusions into question. Note that even great papers that established long-accepted truths contain important errors, see both https://en.wikipedia.org/wiki/Oil_drop_experiment#Controversy https://en.wikipedia.org/wiki/Oil_drop_experiment#Controvers... and https://en.wikipedia.org/wiki/Oil_drop_experiment#Millikan's_experiment_as_an_example_of_psychological_effects_in_scientific_methodology https://en.wikipedia.org/wiki/Oil_drop_experiment#Millikan's... Similarly, the original 3D structure of DNA from W&C was actually "wrong" but I truly doubt any laymen (laypeople) could have determined that by reading the original paper (which is a paragon of clear and simple scientific reporting).
- mike_hearn 3y ago
  Although it's true I wasn't a layman, it turned out my own expertise in bot fighting had no utility to reviewing the paper. The review was done as a "layman", because the sociological subfield of twitter bot research has nothing to do with bots :( Anyone could have written that and actually most of the paper takedowns in that subfield are done by a team of a CS prof (with no specialism in bots) and a data journalist.
  - dekhn 3y ago
    Understood. At best, though, I think these takedowns are based primarily on "technical invalidation": identification of one or more faults in logic, presumption, study design, or execution, which brings enough question to the results/conclusions that it's best just to assume the paper isn't worth reading. (put another way: many papers that contributed what appears to be new, true knowledge contain factual errors, some of them invalidating, yet the paper turned out to be correct, in terms of its conclusions, in the long run)

thorncorona 3y ago

> 1. Report the fact that disagreement exists. Phrased differently, stop taking academics at their word when they claim there's a consensus. Do some basic web searches to find people who disagree. Get quotes from them. Stress on the word people; not just other academics but literally anyone disagreeing on scientific grounds eliminates the claim of consensus. Bloggers are fine. This is how you get journalists to report disagreements that don't exist in reality. For example, whether the earth is flat, whether climate change is real, etc. > 2. READ the papers. Literally nobody reads the papers. A huge amount of news isn't even investigated. A large number of news these days is regurgitated from other sources.

mike_hearn 3y ago
Journalists love reporting on flat Earthers even though their beliefs have no impact on anything in the real world: https://news.google.com/search?q=%22flat%20earth%22&hl=en-US&gl=US&ceid=US%3Aen https://news.google.com/search?q=%22flat%20earth%22&hl=en-US... Realistically, journalists like to report on fringe or weird beliefs so they can laugh at the people holding them, and dislike reporting on serious disagreement with things they want to be true. Agree that almost nobody is reading the papers, outside of random tweeters and bloggers. Journalists might as well start, though. Reading obscure documents is a part of the job, classically at least.
SoftTalker 3y ago
You can still look at credentials. If there is disagreement among people with appropriate background, experience, and education, it can be reported as a legitimate disagreement. The journalist should be evaluating the credentials of sources, but not what they say, and not on the basis of whether the journalist personally agrees with them.
hackyhacky 3y ago
> This is how you get journalists to report disagreements that don't exist in reality. For example, whether the earth is flat, whether climate change is real, etc. Correct. Part of the problem is that there is disagreement about whether a disagreement exists. Cranks believe that there is a vigorous debate about flat-earth/evolution/climate change, and scientists don't. Do how does one (journalist) objectively determine whether an issue is settled or not?

timr 3y ago

Great comment. One thing I'll add: if you don't have specific training in the field in question, just ignore any use of "scientific consensus" to justify an argument. The "scientific consensus" trope is just dressed-up appeal to authority, and even if there is such a "consensus", it's almost never broad enough to be applicable to whatever pop-science journalism thing you're reading. Also, even within the hallowed halls of academic science, most scientists are just repeating things they've heard other people say. Unless the "consensus" is amongst scientists who have spent their entire career studying the specific question (and by "specific", I mean...hyper specific, not just "in the same field", and certainly not something meaningless like "epidemiology"), this kind of thing just devolves into a popularity contest. You'd be shocked by how many PhDs just confidently repeat whatever silly thing they saw that morning in the New York Times. Also, since I'm already seeing the meme appear...people are waaaaay too worried about "amplifying fringe voices" these days. News flash: if you don't know what you're talking about, then you can't possibly know what you should or should not be "amplifying". Stick to what you know, be modest about what you don't know (which is most things), and let the facts sort themselves out over time. Science only works if contrarians get a voice.

giantrobot 3y ago
> Science only works if contrarians get a voice. This is more of the rather tired "contrarians are always right" meme that seems to crop up constantly on HN. Sometimes a contrarian is right and the accepted consensus is wrong. But that doesn't happen only because the contrarian position is contrarian, it's because the contrarians brought receipts. They applied proper scientific rigor and came up with a falsifiable theory that fits empirical observations and is sufficiently predictive. They also set out to disprove their hypothesis. Not all contrarians need a "voice". It's not worth anyone's time to rebut yet another unfounded and stupid perpetual motion theorem or electric universe bullshit. It's far easier to spam stupid contrarian ideas than to produce real rigorous scientific output.
- runesofdoom 3y ago
  As Carl Sagan said, "They laughed at Columbus, they laughed at Fulton, they laughed at the Wright brothers. But they also laughed at Bozo the Clown."

ke88y 3y ago

I agree with your prescriptions, but worry the point might've slipped by. > READ the papers The point of my original comment is that most journalists don't have the educational background required to do this. And everything else flows downstream of that problem.

mike_hearn 3y ago
The issue is not that they can't do it, they just don't want to. Journalists with no science background are happy to write quite technical fact checks of articles - even articles written by scientists - when those articles are contradicting something the journalists are already invested in. And often the problems don't need specialist knowledge to spot. The before/after images purporting to be of surgery in this article can be detected as fraud by anyone: https://blog.plan99.net/fake-science-part-i-7e9764571422 https://blog.plan99.net/fake-science-part-i-7e9764571422 No expertise needed. Blatant stuff like that is more obvious than we'd hope.
- mcpackieh 3y ago
  The issue is that journalists could do better if they wanted to, but they don't want to and there's no system of incentives in place that would make them want to. There's no such thing as a journalism license they could lose, and publishing nonsensical science articles doesn't hurt their careers in any way because nobody expects better of them. After all, they're just journalists with no STEM education... Even if you somehow forced science journalists to all get STEM dual majors, it still wouldn't make them care. They'd still take the path of least resistance and pump out slop. The only way to make them care is to put them under editors that enforce standards. But how do you make a publication care enough to hire editors that care? Even state funded university press departments notoriously sloppy. If they can't uphold standards, I doubt any organization can.
  - _djo_ 3y ago
    I don't think you have any idea of the kind of time pressures journalists are under, and how little time they have to devote hours or days of deep research to each and every story they work on. At best you can get that for a large story if you're an investigative journalist, but very few have that privilege. All of the journalists I know personally would love to have more time to dig into stories, to do more background research, to spend more time following them up, and to get to understand a subject more deeply. None are given that opportunity in today's high pressure, short-staffed, and cut-to-the-bones newsrooms.

hooverd 3y ago

wrt 1., there are an unfortunate number of cranks for every discipline. It would be funny to see journos publish the emails that every faculty member who releases their email in the university phonebook gets as Serious Disagreement* though.

dekhn 3y ago

it's rare for "layman" to find true errors in papers, and you weren't a layman when you did your investigation, as you say in the article. That said, 90% of all papers contain at least one important error that brings the conclusions into question. Note that even great papers that established long-accepted truths contain important errors, see both https://en.wikipedia.org/wiki/Oil_drop_experiment#Controversy https://en.wikipedia.org/wiki/Oil_drop_experiment#Controvers... and https://en.wikipedia.org/wiki/Oil_drop_experiment#Millikan's_experiment_as_an_example_of_psychological_effects_in_scientific_methodology https://en.wikipedia.org/wiki/Oil_drop_experiment#Millikan's... Similarly, the original 3D structure of DNA from W&C was actually "wrong" but I truly doubt any laymen (laypeople) could have determined that by reading the original paper (which is a paragon of clear and simple scientific reporting).

mike_hearn 3y ago
Although it's true I wasn't a layman, it turned out my own expertise in bot fighting had no utility to reviewing the paper. The review was done as a "layman", because the sociological subfield of twitter bot research has nothing to do with bots :( Anyone could have written that and actually most of the paper takedowns in that subfield are done by a team of a CS prof (with no specialism in bots) and a data journalist.
- dekhn 3y ago
  Understood. At best, though, I think these takedowns are based primarily on "technical invalidation": identification of one or more faults in logic, presumption, study design, or execution, which brings enough question to the results/conclusions that it's best just to assume the paper isn't worth reading. (put another way: many papers that contributed what appears to be new, true knowledge contain factual errors, some of them invalidating, yet the paper turned out to be correct, in terms of its conclusions, in the long run)