4 ms·

If you were a mapmaker hired to make maps of the city for people, yes you should absolutely be shamed for trusting some random guy on the street without verifyi

by imgabe 4y ago

If you were a mapmaker hired to make maps of the city for people, yes you should absolutely be shamed for trusting some random guy on the street without verifying it. What the study exposed is that science journalists, whose literal job is to understand this stuff and explain it to the general public, suck at their jobs. You absolutely can blame them for just publishing whatever anybody says without verifying it.

dataflow 4y ago
I think you missed my 2nd paragraph. What I was saying was: no, this article didn't show that. The only thing it's exposed so far is that "science journalists don't watch out for this vulnerability". Well, so what if they don't? There are a million things people don't watch out for. Is this one actually causing problems in reality, or is this a hypothetical thing that doesn't occur in practice? To show this is a real problem requires providing evidence of real-world p-hacking in the wild that has had significant actual adverse consequences.
- mixologic 4y ago
  That was the whole point. Science journalists dont do the jobs we think they do. Our presumption is that they do not trust anything as true facts until they verify and vet the information. I.e do what a journalists do. Report whats true But thats not actually their job anymore. Their job is to get eyeballs. Doesnt really matter whats true.
  - lupire 4y ago
    In no sense is this limited to science journalism. Bizarrely, the most reputable journalist are probably at TMZ, because they get sued for libel if they are wrong. When there is no cost for being wrong, there is no incentive for truth.
- imgabe 4y ago
  No, the prank pointed out multiple failures in different stages of the process. > But even if we had been careful to avoid p-hacking, our study was doomed by the tiny number of subjects, which amplifies the effects of uncontrolled factors. ... Which is why you need to use a large number of people, and balance age and gender across treatment groups. (We didn’t bother.) This poor design should have been reason enough not to publish the study - p-hacking or not. But... > Our paper was accepted for publication by multiple journals within 24 hours. Needless to say, we faced no peer review at all. The eager suitor we ultimately chose was the the International Archives of Medicine. It got published anyway in a "journal" with a fancy sounding name that doesn't actually do any rigorous peer review. The team then issued a press release that the press regurgitated without any scrutiny. If they had bothered to look and if they were good at their jobs, they might have noticed that the journal was not reputable. Anybody familiar with any academic field can recite off the top of their head what the reputable journals they'd want to be published in are. If it's not one of those, it deserves a closer look. That's without even doing some basic googling about the team behind the research which should have been able to uncover that they were nothing but a website.
  - lupire 4y ago
    The problem here is a breakdown in society. Why do sham journals exist? Why do their press release reach the ears of journalists? Why is there no web of trust of credibility? Why have all our institutions (flawed as they may have been) collapsed?
meowkit 4y ago
> What the study exposed is that science journalists, whose literal job is to understand this stuff and explain it to the general public, suck at their jobs. I was under their impression the job of a journalist was to get attention of readers to generate ad and subscription revenue. Not that I disagree with the claim that they should explain things to a general public to accomplish the above, but that's not how our system is currently set up. If doing a poor job makes them more money - that's the real issue.
- orzig 4y ago
  I’m related to a well-known science journalist. They absolutely consider it their job to verify. There are irresponsible people who try to cram in more visible output by skimping on less visible quality. Everyone else grumbles about those reporters behind their back, like a developer whose code is undocumented and insecure. Setting incentives to deter that behavior is management’s responsibility, as in any industry. Since it’s an open secret in the business: The New York Times Science section has failed to do this in notable situations over the last decade.
- coldtea 4y ago
  >I was under their impression the job of a journalist was to get attention of readers to generate ad and subscription revenue. That's because cynicism and acceptance of brokeness has become the norm. But while journalism did start out as the "wild west" of yellow press and cheap wriging, and while it's that way today... ... there was also a period of many decades where people expected better of journalists, and they did get considerable better results...
ouid 4y ago
This is not the job of science journalists. The job of science journalists is to make money for someone. The honor system is just the name that people who dont understand corruption give to no system at all. If you want journalism to be better, you probably have to make advertising and PR illegal.
- coldtea 4y ago
  >This is not the job of science journalists. The job of science journalists is to make money for someone Well, if it isn't their actual job, it should be. And they should get no sympathy when they get it wrong.

by imgabe 4y ago

dataflow 4y ago

I think you missed my 2nd paragraph. What I was saying was: no, this article didn't show that. The only thing it's exposed so far is that "science journalists don't watch out for this vulnerability". Well, so what if they don't? There are a million things people don't watch out for. Is this one actually causing problems in reality, or is this a hypothetical thing that doesn't occur in practice? To show this is a real problem requires providing evidence of real-world p-hacking in the wild that has had significant actual adverse consequences.

mixologic 4y ago
That was the whole point. Science journalists dont do the jobs we think they do. Our presumption is that they do not trust anything as true facts until they verify and vet the information. I.e do what a journalists do. Report whats true But thats not actually their job anymore. Their job is to get eyeballs. Doesnt really matter whats true.
- lupire 4y ago
  In no sense is this limited to science journalism. Bizarrely, the most reputable journalist are probably at TMZ, because they get sued for libel if they are wrong. When there is no cost for being wrong, there is no incentive for truth.
imgabe 4y ago
No, the prank pointed out multiple failures in different stages of the process. > But even if we had been careful to avoid p-hacking, our study was doomed by the tiny number of subjects, which amplifies the effects of uncontrolled factors. ... Which is why you need to use a large number of people, and balance age and gender across treatment groups. (We didn’t bother.) This poor design should have been reason enough not to publish the study - p-hacking or not. But... > Our paper was accepted for publication by multiple journals within 24 hours. Needless to say, we faced no peer review at all. The eager suitor we ultimately chose was the the International Archives of Medicine. It got published anyway in a "journal" with a fancy sounding name that doesn't actually do any rigorous peer review. The team then issued a press release that the press regurgitated without any scrutiny. If they had bothered to look and if they were good at their jobs, they might have noticed that the journal was not reputable. Anybody familiar with any academic field can recite off the top of their head what the reputable journals they'd want to be published in are. If it's not one of those, it deserves a closer look. That's without even doing some basic googling about the team behind the research which should have been able to uncover that they were nothing but a website.
- lupire 4y ago
  The problem here is a breakdown in society. Why do sham journals exist? Why do their press release reach the ears of journalists? Why is there no web of trust of credibility? Why have all our institutions (flawed as they may have been) collapsed?

meowkit 4y ago

> What the study exposed is that science journalists, whose literal job is to understand this stuff and explain it to the general public, suck at their jobs. I was under their impression the job of a journalist was to get attention of readers to generate ad and subscription revenue. Not that I disagree with the claim that they should explain things to a general public to accomplish the above, but that's not how our system is currently set up. If doing a poor job makes them more money - that's the real issue.

orzig 4y ago
I’m related to a well-known science journalist. They absolutely consider it their job to verify. There are irresponsible people who try to cram in more visible output by skimping on less visible quality. Everyone else grumbles about those reporters behind their back, like a developer whose code is undocumented and insecure. Setting incentives to deter that behavior is management’s responsibility, as in any industry. Since it’s an open secret in the business: The New York Times Science section has failed to do this in notable situations over the last decade.
coldtea 4y ago
>I was under their impression the job of a journalist was to get attention of readers to generate ad and subscription revenue. That's because cynicism and acceptance of brokeness has become the norm. But while journalism did start out as the "wild west" of yellow press and cheap wriging, and while it's that way today... ... there was also a period of many decades where people expected better of journalists, and they did get considerable better results...

ouid 4y ago

This is not the job of science journalists. The job of science journalists is to make money for someone. The honor system is just the name that people who dont understand corruption give to no system at all. If you want journalism to be better, you probably have to make advertising and PR illegal.

coldtea 4y ago
>This is not the job of science journalists. The job of science journalists is to make money for someone Well, if it isn't their actual job, it should be. And they should get no sympathy when they get it wrong.