4 ms·

It's true that we have lost trust in their analysis, but there is an important thing to remember here: fivethirtyeight is not a peer reviewed journal. You shou

It's true that we have lost trust in their analysis, but there is an important thing to remember here: fivethirtyeight is not a peer reviewed journal. You should treat their articles in roughly the same category as a CNN article on something Trump tweeted, not in the category as an article published in Nature. While they do tend to do better data analysis than the Associated Press, 538 articles are not refereed and should not be treated as such. The assumption of a "highly defensible" data analysis is a little strong for them. This is very apparent on their sports section, which should give you a clue about the actual rigor of their analyses. Don't cite a 538 article in your academic research unless the 538 article is directly citing peer-reviewed research (and even then you should probably just cite the original research). If you think of them as a news organization that uses data as it's gimmick to sell page views, you will be less surprised at events like this (disappointed, yes- surprised, no). They have the same incentive to sensationalize things as a regular news organization. Their mission is not to increase the knowledge of the human race, their mission is to bring in page views to sell ads and make money. It may sound like I'm coming down harsh on fivethirtyeight. I genuinely enjoy reading some of their articles, but I make sure to remember what kind of organization they actually are and don't fall for the trap of thinking they are a think tank staffed with postdocs.

TheCowboy 9y ago
This anchors motives too much in economic determinism. It's a useful mental model to remember this is a factor, and it can come to dominate and cause problems, but it is not always the one true factor to rule them all through which we should view all motives. You basically end up arguing that it is all about money, and real journalism cannot happen under profit-seeking organizations. It also trivializes that journalism's big challenge right now is how to do real journalism when the big tech players have vacuumed up their ad revenue. I also wouldn't treat 538 as the same as CNN writing about a thing Trump tweeted. They actively talk about and discuss their journalistic goals. They try to be openly self-critical about what and how they cover topics. They are trying to compete by not doing the same thing as other organizations.
- olympus 9y ago
  I think that real journalism can happen under profit seeking organizations, because people find value in real journalism. However, real journalists don't do original research, they compile insights from experts. Journalists have a tendency to get it wrong when they try to add original insight in an article. Just about every person who is an expert in a field has a story about when their field made it into the news cycle and the journalists butchered some important concept. I'm a statistician, and we always work with investigators that are experts in their field- unless we are researching statistical methods, then we act as our own experts. The statisticians handle the data analysis and make sure that the investigators don't make silly data mistakes. The investigators handle the reasoning and mechanisms behind the research. When they work together they can collect good data that they are familiar with and know how to interpret correctly. When they work separately they are prone to mistakes. Fivethirtyeight (and also the ASU researchers) fell into this trap. They were not involved in the data collection, so they don't really know what each variable means, and just took someone's word for it who, (back to the economic motives) has an incentive to sell their dataset and not to tell the researchers to go look elsewhere. I'll admit I was too harsh in comparing fivethirtyeight to an article about Trump's tweet, 538 is typically decent investigative journalism. However, I maintain that it isn't on the level of peer-reviewed academic research. Articles on 538 don't go through peer review. They aren't submitted and languish for months with revisions and answering follow up questions. I'm not saying that this mistake would have been caught by a referee, but in the peer-review process a referee could have caught it by asking questions about the analysis process and if it's valid to use a variable named "X" was being used as a proxy for a variable named "broadband access."
  - azeotropic 9y ago
    >I think that real journalism can happen under profit seeking organizations, because people find value in real journalism. However, real journalists don't do original research, they compile insights from experts. Journalists have a tendency to get it wrong when they try to add original insight in an article. Just about every person who is an expert in a field has a story about when their field made it into the news cycle and the journalists butchered some important concept. It's much much worse than this. I was once interviewed about my research by an NPR reporter. He had already decided what his story was about and tried a variety of tricks get me to say some pithy quote he had devised so he could use it on air. The problem was that my research actually debunked, rather that supported, the story he wanted to run, and the pithy quote was scientifically unsupportable. I often wonder whether some of the quotes I hear on NPR are edited versions of the interviewees saying "no it wouldn't be accurate to say x" cut down to "x".
cpsempek 9y ago
I think this is fair and is a sound refutation to my stance on how rigorous they should be. Your reply reads like a debearding-of-santa, which is maybe what I needed in this case.
VintageCool 9y ago
I expect way more of FiveThirtyEight articles than I expect of CNN articles, and I value FiveThirtyEight articles much more than I value CNN articles.

TheCowboy 9y ago

This anchors motives too much in economic determinism. It's a useful mental model to remember this is a factor, and it can come to dominate and cause problems, but it is not always the one true factor to rule them all through which we should view all motives. You basically end up arguing that it is all about money, and real journalism cannot happen under profit-seeking organizations. It also trivializes that journalism's big challenge right now is how to do real journalism when the big tech players have vacuumed up their ad revenue. I also wouldn't treat 538 as the same as CNN writing about a thing Trump tweeted. They actively talk about and discuss their journalistic goals. They try to be openly self-critical about what and how they cover topics. They are trying to compete by not doing the same thing as other organizations.

olympus 9y ago
I think that real journalism can happen under profit seeking organizations, because people find value in real journalism. However, real journalists don't do original research, they compile insights from experts. Journalists have a tendency to get it wrong when they try to add original insight in an article. Just about every person who is an expert in a field has a story about when their field made it into the news cycle and the journalists butchered some important concept. I'm a statistician, and we always work with investigators that are experts in their field- unless we are researching statistical methods, then we act as our own experts. The statisticians handle the data analysis and make sure that the investigators don't make silly data mistakes. The investigators handle the reasoning and mechanisms behind the research. When they work together they can collect good data that they are familiar with and know how to interpret correctly. When they work separately they are prone to mistakes. Fivethirtyeight (and also the ASU researchers) fell into this trap. They were not involved in the data collection, so they don't really know what each variable means, and just took someone's word for it who, (back to the economic motives) has an incentive to sell their dataset and not to tell the researchers to go look elsewhere. I'll admit I was too harsh in comparing fivethirtyeight to an article about Trump's tweet, 538 is typically decent investigative journalism. However, I maintain that it isn't on the level of peer-reviewed academic research. Articles on 538 don't go through peer review. They aren't submitted and languish for months with revisions and answering follow up questions. I'm not saying that this mistake would have been caught by a referee, but in the peer-review process a referee could have caught it by asking questions about the analysis process and if it's valid to use a variable named "X" was being used as a proxy for a variable named "broadband access."
- azeotropic 9y ago
  >I think that real journalism can happen under profit seeking organizations, because people find value in real journalism. However, real journalists don't do original research, they compile insights from experts. Journalists have a tendency to get it wrong when they try to add original insight in an article. Just about every person who is an expert in a field has a story about when their field made it into the news cycle and the journalists butchered some important concept. It's much much worse than this. I was once interviewed about my research by an NPR reporter. He had already decided what his story was about and tried a variety of tricks get me to say some pithy quote he had devised so he could use it on air. The problem was that my research actually debunked, rather that supported, the story he wanted to run, and the pithy quote was scientifically unsupportable. I often wonder whether some of the quotes I hear on NPR are edited versions of the interviewees saying "no it wouldn't be accurate to say x" cut down to "x".

cpsempek 9y ago

I think this is fair and is a sound refutation to my stance on how rigorous they should be. Your reply reads like a debearding-of-santa, which is maybe what I needed in this case.

VintageCool 9y ago

I expect way more of FiveThirtyEight articles than I expect of CNN articles, and I value FiveThirtyEight articles much more than I value CNN articles.