4 ms·

I'm probably being too harsh but... Good on them for writing this, it's important to admit when you're wrong. However, I feel like this outlet has a larger res

I'm probably being too harsh but... Good on them for writing this, it's important to admit when you're wrong. However, I feel like this outlet has a larger responsibility to be actual data analysts as well as journalists (over say a more traditional journalist for a more traditional news outlet). As such, why was the analysis done in the postmortem article not done prior to publishing the original articles? A good analyst is one you can trust, and trust for an analyst is built by drawing conclusions from highly defensible data, and highly defensible data is data which has undergone sever scrutiny of the analyst before conclusions are drawn, not after. Also, they should probably update the now erroneous articles with a disclaimer indicating that much of the research is now invalid.

misterbowfinger 9y ago
I think the idea is that we all make mistakes. The best we can do is admit them and learn from them, just like any other postmortem. Also, they did update the erroneous articles: https://fivethirtyeight.com/features/lots-of-people-in-cities-still-cant-afford-broadband/ https://fivethirtyeight.com/features/lots-of-people-in-citie... https://fivethirtyeight.com/features/the-worst-internet-in-america/ https://fivethirtyeight.com/features/the-worst-internet-in-a...
- cpsempek 9y ago
  Also, they did update the erroneous articles: Ah so they did, my mistake. I think the idea is that we all make mistakes. I am in total agreement, we all do make mistakes, and it's refreshing when people own up to them. I guess my main question still stands though. As a top-tier data journalism outlet, why was a data QA not performed prior to analysis, or, before publishing results? And if there was data QA performed, where did it fail? It just seems that they got bit by a time crunch to publish content amid the FCC repeal and sloppy analysis was the result.
  - fjsolwmv 9y ago
    Everyone can do better. It's impossible to research every story from first principles of physics.
  - taoistextremist 9y ago
    They probably just didn't think of this possible flaw in data. I mean, it's not like the university researchers caught wind of any problem, either.
olympus 9y ago
It's true that we have lost trust in their analysis, but there is an important thing to remember here: fivethirtyeight is not a peer reviewed journal. You should treat their articles in roughly the same category as a CNN article on something Trump tweeted, not in the category as an article published in Nature. While they do tend to do better data analysis than the Associated Press, 538 articles are not refereed and should not be treated as such. The assumption of a "highly defensible" data analysis is a little strong for them. This is very apparent on their sports section, which should give you a clue about the actual rigor of their analyses. Don't cite a 538 article in your academic research unless the 538 article is directly citing peer-reviewed research (and even then you should probably just cite the original research). If you think of them as a news organization that uses data as it's gimmick to sell page views, you will be less surprised at events like this (disappointed, yes- surprised, no). They have the same incentive to sensationalize things as a regular news organization. Their mission is not to increase the knowledge of the human race, their mission is to bring in page views to sell ads and make money. It may sound like I'm coming down harsh on fivethirtyeight. I genuinely enjoy reading some of their articles, but I make sure to remember what kind of organization they actually are and don't fall for the trap of thinking they are a think tank staffed with postdocs.
- TheCowboy 9y ago
  This anchors motives too much in economic determinism. It's a useful mental model to remember this is a factor, and it can come to dominate and cause problems, but it is not always the one true factor to rule them all through which we should view all motives. You basically end up arguing that it is all about money, and real journalism cannot happen under profit-seeking organizations. It also trivializes that journalism's big challenge right now is how to do real journalism when the big tech players have vacuumed up their ad revenue. I also wouldn't treat 538 as the same as CNN writing about a thing Trump tweeted. They actively talk about and discuss their journalistic goals. They try to be openly self-critical about what and how they cover topics. They are trying to compete by not doing the same thing as other organizations.
  - olympus 9y ago

misterbowfinger 9y ago

I think the idea is that we all make mistakes. The best we can do is admit them and learn from them, just like any other postmortem. Also, they did update the erroneous articles: https://fivethirtyeight.com/features/lots-of-people-in-cities-still-cant-afford-broadband/ https://fivethirtyeight.com/features/lots-of-people-in-citie... https://fivethirtyeight.com/features/the-worst-internet-in-america/ https://fivethirtyeight.com/features/the-worst-internet-in-a...

cpsempek 9y ago
Also, they did update the erroneous articles: Ah so they did, my mistake. I think the idea is that we all make mistakes. I am in total agreement, we all do make mistakes, and it's refreshing when people own up to them. I guess my main question still stands though. As a top-tier data journalism outlet, why was a data QA not performed prior to analysis, or, before publishing results? And if there was data QA performed, where did it fail? It just seems that they got bit by a time crunch to publish content amid the FCC repeal and sloppy analysis was the result.
- fjsolwmv 9y ago
  Everyone can do better. It's impossible to research every story from first principles of physics.
- taoistextremist 9y ago
  They probably just didn't think of this possible flaw in data. I mean, it's not like the university researchers caught wind of any problem, either.

olympus 9y ago

It's true that we have lost trust in their analysis, but there is an important thing to remember here: fivethirtyeight is not a peer reviewed journal. You should treat their articles in roughly the same category as a CNN article on something Trump tweeted, not in the category as an article published in Nature. While they do tend to do better data analysis than the Associated Press, 538 articles are not refereed and should not be treated as such. The assumption of a "highly defensible" data analysis is a little strong for them. This is very apparent on their sports section, which should give you a clue about the actual rigor of their analyses. Don't cite a 538 article in your academic research unless the 538 article is directly citing peer-reviewed research (and even then you should probably just cite the original research). If you think of them as a news organization that uses data as it's gimmick to sell page views, you will be less surprised at events like this (disappointed, yes- surprised, no). They have the same incentive to sensationalize things as a regular news organization. Their mission is not to increase the knowledge of the human race, their mission is to bring in page views to sell ads and make money. It may sound like I'm coming down harsh on fivethirtyeight. I genuinely enjoy reading some of their articles, but I make sure to remember what kind of organization they actually are and don't fall for the trap of thinking they are a think tank staffed with postdocs.

TheCowboy 9y ago
This anchors motives too much in economic determinism. It's a useful mental model to remember this is a factor, and it can come to dominate and cause problems, but it is not always the one true factor to rule them all through which we should view all motives. You basically end up arguing that it is all about money, and real journalism cannot happen under profit-seeking organizations. It also trivializes that journalism's big challenge right now is how to do real journalism when the big tech players have vacuumed up their ad revenue. I also wouldn't treat 538 as the same as CNN writing about a thing Trump tweeted. They actively talk about and discuss their journalistic goals. They try to be openly self-critical about what and how they cover topics. They are trying to compete by not doing the same thing as other organizations.
- olympus 9y ago