5 ms·

It would be telling, however, to quantify the reliability of those “years of reliable journalism” by fact checking a random sample of the stories told over thos

It would be telling, however, to quantify the reliability of those “years of reliable journalism” by fact checking a random sample of the stories told over those years. According to the article we are discussing, TAL started using professional fact checkers only after the discovery of the Daisey incident. We’re assuming that stories aired prior to that event are reliable, but we haven’t verified that belief, have we?

glenstein 2y ago
So that's exactly the kind of over correction in the wrong direction that I'm talking about. I don't think I agree that that's the pertinent extrapolation here. We absolutely would benefit from that spot checking. But I don't think the implication should be that 100% or something near it of the previous articles are fabricated, or under the cloud of deep suspicion until proven otherwise. The same things that led to this particular story unraveling, are vulnerabilities that could have led to other stories unraveling. If we get a second and a third, I think you might be right to have that cloud of suspicion. That would be like a Shattered Glass scenario and we're not there yet. https://en.m.wikipedia.org/wiki/Shattered_Glass_(film) https://en.m.wikipedia.org/wiki/Shattered_Glass_(film)
- II2II 2y ago
  Suggesting that something is unreliable is not the same as saying that it is consistently fabricated. As they say, a broken clock is still correct twice a day. To choose something less extreme: an unreliable employee may still show up for work 80% of the time. When their methodology prior to a particular point of time was shown to be weak, and the programs from that time are still available (the archives go back 30 years), I think that asking for spot checking of those old episodes is legitimate. The key thing here are the archives. If the archives weren't available it would be much easier to shrug and say, "live and learn."
  - stickfigure 2y ago
    There's reason to be more optimistic than this. There is, to some extent, an automatic post-hoc fact checking process built in to being such a high profile publication. For example, the Daisy narrative was challenged because someone with some personal knowledge of the subject heard the show. It's much better and less embarrassing to get the fact checking right before publication, but the truth generally comes out one way or another. So I'm willing to give historic TAL... not certainty, but at least the benefit of the doubt.
  - glenstein 2y ago
    >Suggesting that something is unreliable is not the same as saying that it is consistently fabricated. I generally understand that to be true, but that was not the upshot of the point being made by the other commenter. Their extrapolation was a much more along the lines of treating it like an open question whether the other stories were fabricated at a level of elevated suspicion that calls for spot checking. All the other stories were vulnerable to being upended just like this one and seem to have withstood the test of time. I also think that despite this particular story falling apart, TAL has a track record of credibility and vetting that is more legitimate than is being implied by casting doubt over the history of theirs, and I did contrast it to the case of Stephen Glass, which model conditions where that degree of skepticism is more appropriately warranted.
- systemstops 2y ago
  Perhaps instead of fact checking, we could compare the previous narratives with known public knowledge of the events, to determine if the narratives provided an accurate view of reality. It is easy to distort the truth and still tell no lies.
- tmoertel 2y ago
  I made no extrapolation. I said that we haven't measured the reliability of the earlier stories. Instead of having a reliable measurement, we're going with our much-less-reliable assumptions (prior beliefs). The interesting thing, I think, is those prior beliefs. Your prior beliefs, it would seem, include the belief TAL's stories are generally reliable. You believe it strongly enough that you write that doubting the reliability of earlier TAL stories is an "over correction in the wrong direction." I don't think it's an overcorrection. When we find evidence of one fabrication from a trusted source, that source ought to lose trust. If we want to know how much trust it should lose, we have to measure. When we don't measure, when we don't take seriously the responsibility to ground our beliefs, that's how we end up with things like the replication crisis in psychology.
  - glenstein 2y ago
    You're extrapolating from the Daisy incident to doubt of their previous reporting. And believe that the Daisy incident merits responding by holding TAL to a newly escalated standard for verification. From this most recent comment of yours, you seem comfortable with "doubting the reliability of earlier TAL stories" as a position you view to be not an over correction. I think you're underestimating how an extreme a position that is, and kind of equivocating between ordinary skepticism and doubting the veracity of their previous stories. You're right that I believe TAL's stories are generally reliable, that I believe doubting them is an over-correction in the wrong direction. I also don't think I agree that it's simply a matter of checking or not checking because I believe the vast body of work that's been free from error, although exposed to the same conditions of public scrutiny that could have revealed error in just the same way as with the Daisy story, is part of the body of evidence that actively testifies in favor of TAL. And I do think if there was more of a rocky track record, or if there proves to be more of one in the future, it absolutely could merit spot checking. And as I've said twice now already, I've given the example of Stephen Glass as a case where that skepticism was warranted. You seem to be implying that I have a categorical opposition to spot checking which couldn't be further from the truth. I just don't think it's warranted in this instance, because it's not a reasonable extrapolation from what happened with the Daisy story.
    - 2y ago
spondylosaurus 2y ago
So... I can speak to this because I've listened to almost the entire TAL back catalog, lol. The show has a stronger journalistic focus now than it did 10 years ago and _way_ more than it did 20 or (almost!) 30 years ago. It's always been part of the show—people who complain that TAL "didn't used to be political" clearly don't remember how many segments they ran about the Iraq war—but for the first decade or so the show had a very strong focus on the arts; they'd have a lot of guests sharing personal essays, short fiction, etc. The serious journalism-type stories were generally either (1) on-the-ground reporting from their own staff, like a really great episode where they toured an aircraft carrier (https://www.thisamericanlife.org/206/somewhere-in-the-arabian-sea https://www.thisamericanlife.org/206/somewhere-in-the-arabia...), (2) stories sourced from other journalists or professional organizations who had serious reputation to lose if they were caught in the supply chain of misinformation, or (3) "this is a thing that happened to me"-type firsthand accounts and observations. And I guess the now-debunked Apple factory story falls into the third category, but the types of stories that had previously fallen into that category tended to be far smaller in scale and/or were presented as more subjective than the Apple story had been. All of that to say that I think you raise a valid question, and I'm sure some stuff slipped through the cracks over the years, but I also think the implication of that crack-slipping was far less dire in the show's earlier days, and there were just fewer news-ish stories overall.

glenstein 2y ago

So that's exactly the kind of over correction in the wrong direction that I'm talking about. I don't think I agree that that's the pertinent extrapolation here. We absolutely would benefit from that spot checking. But I don't think the implication should be that 100% or something near it of the previous articles are fabricated, or under the cloud of deep suspicion until proven otherwise. The same things that led to this particular story unraveling, are vulnerabilities that could have led to other stories unraveling. If we get a second and a third, I think you might be right to have that cloud of suspicion. That would be like a Shattered Glass scenario and we're not there yet. https://en.m.wikipedia.org/wiki/Shattered_Glass_(film) https://en.m.wikipedia.org/wiki/Shattered_Glass_(film)

II2II 2y ago
Suggesting that something is unreliable is not the same as saying that it is consistently fabricated. As they say, a broken clock is still correct twice a day. To choose something less extreme: an unreliable employee may still show up for work 80% of the time. When their methodology prior to a particular point of time was shown to be weak, and the programs from that time are still available (the archives go back 30 years), I think that asking for spot checking of those old episodes is legitimate. The key thing here are the archives. If the archives weren't available it would be much easier to shrug and say, "live and learn."
- stickfigure 2y ago
  There's reason to be more optimistic than this. There is, to some extent, an automatic post-hoc fact checking process built in to being such a high profile publication. For example, the Daisy narrative was challenged because someone with some personal knowledge of the subject heard the show. It's much better and less embarrassing to get the fact checking right before publication, but the truth generally comes out one way or another. So I'm willing to give historic TAL... not certainty, but at least the benefit of the doubt.
- glenstein 2y ago
  >Suggesting that something is unreliable is not the same as saying that it is consistently fabricated. I generally understand that to be true, but that was not the upshot of the point being made by the other commenter. Their extrapolation was a much more along the lines of treating it like an open question whether the other stories were fabricated at a level of elevated suspicion that calls for spot checking. All the other stories were vulnerable to being upended just like this one and seem to have withstood the test of time. I also think that despite this particular story falling apart, TAL has a track record of credibility and vetting that is more legitimate than is being implied by casting doubt over the history of theirs, and I did contrast it to the case of Stephen Glass, which model conditions where that degree of skepticism is more appropriately warranted.
systemstops 2y ago
Perhaps instead of fact checking, we could compare the previous narratives with known public knowledge of the events, to determine if the narratives provided an accurate view of reality. It is easy to distort the truth and still tell no lies.
tmoertel 2y ago
I made no extrapolation. I said that we haven't measured the reliability of the earlier stories. Instead of having a reliable measurement, we're going with our much-less-reliable assumptions (prior beliefs). The interesting thing, I think, is those prior beliefs. Your prior beliefs, it would seem, include the belief TAL's stories are generally reliable. You believe it strongly enough that you write that doubting the reliability of earlier TAL stories is an "over correction in the wrong direction." I don't think it's an overcorrection. When we find evidence of one fabrication from a trusted source, that source ought to lose trust. If we want to know how much trust it should lose, we have to measure. When we don't measure, when we don't take seriously the responsibility to ground our beliefs, that's how we end up with things like the replication crisis in psychology.
- glenstein 2y ago
  You're extrapolating from the Daisy incident to doubt of their previous reporting. And believe that the Daisy incident merits responding by holding TAL to a newly escalated standard for verification. From this most recent comment of yours, you seem comfortable with "doubting the reliability of earlier TAL stories" as a position you view to be not an over correction. I think you're underestimating how an extreme a position that is, and kind of equivocating between ordinary skepticism and doubting the veracity of their previous stories. You're right that I believe TAL's stories are generally reliable, that I believe doubting them is an over-correction in the wrong direction. I also don't think I agree that it's simply a matter of checking or not checking because I believe the vast body of work that's been free from error, although exposed to the same conditions of public scrutiny that could have revealed error in just the same way as with the Daisy story, is part of the body of evidence that actively testifies in favor of TAL. And I do think if there was more of a rocky track record, or if there proves to be more of one in the future, it absolutely could merit spot checking. And as I've said twice now already, I've given the example of Stephen Glass as a case where that skepticism was warranted. You seem to be implying that I have a categorical opposition to spot checking which couldn't be further from the truth. I just don't think it's warranted in this instance, because it's not a reasonable extrapolation from what happened with the Daisy story.
  - 2y ago

spondylosaurus 2y ago

So... I can speak to this because I've listened to almost the entire TAL back catalog, lol. The show has a stronger journalistic focus now than it did 10 years ago and _way_ more than it did 20 or (almost!) 30 years ago. It's always been part of the show—people who complain that TAL "didn't used to be political" clearly don't remember how many segments they ran about the Iraq war—but for the first decade or so the show had a very strong focus on the arts; they'd have a lot of guests sharing personal essays, short fiction, etc. The serious journalism-type stories were generally either (1) on-the-ground reporting from their own staff, like a really great episode where they toured an aircraft carrier (https://www.thisamericanlife.org/206/somewhere-in-the-arabian-sea https://www.thisamericanlife.org/206/somewhere-in-the-arabia...), (2) stories sourced from other journalists or professional organizations who had serious reputation to lose if they were caught in the supply chain of misinformation, or (3) "this is a thing that happened to me"-type firsthand accounts and observations. And I guess the now-debunked Apple factory story falls into the third category, but the types of stories that had previously fallen into that category tended to be far smaller in scale and/or were presented as more subjective than the Apple story had been. All of that to say that I think you raise a valid question, and I'm sure some stuff slipped through the cracks over the years, but I also think the implication of that crack-slipping was far less dire in the show's earlier days, and there were just fewer news-ish stories overall.