6 ms·

Too bad it employs the PolitiFact database, as that itself is suspect to having its own bias. (http://blog.lib.umn.edu/cspg/smartpolitics/2011/02/selection_bias

Too bad it employs the PolitiFact database, as that itself is suspect to having its own bias. (http://blog.lib.umn.edu/cspg/smartpolitics/2011/02/selection_bias_politifact_rate.php http://blog.lib.umn.edu/cspg/smartpolitics/2011/02/selection...) Using a biased fact-checker to check facts doesn't really work well. Constructing a non-biased fact-checker is the real challenge here. I'm not sure it can be done, due to the great amount of spin that can be put on most anything political. EDIT: BTW, I'm not meaning to take sides on this, just pointing out the very real difficulties in establishing a "fact-checker". It's worth knowing that others have contested PolitiFact's methodologies. If the trustworthiness of the statement is based on a person's judgement, then the fact-checker has bias.

Cushman 15y ago
Uh, it looks like that accusation of bias comes down to PolitiFact grading Republicans as wrong more often that Democrats... I don't want to get all political, but there's another potential explanation for that, yes? (There's also the accusation that they target Republicans more than Democrats, but that's easily explained by their methodology, from that post: "If we look at something and we think that an elected official or talk show host is wrong, then we will fact-check it." If we posit that Republicans are wrong more often than Democrats, we would expect them to show up more often.)
- jerf 15y ago
  Yes, but the parsimonious explanation is that neither Democratic politicians nor Republican politicians nor any other kind of politicians are particularly careful with their facts, and which you think are more prone to "lying" has more to do with the listeners than the speakers. The idea that any politician is routinely dealing in truth is laughable... and I don't mean that so much in terms of cynicism per se, but because they really have very little incentive to do the necessary huge amounts of work it would take that would bring them no gain, or even less than no gain inasmuch as it would limit their verbal weaponry. Just as one example of the sort of thing that makes it difficult for me to take Politifact ratings seriously in 30 seconds of browsing, look at: http://www.politifact.com/truth-o-meter/statements/2011/sep/22/mitt-romney/mitt-romney-repeats-claim-obama-went-around-world-/ http://www.politifact.com/truth-o-meter/statements/2011/sep/... Mitt Romney gets a "pants on fire" rating for claiming that Obama has apologized to the world for the US on various occasions. To my eyes, it's a desperate attempt to avoid the accusation by deliberately and stubbornly being as naive as possible, insisting that it wasn't anything remotely resembling an apology ("pants on fire" remember, not just "half truth", which I would actually accept just fine on this matter) because it doesn't contain the word "sorry" and never quite fully says "apology". I call it "naive" because in politics everybody knows that there's more to what's going on that just what is being said; would Politifact expect me to be naive enough that when diplomats say they had the traditional "frank exchange of views" that means the negotiations are going well, because nobody said outright they were going poorly? Of course Obama never said "I apologize for the US", but there's rather more wiggle room for interpretation than giving someone who says there were an apology a "pants on fire" rating. And to be perfectly clear, I see equally dubious straining-at-gnats attacks from conservatives on liberals as well routinely. The only way these "truth goggles" could actually work is to draw from a wide variety of sources and hope the bias cancels.
  - andrewcooke 15y ago
    no, the parsimonious explanation is that the person caught lying most lies most. really. it sounds trivially obvious because, well, it's trivially obvious. sure, politics is complicated and everyone bends the rules. but that would give a "background noise" that was the same for everyone. sometimes anyone can make a mistake. sometimes anyone can get away with it. there's no bias there. if you catch someone who lies more than the others then the simple, obvious explanation is that they lie more than the others. if you want long words, then it's similar to the difference between systematic and stochastic errors. in this context it's a little ironic to say this, but surely only a liberal who was so completely disconnected from reality that they have no idea of right and wrong would argue otherwise: some people lie more than others; you can tell who they are because they are caught lying more. to argue that there is no right and wrong in people's statements is "political correctness" taken to crazy extremes. of course, there are exceptions: if you only check one person then it's unfair. but as far as i can tell (see my other comment) that is not happening here.
    - adbge 15y ago
      The person caught lying most is probably the person who is the worst at allaying suspicion. PolitiFact's fatal flaw is that they only fact-check a subset of statements: those which sound suspicious. Further, it's important to distinguish lying from being factually incorrect. A lie implies deceit, being incorrect does not.
    - bobwaycott 15y ago
      In both this and the parent comment, is parsimonious really the adjective you were looking for? In what way is either of these a parsimonious explanation? Simple, perhaps. Reductionist or nominal, certainly. But the cheaper, stingy explanation? I'm not so sure that was correct.
andrewcooke 15y ago
have you read that article in detail? i have spent some time on it, and am having trouble following the argument. at the start they say that approximately equal numbers of statements are taken from either side. then later they seem to be saying that a bias exists because the republicans lie more than the democrats. what i'm having a hard time understanding is why that is a bias - perhaps some republicans simply lie more? again: it's not they are focusing more on one party than on another (they don't: "the number of public officials subjected to PolitiFact's Truth-O-Meter lens from each party is fairly even during the period under analysis"), but that they catch more lies from one party (the republicans) than the other. in particular, it appears that palin and bachmann are responsible for raising the average number of lies for the republicans. so why is this bias? why isn't it simply that palin and bachmann are liars? what am i missing (i am not american, so don't know much of the context - i just read the article and couldn't follow the logic of the argument)? one possible alternative is that they are ignoring lies by democrats and instead checking only true statements. but in that case you'd think there would be a smoking gun - notorious statements from democrats that weren't checked. the article doesn't give any evidence for that. ps also, i take issue with your final disclaimer. you can make judgement calls and still be better than coin flipping. just because it's not an automated, exact science doesn't mean that it cannot edge closer to the truth. almost everything in life is not exact, but we still know some people lie more than others. one way to do this is by looking at what is given (for example, this article) and checking whether it makes sense. maybe i have made a mistake - if so, hopefully someone will correct me. eventually this thread will lend weight to one conclusion or another...
- kls 15y ago
  you can make judgement calls and still be better than coin flipping. just because it's not an automated, exact science doesn't mean that it cannot edge closer to the truth. almost everything in life is not exact, but we still know some people lie more than others. I agree with this in the fuzzy world of politics you have no choice but to bring in probabilities. I would think that this system could be automated by software to aid the fact checker in coming to a decision. I would say that aggregating a politicians history of lying and using that as a weight as to whether a suspect statement could be untruthful would help. If a candidate has a history of being less than truthful and a certainty of fact cannot be deduced one could look at weight factors such as history of truthfulness on a subject to help them come to a conclusion. These methods and the algorithms could them be published for scrutiny. This way it is not arbitrary, it may not be 100% accurate but at least the system is open and the method reproducible. From there it can be refined to help aid in accuracy.
slifty 15y ago
You are absolutely right -- and I'm not coming into this project with the illusion that any given source will be universally trusted (or trusted at all). I don't want to create a tool that people will look at and say "all righty then! Time to believe anything this tells me!" In fact, that would be counterproductive. The real goal is a tool that will trigger critical thinking when it is most needed -- flag pieces of content that are worth spending an extra few seconds / minutes / etc looking into and then help them look into it. I want people to think carefully before infusing controversial information into their world view. (btw I'm Dan in case you didn't pick up on that)
jaylevitt 15y ago
I've got a strong (but untrained) interest in the psychology of bias, and I don't think it is possible. Humans, even smart humans, even smart humans who study bias are notoriously unaware of their own biases and self-deceptions. We tune out information that challenges our belief system, we form alliances, we make the fundamental attribution error, and we attach meanings that tend to reinforce our beliefs. It's really good to try to be unbiased, but understand that it's an asymptotic kinda of thing.
jaylevitt 15y ago
I've got a strong (but untrained) interest in the psychology of bias, and I don't think it is possible. Humans, even smart humans, even smart humans who study bias are notoriously unaware of their own biases and self-deceptions. We tune out information that challenges our belief system, we form alliances, we make the fundamental attribution error, and we attach meanings that tend to reinforce our beliefs. It's really good to try to be unbiased, but understand that it's an asymptotic kinda of thing.
stfu 15y ago
Very valid points. I think that "facts" in the political dialogue are an illusion and politics has always been interpreting reality. Of cause some "truth machine" could spot misconceptions of obvious facts (e.g. how many states are in the Union) but with anything slightly more sophisticated it is going to be a subjective mine field. Just take some basic question and get studies drawn up by different ideological thinktanks and I am sure they are able to produce the most beautiful statistics arguing that their ideological position always has been and always will be the right one. Especially since PolitiFact attempts to "rate" gray areas beyond just true and false their neutrality will, and has to be questioned.

Cushman 15y ago

Uh, it looks like that accusation of bias comes down to PolitiFact grading Republicans as wrong more often that Democrats... I don't want to get all political, but there's another potential explanation for that, yes? (There's also the accusation that they target Republicans more than Democrats, but that's easily explained by their methodology, from that post: "If we look at something and we think that an elected official or talk show host is wrong, then we will fact-check it." If we posit that Republicans are wrong more often than Democrats, we would expect them to show up more often.)

jerf 15y ago
Yes, but the parsimonious explanation is that neither Democratic politicians nor Republican politicians nor any other kind of politicians are particularly careful with their facts, and which you think are more prone to "lying" has more to do with the listeners than the speakers. The idea that any politician is routinely dealing in truth is laughable... and I don't mean that so much in terms of cynicism per se, but because they really have very little incentive to do the necessary huge amounts of work it would take that would bring them no gain, or even less than no gain inasmuch as it would limit their verbal weaponry. Just as one example of the sort of thing that makes it difficult for me to take Politifact ratings seriously in 30 seconds of browsing, look at: http://www.politifact.com/truth-o-meter/statements/2011/sep/22/mitt-romney/mitt-romney-repeats-claim-obama-went-around-world-/ http://www.politifact.com/truth-o-meter/statements/2011/sep/... Mitt Romney gets a "pants on fire" rating for claiming that Obama has apologized to the world for the US on various occasions. To my eyes, it's a desperate attempt to avoid the accusation by deliberately and stubbornly being as naive as possible, insisting that it wasn't anything remotely resembling an apology ("pants on fire" remember, not just "half truth", which I would actually accept just fine on this matter) because it doesn't contain the word "sorry" and never quite fully says "apology". I call it "naive" because in politics everybody knows that there's more to what's going on that just what is being said; would Politifact expect me to be naive enough that when diplomats say they had the traditional "frank exchange of views" that means the negotiations are going well, because nobody said outright they were going poorly? Of course Obama never said "I apologize for the US", but there's rather more wiggle room for interpretation than giving someone who says there were an apology a "pants on fire" rating. And to be perfectly clear, I see equally dubious straining-at-gnats attacks from conservatives on liberals as well routinely. The only way these "truth goggles" could actually work is to draw from a wide variety of sources and hope the bias cancels.
- andrewcooke 15y ago
  no, the parsimonious explanation is that the person caught lying most lies most. really. it sounds trivially obvious because, well, it's trivially obvious. sure, politics is complicated and everyone bends the rules. but that would give a "background noise" that was the same for everyone. sometimes anyone can make a mistake. sometimes anyone can get away with it. there's no bias there. if you catch someone who lies more than the others then the simple, obvious explanation is that they lie more than the others. if you want long words, then it's similar to the difference between systematic and stochastic errors. in this context it's a little ironic to say this, but surely only a liberal who was so completely disconnected from reality that they have no idea of right and wrong would argue otherwise: some people lie more than others; you can tell who they are because they are caught lying more. to argue that there is no right and wrong in people's statements is "political correctness" taken to crazy extremes. of course, there are exceptions: if you only check one person then it's unfair. but as far as i can tell (see my other comment) that is not happening here.
  - adbge 15y ago
    The person caught lying most is probably the person who is the worst at allaying suspicion. PolitiFact's fatal flaw is that they only fact-check a subset of statements: those which sound suspicious. Further, it's important to distinguish lying from being factually incorrect. A lie implies deceit, being incorrect does not.
  - bobwaycott 15y ago
    In both this and the parent comment, is parsimonious really the adjective you were looking for? In what way is either of these a parsimonious explanation? Simple, perhaps. Reductionist or nominal, certainly. But the cheaper, stingy explanation? I'm not so sure that was correct.

andrewcooke 15y ago

have you read that article in detail? i have spent some time on it, and am having trouble following the argument. at the start they say that approximately equal numbers of statements are taken from either side. then later they seem to be saying that a bias exists because the republicans lie more than the democrats. what i'm having a hard time understanding is why that is a bias - perhaps some republicans simply lie more? again: it's not they are focusing more on one party than on another (they don't: "the number of public officials subjected to PolitiFact's Truth-O-Meter lens from each party is fairly even during the period under analysis"), but that they catch more lies from one party (the republicans) than the other. in particular, it appears that palin and bachmann are responsible for raising the average number of lies for the republicans. so why is this bias? why isn't it simply that palin and bachmann are liars? what am i missing (i am not american, so don't know much of the context - i just read the article and couldn't follow the logic of the argument)? one possible alternative is that they are ignoring lies by democrats and instead checking only true statements. but in that case you'd think there would be a smoking gun - notorious statements from democrats that weren't checked. the article doesn't give any evidence for that. ps also, i take issue with your final disclaimer. you can make judgement calls and still be better than coin flipping. just because it's not an automated, exact science doesn't mean that it cannot edge closer to the truth. almost everything in life is not exact, but we still know some people lie more than others. one way to do this is by looking at what is given (for example, this article) and checking whether it makes sense. maybe i have made a mistake - if so, hopefully someone will correct me. eventually this thread will lend weight to one conclusion or another...

kls 15y ago
you can make judgement calls and still be better than coin flipping. just because it's not an automated, exact science doesn't mean that it cannot edge closer to the truth. almost everything in life is not exact, but we still know some people lie more than others. I agree with this in the fuzzy world of politics you have no choice but to bring in probabilities. I would think that this system could be automated by software to aid the fact checker in coming to a decision. I would say that aggregating a politicians history of lying and using that as a weight as to whether a suspect statement could be untruthful would help. If a candidate has a history of being less than truthful and a certainty of fact cannot be deduced one could look at weight factors such as history of truthfulness on a subject to help them come to a conclusion. These methods and the algorithms could them be published for scrutiny. This way it is not arbitrary, it may not be 100% accurate but at least the system is open and the method reproducible. From there it can be refined to help aid in accuracy.

slifty 15y ago

You are absolutely right -- and I'm not coming into this project with the illusion that any given source will be universally trusted (or trusted at all). I don't want to create a tool that people will look at and say "all righty then! Time to believe anything this tells me!" In fact, that would be counterproductive. The real goal is a tool that will trigger critical thinking when it is most needed -- flag pieces of content that are worth spending an extra few seconds / minutes / etc looking into and then help them look into it. I want people to think carefully before infusing controversial information into their world view. (btw I'm Dan in case you didn't pick up on that)

jaylevitt 15y ago

I've got a strong (but untrained) interest in the psychology of bias, and I don't think it is possible. Humans, even smart humans, even smart humans who study bias are notoriously unaware of their own biases and self-deceptions. We tune out information that challenges our belief system, we form alliances, we make the fundamental attribution error, and we attach meanings that tend to reinforce our beliefs. It's really good to try to be unbiased, but understand that it's an asymptotic kinda of thing.

jaylevitt 15y ago

stfu 15y ago

Very valid points. I think that "facts" in the political dialogue are an illusion and politics has always been interpreting reality. Of cause some "truth machine" could spot misconceptions of obvious facts (e.g. how many states are in the Union) but with anything slightly more sophisticated it is going to be a subjective mine field. Just take some basic question and get studies drawn up by different ideological thinktanks and I am sure they are able to produce the most beautiful statistics arguing that their ideological position always has been and always will be the right one. Especially since PolitiFact attempts to "rate" gray areas beyond just true and false their neutrality will, and has to be questioned.