5 ms·

It's such a pity that they chose to use PolitFact as labelling "experts" in one of the underlying datasets. Given how extreme the political bias in academia is

It's such a pity that they chose to use PolitFact as labelling "experts" in one of the underlying datasets. Given how extreme the political bias in academia is they should really know better than to try and make claims about politics in what would otherwise be reasonable ML research. The assumption that fact checkers are reliable is going to be an automatic strike against the paper for a large number of readers because it's so self-evidently wrong, and calls their judgement into question on everything else. Even though they only give three examples in the paper itself the problems are already obvious. If we look at table 6 then the latter two statements are genuinely ambiguous, but they also provide this as an example of an ambiguous statement: "When President Obama was elected, the market crashed". This is called ambiguous because "the claim implies a causal relationship". That statement isn't ambiguous. It has exactly one possible parse and meaning, asserting temporal alignment. "When A happened, B happened" doesn't automatically imply causality in English, and when people assume it does that's the famous "correlation is not causation" fallacy. Yet they classify it as ambiguous because an LLM rewrote it in one case to contain an implied assertion of causality that doesn't actually appear in the original text. In effect the political bias of the LLM or possibly the exact data fed in has been used to condemn the original speaker - exactly the sort of thing that makes self-proclaimed "professional fact checkers" such a joke. This sort of problem is inevitable if you use politics as a verification dataset because the internet is full of people reading implied-but-unstated things into factual statements and then "fact checkers" proceed to debunk what they think people would have said, instead of what they actually said. Although politics will have this problem frequently, a few of the other examples in their dev set show similar issues. For example, "Ralph knows that someone here is a spy" is considered ambiguous because you could potentially infer that he either does or doesn't know who it is, but you could also argue that the statement is unambiguous and it's only your attempt to read more into the statement than actually exists that makes it appear ambiguous. If the original statement was surrounded with more context then that would let you pick between the two possible outcomes but the statement by itself is not ambiguous - it just doesn't give as much information as the labeller might have wanted.

8organicbits 4y ago
> "When President Obama was elected, the market crashed". This is called ambiguous because "the claim implies a causal relationship". You really don't think its ambiguous? The abstract says that they want to "anticipate misunderstanding". Knowing that people have difficulty understanding "correlation is not causation", you'd expect that people would misunderstand a statement like this. If your goal is clear communication, statements like this should be avoided. It's ambiguous in practice. People intentionally exploit this by putting two unrelated facts together to trick people into believing a third thing. Identifying those misleading statements is helpful. Separately, is that statement even true? The election was Nov 4th, Obama was elected on the 4th and McCain gave his concessions speech on the 5th before markets opened making it clear Obama had won. On the 5th SP500 was slightly up. SP500 was down 20%+ later in the month, but I'm not sure that was "when he was elected". The market was also crashing since September. Claiming the market crashed at the election feels off.
- revelio 4y ago
  I think what they mean by ambiguous could itself be considered ambiguous. Let's view this from the context of a classical AI problem, trying to be as neutral and computer-like as possible. In that case, the statement is not ambiguous, no. It consists of two events with a connective "when" indicating they happened simultaneously or near simultaneously. We can argue about how simultaneous they need to be for a "when" to be justified, but this paper isn't about factuality, just ambiguity. Now the choice of politics primes people for culture war and so they will read into that statement more than exists. They will assume the writer is attempting to make you infer something, without outright stating it. They might be right! And with additional surrounding context, perhaps it would become completely certain. But it might also not be the case, for example the surrounding context could be something like this: "The election was at a time of unprecedented economic turmoil. A loose regulatory approach had left the banking sector unconstrained and primed to fail. As such when President Obama was elected the markets crashed, proving his predictions correct". Well, now it's unlikely that any causality is being implied even though the words are the same. The paper authors say they want to improve clarity in communication, a worthy goal. But a serious problem in all political communication is the frequency with which people make statements that are factually correct, and which then get attacked anyway because some listener would rather attack a straw man than the actual claim (or hypothetical listener, in the frequent case of left-wing journalists writing about it under the guise of fact checking). If they want to avoid this problem then at the very least they should have focused on claims that are syntactically ambiguous. Otherwise it opens them up to this very problem where they infer a straw man then blame the original speaker. No, the blame lies with the person committing the fallacy. Once you go down the route of blaming someone for your own interpretations you end up with micro-aggressions and other stuff that left-wing academics surely assume is obvious and normal, but isn't.

8organicbits 4y ago

> "When President Obama was elected, the market crashed". This is called ambiguous because "the claim implies a causal relationship". You really don't think its ambiguous? The abstract says that they want to "anticipate misunderstanding". Knowing that people have difficulty understanding "correlation is not causation", you'd expect that people would misunderstand a statement like this. If your goal is clear communication, statements like this should be avoided. It's ambiguous in practice. People intentionally exploit this by putting two unrelated facts together to trick people into believing a third thing. Identifying those misleading statements is helpful. Separately, is that statement even true? The election was Nov 4th, Obama was elected on the 4th and McCain gave his concessions speech on the 5th before markets opened making it clear Obama had won. On the 5th SP500 was slightly up. SP500 was down 20%+ later in the month, but I'm not sure that was "when he was elected". The market was also crashing since September. Claiming the market crashed at the election feels off.

revelio 4y ago
I think what they mean by ambiguous could itself be considered ambiguous. Let's view this from the context of a classical AI problem, trying to be as neutral and computer-like as possible. In that case, the statement is not ambiguous, no. It consists of two events with a connective "when" indicating they happened simultaneously or near simultaneously. We can argue about how simultaneous they need to be for a "when" to be justified, but this paper isn't about factuality, just ambiguity. Now the choice of politics primes people for culture war and so they will read into that statement more than exists. They will assume the writer is attempting to make you infer something, without outright stating it. They might be right! And with additional surrounding context, perhaps it would become completely certain. But it might also not be the case, for example the surrounding context could be something like this: "The election was at a time of unprecedented economic turmoil. A loose regulatory approach had left the banking sector unconstrained and primed to fail. As such when President Obama was elected the markets crashed, proving his predictions correct". Well, now it's unlikely that any causality is being implied even though the words are the same. The paper authors say they want to improve clarity in communication, a worthy goal. But a serious problem in all political communication is the frequency with which people make statements that are factually correct, and which then get attacked anyway because some listener would rather attack a straw man than the actual claim (or hypothetical listener, in the frequent case of left-wing journalists writing about it under the guise of fact checking). If they want to avoid this problem then at the very least they should have focused on claims that are syntactically ambiguous. Otherwise it opens them up to this very problem where they infer a straw man then blame the original speaker. No, the blame lies with the person committing the fallacy. Once you go down the route of blaming someone for your own interpretations you end up with micro-aggressions and other stuff that left-wing academics surely assume is obvious and normal, but isn't.