3 ms·

Identifying Twitter users who repost unreliable news sources

43 points by laurex 6y ago

sieabahlpark 6y ago
You know how I know this study is effectively useless? > (1) those that have reposted content from unreliable news sources; and (2) those that repost content only from reliable sources. I'm glad this study basically in definition will centralize news authority. A single tweet can prevent you being "reliable". Which means whoever decides what is the "reliable" sources can determine exactly who is reliable.
11thEarlOfMar 6y ago
I'd be much more interested in approaches to identifying whether a news source is reliable. First question would be, reliable in what way? Accurate? Timely? Relevant? Unbiased? Then, how would you objectively measure those attributes and score them? Then, what would the thresholds be to separate reliable vs. unreliable?
- 11thEarlOfMar 6y ago
  Note to self: One of the sites the authors used to identify whether a source is unreliable is for sale: www.fakenewswatch.com
- jka 6y ago
  In the worst case, it might not be possible to assign a reliability score that you can rely on (reliability squared) to a news source; let's consider some edge cases: - The news source has a biased author. Articles on related topics by that author may be untrustworthy and deviate from the score. - A particular author and their team have been worked off their feet for the past two weeks due to internal or external pressure. Their next article may not be as reliable as it should be. - The management of a news source may have changed (potentially months or years ago) and now they have decided that it is time to alter editorial policy. Subsequent articles may differ in accuracy. I think that news workers understand these concerns and take them seriously since it affects their reputations and career prospects. They probably also understand that there's a certain extent to which they can get away with small errors and mistakes. Demanding complete truth and accuracy might make the job impossible, or so time-consuming that the results become irrelevant. So there may be some element of a time-accuracy tradeoff. Perhaps fact-checkers and AI can rise above these challenges, or at least guide them, but we should be wary and allow for corrections, since much of what they do builds upon previously-seen data and consensus on what has previously been accepted as true.
- Bancakes 6y ago
  People just want someone else to do the work for free, I guess. How many do you think would be interested in the workings?
auganov 6y ago
If you look at table 4 it's pretty obvious all the model does is separate people who post about politics from those who don't. Given the methodology [0] this result is unsurprising. Those labeled as "unreliable" seem to be universally political, while those deemed "reliable" are mainstream sources that may do stories on a variety of topics. [0] https://www.aclweb.org/anthology/P17-2102.pdf https://www.aclweb.org/anthology/P17-2102.pdf
motohagiography 6y ago
Super interesting. I wonder if the metamodel of the differences between reliable and unreliable could be related to whether the overall lexicon (and implied ontology) of a given article is more focused on the endogenous logic of the subject and the events it describes, or relies on "callouts" to exogenous ideas. ( for exmaple, this comment is a callout, instead of a discussion of their methodology) It would be a kind of subtext analysis, where the question is whether the article has more of an internal logical consistency, or if it resembles a form that is a vehicle for an external idea. The concept worth exploring might not be "reliability," but "value." Sorting news into tribes is pretty simple and doesn't really require ML, so their examples of unreliable news sites are useful to prove the apparatus tech for a surveillance scheme, but I'd suggest less scientifically meaningful. Their sample sites were more about social class than reliability (the same thing to them, probably), as there are definitely news sources that are both extremely unreliable and high value. That unreliable/high-value quadrant is probably more useful to study for determining how to discover value than just numbering people for their class and beliefs. Instead of a sentiment analysis, the question is, "is this article about a subject, or is the subject of this article a vehicle for another idea?" Wonder if that's possible. It would be like determining sentiment to separate reporting from criticism.
kipchak 6y ago
The list of trustworthy and untrustworthy sources seems to have been manually compiled in a prior paper. >For that purpose, we use a widely-used and publicly available list of English news media Twitter accounts provided by Volkova et al. (2017) and Glenski, Weninger & Volkova (2018a), which contains 424 English news media sources categorized in unreliable (satire, propaganda, hoax, clickbait) and reliable, following Rubin, Chen & Conroy (2015). The list is linked in the referenced paper, but unfortunately doesn't seem to be available anymore outside of a few examples given in the paper, such as the Drudge Report being under the hoax category. The list seems to have been compiled based on Propornot's list[1], which doesn't seem to have updated since 2017, (though their Twitter account is active[6]) and fakenewswatch[2] which no longer is active, based on one of the cited papers.[3] Some of the inclusions on those sites' lists I'm not completely sure about, such as putting "ronpaulinstitute.org" under the Russian propaganda category. I'm not sure what made a source trusted. [1]http://www.propornot.com/p/the-list.html http://www.propornot.com/p/the-list.html [2]https://webarchive.loc.gov/all/20170301201107/http://www.fakenewswatch.com/ https://webarchive.loc.gov/all/20170301201107/http://www.fak... [3]https://arxiv.org/pdf/1805.12032.pdf https://arxiv.org/pdf/1805.12032.pdf [4]https://www.cs.jhu.edu/~svitlana/posters/FakeNewsPoster.pdf https://www.cs.jhu.edu/~svitlana/posters/FakeNewsPoster.pdf [5]https://webarchive.loc.gov/all/20170301201107/http://www.fakenewswatch.com/ https://webarchive.loc.gov/all/20170301201107/http://www.fak... [6]https://twitter.com/propornot/status/1295531214862815232 https://twitter.com/propornot/status/1295531214862815232
- sieabahlpark 6y ago
  > Some of the inclusions on those sites' lists I'm not completely sure about, such as putting "ronpaulinstitute.org" under the Russian propaganda category. I'm not sure what made a source trusted Thanks for proving my point that this study is actually worthless.
- StanislavPetrov 6y ago
  Frightening, but extremely apropos that propaganda outlet Propornot is used as the benchmark to determine what is "reliable". Propornot first gained credibility as a source in a 2016 Washington Post article about "Russian disinformation". It was so hilariously bad that The Washington Post was quickly forced to amend the article with an editorial disclaimer about Propornot. A detailed writeup of this debacle was done at the time by The Intercept: https://theintercept.com/2016/11/26/washington-post-disgracefully-promotes-a-mccarthyite-blacklist-from-a-new-hidden-and-very-shady-group/ https://theintercept.com/2016/11/26/washington-post-disgrace... Matt Taibi also chimed in at the time: https://www.rollingstone.com/politics/politics-features/the-washington-post-blacklist-story-is-shameful-and-disgusting-115978/ https://www.rollingstone.com/politics/politics-features/the-...
Simulacra 6y ago
Do we really need MORE lists to put people on? Even the sex offender list has proven to be a waste of time. https://www.aclu-nj.org/theissues/criminaljustice/whysexoffenderlawsdomoreha https://www.aclu-nj.org/theissues/criminaljustice/whysexoffe...
mhh__ 6y ago
As a basic approach I think these efforts are better than nothing but I'm reminded of a manually compiled list that contained the Private Eye as a fake news site. For those unfamiliar the private eyes back pages are literally filled with hard-won stories about malpractice, corruption, fraud, (and jokes).
rainybrisb 6y ago
How do they identity unreliable news sources? Do they just remove all publications that took part in Covid hoaxes, WMDs in Iraq hoaxes, etc.?

43 points by laurex 6y ago

You know how I know this study is effectively useless? > (1) those that have reposted content from unreliable news sources; and (2) those that repost content only from reliable sources. I'm glad this study basically in definition will centralize news authority. A single tweet can prevent you being "reliable". Which means whoever decides what is the "reliable" sources can determine exactly who is reliable.

11thEarlOfMar 6y ago

I'd be much more interested in approaches to identifying whether a news source is reliable. First question would be, reliable in what way? Accurate? Timely? Relevant? Unbiased? Then, how would you objectively measure those attributes and score them? Then, what would the thresholds be to separate reliable vs. unreliable?

11thEarlOfMar 6y ago
Note to self: One of the sites the authors used to identify whether a source is unreliable is for sale: www.fakenewswatch.com
jka 6y ago
In the worst case, it might not be possible to assign a reliability score that you can rely on (reliability squared) to a news source; let's consider some edge cases: - The news source has a biased author. Articles on related topics by that author may be untrustworthy and deviate from the score. - A particular author and their team have been worked off their feet for the past two weeks due to internal or external pressure. Their next article may not be as reliable as it should be. - The management of a news source may have changed (potentially months or years ago) and now they have decided that it is time to alter editorial policy. Subsequent articles may differ in accuracy. I think that news workers understand these concerns and take them seriously since it affects their reputations and career prospects. They probably also understand that there's a certain extent to which they can get away with small errors and mistakes. Demanding complete truth and accuracy might make the job impossible, or so time-consuming that the results become irrelevant. So there may be some element of a time-accuracy tradeoff. Perhaps fact-checkers and AI can rise above these challenges, or at least guide them, but we should be wary and allow for corrections, since much of what they do builds upon previously-seen data and consensus on what has previously been accepted as true.
Bancakes 6y ago
People just want someone else to do the work for free, I guess. How many do you think would be interested in the workings?

auganov 6y ago

If you look at table 4 it's pretty obvious all the model does is separate people who post about politics from those who don't. Given the methodology [0] this result is unsurprising. Those labeled as "unreliable" seem to be universally political, while those deemed "reliable" are mainstream sources that may do stories on a variety of topics. [0] https://www.aclweb.org/anthology/P17-2102.pdf https://www.aclweb.org/anthology/P17-2102.pdf

motohagiography 6y ago

Super interesting. I wonder if the metamodel of the differences between reliable and unreliable could be related to whether the overall lexicon (and implied ontology) of a given article is more focused on the endogenous logic of the subject and the events it describes, or relies on "callouts" to exogenous ideas. ( for exmaple, this comment is a callout, instead of a discussion of their methodology) It would be a kind of subtext analysis, where the question is whether the article has more of an internal logical consistency, or if it resembles a form that is a vehicle for an external idea. The concept worth exploring might not be "reliability," but "value." Sorting news into tribes is pretty simple and doesn't really require ML, so their examples of unreliable news sites are useful to prove the apparatus tech for a surveillance scheme, but I'd suggest less scientifically meaningful. Their sample sites were more about social class than reliability (the same thing to them, probably), as there are definitely news sources that are both extremely unreliable and high value. That unreliable/high-value quadrant is probably more useful to study for determining how to discover value than just numbering people for their class and beliefs. Instead of a sentiment analysis, the question is, "is this article about a subject, or is the subject of this article a vehicle for another idea?" Wonder if that's possible. It would be like determining sentiment to separate reporting from criticism.

kipchak 6y ago

The list of trustworthy and untrustworthy sources seems to have been manually compiled in a prior paper. >For that purpose, we use a widely-used and publicly available list of English news media Twitter accounts provided by Volkova et al. (2017) and Glenski, Weninger & Volkova (2018a), which contains 424 English news media sources categorized in unreliable (satire, propaganda, hoax, clickbait) and reliable, following Rubin, Chen & Conroy (2015). The list is linked in the referenced paper, but unfortunately doesn't seem to be available anymore outside of a few examples given in the paper, such as the Drudge Report being under the hoax category. The list seems to have been compiled based on Propornot's list[1], which doesn't seem to have updated since 2017, (though their Twitter account is active[6]) and fakenewswatch[2] which no longer is active, based on one of the cited papers.[3] Some of the inclusions on those sites' lists I'm not completely sure about, such as putting "ronpaulinstitute.org" under the Russian propaganda category. I'm not sure what made a source trusted. [1]http://www.propornot.com/p/the-list.html http://www.propornot.com/p/the-list.html [2]https://webarchive.loc.gov/all/20170301201107/http://www.fakenewswatch.com/ https://webarchive.loc.gov/all/20170301201107/http://www.fak... [3]https://arxiv.org/pdf/1805.12032.pdf https://arxiv.org/pdf/1805.12032.pdf [4]https://www.cs.jhu.edu/~svitlana/posters/FakeNewsPoster.pdf https://www.cs.jhu.edu/~svitlana/posters/FakeNewsPoster.pdf [5]https://webarchive.loc.gov/all/20170301201107/http://www.fakenewswatch.com/ https://webarchive.loc.gov/all/20170301201107/http://www.fak... [6]https://twitter.com/propornot/status/1295531214862815232 https://twitter.com/propornot/status/1295531214862815232

sieabahlpark 6y ago
> Some of the inclusions on those sites' lists I'm not completely sure about, such as putting "ronpaulinstitute.org" under the Russian propaganda category. I'm not sure what made a source trusted Thanks for proving my point that this study is actually worthless.
StanislavPetrov 6y ago
Frightening, but extremely apropos that propaganda outlet Propornot is used as the benchmark to determine what is "reliable". Propornot first gained credibility as a source in a 2016 Washington Post article about "Russian disinformation". It was so hilariously bad that The Washington Post was quickly forced to amend the article with an editorial disclaimer about Propornot. A detailed writeup of this debacle was done at the time by The Intercept: https://theintercept.com/2016/11/26/washington-post-disgracefully-promotes-a-mccarthyite-blacklist-from-a-new-hidden-and-very-shady-group/ https://theintercept.com/2016/11/26/washington-post-disgrace... Matt Taibi also chimed in at the time: https://www.rollingstone.com/politics/politics-features/the-washington-post-blacklist-story-is-shameful-and-disgusting-115978/ https://www.rollingstone.com/politics/politics-features/the-...

Simulacra 6y ago

Do we really need MORE lists to put people on? Even the sex offender list has proven to be a waste of time. https://www.aclu-nj.org/theissues/criminaljustice/whysexoffenderlawsdomoreha https://www.aclu-nj.org/theissues/criminaljustice/whysexoffe...

mhh__ 6y ago

As a basic approach I think these efforts are better than nothing but I'm reminded of a manually compiled list that contained the Private Eye as a fake news site. For those unfamiliar the private eyes back pages are literally filled with hard-won stories about malpractice, corruption, fraud, (and jokes).

rainybrisb 6y ago

How do they identity unreliable news sources? Do they just remove all publications that took part in Covid hoaxes, WMDs in Iraq hoaxes, etc.?