6 ms·

A Stylometric Inquiry into Hyperpartisan and Fake News

30 points by laretluval 10y ago

failrate 10y ago
I find it refreshing that they admit it isn't good enough right in the abstract (although they do point out that it is good enough to supplement an additional method).
mafribe 10y ago
manually fact-checked by professional journalists from BuzzFeed. The article seems to accept -- uncritically -- "professional journalists from BuzzFeed" as arbiters of fake news. Not a good start. That said, and anecdotally, style is often a fair heuristic for political agenda. It should be possible to improve the methodology of such research so it is insightful without such arbiters. As an aside, it is not surprising that "that the style of left-wing and right-wing news have a lot more in common than any of the two have with the mainstream", for they often use the same, or similar tools to undermine society, a goal both have in common. They read the same literature about how do this, typically following a Lenisist approach.
- ivraatiems 10y ago
  I think it's less style and more rhetoric. There are key phrases (ironically, "fake news" is one of them) that I think might be usable to both indicate how partisan something is, and potentially in what direction.
  - mafribe 10y ago
    I'm not quite sure what your style/rhetoric distinction refers to. I conjecture that the underlying mechanism why we can often easily predict political content from style is that most writers in general, and journalists in particular, don't write their texts from scratch, but mostly copy existing material and adapt it a bit. You can think of it along the lines of replication of DNA. The economics of modern journalism have made the copying worse. As an aside, I once developed software for an old telecoms protocol, and found a large number descriptions of this protocol in the trade press. They were all terrible, but felt quite similar. Eventually I managed to track down the origin: teaching material for a course developed by Cisco. As far as I can tell, not one article in the trade press was original. Yet, clearly some authors did quite a bit of work to disguise their copying, including -- bizarrely -- to redraw (badly) the figures in the original material.
    - ivraatiems 10y ago
      The distinction I see would be this: Style is how you lay out the article and make the argument. Rhetoric is more about the content of the article itself - what phrases you use, what slogans, terms, etc; how you frame certain issues. Most publications, fringe or not, have journalistic style that would, in my opinion, make them hard to distinguish. That is, if you took a Brietbart article and a Vox article on two different issues and compared the format/length/general tone and language, they wouldn't be all that different. However, I think these outlets do vary in the specific rhetoric they use, the people they rely on for comment/as sources, etc. I would therefore focus on specific terms or language rather than general style.
- ubernostrum 10y ago
  1. If anybody would recognize what manufactured clickbait looks like, it would be Buzzfeed. 2. Buzzfeed, and a number of other traditionally disreputable sites, have really stepped up their journalism game over the last year or so. If all you know about them is listicles, you might be surprised at some of the stuff they turn out nowadays.
  - mafribe 10y ago
    Your (1) is a very good point ... as the saying goes "It takes one to know one!". But I find "were manually fact-checked by professional journalists..." a bit of an unfortunate way of describing BF's expertise. I would not be familiar with (2) as I gave up on BF a long time ago.
    - deleted 10y ago
      [deleted]
    - maxerickson 10y ago
      They put their names on the work: https://www.buzzfeed.com/craigsilverman/partisan-fb-pages-analysis https://www.buzzfeed.com/craigsilverman/partisan-fb-pages-an... Which of the journalists there do you think is just the worst? I mean, they work for Buzzfeed right?
  - trueSlav 10y ago
    Buzzfeed is one of the prominent "progressive" left sites. Everything it does is skewed by that lens. Asking them anything is about as useful as asking Alex Jones.
    - scholia 10y ago
      No, it isn't. Its news is mainstream, and not consistently more left wing than the New York Times, LA Times or the Boston Globe. (All the mainstream news sources include stories that trend both left and right.) Infowars is rabid right wing nut-job stuff. By Alex Jones standards, Fox News is left wing biased.
- cecilpl 10y ago
  Buzzfeed News is actually a very respectable investigative journalism team led by a Pulitzer-winning journalist. It's funded by their clickbait site.
  - laretluval 10y ago
    Is the Pulitzer prize evidence of reliability? https://en.wikipedia.org/wiki/Janet_Cooke https://en.wikipedia.org/wiki/Janet_Cooke
    - noobiemcfoob 10y ago
      Evidence? No, but a good indication.
- Pokepokalypse 10y ago
  Given that the goal of "partisan" news-style is emotional manipulation, it is fairly easy to spot; and I think that even if authors try to adjust style to avoid pre-screen filters, they'll lose some of that ability to manipulate.
- webmaven 10y ago
  It is worth noting that BuzzFeed's (considerable) clickbait revenues are being used to fund serious journalism (and that their investigative team is one of the few in the industry that has been expanding instead of downsizing): https://www.poynter.org/2016/how-buzzfeed-built-an-investigative-team-from-the-ground-up/396656/ https://www.poynter.org/2016/how-buzzfeed-built-an-investiga...
ivraatiems 10y ago
"Fake news detectors" just sounds like a way to abrogate the responsibility of the individual to understand and analyze sources. It's not a good idea to mentally outsource this kind of critical thinking. Also, Buzzfeed, while sometimes capable of strong journalism, is not the organization I'd pick to help with this. Why not Politifact, Reuters, 538, etc.?
- peterlk 10y ago
  I totally agree. As an academic exercise, this is an interesting problem, but actually trying to ship this is absurd. "Fake news" is a cultural/human problem. Adding a fake news indicator to Facebook (etc.) will just create further polarization: the tech industry sells out to the illuminati to bury the truth that they don't want you to know. Critical thinking can go a long way to identifying fake news. Our brains are pretty good at detecting bullshit if we filter out our outrage. Though, it might take a little bit of effort in digging through news. A better solution might be to identify articles that oppose one another and pin them together.
- scholia 10y ago
  It's really not hard to identify partizan news sources, which is what this is based on. They often self-identify themselves. It's really not hard for professional journalists to fact-check partizan articles, and I'd assume the ones hired by Buzzfeed used several well-known fact-checking sites, including Politifact. > sounds like a way to abrogate the responsibility of the individual to understand and analyze sources But of course it doesn't do that. It's just a way of getting a machine to identify partizan stories that are statistically likely to contain a higher proportion of lies, especially if they are from right-wing sources (Table 1). As we know all too well, individuals commonly abrogate the responsibility to understand and analyze sources all by themselves.
  - ivraatiems 10y ago
    > But of course it doesn't do that. It's just a way of getting a machine to identify partizan stories that are statistically likely to contain a higher proportion of lies, especially if they are from right-wing sources (Table 1). I think if a system like this got traction, then like many things academics produce, it will be misconstrued and misused. > It's really not hard for professional journalists to fact-check partizan articles, and I'd assume the ones hired by Buzzfeed used several well-known fact-checking sites, including Politifact. You should not assume things like that. And again, if they were just gonna look stuff up on Politifact, why couldn't the researcher do that themselves? It's clear there was a more in-depth process here. I think it is totally fair to say that the process could be a good one even if Buzzfeed is a bad choice of organization to run the process with, but further research would have to bear that out. Because only Buzzfeed was involved, it's impossible for us to determine whether there was a bias in the data because of who produced it. That is, we can't be sure if that data would be similar if we had, say, the National Review doing fact-checking. That's why we need either (preferably) more than one fact-checking organization, or at least a single but highly neutral one. > As we know all too well, individuals commonly abrogate the responsibility to understand and analyze sources all by themselves. Which does not mean we shouldn't be careful about enabling it. My fear here is the common trope of "X new discovery is an oracle," where we take some interesting tidbit about patterns (in in this case, in linguistics) and decide it is absolute truth or applies everywhere.
uuilly 10y ago
From the article: "It presents a large corpus of 1,627 articles that were manually fact-checked by professional journalists from BuzzFeed." At first I thought this was an onion article. It turns out they're serious.
- webmaven 10y ago
  Much as I dislike BuzzFeed's clickbait headlines and articles, they are using their considerable profits to fund serious journalism: https://www.poynter.org/2016/how-buzzfeed-built-an-investigative-team-from-the-ground-up/396656/ https://www.poynter.org/2016/how-buzzfeed-built-an-investiga...
draugadrotten 10y ago
I wonder how well the Amazon Mechanical Turk would do if you put it to the test to determine what is fake news or not.
trueSlav 10y ago
This is exactly the type of absolute garbage I want to see when checking out HN... wonderful.
zachrose 10y ago
The next step here is that the marketplace adapts gets to some kind of stylometric equilibrium where Breitbart sounds like NPR.
deleted 10y ago
[deleted]
uncletaco 10y ago
In my opinion a lot of comments here are too hung up on the fact that the researchers used data collected by Buzzfeed. I'm not a fan of clickbait either, but their corpus is publicly available here: https://docs.google.com/spreadsheets/d/1ysnzawW6pDGBEqbXqeYuzWa7Rx2mQUip6CXUUUk4jIk/edit#gid=1756764129 https://docs.google.com/spreadsheets/d/1ysnzawW6pDGBEqbXqeYu... So as far as I'm concerned, so long as the authors acknowledge their source (which they did) then there's nothing wrong here. Even FiveThirtyEight builds most of their models from datasets obtained from third parties. A lot of data science research is carried out using public datasets because collection is often an expensive or time-consuming endeavor, especially for social science research. The authors even say in their acknowledgement "We thank Craig Silverman, Lauren Strapagiel, Hamza Shaban, Ellie Hall, and Jeremy Singer-Vine from BuzzFeed for making their data available, enabling our research".
rdtsc 10y ago
> were manually fact-checked by professional journalists from BuzzFeed. Anyone else stopped reading there? Because I did.
- webmaven 10y ago
  Understandable, but BuzzFeed's considerable profits from clickbait articles are being used to fund serious journalism: https://www.poynter.org/2016/how-buzzfeed-built-an-investigative-team-from-the-ground-up/396656/ https://www.poynter.org/2016/how-buzzfeed-built-an-investiga...

30 points by laretluval 10y ago

I find it refreshing that they admit it isn't good enough right in the abstract (although they do point out that it is good enough to supplement an additional method).

mafribe 10y ago

manually fact-checked by professional journalists from BuzzFeed. The article seems to accept -- uncritically -- "professional journalists from BuzzFeed" as arbiters of fake news. Not a good start. That said, and anecdotally, style is often a fair heuristic for political agenda. It should be possible to improve the methodology of such research so it is insightful without such arbiters. As an aside, it is not surprising that "that the style of left-wing and right-wing news have a lot more in common than any of the two have with the mainstream", for they often use the same, or similar tools to undermine society, a goal both have in common. They read the same literature about how do this, typically following a Lenisist approach.

ivraatiems 10y ago
I think it's less style and more rhetoric. There are key phrases (ironically, "fake news" is one of them) that I think might be usable to both indicate how partisan something is, and potentially in what direction.
- mafribe 10y ago
  I'm not quite sure what your style/rhetoric distinction refers to. I conjecture that the underlying mechanism why we can often easily predict political content from style is that most writers in general, and journalists in particular, don't write their texts from scratch, but mostly copy existing material and adapt it a bit. You can think of it along the lines of replication of DNA. The economics of modern journalism have made the copying worse. As an aside, I once developed software for an old telecoms protocol, and found a large number descriptions of this protocol in the trade press. They were all terrible, but felt quite similar. Eventually I managed to track down the origin: teaching material for a course developed by Cisco. As far as I can tell, not one article in the trade press was original. Yet, clearly some authors did quite a bit of work to disguise their copying, including -- bizarrely -- to redraw (badly) the figures in the original material.
  - ivraatiems 10y ago
    The distinction I see would be this: Style is how you lay out the article and make the argument. Rhetoric is more about the content of the article itself - what phrases you use, what slogans, terms, etc; how you frame certain issues. Most publications, fringe or not, have journalistic style that would, in my opinion, make them hard to distinguish. That is, if you took a Brietbart article and a Vox article on two different issues and compared the format/length/general tone and language, they wouldn't be all that different. However, I think these outlets do vary in the specific rhetoric they use, the people they rely on for comment/as sources, etc. I would therefore focus on specific terms or language rather than general style.
ubernostrum 10y ago
1. If anybody would recognize what manufactured clickbait looks like, it would be Buzzfeed. 2. Buzzfeed, and a number of other traditionally disreputable sites, have really stepped up their journalism game over the last year or so. If all you know about them is listicles, you might be surprised at some of the stuff they turn out nowadays.
- mafribe 10y ago
  Your (1) is a very good point ... as the saying goes "It takes one to know one!". But I find "were manually fact-checked by professional journalists..." a bit of an unfortunate way of describing BF's expertise. I would not be familiar with (2) as I gave up on BF a long time ago.
  - deleted 10y ago
    [deleted]
  - maxerickson 10y ago
    They put their names on the work: https://www.buzzfeed.com/craigsilverman/partisan-fb-pages-analysis https://www.buzzfeed.com/craigsilverman/partisan-fb-pages-an... Which of the journalists there do you think is just the worst? I mean, they work for Buzzfeed right?
- trueSlav 10y ago
  Buzzfeed is one of the prominent "progressive" left sites. Everything it does is skewed by that lens. Asking them anything is about as useful as asking Alex Jones.
  - scholia 10y ago
    No, it isn't. Its news is mainstream, and not consistently more left wing than the New York Times, LA Times or the Boston Globe. (All the mainstream news sources include stories that trend both left and right.) Infowars is rabid right wing nut-job stuff. By Alex Jones standards, Fox News is left wing biased.
cecilpl 10y ago
Buzzfeed News is actually a very respectable investigative journalism team led by a Pulitzer-winning journalist. It's funded by their clickbait site.
- laretluval 10y ago
  Is the Pulitzer prize evidence of reliability? https://en.wikipedia.org/wiki/Janet_Cooke https://en.wikipedia.org/wiki/Janet_Cooke
  - noobiemcfoob 10y ago
    Evidence? No, but a good indication.
Pokepokalypse 10y ago
Given that the goal of "partisan" news-style is emotional manipulation, it is fairly easy to spot; and I think that even if authors try to adjust style to avoid pre-screen filters, they'll lose some of that ability to manipulate.
webmaven 10y ago
It is worth noting that BuzzFeed's (considerable) clickbait revenues are being used to fund serious journalism (and that their investigative team is one of the few in the industry that has been expanding instead of downsizing): https://www.poynter.org/2016/how-buzzfeed-built-an-investigative-team-from-the-ground-up/396656/ https://www.poynter.org/2016/how-buzzfeed-built-an-investiga...

ivraatiems 10y ago

"Fake news detectors" just sounds like a way to abrogate the responsibility of the individual to understand and analyze sources. It's not a good idea to mentally outsource this kind of critical thinking. Also, Buzzfeed, while sometimes capable of strong journalism, is not the organization I'd pick to help with this. Why not Politifact, Reuters, 538, etc.?

peterlk 10y ago
I totally agree. As an academic exercise, this is an interesting problem, but actually trying to ship this is absurd. "Fake news" is a cultural/human problem. Adding a fake news indicator to Facebook (etc.) will just create further polarization: the tech industry sells out to the illuminati to bury the truth that they don't want you to know. Critical thinking can go a long way to identifying fake news. Our brains are pretty good at detecting bullshit if we filter out our outrage. Though, it might take a little bit of effort in digging through news. A better solution might be to identify articles that oppose one another and pin them together.
scholia 10y ago
It's really not hard to identify partizan news sources, which is what this is based on. They often self-identify themselves. It's really not hard for professional journalists to fact-check partizan articles, and I'd assume the ones hired by Buzzfeed used several well-known fact-checking sites, including Politifact. > sounds like a way to abrogate the responsibility of the individual to understand and analyze sources But of course it doesn't do that. It's just a way of getting a machine to identify partizan stories that are statistically likely to contain a higher proportion of lies, especially if they are from right-wing sources (Table 1). As we know all too well, individuals commonly abrogate the responsibility to understand and analyze sources all by themselves.
- ivraatiems 10y ago
  > But of course it doesn't do that. It's just a way of getting a machine to identify partizan stories that are statistically likely to contain a higher proportion of lies, especially if they are from right-wing sources (Table 1). I think if a system like this got traction, then like many things academics produce, it will be misconstrued and misused. > It's really not hard for professional journalists to fact-check partizan articles, and I'd assume the ones hired by Buzzfeed used several well-known fact-checking sites, including Politifact. You should not assume things like that. And again, if they were just gonna look stuff up on Politifact, why couldn't the researcher do that themselves? It's clear there was a more in-depth process here. I think it is totally fair to say that the process could be a good one even if Buzzfeed is a bad choice of organization to run the process with, but further research would have to bear that out. Because only Buzzfeed was involved, it's impossible for us to determine whether there was a bias in the data because of who produced it. That is, we can't be sure if that data would be similar if we had, say, the National Review doing fact-checking. That's why we need either (preferably) more than one fact-checking organization, or at least a single but highly neutral one. > As we know all too well, individuals commonly abrogate the responsibility to understand and analyze sources all by themselves. Which does not mean we shouldn't be careful about enabling it. My fear here is the common trope of "X new discovery is an oracle," where we take some interesting tidbit about patterns (in in this case, in linguistics) and decide it is absolute truth or applies everywhere.

uuilly 10y ago

From the article: "It presents a large corpus of 1,627 articles that were manually fact-checked by professional journalists from BuzzFeed." At first I thought this was an onion article. It turns out they're serious.

webmaven 10y ago
Much as I dislike BuzzFeed's clickbait headlines and articles, they are using their considerable profits to fund serious journalism: https://www.poynter.org/2016/how-buzzfeed-built-an-investigative-team-from-the-ground-up/396656/ https://www.poynter.org/2016/how-buzzfeed-built-an-investiga...

draugadrotten 10y ago

I wonder how well the Amazon Mechanical Turk would do if you put it to the test to determine what is fake news or not.

trueSlav 10y ago

This is exactly the type of absolute garbage I want to see when checking out HN... wonderful.

zachrose 10y ago

The next step here is that the marketplace adapts gets to some kind of stylometric equilibrium where Breitbart sounds like NPR.

deleted 10y ago

[deleted]

uncletaco 10y ago

In my opinion a lot of comments here are too hung up on the fact that the researchers used data collected by Buzzfeed. I'm not a fan of clickbait either, but their corpus is publicly available here: https://docs.google.com/spreadsheets/d/1ysnzawW6pDGBEqbXqeYuzWa7Rx2mQUip6CXUUUk4jIk/edit#gid=1756764129 https://docs.google.com/spreadsheets/d/1ysnzawW6pDGBEqbXqeYu... So as far as I'm concerned, so long as the authors acknowledge their source (which they did) then there's nothing wrong here. Even FiveThirtyEight builds most of their models from datasets obtained from third parties. A lot of data science research is carried out using public datasets because collection is often an expensive or time-consuming endeavor, especially for social science research. The authors even say in their acknowledgement "We thank Craig Silverman, Lauren Strapagiel, Hamza Shaban, Ellie Hall, and Jeremy Singer-Vine from BuzzFeed for making their data available, enabling our research".

rdtsc 10y ago

> were manually fact-checked by professional journalists from BuzzFeed. Anyone else stopped reading there? Because I did.

webmaven 10y ago
Understandable, but BuzzFeed's considerable profits from clickbait articles are being used to fund serious journalism: https://www.poynter.org/2016/how-buzzfeed-built-an-investigative-team-from-the-ground-up/396656/ https://www.poynter.org/2016/how-buzzfeed-built-an-investiga...