6 ms·

“Fake News” dataset on Kaggle

32 points by jo_kruger 10y ago

trendia 10y ago
ronpaulinstitute.org is listed as fake news? How did they determine what was fake or not?
- rusew 10y ago
  That list is not just fake news but also ones with extreme bias. Source: the domain classifications in the project readme.
- ominous 10y ago
  Following the link, here's the source: https://github.com/selfagency/bs-detector/blob/master/chrome/data/data.json https://github.com/selfagency/bs-detector/blob/master/chrome... As per the author of the dataset on kaggle: contains text and metadata scraped from 244 websites tagged as "bullshit" here by the BS Detector Chrome Extension by Daniel Sieradski. Warning: I did not modify the list of news sources from the BS Detector so as not to introduce my (useless) layer of bias
- inimino 10y ago
  Just a list of domains compiled by a guy. If you want to know more about the level of thinking behind the list of domains, this[1] page linked from the BS detector readme gives the general idea. I would say it's the product of a very earnest idealist who thinks he has solved the problem because he hasn't spent enough time thinking about the problem to understand the ways in which it is non-trivial. https://www.inverse.com/article/23781-bs-detector-facebook-fake-news-daniel-sieradski https://www.inverse.com/article/23781-bs-detector-facebook-f...
gadders 10y ago
It includes Breitbart and Drudge Report as fake. Those sites may have a bias you disagree with but no more so than, say, The Daily Kos or Mother Jones. Can anyone identify any left-of-centre site urls in the list
- akerro 10y ago
  Fake news is only what we disagree with.
  - derrickdirge 10y ago
    No. Completely unsourced material is not news, regardless of if we 'agree' with it. If completely unsourced material is labeled 'news' then this material could reasonably be called 'fake news'.
- hamax 10y ago
  It includes both fake and biased news (and a couple of other categories). Neither Breitbart or Drudge Report are marked as fake in the dataset.
  - gadders 10y ago
    Does it include The Guardian and the New York Times as biassed?
  - 551199 10y ago
    There isn't news that aren't biased.
- ZeroGravitas 10y ago
  Breitbart's ex-editor-at-large quit because they "shaped the company into Trump’s personal Pravda, to the extent that he abandoned and undercut his own reporter, Breitbart News’ Michelle Fields, in order to protect Trump’s bully campaign manager, Corey Lewandowski, who allegedly assaulted Michelle." Did anything like that happen at the sites you name?
  - trendia 10y ago
    Yes. At the NYT: "By and large, talented reporters scrambled to match stories with what internally was often called “the narrative.” We were occasionally asked to map a narrative for our various beats a year in advance, square the plan with editors, then generate stories that fit the pre-designated line." http://deadline.com/2016/11/shocked-by-trump-new-york-times-finds-time-for-soul-searching-1201852490/ http://deadline.com/2016/11/shocked-by-trump-new-york-times-...
ramblenode 10y ago
This data was collected using a recently developed tool called the BS Detector [0] which aims to automatically classify fake news. As the labels have not been validated by humans, I question how much utility this will have for developing additional fake news detection systems. Any model trained on this data is really just learning the parameters of the BS Detector, not ground truth fake news. [0] https://github.com/selfagency/bs-detector https://github.com/selfagency/bs-detector
- ManlyBread 10y ago
  Classify? I'd expect classification to be able to detect whether something is fake or not regardless of the source. Meanwhile this tool is nothing more than just a simple lookup of sites that the author deemed to be biased or fake. None of these are backed up with any proof regarding the claim that these sites provide "fake news" - most of these doesn't even have a reason on why these sites are there. If the author cannot provide a reason for these sites to be there then for what reason should I trust the author?
- dexen 10y ago
  To re-iterate what ManlyBread wrote above, the BS Detector tool uses a fixed list [1] of domains that are deemed to convey 'fake news'; there is no neural network / algorithm / heuristics being used in the tool. Just a blacklist with several categories, curated by humans. For example: {"url": "wikileaks.org", "type": "rumors"}, [1] https://github.com/selfagency/bs-detector/blob/master/chrome/data/data.json https://github.com/selfagency/bs-detector/blob/master/chrome...
- cheald 10y ago
  Yeah, it'll definitely need human filtering/tagging prior to training any serious model, but it's neat to have a corpus of stories to start to work from. I think the biggest issue in "fake news" detection is going to be the model-builders leaking as little of their biases into the model as possible. Objective truth is a tricky thing, and any model which proclaims to be able to represent The Truth should be viewed with extreme suspicion.
shadowban_me 10y ago
Thanks for this dataset. I'm glad to help establishment with free solutions. Maybe I'll build tools for myself that help me be more unaware.
AdamSC1 10y ago
I'm livid about the abundant existence and spread of 'fake news' and the inability people have to fact-check, but, I also want to play devil's advocate here. 1) This data set mostly relies on a data-set called the "BS Detector" which describes itself as "[a] hastily assembled...proof of concept...[that] searches all links on a given webpage for references to unreliable sources, checking against a manually compiled list of domains...[to] address the proliferation of fake news". While I agree with their choices in sites, the concept of manual curation deciding the legitimacy of news is horrifying and something that we hoped the internet would move us away from. 2) We run into this problem where we want places like Facebook to censor 'fake news'. But, I imagine most of us can agree that as autonomous moral agents, we'd never want out access to information censored. Rather we'd want the tools to make an informed decision ourselves. In accepting that, you'd have to extend that same right of autonomy to other individuals no matter how 'wrong' you think they may be. A) We can't manually curate legitimacy list. As that leads to more abuse, and let's face it people who read fake news wouldn't trust your list anyway. B) We can't let tech companies filter out websites based on their interpretation that leads to more bias especially politically. C) We can't let crowd-sourced intelligence do it without being locked into the tyranny of the majority, which as we've seen, spreads fake news. In a time when traditional news agencies are hurting for views, lost in the digital noise and often slower to respond to breaking news than niche outlets, we can't simply give them the benefit of the doubt either. (That would also enshrine them as some sort of news aristocracy which emerging outlets and voices couldn't compete with). It seems rather than try and interpret the news as fake based on the piece/site itself, we need to take a step back and look and how on the consumption platform (browser, news channel, social media) you can display relevant facts to people. [Unsubstantiated claim warning] I imagine most credible journalists have at one time another cited an irrefutable fact. A raw figure from a report, a percent by which a poll was won, or the weight of the Eiffel Tower. It seems that with data sets like WikiData and Firebase (old dump, not the new mobile platform) we would be able to locate claims of fact within the writings of journalists and outlets and compare those against the irrefutable fact. If a journalist or outlet has a number of facts that are incorrectly stated well outside a margin of error, a little pop-up displays those to the user letting them decide how to interpret that. I know we want to label things as 'Fake' or 'Manipulative' media, and I do think it should be a criminal offense to knowingly publish false information outside the protections of entertainment/satire/art etc. But, no matter how we spin it, categorizing media is more destructive than it is helpful. We need to find ways to put the educated decision back in the hands of moral agents who are struggling to find facts in an overwhelming sea of noise.
web64 10y ago
The dataset doesn't contain the article URLs which makes further analysis harder. With the URLs you could look at who is linking to the article or website and use that as an indication of authority.
solotronics 10y ago
nice! the digital equivalent of safe spaces. perfect for when I want to only hear opinions that conform to my own.
garysieling 10y ago
I'd like to see a dataset like this that includes email forwards, e.g. from Snopes.

32 points by jo_kruger 10y ago

trendia 10y ago

ronpaulinstitute.org is listed as fake news? How did they determine what was fake or not?

rusew 10y ago
That list is not just fake news but also ones with extreme bias. Source: the domain classifications in the project readme.
ominous 10y ago
Following the link, here's the source: https://github.com/selfagency/bs-detector/blob/master/chrome/data/data.json https://github.com/selfagency/bs-detector/blob/master/chrome... As per the author of the dataset on kaggle: contains text and metadata scraped from 244 websites tagged as "bullshit" here by the BS Detector Chrome Extension by Daniel Sieradski. Warning: I did not modify the list of news sources from the BS Detector so as not to introduce my (useless) layer of bias
inimino 10y ago
Just a list of domains compiled by a guy. If you want to know more about the level of thinking behind the list of domains, this[1] page linked from the BS detector readme gives the general idea. I would say it's the product of a very earnest idealist who thinks he has solved the problem because he hasn't spent enough time thinking about the problem to understand the ways in which it is non-trivial. https://www.inverse.com/article/23781-bs-detector-facebook-fake-news-daniel-sieradski https://www.inverse.com/article/23781-bs-detector-facebook-f...

gadders 10y ago

It includes Breitbart and Drudge Report as fake. Those sites may have a bias you disagree with but no more so than, say, The Daily Kos or Mother Jones. Can anyone identify any left-of-centre site urls in the list

akerro 10y ago
Fake news is only what we disagree with.
- derrickdirge 10y ago
  No. Completely unsourced material is not news, regardless of if we 'agree' with it. If completely unsourced material is labeled 'news' then this material could reasonably be called 'fake news'.
hamax 10y ago
It includes both fake and biased news (and a couple of other categories). Neither Breitbart or Drudge Report are marked as fake in the dataset.
- gadders 10y ago
  Does it include The Guardian and the New York Times as biassed?
- 551199 10y ago
  There isn't news that aren't biased.
ZeroGravitas 10y ago
Breitbart's ex-editor-at-large quit because they "shaped the company into Trump’s personal Pravda, to the extent that he abandoned and undercut his own reporter, Breitbart News’ Michelle Fields, in order to protect Trump’s bully campaign manager, Corey Lewandowski, who allegedly assaulted Michelle." Did anything like that happen at the sites you name?
- trendia 10y ago
  Yes. At the NYT: "By and large, talented reporters scrambled to match stories with what internally was often called “the narrative.” We were occasionally asked to map a narrative for our various beats a year in advance, square the plan with editors, then generate stories that fit the pre-designated line." http://deadline.com/2016/11/shocked-by-trump-new-york-times-finds-time-for-soul-searching-1201852490/ http://deadline.com/2016/11/shocked-by-trump-new-york-times-...

ramblenode 10y ago

This data was collected using a recently developed tool called the BS Detector [0] which aims to automatically classify fake news. As the labels have not been validated by humans, I question how much utility this will have for developing additional fake news detection systems. Any model trained on this data is really just learning the parameters of the BS Detector, not ground truth fake news. [0] https://github.com/selfagency/bs-detector https://github.com/selfagency/bs-detector

ManlyBread 10y ago
Classify? I'd expect classification to be able to detect whether something is fake or not regardless of the source. Meanwhile this tool is nothing more than just a simple lookup of sites that the author deemed to be biased or fake. None of these are backed up with any proof regarding the claim that these sites provide "fake news" - most of these doesn't even have a reason on why these sites are there. If the author cannot provide a reason for these sites to be there then for what reason should I trust the author?
dexen 10y ago
To re-iterate what ManlyBread wrote above, the BS Detector tool uses a fixed list [1] of domains that are deemed to convey 'fake news'; there is no neural network / algorithm / heuristics being used in the tool. Just a blacklist with several categories, curated by humans. For example: {"url": "wikileaks.org", "type": "rumors"}, [1] https://github.com/selfagency/bs-detector/blob/master/chrome/data/data.json https://github.com/selfagency/bs-detector/blob/master/chrome...
cheald 10y ago
Yeah, it'll definitely need human filtering/tagging prior to training any serious model, but it's neat to have a corpus of stories to start to work from. I think the biggest issue in "fake news" detection is going to be the model-builders leaking as little of their biases into the model as possible. Objective truth is a tricky thing, and any model which proclaims to be able to represent The Truth should be viewed with extreme suspicion.

shadowban_me 10y ago

Thanks for this dataset. I'm glad to help establishment with free solutions. Maybe I'll build tools for myself that help me be more unaware.

AdamSC1 10y ago

I'm livid about the abundant existence and spread of 'fake news' and the inability people have to fact-check, but, I also want to play devil's advocate here. 1) This data set mostly relies on a data-set called the "BS Detector" which describes itself as "[a] hastily assembled...proof of concept...[that] searches all links on a given webpage for references to unreliable sources, checking against a manually compiled list of domains...[to] address the proliferation of fake news". While I agree with their choices in sites, the concept of manual curation deciding the legitimacy of news is horrifying and something that we hoped the internet would move us away from. 2) We run into this problem where we want places like Facebook to censor 'fake news'. But, I imagine most of us can agree that as autonomous moral agents, we'd never want out access to information censored. Rather we'd want the tools to make an informed decision ourselves. In accepting that, you'd have to extend that same right of autonomy to other individuals no matter how 'wrong' you think they may be. A) We can't manually curate legitimacy list. As that leads to more abuse, and let's face it people who read fake news wouldn't trust your list anyway. B) We can't let tech companies filter out websites based on their interpretation that leads to more bias especially politically. C) We can't let crowd-sourced intelligence do it without being locked into the tyranny of the majority, which as we've seen, spreads fake news. In a time when traditional news agencies are hurting for views, lost in the digital noise and often slower to respond to breaking news than niche outlets, we can't simply give them the benefit of the doubt either. (That would also enshrine them as some sort of news aristocracy which emerging outlets and voices couldn't compete with). It seems rather than try and interpret the news as fake based on the piece/site itself, we need to take a step back and look and how on the consumption platform (browser, news channel, social media) you can display relevant facts to people. [Unsubstantiated claim warning] I imagine most credible journalists have at one time another cited an irrefutable fact. A raw figure from a report, a percent by which a poll was won, or the weight of the Eiffel Tower. It seems that with data sets like WikiData and Firebase (old dump, not the new mobile platform) we would be able to locate claims of fact within the writings of journalists and outlets and compare those against the irrefutable fact. If a journalist or outlet has a number of facts that are incorrectly stated well outside a margin of error, a little pop-up displays those to the user letting them decide how to interpret that. I know we want to label things as 'Fake' or 'Manipulative' media, and I do think it should be a criminal offense to knowingly publish false information outside the protections of entertainment/satire/art etc. But, no matter how we spin it, categorizing media is more destructive than it is helpful. We need to find ways to put the educated decision back in the hands of moral agents who are struggling to find facts in an overwhelming sea of noise.

web64 10y ago

The dataset doesn't contain the article URLs which makes further analysis harder. With the URLs you could look at who is linking to the article or website and use that as an indication of authority.

solotronics 10y ago

nice! the digital equivalent of safe spaces. perfect for when I want to only hear opinions that conform to my own.

garysieling 10y ago

I'd like to see a dataset like this that includes email forwards, e.g. from Snopes.