4 ms·

This data was collected using a recently developed tool called the BS Detector [0] which aims to automatically classify fake news. As the labels have not been v

This data was collected using a recently developed tool called the BS Detector [0] which aims to automatically classify fake news. As the labels have not been validated by humans, I question how much utility this will have for developing additional fake news detection systems. Any model trained on this data is really just learning the parameters of the BS Detector, not ground truth fake news. [0] https://github.com/selfagency/bs-detector https://github.com/selfagency/bs-detector

ManlyBread 10y ago
Classify? I'd expect classification to be able to detect whether something is fake or not regardless of the source. Meanwhile this tool is nothing more than just a simple lookup of sites that the author deemed to be biased or fake. None of these are backed up with any proof regarding the claim that these sites provide "fake news" - most of these doesn't even have a reason on why these sites are there. If the author cannot provide a reason for these sites to be there then for what reason should I trust the author?
dexen 10y ago
To re-iterate what ManlyBread wrote above, the BS Detector tool uses a fixed list [1] of domains that are deemed to convey 'fake news'; there is no neural network / algorithm / heuristics being used in the tool. Just a blacklist with several categories, curated by humans. For example: {"url": "wikileaks.org", "type": "rumors"}, [1] https://github.com/selfagency/bs-detector/blob/master/chrome/data/data.json https://github.com/selfagency/bs-detector/blob/master/chrome...
cheald 10y ago
Yeah, it'll definitely need human filtering/tagging prior to training any serious model, but it's neat to have a corpus of stories to start to work from. I think the biggest issue in "fake news" detection is going to be the model-builders leaking as little of their biases into the model as possible. Objective truth is a tricky thing, and any model which proclaims to be able to represent The Truth should be viewed with extreme suspicion.

ManlyBread 10y ago

Classify? I'd expect classification to be able to detect whether something is fake or not regardless of the source. Meanwhile this tool is nothing more than just a simple lookup of sites that the author deemed to be biased or fake. None of these are backed up with any proof regarding the claim that these sites provide "fake news" - most of these doesn't even have a reason on why these sites are there. If the author cannot provide a reason for these sites to be there then for what reason should I trust the author?

dexen 10y ago

To re-iterate what ManlyBread wrote above, the BS Detector tool uses a fixed list [1] of domains that are deemed to convey 'fake news'; there is no neural network / algorithm / heuristics being used in the tool. Just a blacklist with several categories, curated by humans. For example: {"url": "wikileaks.org", "type": "rumors"}, [1] https://github.com/selfagency/bs-detector/blob/master/chrome/data/data.json https://github.com/selfagency/bs-detector/blob/master/chrome...

cheald 10y ago

Yeah, it'll definitely need human filtering/tagging prior to training any serious model, but it's neat to have a corpus of stories to start to work from. I think the biggest issue in "fake news" detection is going to be the model-builders leaking as little of their biases into the model as possible. Objective truth is a tricky thing, and any model which proclaims to be able to represent The Truth should be viewed with extreme suspicion.