4 ms·

What the....? The author describes a "fake news detector AI", that is actually a "typically legitimate source of news" data model, combined with a fake news do

What the....? The author describes a "fake news detector AI", that is actually a "typically legitimate source of news" data model, combined with a fake news domain blacklist. It doesn't detect fake news. It detects whether a story possibly came from a source you find to typically be legitimate. This article is fake news.

dantillberg 9y ago
Yeah; I don't doubt that there's a lot of value in the collection of data they built, but it's hard to judge the value of "95%" accuracy without comparison to a baseline like `is_fake = !whitelist.includes(article_domain)`. That whitelist is basically how our brains work currently, and my own (biased, perhaps) accuracy rate is something close to 100%. A "true fake news detector" would use only the text of the article -- without the URL. And so I agree that this article is kind of like fake news of the "misleading" variety. :)
- eropple 9y ago
  Why would reputational analysis of sources not be part of analyzing news trustworthiness?
  - wu-ikkyu 9y ago
    Just because someone is wrong most of the time doesn't mean they're wrong all the time. It's ad hominem
    - eropple 9y ago
      OK. And? Is the point to be perfect, or is the point to be useful? False fakes--and a wisely designed system would probably say "probably fake" in the first place, not outright "fake"--are less damaging than false truths overall.
      - wu-ikkyu 9y ago
        >False fakes--and a wisely designed system would probably say "probably fake" in the first place, not outright "fake"--are less damaging than false truths overall. That depends on what your goal is: to discover groundbreaking truths (which are almost always considered taboo/false at first) in order to improve society, or merely attempt to cover up apparent falsehoods as damage control. In the latter case, such a system would be highly prone to enforcing the prexisting biases and prejudices of its developers onto the users, making it much more difficult for them to discover truths that the developers are ignorant of.
throwawayjava 9y ago
Wait, is that true? From the article, it does seem that the training and test sets were both generating from a dubious data collection process: scraping URLs they knew were fake/satire/real and then applying a label based upon the domain. But, they had more features than just "source" in their model. so while it's possible their data collection method means that their model is way over-fit and basically just tests of proxies of "source", it's not prime facie obvious that this is the case...? Or am I missing something?
- minimaxir 9y ago
  The website lists domain name as a model feature: https://machinebox.io/docs/fakebox https://machinebox.io/docs/fakebox > Domain name - Some domains are known for hosting certain types of content, Fakebox knows about the most popular sites Fake news sites typically register bespoke domains, though.
  - throwawayjava 9y ago
    Oh, I misinterpreted the article. If the test data was collected in the same way that the training data was collected, this was a rather ridiculously round-about way of writing what should've been a 5 line perl script, and I'm really curious where the 5% error could've possibly even come from :-\
kingmanaz 9y ago
Personally speaking: family, friends, and coworkers have long equated the "Fake News" label with the pap put out by mainstream media monopolies, and, the agendas pushed by trendy social networking corporations. Many of the people included this cohort are working class democrats and self avowed "liberals". I find it odd that some think the problem with "fakeness" in news reporting is being caused by anything but monopolies over media outlets; curating sources and blacklisting hardly seems a solution. Try antitrust. https://en.wikipedia.org/wiki/Concentration_of_media_ownership https://en.wikipedia.org/wiki/Concentration_of_media_ownersh...
jhanschoo 9y ago
Nevertheless, this is of value since it distinguishes between articles with a lot of opinion and articles with little opinion. Even reputable newspapers like the NYT regularly publish news with a lot of opinion.

dantillberg 9y ago

Yeah; I don't doubt that there's a lot of value in the collection of data they built, but it's hard to judge the value of "95%" accuracy without comparison to a baseline like `is_fake = !whitelist.includes(article_domain)`. That whitelist is basically how our brains work currently, and my own (biased, perhaps) accuracy rate is something close to 100%. A "true fake news detector" would use only the text of the article -- without the URL. And so I agree that this article is kind of like fake news of the "misleading" variety. :)

eropple 9y ago
Why would reputational analysis of sources not be part of analyzing news trustworthiness?
- wu-ikkyu 9y ago
  Just because someone is wrong most of the time doesn't mean they're wrong all the time. It's ad hominem
  - eropple 9y ago
    OK. And? Is the point to be perfect, or is the point to be useful? False fakes--and a wisely designed system would probably say "probably fake" in the first place, not outright "fake"--are less damaging than false truths overall.
    - wu-ikkyu 9y ago
      >False fakes--and a wisely designed system would probably say "probably fake" in the first place, not outright "fake"--are less damaging than false truths overall. That depends on what your goal is: to discover groundbreaking truths (which are almost always considered taboo/false at first) in order to improve society, or merely attempt to cover up apparent falsehoods as damage control. In the latter case, such a system would be highly prone to enforcing the prexisting biases and prejudices of its developers onto the users, making it much more difficult for them to discover truths that the developers are ignorant of.

throwawayjava 9y ago

Wait, is that true? From the article, it does seem that the training and test sets were both generating from a dubious data collection process: scraping URLs they knew were fake/satire/real and then applying a label based upon the domain. But, they had more features than just "source" in their model. so while it's possible their data collection method means that their model is way over-fit and basically just tests of proxies of "source", it's not prime facie obvious that this is the case...? Or am I missing something?

minimaxir 9y ago
The website lists domain name as a model feature: https://machinebox.io/docs/fakebox https://machinebox.io/docs/fakebox > Domain name - Some domains are known for hosting certain types of content, Fakebox knows about the most popular sites Fake news sites typically register bespoke domains, though.
- throwawayjava 9y ago
  Oh, I misinterpreted the article. If the test data was collected in the same way that the training data was collected, this was a rather ridiculously round-about way of writing what should've been a 5 line perl script, and I'm really curious where the 5% error could've possibly even come from :-\

kingmanaz 9y ago

Personally speaking: family, friends, and coworkers have long equated the "Fake News" label with the pap put out by mainstream media monopolies, and, the agendas pushed by trendy social networking corporations. Many of the people included this cohort are working class democrats and self avowed "liberals". I find it odd that some think the problem with "fakeness" in news reporting is being caused by anything but monopolies over media outlets; curating sources and blacklisting hardly seems a solution. Try antitrust. https://en.wikipedia.org/wiki/Concentration_of_media_ownership https://en.wikipedia.org/wiki/Concentration_of_media_ownersh...

jhanschoo 9y ago

Nevertheless, this is of value since it distinguishes between articles with a lot of opinion and articles with little opinion. Even reputable newspapers like the NYT regularly publish news with a lot of opinion.