4 ms·

The unfortunate reality is most people will not apply such guidance regularly given the torrent of information we are faced with. Pardon the self-promotion but

The unfortunate reality is most people will not apply such guidance regularly given the torrent of information we are faced with. Pardon the self-promotion but my company, The Factual, has built a chrome extension that evaluates how opinionated and well-sourced a news article is and gives you a simple red/yellow/green rating. We're transparent with details behind the rating. Not as good as The Verge's advice but hopefully more pragmatic to apply regularly. Feedback appreciated. https://chrome.google.com/webstore/detail/civikowl/clbbiejjicefdjlblgnojolgbideklkp?hl=en https://chrome.google.com/webstore/detail/civikowl/clbbiejji...

mistermann 7y ago
It's good to see anyone working in this space, but I'm skeptical about the trustworthiness of the results one can expect from this sort of approach, with current technology. From https://www.thefactual.com/static/extfaq.html https://www.thefactual.com/static/extfaq.html -------------------------------- The Factual automatically calculates the credibility of an article based on four factors: a) The diversity and quality of its sources b) The factual tone of the article's writing style c) The expertise of the journalist on the topic based on historical focus d) The site reputation based on historical scores of every article on the site Because the calculation is automated, without human involvement, criteria are consistently applied across articles and sources. -------------------------------- a) Value can be derived this way, but "quality" seems subject to bias, intentional or not b) I'm curious how this would be accomplished - it's a huge problem, but most human minds do extremely poorly at the task in my experience c) "Expertise" often has the same issues as "quality" d) This seems fairly straightforward That such issues are not noted in "How reliable are the scores on articles?" is also somewhat unsettling. I might install this and give it a spin, because I would have great fun dissecting articles that get a high score but are guilty of ideological framing/bias and other popular rhetorical techniques.
- amoorthy 7y ago
  Thanks for the thoughtful feedback. Answers to your questions/comments: 1. "Quality" of each source is a historical rating for each site. This varies as sites write higher or lower scoring articles over time. So not a judgment call we are making. 2. Factual tone is a set of NLP algorithms evaluating the degree and extent of emotional words used as a percentage of the total text. There are also other indicators like how many first person statements are present, unnecessary adverbs etc. Not perfect but directionally accurate and useful as part of the overall rubric. 3. Author expertise does not have subjectivity. Articles are classified into one of several subject areas and we look at how often the author's prior work falls in the same subject area and how well those prior articles rated. I'll add to our FAQ after your response to see which issues remain unresolved after my comments above. Thanks again.
  - mistermann 7y ago
    > 1. "Quality" of each source is a historical rating for each site ("cited" third site"). This varies as sites write higher or lower scoring articles over time. So an average of the score of all articles you're previously processed, that cite that third site as a reference? I can certainly think of plenty of sites where this would be non-problematic, but some sites I could see having very high variance. Do you account for this in any way? > 2. Factual tone is a set of NLP algorithms evaluating the degree and extent of emotional words used as a percentage of the total text. There are also other indicators like how many first person statements are present, unnecessary adverbs etc. Not perfect but directionally accurate and useful as part of the overall rubric. My intuition tells me a skilled writer may score ok on this, whereas a human reviewer might catch things. > 3. Author expertise does not have subjectivity. Articles are classified into one of several subject areas and we look at how often the author's prior work falls in the same subject area and how well those prior articles rated. I'm not saying this is necessarily a big deal, but there are some obvious shortcomings. Overall, this seems like the type of tool that is sorely needed, so I hope you find success. I'm particularly interested in it for how contentious of an issue it is. It would be cool (and good for credibility I'd think, as well as potentially providing you some free labor if you allowed feedback) if you had a public accessible page on your website where people could manually submit links and see how your engine ranks each of the 4 attributes. Any chance you might do such a thing?

mistermann 7y ago

It's good to see anyone working in this space, but I'm skeptical about the trustworthiness of the results one can expect from this sort of approach, with current technology. From https://www.thefactual.com/static/extfaq.html https://www.thefactual.com/static/extfaq.html -------------------------------- The Factual automatically calculates the credibility of an article based on four factors: a) The diversity and quality of its sources b) The factual tone of the article's writing style c) The expertise of the journalist on the topic based on historical focus d) The site reputation based on historical scores of every article on the site Because the calculation is automated, without human involvement, criteria are consistently applied across articles and sources. -------------------------------- a) Value can be derived this way, but "quality" seems subject to bias, intentional or not b) I'm curious how this would be accomplished - it's a huge problem, but most human minds do extremely poorly at the task in my experience c) "Expertise" often has the same issues as "quality" d) This seems fairly straightforward That such issues are not noted in "How reliable are the scores on articles?" is also somewhat unsettling. I might install this and give it a spin, because I would have great fun dissecting articles that get a high score but are guilty of ideological framing/bias and other popular rhetorical techniques.

amoorthy 7y ago
Thanks for the thoughtful feedback. Answers to your questions/comments: 1. "Quality" of each source is a historical rating for each site. This varies as sites write higher or lower scoring articles over time. So not a judgment call we are making. 2. Factual tone is a set of NLP algorithms evaluating the degree and extent of emotional words used as a percentage of the total text. There are also other indicators like how many first person statements are present, unnecessary adverbs etc. Not perfect but directionally accurate and useful as part of the overall rubric. 3. Author expertise does not have subjectivity. Articles are classified into one of several subject areas and we look at how often the author's prior work falls in the same subject area and how well those prior articles rated. I'll add to our FAQ after your response to see which issues remain unresolved after my comments above. Thanks again.
- mistermann 7y ago
  > 1. "Quality" of each source is a historical rating for each site ("cited" third site"). This varies as sites write higher or lower scoring articles over time. So an average of the score of all articles you're previously processed, that cite that third site as a reference? I can certainly think of plenty of sites where this would be non-problematic, but some sites I could see having very high variance. Do you account for this in any way? > 2. Factual tone is a set of NLP algorithms evaluating the degree and extent of emotional words used as a percentage of the total text. There are also other indicators like how many first person statements are present, unnecessary adverbs etc. Not perfect but directionally accurate and useful as part of the overall rubric. My intuition tells me a skilled writer may score ok on this, whereas a human reviewer might catch things. > 3. Author expertise does not have subjectivity. Articles are classified into one of several subject areas and we look at how often the author's prior work falls in the same subject area and how well those prior articles rated. I'm not saying this is necessarily a big deal, but there are some obvious shortcomings. Overall, this seems like the type of tool that is sorely needed, so I hope you find success. I'm particularly interested in it for how contentious of an issue it is. It would be cool (and good for credibility I'd think, as well as potentially providing you some free labor if you allowed feedback) if you had a public accessible page on your website where people could manually submit links and see how your engine ranks each of the 4 attributes. Any chance you might do such a thing?