4 ms·

Good point regarding pervasive bias in journalism, intentional or not. I've been thinking about how to help people identify bias and poor quality citations and

Good point regarding pervasive bias in journalism, intentional or not. I've been thinking about how to help people identify bias and poor quality citations and built a chrome plug-in to do so. This is Beta but would appreciate any feedback you all have. https://chrome.google.com/webstore/detail/civikowl/clbbiejjicefdjlblgnojolgbideklkp?hl=en https://chrome.google.com/webstore/detail/civikowl/clbbiejji...

KGIII 9y ago
I am tablet-bound, at the moment. I'll check it out, probably over the weekend. I've emailed myself a link. I will check to see if I can contact you through the site - though I will be using it in Opera. It does make me question this, however. How are you certain you're not introducing your own biases with it? Or, perhaps not even biases of your own? Sometimes bias exists simply by virtue of externalities. An example might be that, when differentiated by economic ability, black people don't actually do more drugs than white people - though crime stats would indicate otherwise. The difference is their neighborhoods are policed more heavily and the biases in enforcement. Notably, even black officers may be biased in their enforcement. It really seems like a tough nut to crack. I'm a mathematician, not a social scientist. I don't actually have any idea how to fix this.
- amoorthy 9y ago
  Thank you very much for offering to take a look. Btw, you can email feedback directly to me at amoorthy at civikowl dot com. I'll also check this thread regularly if that's easier for you. (Oh, and only works on Chrome for now; sorry) We may introduce some bias through our algorithm, insofar as the weights we assign various factors favor one type of article over another. But we haven't used a training set yet (which is where ingrained biases would be factored in) so the algorithm is fairly simple for now. I agree with you this isn't an easy problem to crack. We wanted to see if (a) there's interest in a solution and (b) reservations about our approach before investing more. So this is exactly the kind of feedback that's helpful. Thank you again.
  - KGIII 9y ago
    I have it working in Opera. I am unavailable for a few hours but I will try to give it some unbiased testing, later tonight and over the weekend. I'm not heavy into the soft sciences. I'm a mathematician (complete with Ph.D.!) if that'll help you? I'm not sure if you're open source. To be truly objective, I may actually need to see that. I'm very much retired - and willing to sign both an NDA and a non-compete, if you want. If you don't want, I'm not sure how much I can offer other than observations. It will also take a bunch of research to the methodology, but I'm not actually doing anything better.
    - amoorthy 9y ago
      Wow! Thanks for offering to help. Can you email me at amoorthy at civikowl dot com and we can figure out how best to work together?
croon 9y ago
I just tried it out for a bit, and went around a few articles. I'm going to approach this from assumptions, and then thoughts based on those, and you can correct me if I'm wrong. * It grades based (purely?) on linked sources. * It grades those sources based on their host's bias and credibility ranking (sourced by mediabiasfactcheck.com?). Given those, the extension's rating of an "objectively good" outlet and an "objectively bad" outlet referencing themselves in a followup article would differ only based on some databases credibility ranking? If the user only trusted this extension, I could as an outlet easily trick it by linking in external sources such as quackblog.biz and truthiness.zone, while decrying a link from biasedrag.com to present the veneer of offering counterpoints. I applaud the effort, but I don't think looking at sources for news works, as outside of science news, they are usually the first to post on things, and thus the only "news" linking to sources, are second-hand outlets or blogs. The first reuters article I tried for example didn't get a rating: https://www.reuters.com/article/us-brazil-amazon-mining/brazil-judge-blocks-decree-opening-amazon-area-to-mining-idUSKCN1BA266 https://www.reuters.com/article/us-brazil-amazon-mining/braz... Without transparency, you as the extension author becomes the central arbiter of news, which is IMHO more dangerous than a status quo of outlets. A lot of news that is factual, unbiased, etc don't require a counterpoint or opposing views as a fact at its core is a fact. A lot of topics are of course aggregates where things get muddy, and statistics involve a lot of assumptions. I like the idea, but I'm unsure of the possibility of the venture.
- amoorthy 9y ago
  Wow, thank you for writing out such detailed feedback! Your initial assumptions are correct on the inputs to the algorithm. >the extension's rating of an "objectively good" outlet and an "objectively bad" outlet referencing themselves in a followup article would differ only based on some databases credibility ranking? I may not understand your question. But there's an algorithm that factors in count of sources, diversity of sources, political leaning of sources, credibility of sources, and (soon) author depth of experience in that topic. So it's more than simply referencing a database. Does that give you more confidence in what the rating or not enough? >If the user only trusted this extension, I could as an outlet easily trick it by linking in external sources such as quackblog.biz and truthiness.zone, while decrying a link from biasedrag.com to present the veneer of offering counterpoints. True. You'd have to link to sites that are rated by Mediabiasfactcheck (MBFC) so you can't link to random stuff and get a high rating. And if the author was trying to game the rating at least they did decry the points from an alternate viewpoint; that's better than what happens with the news today. MBFC is not perfect of course but their methodology follows the Fact-checkers’ Code of Principles developed by Poynter Institute so is generally consistent and high quality. >I applaud the effort, but I don't think looking at sources for news works, as outside of science news, they are usually the first to post on things, and thus the only "news" linking to sources, are second-hand outlets or blogs. Fair point on breaking news. We wanted to first solve for non-breaking-news (i.e. issue centric), which do have sources and those range widely in quality. i.e. most things discussed on HN are not breaking news but deeper news. On breaking news we plan to have some calculations based on source info like tweets/posts/videos as well as author's expertise in that topic. What do you think? The Reuters article has no links hence we cannot rate it. Not great but maybe helping people to get accustom to looking at sources, or lack there of, is good in and of itself? >Without transparency, you as the extension author becomes the central arbiter of news, which is IMHO more dangerous than a status quo of outlets. Agreed. I'll update our FAQ to be clearer on how we do the rating. (Just rolled out yesterday so a bit slow on the documentation; sorry). I hope we can make clear that there is no objective arbiter of bias but rather such tools can help you get smarter about evaluating the news you read. >A lot of news that is factual, unbiased, etc don't require a counterpoint or opposing views as a fact at its core is a fact. A lot of topics are of course aggregates where things get muddy, and statistics involve a lot of assumptions. Agreed on both points, though I might counter that there's no such thing as unbiased news just because of topic selection and language used. Anyway, our algorithm doesn't require that counter-points be considered though it does evaluate that. So an article with high quality sources that lean one way politically can score ok but we'll notify the user of this lean preference. We are unlikely to be able to say a claim is true or false, or that statistics are selectively being used to prove a point. That's pretty tough to do with an algorithm. But maybe the counterpoint sourcing can at least hint at if the author has considered this? >I like the idea, but I'm unsure of the possibility of the venture. We wanted to see if the idea has merit and your sentence is exactly what I was looking for. So thank you and thank you for all the feedback. As for the venture... I hear you, we have more to do. Please let me know of any other comments. I'm at amoorthy at civikowl dot com.

KGIII 9y ago

I am tablet-bound, at the moment. I'll check it out, probably over the weekend. I've emailed myself a link. I will check to see if I can contact you through the site - though I will be using it in Opera. It does make me question this, however. How are you certain you're not introducing your own biases with it? Or, perhaps not even biases of your own? Sometimes bias exists simply by virtue of externalities. An example might be that, when differentiated by economic ability, black people don't actually do more drugs than white people - though crime stats would indicate otherwise. The difference is their neighborhoods are policed more heavily and the biases in enforcement. Notably, even black officers may be biased in their enforcement. It really seems like a tough nut to crack. I'm a mathematician, not a social scientist. I don't actually have any idea how to fix this.

amoorthy 9y ago
Thank you very much for offering to take a look. Btw, you can email feedback directly to me at amoorthy at civikowl dot com. I'll also check this thread regularly if that's easier for you. (Oh, and only works on Chrome for now; sorry) We may introduce some bias through our algorithm, insofar as the weights we assign various factors favor one type of article over another. But we haven't used a training set yet (which is where ingrained biases would be factored in) so the algorithm is fairly simple for now. I agree with you this isn't an easy problem to crack. We wanted to see if (a) there's interest in a solution and (b) reservations about our approach before investing more. So this is exactly the kind of feedback that's helpful. Thank you again.
- KGIII 9y ago
  I have it working in Opera. I am unavailable for a few hours but I will try to give it some unbiased testing, later tonight and over the weekend. I'm not heavy into the soft sciences. I'm a mathematician (complete with Ph.D.!) if that'll help you? I'm not sure if you're open source. To be truly objective, I may actually need to see that. I'm very much retired - and willing to sign both an NDA and a non-compete, if you want. If you don't want, I'm not sure how much I can offer other than observations. It will also take a bunch of research to the methodology, but I'm not actually doing anything better.
  - amoorthy 9y ago
    Wow! Thanks for offering to help. Can you email me at amoorthy at civikowl dot com and we can figure out how best to work together?

croon 9y ago

I just tried it out for a bit, and went around a few articles. I'm going to approach this from assumptions, and then thoughts based on those, and you can correct me if I'm wrong. * It grades based (purely?) on linked sources. * It grades those sources based on their host's bias and credibility ranking (sourced by mediabiasfactcheck.com?). Given those, the extension's rating of an "objectively good" outlet and an "objectively bad" outlet referencing themselves in a followup article would differ only based on some databases credibility ranking? If the user only trusted this extension, I could as an outlet easily trick it by linking in external sources such as quackblog.biz and truthiness.zone, while decrying a link from biasedrag.com to present the veneer of offering counterpoints. I applaud the effort, but I don't think looking at sources for news works, as outside of science news, they are usually the first to post on things, and thus the only "news" linking to sources, are second-hand outlets or blogs. The first reuters article I tried for example didn't get a rating: https://www.reuters.com/article/us-brazil-amazon-mining/brazil-judge-blocks-decree-opening-amazon-area-to-mining-idUSKCN1BA266 https://www.reuters.com/article/us-brazil-amazon-mining/braz... Without transparency, you as the extension author becomes the central arbiter of news, which is IMHO more dangerous than a status quo of outlets. A lot of news that is factual, unbiased, etc don't require a counterpoint or opposing views as a fact at its core is a fact. A lot of topics are of course aggregates where things get muddy, and statistics involve a lot of assumptions. I like the idea, but I'm unsure of the possibility of the venture.

amoorthy 9y ago
Wow, thank you for writing out such detailed feedback! Your initial assumptions are correct on the inputs to the algorithm. >the extension's rating of an "objectively good" outlet and an "objectively bad" outlet referencing themselves in a followup article would differ only based on some databases credibility ranking? I may not understand your question. But there's an algorithm that factors in count of sources, diversity of sources, political leaning of sources, credibility of sources, and (soon) author depth of experience in that topic. So it's more than simply referencing a database. Does that give you more confidence in what the rating or not enough? >If the user only trusted this extension, I could as an outlet easily trick it by linking in external sources such as quackblog.biz and truthiness.zone, while decrying a link from biasedrag.com to present the veneer of offering counterpoints. True. You'd have to link to sites that are rated by Mediabiasfactcheck (MBFC) so you can't link to random stuff and get a high rating. And if the author was trying to game the rating at least they did decry the points from an alternate viewpoint; that's better than what happens with the news today. MBFC is not perfect of course but their methodology follows the Fact-checkers’ Code of Principles developed by Poynter Institute so is generally consistent and high quality. >I applaud the effort, but I don't think looking at sources for news works, as outside of science news, they are usually the first to post on things, and thus the only "news" linking to sources, are second-hand outlets or blogs. Fair point on breaking news. We wanted to first solve for non-breaking-news (i.e. issue centric), which do have sources and those range widely in quality. i.e. most things discussed on HN are not breaking news but deeper news. On breaking news we plan to have some calculations based on source info like tweets/posts/videos as well as author's expertise in that topic. What do you think? The Reuters article has no links hence we cannot rate it. Not great but maybe helping people to get accustom to looking at sources, or lack there of, is good in and of itself? >Without transparency, you as the extension author becomes the central arbiter of news, which is IMHO more dangerous than a status quo of outlets. Agreed. I'll update our FAQ to be clearer on how we do the rating. (Just rolled out yesterday so a bit slow on the documentation; sorry). I hope we can make clear that there is no objective arbiter of bias but rather such tools can help you get smarter about evaluating the news you read. >A lot of news that is factual, unbiased, etc don't require a counterpoint or opposing views as a fact at its core is a fact. A lot of topics are of course aggregates where things get muddy, and statistics involve a lot of assumptions. Agreed on both points, though I might counter that there's no such thing as unbiased news just because of topic selection and language used. Anyway, our algorithm doesn't require that counter-points be considered though it does evaluate that. So an article with high quality sources that lean one way politically can score ok but we'll notify the user of this lean preference. We are unlikely to be able to say a claim is true or false, or that statistics are selectively being used to prove a point. That's pretty tough to do with an algorithm. But maybe the counterpoint sourcing can at least hint at if the author has considered this? >I like the idea, but I'm unsure of the possibility of the venture. We wanted to see if the idea has merit and your sentence is exactly what I was looking for. So thank you and thank you for all the feedback. As for the venture... I hear you, we have more to do. Please let me know of any other comments. I'm at amoorthy at civikowl dot com.