3 ms·

> Easy != Informative. Oh sure, absolutely, but note I said "easiest to implement". And there's that cost measure again, as if it's the only KPI. How do we mea

by kitd 2y ago

> Easy != Informative. Oh sure, absolutely, but note I said "easiest to implement". And there's that cost measure again, as if it's the only KPI. How do we measure the quality of the "quality" system? :)

dredmorbius 2y ago
My thoughts on assessment of a rating system leans strongly on audit methodologies. It also starts with what the goals of a rating system are, and how it might be subverted or attacked. I see four key principles or assumptions: 1. The reported ratings for overall product performance should correspond with some reasonable informed-experts' ratings. 2. Issues with product support (returns, refunds, repairs, replacements, etc.) should correspond with identifiable events. 3. Solicitation of ratings, false positive ratings, or false negative ratings as forms of harassment or industrial subversion are all obvious (and extant) forms of abuse, as would be the ratings platform itself predicating positive reviews on, say, advertising purchases by the reviewed site, as has been alleged in numerous cases. 4. Systematic abuse will tend to be orchestrated by a given entity, and should exhibit some form of behavioural or networked anomalies. For key products (high volume, high-price, and/or both), independent assessments should be performed on at least some sample of products and conformity with assessed ratings shown. Where expert opinions can be reasonably attained, they should generally take priority over crowdsourced reivews, especially where specific expertise and tangible distinction in results exists. This is guided by both the notion that you'd rather have a skilled pilot than a democratic representation of passengers flying an aeroplane (a notion dating back to Plato, though he somewhat unambitiously used the example of a ship at sea), and awareness that in some cases there isn't much consistency even amongst experts, as with wine tastings and ratings. Third parties (e.g., Consumers' Union, Underwriters Laboratories) might also be compensated for unbiased expert assessments. Individual raters with high volumes, anomalous ratings, or patterns of coordination with other raters (e.g., similar ratings on the same products) should be flagged for review, and potentially voided from the system. Any indications of suppliers soliciting ratings, providing kickbacks or rebates, or otherwise inducing positive ratings should be addressed severely. I'd be in favour of a quick and long-term minimum-rating consequence. This of course raises the risk of suppliers "joe-jobbing" competitors' offerings to create the appearance of manipulation (by the target). That activity should result in an even harsher consequence. The overall rating system and activity should be constantly monitored for anomolous patterns. Highly-active individual accounts are probably the easiest to detect. Large groups of low-activity accounts with similar activity are probably the hardest to detect, but would be another key issue. Audits of such behaviours should be a constant part of a system, and the subject of regular transparency reports.
dredmorbius 2y ago
What elements of the quality system's quality would you think need assessing? (I think your question's a valid one, and a quite good one ... a high-quality question, if you will. I also Have Thoughts, but would prefer to see you expand your question before polluting the idea space with my own biases and opinions. Though if I may steer the conversation somewhat: audits. Of what and how would be informed to your response to my first question though.)

by kitd 2y ago

dredmorbius 2y ago

My thoughts on assessment of a rating system leans strongly on audit methodologies. It also starts with what the goals of a rating system are, and how it might be subverted or attacked. I see four key principles or assumptions: 1. The reported ratings for overall product performance should correspond with some reasonable informed-experts' ratings. 2. Issues with product support (returns, refunds, repairs, replacements, etc.) should correspond with identifiable events. 3. Solicitation of ratings, false positive ratings, or false negative ratings as forms of harassment or industrial subversion are all obvious (and extant) forms of abuse, as would be the ratings platform itself predicating positive reviews on, say, advertising purchases by the reviewed site, as has been alleged in numerous cases. 4. Systematic abuse will tend to be orchestrated by a given entity, and should exhibit some form of behavioural or networked anomalies. For key products (high volume, high-price, and/or both), independent assessments should be performed on at least some sample of products and conformity with assessed ratings shown. Where expert opinions can be reasonably attained, they should generally take priority over crowdsourced reivews, especially where specific expertise and tangible distinction in results exists. This is guided by both the notion that you'd rather have a skilled pilot than a democratic representation of passengers flying an aeroplane (a notion dating back to Plato, though he somewhat unambitiously used the example of a ship at sea), and awareness that in some cases there isn't much consistency even amongst experts, as with wine tastings and ratings. Third parties (e.g., Consumers' Union, Underwriters Laboratories) might also be compensated for unbiased expert assessments. Individual raters with high volumes, anomalous ratings, or patterns of coordination with other raters (e.g., similar ratings on the same products) should be flagged for review, and potentially voided from the system. Any indications of suppliers soliciting ratings, providing kickbacks or rebates, or otherwise inducing positive ratings should be addressed severely. I'd be in favour of a quick and long-term minimum-rating consequence. This of course raises the risk of suppliers "joe-jobbing" competitors' offerings to create the appearance of manipulation (by the target). That activity should result in an even harsher consequence. The overall rating system and activity should be constantly monitored for anomolous patterns. Highly-active individual accounts are probably the easiest to detect. Large groups of low-activity accounts with similar activity are probably the hardest to detect, but would be another key issue. Audits of such behaviours should be a constant part of a system, and the subject of regular transparency reports.

dredmorbius 2y ago

What elements of the quality system's quality would you think need assessing? (I think your question's a valid one, and a quite good one ... a high-quality question, if you will. I also Have Thoughts, but would prefer to see you expand your question before polluting the idea space with my own biases and opinions. Though if I may steer the conversation somewhat: audits. Of what and how would be informed to your response to my first question though.)