3 ms·

Hence the obsession with customer reviews. In the absence of an objective measure of quality, the easiest to implement is a star rating of "what everyone else t

by kitd 2y ago

Hence the obsession with customer reviews. In the absence of an objective measure of quality, the easiest to implement is a star rating of "what everyone else thinks about this product".

dredmorbius 2y ago
Easy != Informative. Such review systems are trivially gamed, with exceedingly high incentives to do so. One of the underappreciated benefits of an established retailier with limited suppliers and a centralised buying operation, as with traditional brick-and-mortar establishments (Harrods, Costco, Macy's, etc.) is that those establishments can set their own minimum standards and select merchandise which is highly likely to provide high value to the end customer. The notion that "choice is good" falls flat in the reality that uninformed choice over a range of intentionally insufficient products is not good, and is in fact a form of gambling with low payoffs. Some retailers target high-end products (e.g., Harrods), some mid- or low-market (e.g., Costco). But by restricting upstream vendors, tracking satisfaction, and having their own testing and development of products or offerings, as well as return/warrantee programmes, they can assure a reasonable rate of satisfaction. I'm also finding retailers who sell manifest crap (o hai any Samsung home appliances) can be eliminated from my own list of acceptable outlets for future purchases.
- kitd 2y ago
  > Easy != Informative. Oh sure, absolutely, but note I said "easiest to implement". And there's that cost measure again, as if it's the only KPI. How do we measure the quality of the "quality" system? :)
  - dredmorbius 2y ago
    My thoughts on assessment of a rating system leans strongly on audit methodologies. It also starts with what the goals of a rating system are, and how it might be subverted or attacked. I see four key principles or assumptions: 1. The reported ratings for overall product performance should correspond with some reasonable informed-experts' ratings. 2. Issues with product support (returns, refunds, repairs, replacements, etc.) should correspond with identifiable events. 3. Solicitation of ratings, false positive ratings, or false negative ratings as forms of harassment or industrial subversion are all obvious (and extant) forms of abuse, as would be the ratings platform itself predicating positive reviews on, say, advertising purchases by the reviewed site, as has been alleged in numerous cases. 4. Systematic abuse will tend to be orchestrated by a given entity, and should exhibit some form of behavioural or networked anomalies. For key products (high volume, high-price, and/or both), independent assessments should be performed on at least some sample of products and conformity with assessed ratings shown. Where expert opinions can be reasonably attained, they should generally take priority over crowdsourced reivews, especially where specific expertise and tangible distinction in results exists. This is guided by both the notion that you'd rather have a skilled pilot than a democratic representation of passengers flying an aeroplane (a notion dating back to Plato, though he somewhat unambitiously used the example of a ship at sea), and awareness that in some cases there isn't much consistency even amongst experts, as with wine tastings and ratings. Third parties (e.g., Consumers' Union, Underwriters Laboratories) might also be compensated for unbiased expert assessments. Individual raters with high volumes, anomalous ratings, or patterns of coordination with other raters (e.g., similar ratings on the same products) should be flagged for review, and potentially voided from the system. Any indications of suppliers soliciting ratings, providing kickbacks or rebates, or otherwise inducing positive ratings should be addressed severely. I'd be in favour of a quick and long-term minimum-rating consequence. This of course raises the risk of suppliers "joe-jobbing" competitors' offerings to create the appearance of manipulation (by the target). That activity should result in an even harsher consequence. The overall rating system and activity should be constantly monitored for anomolous patterns. Highly-active individual accounts are probably the easiest to detect. Large groups of low-activity accounts with similar activity are probably the hardest to detect, but would be another key issue. Audits of such behaviours should be a constant part of a system, and the subject of regular transparency reports.
  - dredmorbius 2y ago
    What elements of the quality system's quality would you think need assessing? (I think your question's a valid one, and a quite good one ... a high-quality question, if you will. I also Have Thoughts, but would prefer to see you expand your question before polluting the idea space with my own biases and opinions. Though if I may steer the conversation somewhat: audits. Of what and how would be informed to your response to my first question though.)

by kitd 2y ago

Hence the obsession with customer reviews. In the absence of an objective measure of quality, the easiest to implement is a star rating of "what everyone else thinks about this product".

dredmorbius 2y ago

Easy != Informative. Such review systems are trivially gamed, with exceedingly high incentives to do so. One of the underappreciated benefits of an established retailier with limited suppliers and a centralised buying operation, as with traditional brick-and-mortar establishments (Harrods, Costco, Macy's, etc.) is that those establishments can set their own minimum standards and select merchandise which is highly likely to provide high value to the end customer. The notion that "choice is good" falls flat in the reality that uninformed choice over a range of intentionally insufficient products is not good, and is in fact a form of gambling with low payoffs. Some retailers target high-end products (e.g., Harrods), some mid- or low-market (e.g., Costco). But by restricting upstream vendors, tracking satisfaction, and having their own testing and development of products or offerings, as well as return/warrantee programmes, they can assure a reasonable rate of satisfaction. I'm also finding retailers who sell manifest crap (o hai any Samsung home appliances) can be eliminated from my own list of acceptable outlets for future purchases.

kitd 2y ago
> Easy != Informative. Oh sure, absolutely, but note I said "easiest to implement". And there's that cost measure again, as if it's the only KPI. How do we measure the quality of the "quality" system? :)
- dredmorbius 2y ago
  My thoughts on assessment of a rating system leans strongly on audit methodologies. It also starts with what the goals of a rating system are, and how it might be subverted or attacked. I see four key principles or assumptions: 1. The reported ratings for overall product performance should correspond with some reasonable informed-experts' ratings. 2. Issues with product support (returns, refunds, repairs, replacements, etc.) should correspond with identifiable events. 3. Solicitation of ratings, false positive ratings, or false negative ratings as forms of harassment or industrial subversion are all obvious (and extant) forms of abuse, as would be the ratings platform itself predicating positive reviews on, say, advertising purchases by the reviewed site, as has been alleged in numerous cases. 4. Systematic abuse will tend to be orchestrated by a given entity, and should exhibit some form of behavioural or networked anomalies. For key products (high volume, high-price, and/or both), independent assessments should be performed on at least some sample of products and conformity with assessed ratings shown. Where expert opinions can be reasonably attained, they should generally take priority over crowdsourced reivews, especially where specific expertise and tangible distinction in results exists. This is guided by both the notion that you'd rather have a skilled pilot than a democratic representation of passengers flying an aeroplane (a notion dating back to Plato, though he somewhat unambitiously used the example of a ship at sea), and awareness that in some cases there isn't much consistency even amongst experts, as with wine tastings and ratings. Third parties (e.g., Consumers' Union, Underwriters Laboratories) might also be compensated for unbiased expert assessments. Individual raters with high volumes, anomalous ratings, or patterns of coordination with other raters (e.g., similar ratings on the same products) should be flagged for review, and potentially voided from the system. Any indications of suppliers soliciting ratings, providing kickbacks or rebates, or otherwise inducing positive ratings should be addressed severely. I'd be in favour of a quick and long-term minimum-rating consequence. This of course raises the risk of suppliers "joe-jobbing" competitors' offerings to create the appearance of manipulation (by the target). That activity should result in an even harsher consequence. The overall rating system and activity should be constantly monitored for anomolous patterns. Highly-active individual accounts are probably the easiest to detect. Large groups of low-activity accounts with similar activity are probably the hardest to detect, but would be another key issue. Audits of such behaviours should be a constant part of a system, and the subject of regular transparency reports.
- dredmorbius 2y ago
  What elements of the quality system's quality would you think need assessing? (I think your question's a valid one, and a quite good one ... a high-quality question, if you will. I also Have Thoughts, but would prefer to see you expand your question before polluting the idea space with my own biases and opinions. Though if I may steer the conversation somewhat: audits. Of what and how would be informed to your response to my first question though.)