4 ms·

Consumerlabs, consumer reports, rtings, and which.co.uk seem to be the only standardized independent review bodies I'm aware of.

bombcar 3y ago
Consumer reports at least is known to be gamed (and they have specific review methodology you need to understand and interpret). It’s hard
- rootusrootus 3y ago
  > Consumer reports at least is known to be gamed I'd be interested to know more about this, but I haven't found much discussion of it when I search. My google fu may be weak today.
  - gandalfgreybeer 3y ago
    Not related to the topic on hand but I’ve felt my google fu diminish significantly over the past years. Not sure if it’s me or my google intuition just not matching the current algorithms
    - Baeocystin 3y ago
      It's not you, it's google. Seriously.
    - seventytwo 3y ago
      It’s google. Try Kagi.
  - somat 3y ago
    On the whole I think consumer reports is one of the better review organizations out there. but for a hint of reader betrayal read up on what they did to the suzuki samurai. It is not exactly cut and dried, but it was found that consumer reports was testing to get a specific result. And due to the trust that people put in consumer reports that result killed the market for the samurai in the US.
    - pseudalopex 3y ago
      Any examples after 1988?
      - somat 3y ago
        Not really, Like I said, I think they are one of the good guys, I am not sure if their testing methodology is any better than any one else. But they do try hard to remain independent from the manufacturers. I think of it as one black mark on their record and a reminder that even our heroes can slip up. By good methodology I think the best is the person who has a fleet of the things and a deep understanding of how they break, the next best is to take the thing apart and apply some industry specific analysis to it, then test it to destruction to see how it breaks. Ether one is beyond the grasp of the average joe, who only wants to buy one of the things and have it work. And almost no reviewers do this which is why the ones that do are so valuable.
        bombcar 3y ago
        Exactly this - they do try to remain independent but they sometimes obviously have biases towards certain operations of things. For example, the only thing I care about with toilets are that they don’t break and they always flush and clear. Literally nothing else matters, but to test that you gotta poop in them and that’s hard to reproduce scientifically. So weird substitutes like golf balls and whatnot are used instead. It’s also an integral problem with new car reviews, because you can’t review the durability of a new car, it’s an unknown. Substitutes end up dinging a car for a body panel alignment as much as a transmission replacement.
        jstarfish 3y ago
        > to test that you gotta poop in them and that’s hard to reproduce scientifically. So weird substitutes like golf balls and whatnot are used instead. Nah, just bring in some large dogs. You'll never be without mighty turds to dispose of. Beyond that, entire rolls of toilet paper, socks, makeup cassettes, cigarettes, toy cars, bleach wipes, pads, tampons (with and without plastic applicators) and other nonfecal items are the things you should be testing your toilet with, since those are what will go down your pipe in production. My FIL is the type to insist on buying whichever toilet can suck down the most golf balls. Pooping at his house means never having to sheepishly ask for a plunger. His is like an airplane toilet.
- wolverine876 3y ago
  > Consumer reports at least is known to be gamed I haven't heard of that. Could you tell us what you're referring to? > and they have specific review methodology you need to understand and interpret I've ready many Consumer Reports reviews; I don't know what this comment refers to. The style of the graphics in their tables? Their methodology is simply scientific method.
  - bombcar 3y ago
    The one I know best is the toilets, as the “scientific method” has nothing to do with actually using the toilet to eliminate human waste. And since the testing is known and not realistic, manufacturers can build for the test. https://seclists.org/politech/2000/Mar/73 https://seclists.org/politech/2000/Mar/73 You have to understand their methodology because they may be ascribing “points” to things you don’t care about at all - especially if you’re looking for longevity.
    - wolverine876 3y ago
      > https://seclists.org/politech/2000/Mar/73 https://seclists.org/politech/2000/Mar/73 The link is a random Internet person's post from 2000 about a single 1995 Consumer Reports review. > You have to understand their methodology because they may be ascribing “points” to things you don’t care about at all - especially if you’re looking for longevity. Every review everywhere, unless I hire the reviewer personally, will examine factors I am more and factors I am less interested in. They can't simultaneously, precisely serve everyone's varied interests (ok, I guess they could try some social media feed-type algorithm). > the “scientific method” has nothing to do with actually using the toilet to eliminate human waste. That's confusing. The scientific method is to actually do it and report what happens, not talk theoretically or speculatively about it from our basements like the rest of the Internet. CR generally does that (the scientific method) in extensive labs, staffed with experienced scientists and engineers. If you read their toilet reviews, that's what they do - put something in and flush. CR's core product is science, applied to consumer products. > especially if you’re looking for longevity. CR looks at longevity for many products, including extensive surveys of product owners.
zoltar 3y ago
I've found smaller categories of products with decent reviews on youtube. Project Farm and America's Test Kitchen spring to mind. I'm not sure they're as rigorous, but it's beyond what I'm willing to do at home by myself.
- Baeocystin 3y ago
  I wish other YouTubers would take note of the information density that Project Farm packs in to his videos. It's really something else.
lars_francke 3y ago
Similar organizations exist in other countries as well https://en.wikipedia.org/wiki/Stiftung_Warentest https://en.wikipedia.org/wiki/Stiftung_Warentest https://en.wikipedia.org/wiki/International_Consumer_Research_%26_Testing https://en.wikipedia.org/wiki/International_Consumer_Researc... I'm subscribed to the former but I found that at least some of my circle of friends and family don't like their reviews because they don't focus on things like "coolness" or aesthetics. Instead they take into account things like safety, labeling and other things.
- yread 3y ago
  wow that's a lot. It would be cool if signing up for one would give you access to reviews from the other ones too.
IndySun 3y ago
A family member has which.co.uk and I find its marking system wildly inconsistent. A good battery life marked well on several products can arbitrarily lift one of those products to the top for example... and as such I don't read it anymore. And secondly, and this is common across all such review sites and comments, is that you suddenly discover how much bumpf is being written when you happen upon a review for a product or department you own or know well.

bombcar 3y ago

Consumer reports at least is known to be gamed (and they have specific review methodology you need to understand and interpret). It’s hard

rootusrootus 3y ago
> Consumer reports at least is known to be gamed I'd be interested to know more about this, but I haven't found much discussion of it when I search. My google fu may be weak today.
- gandalfgreybeer 3y ago
  Not related to the topic on hand but I’ve felt my google fu diminish significantly over the past years. Not sure if it’s me or my google intuition just not matching the current algorithms
  - Baeocystin 3y ago
    It's not you, it's google. Seriously.
  - seventytwo 3y ago
    It’s google. Try Kagi.
- somat 3y ago
  On the whole I think consumer reports is one of the better review organizations out there. but for a hint of reader betrayal read up on what they did to the suzuki samurai. It is not exactly cut and dried, but it was found that consumer reports was testing to get a specific result. And due to the trust that people put in consumer reports that result killed the market for the samurai in the US.
  - pseudalopex 3y ago
    Any examples after 1988?
    - somat 3y ago
      Not really, Like I said, I think they are one of the good guys, I am not sure if their testing methodology is any better than any one else. But they do try hard to remain independent from the manufacturers. I think of it as one black mark on their record and a reminder that even our heroes can slip up. By good methodology I think the best is the person who has a fleet of the things and a deep understanding of how they break, the next best is to take the thing apart and apply some industry specific analysis to it, then test it to destruction to see how it breaks. Ether one is beyond the grasp of the average joe, who only wants to buy one of the things and have it work. And almost no reviewers do this which is why the ones that do are so valuable.
      - bombcar 3y ago
        Exactly this - they do try to remain independent but they sometimes obviously have biases towards certain operations of things. For example, the only thing I care about with toilets are that they don’t break and they always flush and clear. Literally nothing else matters, but to test that you gotta poop in them and that’s hard to reproduce scientifically. So weird substitutes like golf balls and whatnot are used instead. It’s also an integral problem with new car reviews, because you can’t review the durability of a new car, it’s an unknown. Substitutes end up dinging a car for a body panel alignment as much as a transmission replacement.
        jstarfish 3y ago
        > to test that you gotta poop in them and that’s hard to reproduce scientifically. So weird substitutes like golf balls and whatnot are used instead. Nah, just bring in some large dogs. You'll never be without mighty turds to dispose of. Beyond that, entire rolls of toilet paper, socks, makeup cassettes, cigarettes, toy cars, bleach wipes, pads, tampons (with and without plastic applicators) and other nonfecal items are the things you should be testing your toilet with, since those are what will go down your pipe in production. My FIL is the type to insist on buying whichever toilet can suck down the most golf balls. Pooping at his house means never having to sheepishly ask for a plunger. His is like an airplane toilet.
wolverine876 3y ago
> Consumer reports at least is known to be gamed I haven't heard of that. Could you tell us what you're referring to? > and they have specific review methodology you need to understand and interpret I've ready many Consumer Reports reviews; I don't know what this comment refers to. The style of the graphics in their tables? Their methodology is simply scientific method.
- bombcar 3y ago
  The one I know best is the toilets, as the “scientific method” has nothing to do with actually using the toilet to eliminate human waste. And since the testing is known and not realistic, manufacturers can build for the test. https://seclists.org/politech/2000/Mar/73 https://seclists.org/politech/2000/Mar/73 You have to understand their methodology because they may be ascribing “points” to things you don’t care about at all - especially if you’re looking for longevity.
  - wolverine876 3y ago
    > https://seclists.org/politech/2000/Mar/73 https://seclists.org/politech/2000/Mar/73 The link is a random Internet person's post from 2000 about a single 1995 Consumer Reports review. > You have to understand their methodology because they may be ascribing “points” to things you don’t care about at all - especially if you’re looking for longevity. Every review everywhere, unless I hire the reviewer personally, will examine factors I am more and factors I am less interested in. They can't simultaneously, precisely serve everyone's varied interests (ok, I guess they could try some social media feed-type algorithm). > the “scientific method” has nothing to do with actually using the toilet to eliminate human waste. That's confusing. The scientific method is to actually do it and report what happens, not talk theoretically or speculatively about it from our basements like the rest of the Internet. CR generally does that (the scientific method) in extensive labs, staffed with experienced scientists and engineers. If you read their toilet reviews, that's what they do - put something in and flush. CR's core product is science, applied to consumer products. > especially if you’re looking for longevity. CR looks at longevity for many products, including extensive surveys of product owners.

zoltar 3y ago

I've found smaller categories of products with decent reviews on youtube. Project Farm and America's Test Kitchen spring to mind. I'm not sure they're as rigorous, but it's beyond what I'm willing to do at home by myself.

Baeocystin 3y ago
I wish other YouTubers would take note of the information density that Project Farm packs in to his videos. It's really something else.

lars_francke 3y ago

Similar organizations exist in other countries as well https://en.wikipedia.org/wiki/Stiftung_Warentest https://en.wikipedia.org/wiki/Stiftung_Warentest https://en.wikipedia.org/wiki/International_Consumer_Research_%26_Testing https://en.wikipedia.org/wiki/International_Consumer_Researc... I'm subscribed to the former but I found that at least some of my circle of friends and family don't like their reviews because they don't focus on things like "coolness" or aesthetics. Instead they take into account things like safety, labeling and other things.

yread 3y ago
wow that's a lot. It would be cool if signing up for one would give you access to reviews from the other ones too.

IndySun 3y ago

A family member has which.co.uk and I find its marking system wildly inconsistent. A good battery life marked well on several products can arbitrarily lift one of those products to the top for example... and as such I don't read it anymore. And secondly, and this is common across all such review sites and comments, is that you suddenly discover how much bumpf is being written when you happen upon a review for a product or department you own or know well.