4 ms·

I'm not certain what you mean by human factors, so correct me if I've misinterpreted. If by human factors you mean basic heuristics based on human intuition, t

I'm not certain what you mean by human factors, so correct me if I've misinterpreted. If by human factors you mean basic heuristics based on human intuition, that fails too. It might satisfy 50% of viewers but there's too much diversity in taste for this to be a good model. I'm not defending the metric as described, but there's obviously good data driven approaches and bad ones. The trick is to find a solid proxy for the business problem and to try to weed out any dark side effects by applying it.

mdoms 5y ago
"Human factors" could include doing something as radical as observing how someone uses your product (no, not with a computer - just watch) and then - and this is the shocking part - speak to them. Ask them why they did that or what they liked and didn't like.
- fumblebee 5y ago
  What fantasy world are you living in where you think Netflix and Facebook don't already do this? They constantly run paid interviews with people to source this information. The issue is many-fold: 1) formalising multiple half hour interviews into something you can A/B test is non-trivial, 2) given the logistics of these interviews the sample you end up with is small and therefore bias.
  - ricardobeat 5y ago
    I worked for the company running the most A/B tests in the industry for most of my career. Having a user testing session is relatively rare, saved for special one-off products/features. Actually having feedback from those sessions distilled into product feature proposals, and getting them into the product roadmap is even rarer. In the meantime, hundreds of A/B tests based on metrics like these would have been ran and resulted in a permanent change to the product. It’s just so much easier for everyone involved to trust “the data” vs something that requires nuanced interpretation.

4 ms·

I'm not certain what you mean by human factors, so correct me if I've misinterpreted. If by human factors you mean basic heuristics based on human intuition, t

by fumblebee 5y ago

mdoms 5y ago
"Human factors" could include doing something as radical as observing how someone uses your product (no, not with a computer - just watch) and then - and this is the shocking part - speak to them. Ask them why they did that or what they liked and didn't like.
- fumblebee 5y ago
  What fantasy world are you living in where you think Netflix and Facebook don't already do this? They constantly run paid interviews with people to source this information. The issue is many-fold: 1) formalising multiple half hour interviews into something you can A/B test is non-trivial, 2) given the logistics of these interviews the sample you end up with is small and therefore bias.
  - ricardobeat 5y ago
    I worked for the company running the most A/B tests in the industry for most of my career. Having a user testing session is relatively rare, saved for special one-off products/features. Actually having feedback from those sessions distilled into product feature proposals, and getting them into the product roadmap is even rarer. In the meantime, hundreds of A/B tests based on metrics like these would have been ran and resulted in a permanent change to the product. It’s just so much easier for everyone involved to trust “the data” vs something that requires nuanced interpretation.