4 ms·

While this is all true, I think the reason why people so often run underpowered tests is not because they "lacks a firm understanding of statistical power" but

by tqi 5y ago

While this is all true, I think the reason why people so often run underpowered tests is not because they "lacks a firm understanding of statistical power" but because we have not made a compelling case for an alternative. UXR surveys? In my experience that is even more underpowered than AB tests. User testing? Often just as subjective as gut calls. Gut calls (ie product sense) seem like the only real choice, but I think there really needs to be an argument made for WHY that is the way, rather than just against other options.

xpe 5y ago
I tend to notice when people write 'the only choice' or 'the only real choice' when they probably mean something more like (1) an option that, when examined, seems best; or (2) an unexamined (sometimes dogmatic) choice. In some cases, a person may have a third intent: by linguistically limiting a choice set, they can frame or limit a discussion (for good or ill).
- tqi 5y ago
  I dunno about all that, but in my case what I mean is "the only choice that would satisfy the author's criteria of being statistically sound"
xpe 5y ago
Yes, in some cases, teams want an experiment to back them up. If a decision maker cannot dedicate the necessary resources to properly power a test, it arguably would be more rational to use another process to make the decision. Perhaps a thoughtfully designed team discussion or even a 'Person P will make the decision because the uncertainty is relatively high, consensus is relatively low, and additional discussion is unlikely to clarify -- meanwhile we could go ahead and try the thing and see what happens!"
- tqi 5y ago
  > If a decision maker cannot dedicate the necessary resources to properly power a test Apologies if this is what you meant, but in a lot of cases it's not a matter of not dedicating enough resources so much as there aren't enough resources (ie visitors, users, posts, etc) to detect reasonable effect size. And in those cases, it seems like the author's point is that running a test is actively worse than a gut call or focus group or whatever else. But to your point, product teams often don't like gut calls because they feel subjective and unilateral, so having a test allows some degree of shared accountability for decisions.
  - xpe 5y ago
    Well said.

by tqi 5y ago

xpe 5y ago

I tend to notice when people write 'the only choice' or 'the only real choice' when they probably mean something more like (1) an option that, when examined, seems best; or (2) an unexamined (sometimes dogmatic) choice. In some cases, a person may have a third intent: by linguistically limiting a choice set, they can frame or limit a discussion (for good or ill).

tqi 5y ago
I dunno about all that, but in my case what I mean is "the only choice that would satisfy the author's criteria of being statistically sound"

xpe 5y ago

Yes, in some cases, teams want an experiment to back them up. If a decision maker cannot dedicate the necessary resources to properly power a test, it arguably would be more rational to use another process to make the decision. Perhaps a thoughtfully designed team discussion or even a 'Person P will make the decision because the uncertainty is relatively high, consensus is relatively low, and additional discussion is unlikely to clarify -- meanwhile we could go ahead and try the thing and see what happens!"

tqi 5y ago
> If a decision maker cannot dedicate the necessary resources to properly power a test Apologies if this is what you meant, but in a lot of cases it's not a matter of not dedicating enough resources so much as there aren't enough resources (ie visitors, users, posts, etc) to detect reasonable effect size. And in those cases, it seems like the author's point is that running a test is actively worse than a gut call or focus group or whatever else. But to your point, product teams often don't like gut calls because they feel subjective and unilateral, so having a test allows some degree of shared accountability for decisions.
- xpe 5y ago
  Well said.