4 ms·

> In short, not many people want to funnel users through N code paths with slightly different behaviors, because not many people have a ton of users, a ton of e

> In short, not many people want to funnel users through N code paths with slightly different behaviors, because not many people have a ton of users, a ton of engineering capacity, and a ton of potential upside from marginal improvements. I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results. I figure by the law of probabilities they would have gotten at least a single significant experiment but most products have such small user bases and make such large changes at a time that it’s completely pointless. All my complaints fell on deaf ears until the PM in charge would get on someone’s bad side and then that metric would be used to push them out. I think they’re largely a political tool like all those management consultants that only come in to justify an executive’s predetermined goals.

rco8786 2y ago
> I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results. Yea, I've been here too. And in every analytics meeting everyone went "well, we know it's not statistically significant but we'll call it the winner anyway". Every. Single. Time. Such a waste of resources.
- hamandcheese 2y ago
  Is it a waste? You proved the change wasn't harmful.
  - zeroCalories 2y ago
    Statistically insignificant means you didn't prove anything by usual standards. I do agree that it's not a waste, as knowing that you have a 70% chance that you're going in the right direction is better than nothing. The 2 sigma crowd can be both too pessimistic and not pessimistic enough.
    - Dylan16807 2y ago
      Statistical significance depends on what you're trying to prove. Looking for substantial harm is a lot easier than figuring out which option is simply better, depending on what level of substantial you're looking for. If your error bar for some change goes from negative 4 percent to positive 6 percent, it may or may not be better, but it's safe to switch to.
  - menaerus 2y ago
    To prove that a change isn't harmful is still a hypothesis.
  - hinkley 2y ago
    You can still enshittify something by degrees this way. I think the disconnect here is some people thinking A/B testing is something you try once a month, and someplace like Amazon where you do it all the time and with hundreds of employees poking things.
eru 2y ago
> I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results. Well, at least it looks like they avoided p-hacking to show more significance than they had! That's ahead of much of science, alas.
dkarl 2y ago
> I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results. What I've seen in practice is that some places trust their designers' decisions and only deploy A/B tests when competent people disagree, or there's no clear, sound reason to choose one design over another. Surprise surprise, those alternatives almost always test very close to each other! Other places remove virtually all friction from A/B testing and then use it religiously for every pixel in their product, and they get results, but often it's things like "we discovered that pink doesn't work as well as red for a warning button," stuff they never would have tried if they didn't have to feed the A/B machine. From all the evidence I've seen in places I've worked, the motivating stories of "we increased revenue 10% by a random change nobody thought would help" may only exist in blog posts.
- zeroCalories 2y ago
  I think trusting your designers is probably the way to go for most teams. Good designers have solid intuitions and design principles for what will increase conversion rates. Many designers will still want a/b tests because they want to be able to justify their impact, but they should probably be denied. For really important projects designers should do small sample size research to validate their designs like we would do in the past. I think a/b tests are still good for measuring stuff like system performance, which can be really hard to predict. Flipping a switch to completely change how you do caching can be scary.
  - Moru 2y ago
    A/B tests for user interface is very annoying when you are on the phone trying to guide someone how to use a website. "Click the green button on the left" - "What do you mean? There is nothing green on the screen." - "Are you on xyz.com? Can you read out the adress to me please?" ... Oh so many hour wasted in tech support.
    - miki123211 2y ago
      Meta apps are like that, particularly around accessibility. It's pretty common for one person to have an issue that no other people have, just because they fell for some feature flag.

rco8786 2y ago

> I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results. Yea, I've been here too. And in every analytics meeting everyone went "well, we know it's not statistically significant but we'll call it the winner anyway". Every. Single. Time. Such a waste of resources.

hamandcheese 2y ago
Is it a waste? You proved the change wasn't harmful.
- zeroCalories 2y ago
  Statistically insignificant means you didn't prove anything by usual standards. I do agree that it's not a waste, as knowing that you have a 70% chance that you're going in the right direction is better than nothing. The 2 sigma crowd can be both too pessimistic and not pessimistic enough.
  - Dylan16807 2y ago
    Statistical significance depends on what you're trying to prove. Looking for substantial harm is a lot easier than figuring out which option is simply better, depending on what level of substantial you're looking for. If your error bar for some change goes from negative 4 percent to positive 6 percent, it may or may not be better, but it's safe to switch to.
- menaerus 2y ago
  To prove that a change isn't harmful is still a hypothesis.
- hinkley 2y ago
  You can still enshittify something by degrees this way. I think the disconnect here is some people thinking A/B testing is something you try once a month, and someplace like Amazon where you do it all the time and with hundreds of employees poking things.

eru 2y ago

> I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results. Well, at least it looks like they avoided p-hacking to show more significance than they had! That's ahead of much of science, alas.

dkarl 2y ago

> I’ve been in companies that have tried dozens if not hundreds of A/B tests with zero statistically significant results. What I've seen in practice is that some places trust their designers' decisions and only deploy A/B tests when competent people disagree, or there's no clear, sound reason to choose one design over another. Surprise surprise, those alternatives almost always test very close to each other! Other places remove virtually all friction from A/B testing and then use it religiously for every pixel in their product, and they get results, but often it's things like "we discovered that pink doesn't work as well as red for a warning button," stuff they never would have tried if they didn't have to feed the A/B machine. From all the evidence I've seen in places I've worked, the motivating stories of "we increased revenue 10% by a random change nobody thought would help" may only exist in blog posts.

zeroCalories 2y ago
I think trusting your designers is probably the way to go for most teams. Good designers have solid intuitions and design principles for what will increase conversion rates. Many designers will still want a/b tests because they want to be able to justify their impact, but they should probably be denied. For really important projects designers should do small sample size research to validate their designs like we would do in the past. I think a/b tests are still good for measuring stuff like system performance, which can be really hard to predict. Flipping a switch to completely change how you do caching can be scary.
- Moru 2y ago
  A/B tests for user interface is very annoying when you are on the phone trying to guide someone how to use a website. "Click the green button on the left" - "What do you mean? There is nothing green on the screen." - "Are you on xyz.com? Can you read out the adress to me please?" ... Oh so many hour wasted in tech support.
  - miki123211 2y ago
    Meta apps are like that, particularly around accessibility. It's pretty common for one person to have an issue that no other people have, just because they fell for some feature flag.