4 ms·

A/B tests, as they are run by current software companies, are inherently flawed. I have never, in my entire career, ever heard of an A/B test that ran for a yea

A/B tests, as they are run by current software companies, are inherently flawed. I have never, in my entire career, ever heard of an A/B test that ran for a year, let alone 3-5 years. That’s where the true power of statistics comes alive, and nobody is financially incentivized to even consider that fact.

eru 3y ago
Gwern runs tests that long.
bonniemuffin 3y ago
You're certainly correct that software companies should do a lot more year+ A/B tests. You can learn really interesting things from it that a shorter test won't capture. I know of this one: https://medium.com/@AnalyticsAtMeta/notifications-why-less-is-more-how-facebook-has-been-increasing-both-user-satisfaction-and-app-9463f7325e7d https://medium.com/@AnalyticsAtMeta/notifications-why-less-i...
makeitdouble 3y ago
I've seen one A/B test in the wild run for a full year. It was on a small part (test of a product name + description across the site), and the most interesting aspect was that it only made a small but measureable difference. Because of that there was no strong incentive to delete the AB test (not much harm to the user) nor make the B side permanent (too low of an effect). In that respect, AB Tests that end early aren't a bad thing IMO: either there's a clear improvement or it's really bad, and the choice is obvious enough to not have to wait much longer.
- mlyle 3y ago
  What I think the grandparent is getting at: You can measure the direct effect of a change now on something like conversions. But you can't measure the second order effects: things like trust from your users, or the effects on community quality and composition, etc. This is a good part of why enshittification happens: lots of changes with immediate "good" impact that can be measured quantifiably, but there's also readily foreseeable negative consequences to them. Of course, just running the test longer doesn't really address this for most possible changes.
  - disgruntledphd2 3y ago
    Generally you'll have a quarterly holdout to measure total impacts but I agree that yearly would be better.
    - mlyle 3y ago
      But even this doesn't work: if you are continually making choices that erode your users' trust in you, there will eventually be an impact. It happens outside of the experiment (e.g. communications between users, general sliding changes in sentiment, etc). And you can't just spot in the time series whether you're going too far or not.
      - strgcmc 3y ago
        Late to this comment thread, but Amazon actually excels at this type of long term measurement, through methodologies internally called HVA/DSI and DSE (to name just a couple). - High Value Action / Downstream Impact == using a "twins" comparison, estimate the 12-month impact of a customer taking a particular action (e.g. sign up for Prime, watch their first Prime Video, etc.), compared to one who doesn't. HVAs are basically those "A's" which turn out to have a high numeric DSI value. - Downstream Expectation == similar but very different - instead of quantifying the impact of a single action, DSE tries to estimate the combined downstream causal impact of a user taking an initial action... there's a sophisticated methodology there that tries to strip away confounding factors like "rich people who would've shopped more anyways, are also naturally more likely to sign up for Prime", because they truly want to measure the causal benefit of Prime itself, separate from the fact that richer customers generally spend more no matter what These are both long-term methodologies, that were explicitly designed in response to the problems of: short term experiments that didn't capture long-term negative effects, and different parts of Amazon having vastly different methodologies for measuring business impact (e.g. page views vs search impressions vs downloads vs orders vs whatever... no, everyone should optimize for the same customer level financial metric which is a flavor of growth-adjusted composite contribution profit (GCCP) that's partly derived from DSE)
        3y ago
eternal_braid 3y ago
Have you heard of long running holdbacks? Even if not, rest assured that for major features, they are very commonly run.
jefftk 3y ago
When I worked on ads at Google we had many A/B tests that had been running that long, generally holdbacks where a feature was almost entirely but not 100% launched. It was relatively rare that the holdback would show markedly different results than the initial A/B test we used in deciding to launch. If that had happened more often we would have run more long tests and been slower to move to launch.
broguinn 3y ago
Your point is a good one - what you're describing I've heard referred to as "the novelty effect". In fact, a lot in this article reminds me of another essay critical of the short-sightedness of many A/B tests: https://www.zumsteg.net/2022/07/05/unchecked-ab-testing-destroys-everything-it-touches/ https://www.zumsteg.net/2022/07/05/unchecked-ab-testing-dest...

eru 3y ago

Gwern runs tests that long.

bonniemuffin 3y ago

You're certainly correct that software companies should do a lot more year+ A/B tests. You can learn really interesting things from it that a shorter test won't capture. I know of this one: https://medium.com/@AnalyticsAtMeta/notifications-why-less-is-more-how-facebook-has-been-increasing-both-user-satisfaction-and-app-9463f7325e7d https://medium.com/@AnalyticsAtMeta/notifications-why-less-i...

makeitdouble 3y ago

I've seen one A/B test in the wild run for a full year. It was on a small part (test of a product name + description across the site), and the most interesting aspect was that it only made a small but measureable difference. Because of that there was no strong incentive to delete the AB test (not much harm to the user) nor make the B side permanent (too low of an effect). In that respect, AB Tests that end early aren't a bad thing IMO: either there's a clear improvement or it's really bad, and the choice is obvious enough to not have to wait much longer.

mlyle 3y ago
What I think the grandparent is getting at: You can measure the direct effect of a change now on something like conversions. But you can't measure the second order effects: things like trust from your users, or the effects on community quality and composition, etc. This is a good part of why enshittification happens: lots of changes with immediate "good" impact that can be measured quantifiably, but there's also readily foreseeable negative consequences to them. Of course, just running the test longer doesn't really address this for most possible changes.
- disgruntledphd2 3y ago
  Generally you'll have a quarterly holdout to measure total impacts but I agree that yearly would be better.
  - mlyle 3y ago
    But even this doesn't work: if you are continually making choices that erode your users' trust in you, there will eventually be an impact. It happens outside of the experiment (e.g. communications between users, general sliding changes in sentiment, etc). And you can't just spot in the time series whether you're going too far or not.
    - strgcmc 3y ago
      Late to this comment thread, but Amazon actually excels at this type of long term measurement, through methodologies internally called HVA/DSI and DSE (to name just a couple). - High Value Action / Downstream Impact == using a "twins" comparison, estimate the 12-month impact of a customer taking a particular action (e.g. sign up for Prime, watch their first Prime Video, etc.), compared to one who doesn't. HVAs are basically those "A's" which turn out to have a high numeric DSI value. - Downstream Expectation == similar but very different - instead of quantifying the impact of a single action, DSE tries to estimate the combined downstream causal impact of a user taking an initial action... there's a sophisticated methodology there that tries to strip away confounding factors like "rich people who would've shopped more anyways, are also naturally more likely to sign up for Prime", because they truly want to measure the causal benefit of Prime itself, separate from the fact that richer customers generally spend more no matter what These are both long-term methodologies, that were explicitly designed in response to the problems of: short term experiments that didn't capture long-term negative effects, and different parts of Amazon having vastly different methodologies for measuring business impact (e.g. page views vs search impressions vs downloads vs orders vs whatever... no, everyone should optimize for the same customer level financial metric which is a flavor of growth-adjusted composite contribution profit (GCCP) that's partly derived from DSE)
      - 3y ago

eternal_braid 3y ago

Have you heard of long running holdbacks? Even if not, rest assured that for major features, they are very commonly run.

jefftk 3y ago

When I worked on ads at Google we had many A/B tests that had been running that long, generally holdbacks where a feature was almost entirely but not 100% launched. It was relatively rare that the holdback would show markedly different results than the initial A/B test we used in deciding to launch. If that had happened more often we would have run more long tests and been slower to move to launch.

broguinn 3y ago

Your point is a good one - what you're describing I've heard referred to as "the novelty effect". In fact, a lot in this article reminds me of another essay critical of the short-sightedness of many A/B tests: https://www.zumsteg.net/2022/07/05/unchecked-ab-testing-destroys-everything-it-touches/ https://www.zumsteg.net/2022/07/05/unchecked-ab-testing-dest...