9 ms·

Interpreting A/B test results: false positives and statistical significance

dmitriid 5y ago
Before interpreting A/B results, the main question that needs to be asked: "what is it that you're A/B testing?" For too many companies, it's testing "engagement" which leads to hiding functionality (more clicks is more engagement), reducing info density (more time spent is more engagement) etc. And coming from Netflix... I don't think there's a single person who likes that when you browse Netflix it autoplays random videos (not even trailers) with audio at full volume. But yeah, A/B tests something something. So I wish Netflix learned from their own teachings.
- type_enthusiast 5y ago
  (disclaimer: I work for Netflix. Edit: I should clarify that I wasn't involved with this article in any way) You can disable the behavior you mentioned. Go to your profile settings, and under "Playback Settings" you can uncheck "Autoplay previews while browsing on all devices".
  - dmitriid 5y ago
    > You can disable the behavior you mentioned. I know I can. But then I can't play trailers for movies/shows where I want to see the trailer.
- dafelst 5y ago
  People may not like that feature (I sure don't), but I would bet a decent sum that feature didn't drive increases in negative metrics like churn, and increased positive metrics like hours watched, perhaps by causing people to scroll through more of the library faster, or perhaps drawing people in with the previews. Or maybe they just saw in improvement in a non-core metric like "distance scrolled" with no other negative effects and said, "meh, ship it". Both seem likely. Of course this is the danger of any sort of behavioral metric driven optimization strategy - you may trade negative customer sentiment for positive business outcomes. That's where the real decision making comes about, i.e. are you willing to make that trade? It seems that in this case, Netflix was.
- mobjack 5y ago
  I've A/B tested hiding functionality and reducing info density increased the number of people spending money on the site. I was completely shocked by seeing those results initially and dove deep to look for any other negative effects from these changes but could not find any. I've repeated similar tests and the results are often similar. From that experience, I've learned that most people are not like me or the HN crowd. The things that you complain about could actually make things easier for the majority.
  - dmitriid 5y ago
    Yes, you're training users to deal with unnecessary complexity, and then the competitors with less complexity swoop in and take your user base. Thankfully for businesses with built-in complexity and unfortunately for everyone, other businesses also have the same mindset.
  - rancar2 5y ago
    I’ve A/B tested this at scale with millions of users across different verticals globally. In many situations, focusing on core functions and clear information presentation helps guide the user to using the site as primarily intended based on their primary intent. It’s tough for most product teams to realize that most users really don’t want all that additional functionality they built.
jonathanbentz 5y ago
I am interested to see what they will be testing in some of the upcoming posts in this series. It would be fun to be scrolling Netflix and have the transparency to know that I'm seeing the 'B' test.
- kristjansson 5y ago
  Like all controlled experiments though, the experimenter wants to hide that information from the subject (user in this instance) to measure how they respond to the change itself, rather than the change and being told about it.
- bertil 5y ago
  It depends on the area of Netflix, but for most recommendation that can fit on a list, Netflix seems to use interleaved testing: * they create two recommendation lists based on distinct principles or parameters; * they pick a first (with an A/B test); * mix those with ‘schoolyard rules’: each side picks the next candidate that wasn’t picked already, creating a new, alternating list. * They show that to the user and track whether you picked an even or odd item on that recommendation, indicating whether A or B is preferable. So most like, you are seeing A and B.
palae 5y ago
It's probably a good idea to remind (or inform) people that at least in scientific research, null hypothesis statistical testing and "statistical significance" in particular have come under fire [1,2]. From the American Statistical Association (ASA) in 2019 [2]: "We conclude, based on our review of the articles in this special issue and the broader literature, that it is time to stop using the term “statistically significant” entirely. Nor should variants such as “significantly different,” “p < 0.05,” and “nonsignificant” survive, whether expressed in words, by asterisks in a table, or in some other way. Regardless of whether it was ever useful, a declaration of “statistical significance” has today become meaningless." [1] The ASA Statement on p-Values: Context, Process, and Purpose - https://www.tandfonline.com/doi/full/10.1080/00031305.2016.1154108 https://www.tandfonline.com/doi/full/10.1080/00031305.2016.1... [2] Moving to a World Beyond “p < 0.05” - https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1583913 https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1...
- antihipocrat 5y ago
  I read through the first link you posted and couldn't find any ideas about what we could use instead of p-Values. Statistical tests are a very useful and objective method of determining whether the outcomes of one thing/activity are more desirable than another when applied correctly. Some solutions could be to set a higher bar for statistical analysis education. Or perhaps a more thorough statistically focussed vetting and peer review process for published material? .. and after reading the second link I note that it includes commentary around the insurmountably difficult challenge of replacing the current paradigm. While not offering a solution, it goes on to provide some great advice for how to improve the quality of statistical research. Following all of the advice to the letter would make it almost impossible to conduct valid research. Maybe this is where economists have the upper hand? Just develop some highly abstract mathematical models representing the topic you're interested in, and abstain from rigorously testing them.
  - brian_spiering 5y ago
    Bayesian hypothesis testing, including Bayes factors, might be more useful.

Before interpreting A/B results, the main question that needs to be asked: "what is it that you're A/B testing?" For too many companies, it's testing "engagement" which leads to hiding functionality (more clicks is more engagement), reducing info density (more time spent is more engagement) etc. And coming from Netflix... I don't think there's a single person who likes that when you browse Netflix it autoplays random videos (not even trailers) with audio at full volume. But yeah, A/B tests something something. So I wish Netflix learned from their own teachings.

type_enthusiast 5y ago
(disclaimer: I work for Netflix. Edit: I should clarify that I wasn't involved with this article in any way) You can disable the behavior you mentioned. Go to your profile settings, and under "Playback Settings" you can uncheck "Autoplay previews while browsing on all devices".
- dmitriid 5y ago
  > You can disable the behavior you mentioned. I know I can. But then I can't play trailers for movies/shows where I want to see the trailer.
dafelst 5y ago
People may not like that feature (I sure don't), but I would bet a decent sum that feature didn't drive increases in negative metrics like churn, and increased positive metrics like hours watched, perhaps by causing people to scroll through more of the library faster, or perhaps drawing people in with the previews. Or maybe they just saw in improvement in a non-core metric like "distance scrolled" with no other negative effects and said, "meh, ship it". Both seem likely. Of course this is the danger of any sort of behavioral metric driven optimization strategy - you may trade negative customer sentiment for positive business outcomes. That's where the real decision making comes about, i.e. are you willing to make that trade? It seems that in this case, Netflix was.
mobjack 5y ago
I've A/B tested hiding functionality and reducing info density increased the number of people spending money on the site. I was completely shocked by seeing those results initially and dove deep to look for any other negative effects from these changes but could not find any. I've repeated similar tests and the results are often similar. From that experience, I've learned that most people are not like me or the HN crowd. The things that you complain about could actually make things easier for the majority.
- dmitriid 5y ago
  Yes, you're training users to deal with unnecessary complexity, and then the competitors with less complexity swoop in and take your user base. Thankfully for businesses with built-in complexity and unfortunately for everyone, other businesses also have the same mindset.
- rancar2 5y ago
  I’ve A/B tested this at scale with millions of users across different verticals globally. In many situations, focusing on core functions and clear information presentation helps guide the user to using the site as primarily intended based on their primary intent. It’s tough for most product teams to realize that most users really don’t want all that additional functionality they built.

jonathanbentz 5y ago

I am interested to see what they will be testing in some of the upcoming posts in this series. It would be fun to be scrolling Netflix and have the transparency to know that I'm seeing the 'B' test.

kristjansson 5y ago
Like all controlled experiments though, the experimenter wants to hide that information from the subject (user in this instance) to measure how they respond to the change itself, rather than the change and being told about it.
bertil 5y ago
It depends on the area of Netflix, but for most recommendation that can fit on a list, Netflix seems to use interleaved testing: * they create two recommendation lists based on distinct principles or parameters; * they pick a first (with an A/B test); * mix those with ‘schoolyard rules’: each side picks the next candidate that wasn’t picked already, creating a new, alternating list. * They show that to the user and track whether you picked an even or odd item on that recommendation, indicating whether A or B is preferable. So most like, you are seeing A and B.

palae 5y ago

It's probably a good idea to remind (or inform) people that at least in scientific research, null hypothesis statistical testing and "statistical significance" in particular have come under fire [1,2]. From the American Statistical Association (ASA) in 2019 [2]: "We conclude, based on our review of the articles in this special issue and the broader literature, that it is time to stop using the term “statistically significant” entirely. Nor should variants such as “significantly different,” “p < 0.05,” and “nonsignificant” survive, whether expressed in words, by asterisks in a table, or in some other way. Regardless of whether it was ever useful, a declaration of “statistical significance” has today become meaningless." [1] The ASA Statement on p-Values: Context, Process, and Purpose - https://www.tandfonline.com/doi/full/10.1080/00031305.2016.1154108 https://www.tandfonline.com/doi/full/10.1080/00031305.2016.1... [2] Moving to a World Beyond “p < 0.05” - https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1583913 https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1...

antihipocrat 5y ago
I read through the first link you posted and couldn't find any ideas about what we could use instead of p-Values. Statistical tests are a very useful and objective method of determining whether the outcomes of one thing/activity are more desirable than another when applied correctly. Some solutions could be to set a higher bar for statistical analysis education. Or perhaps a more thorough statistically focussed vetting and peer review process for published material? .. and after reading the second link I note that it includes commentary around the insurmountably difficult challenge of replacing the current paradigm. While not offering a solution, it goes on to provide some great advice for how to improve the quality of statistical research. Following all of the advice to the letter would make it almost impossible to conduct valid research. Maybe this is where economists have the upper hand? Just develop some highly abstract mathematical models representing the topic you're interested in, and abstain from rigorously testing them.
- brian_spiering 5y ago
  Bayesian hypothesis testing, including Bayes factors, might be more useful.