3 ms·

This is why you have long-term holdbacks: a group of users who never sees a set of experiments. That way you can measure over a longer period of time the true i

This is why you have long-term holdbacks: a group of users who never sees a set of experiments. That way you can measure over a longer period of time the true impact of a set of experiments.

dotancohen 3y ago
You cannot control for lots of other externalities, such as inflation and competitors coming to market. These parameters will affect long-term users as well.
- smachiz 3y ago
  isn't that why it's good? You can't control infinite variables, the only ones here are A/B and you're keeping a subset of them as a control group?
  - dotancohen 3y ago
    I suppose that is a way of looking at it.
- bootsmann 3y ago
  Well presumably these externalities are independent of the variables you are changing so they should affect your hold-out set and your experiment sets equally.
rdca 3y ago
Booking.com does not do it. Once an A/B test shows positive outcome with a high enough confidence level, the experiment goes "full on" and is shown to everybody.
- esafak 3y ago
  And that's how you get dark patterns. Let them enjoy their engagement lifts until they drive their product over the cliff.
- smueller1234 3y ago
  That's correct. Long term holdbacks actually are way way harder to implement and maintain in a fast moving product than meets the eye. We had passionate and ultimately unfruitful conversations about how to do that when I was there in 2016. The only thing we had at the time was continued tracking against the experiment groups after flipping that switch.
jrkatz 3y ago
For very long term effects (e.g., the impact of painting every surface of your website in ads which will certainly drive users away), hold backs aren't enough because it's too hard to consistently identify users. If ads will drive me away over the course of a year, there's a five percent holdback, and I use multiple devices and cash maybe be consistently identified for a few days or weeks at a time, that means that 95% of the time I use the website I get the bad experience, and I eventually bounce. The holdback will show no effect.

This is why you have long-term holdbacks: a group of users who never sees a set of experiments. That way you can measure over a longer period of time the true impact of a set of experiments.

dotancohen 3y ago

You cannot control for lots of other externalities, such as inflation and competitors coming to market. These parameters will affect long-term users as well.

smachiz 3y ago
isn't that why it's good? You can't control infinite variables, the only ones here are A/B and you're keeping a subset of them as a control group?
- dotancohen 3y ago
  I suppose that is a way of looking at it.
bootsmann 3y ago
Well presumably these externalities are independent of the variables you are changing so they should affect your hold-out set and your experiment sets equally.

rdca 3y ago

Booking.com does not do it. Once an A/B test shows positive outcome with a high enough confidence level, the experiment goes "full on" and is shown to everybody.

esafak 3y ago
And that's how you get dark patterns. Let them enjoy their engagement lifts until they drive their product over the cliff.
smueller1234 3y ago
That's correct. Long term holdbacks actually are way way harder to implement and maintain in a fast moving product than meets the eye. We had passionate and ultimately unfruitful conversations about how to do that when I was there in 2016. The only thing we had at the time was continued tracking against the experiment groups after flipping that switch.

jrkatz 3y ago

For very long term effects (e.g., the impact of painting every surface of your website in ads which will certainly drive users away), hold backs aren't enough because it's too hard to consistently identify users. If ads will drive me away over the course of a year, there's a five percent holdback, and I use multiple devices and cash maybe be consistently identified for a few days or weeks at a time, that means that 95% of the time I use the website I get the bad experience, and I eventually bounce. The holdback will show no effect.