3 ms·
> "Test was successful so it's rolling out to all users, minus a 0.5% holdback population for the next 2 years" Man, I couldn't imagine being a user in such a
by SketchySeaBeast 3y ago
> "Test was successful so it's rolling out to all users, minus a 0.5% holdback population for the next 2 years"
Man, I couldn't imagine being a user in such a situation. "Oh, I guess I'm just not getting the better functionality?" Even worse if I were a paying customer.
- Brian_K_White 3y agoYou are probably the lucky elite who got to keep the functionality you wanted.
- adamesque 3y agoIt’s actually usually the paying customers asking via support to be added to the holdback, improved experience or no. This is more true for larger flags that substantially change the experience and may not implement niche or edge-case functionality. Obviously you want to avoid these kinds of tests if possible but it’s not always possible.
- esafak 3y agoUsers should not be allowed to select their treatments; it defeats randomization, which is what allows causal inference.
- mandelbrotwurst 3y agoSure, they'll be more predictive that way, and simultaneously it's valuable to not piss off your customers.
- esafak 3y agoIn that case I would take them out of the experiment and impute the censored data. https://en.wikipedia.org/wiki/Censoring_(statistics) https://en.wikipedia.org/wiki/Censoring_(statistics)
- SketchySeaBeast 3y agoAh, that makes way more sense.