4 ms·
Interesting. At one place I worked, employees were excluded from experiments (they had to enable the flag personally to see them) by default. At one point, we h
by withinboredom 2y ago
Interesting. At one place I worked, employees were excluded from experiments (they had to enable the flag personally to see them) by default. At one point, we had so many experiments that literally nobody except employees were using "that version" of the software. Everyone else was using some slightly different version (if you counted each feature as a version), and there were thousands and thousands of versions in total.
We ended up creating just ~100 versions of our app (~100 experiment buckets), and then you could join a bucket. Teams could even reserve sets of buckets for exclusive experimentation purposes. We also ended up reserving a set of buckets that always got the control group.
You've approached it a different way, and probably a more sustainable way. It's interesting. How do you deal with the bias from your 'more courageous' people?
- dangsux 2y ago[dead]
- tauntz 2y ago> At one point, we had so many experiments that literally nobody except employees were using "that version" of the software. Everyone else was using some slightly different version Was this at Spotify by any chance? :)
- deleted 2y ago[deleted]
- withinboredom 2y agoNo.
- adontz 2y ago>> How do you deal with the bias from your 'more courageous' people? That's a great question. We had no general solution for that. We tried to survey people, but results were inconclusive, not statistically significant.
- withinboredom 2y agoI mean that "courageous" people are more likely to take risks and accept new features and thus probably more likely to be attracted to novelty (see: Novelty Effect) and require longer experiments to understand the actual impact.