18 ms·
No place I worked at ever experimented at the pageload level. We experimented at the user level, so 1% of users would get the new UI. I suppose this is only pos
by squokko 3y ago
No place I worked at ever experimented at the pageload level. We experimented at the user level, so 1% of users would get the new UI. I suppose this is only possible at the millions of users scale which all of them had.
- opportune 3y agoI updated the comment to reflect that. Certainly the signal is stronger because you’re amortizing away the surprise factor of the change, and at least it’s a consistent UX, but the UX tradeoff in the worst case is that experiment-group users get a broken product with no notice or escape hatch. Unless you’re being very careful, meticulous, and transparent it’s just not acceptable if you’re a paying customer.
- reidjs 3y agoIn some cases you’re making the change because the app is already broken for the majority of users and you’re testing the fix