3 ms·
> During software updates, we enable the new version for a subset of users and ... compare how the new version is performing vs the previous version. Any regres
by devmunchies 7y ago
> During software updates, we enable the new version for a subset of users and ... compare how the new version is performing vs the previous version. Any regression in the metrics gives us a signal to abort the update and revert
How do you account for the possibility that the update only performs badly because it’s different than what users are used to, but would actually be an improvement in the long run?
- Joof 7y agoHave you worked at a large company before? You don't!
- catnzhat 7y agoYou acknowledge that fact and if you're making a severe change you either run a longer test or you break up the large change into manageable chunks (Google Maps is a great example of this). For something that is effectively binary and can't be changed incrementally the only way is to extend the evaluation window or make a call with limited information.
- 2rsf 7y agoregressions due to "users are used to" usually goes away after a while, you'll see a temporary dip in the results. If you are convinced it's better for the long tun then what's the point in measuring ?
- Twirrim 7y ago> How do you account for the possibility that the update only performs badly because it’s different than what users are used to, but would actually be an improvement in the long run? That's assuming that the update contains a UI/UX change. A lot of updates that will roll out won't include that, they'll be fixing or optimising things.