7 ms·
So when deploying changes it should just go to every single user with no incremental rollout?
by staticassertion 7y ago
So when deploying changes it should just go to every single user with no incremental rollout?
- benatkin 7y agoYes. The testing should be done on users that opt in (likely less than 0.1% but that should be OK if it's done properly) and it should be rolled out by releasing an update to everyone. These so-called "best practices" often aren't.
- Vinnl 7y agoBy relying on opt-in you won't test whether a feature actually works for the average user, so if you don't test it, you might roll something useless/confusing/actively harmful out to all of them instead of just a few.
- zamalek 7y agoExactly. The recent Windows 10 update fiascos demonstrate this perfectly: they are opt-in (Insider program) and there were too few users experiencing the bugs to overcome the noise of those who weren't. You need a truly random sample.
- benatkin 7y agoI think they updated too many parts of Windows at once, not too many computers at once. And I'm not certain that a delayed rollout would have fixed it. Is it fair that a random sampling of people are given the problem?
- hutzlibu 7y agoSo instead of having enough voluntary or paid testers to detect catastrophic, file deleting bugs in a new version, your solution is to just roll out the new changes to random users expecting a stable system? Something went terrible wrong somewhere, that we got to this.
- enriquto 7y agoAnd that's OK. Being honest to your users is much more important that having the most efficient deploy system. I, for one, as a matter of principle, opt-in to all user tracking systems that are disabled by default, and try to opt-out whenever they are enabled by default. In the case of firefox, it was never clear to me that the tracking/remote install capability was enabled by default.
- staticassertion 7y agoI don't see the purpose. There seems to be an assumption that A/B testing is inherently bad, therefor should be opt-in. A/B testing is necessary for stable rollouts, that much seems self evident. So what is the evidence that A/B testing is harmful to users?
- badsectoracula 7y agoIt isn't A/B testing that is bad, it is running and/or modifying stuff in the user's computer without their knowledge that is bad. If a site modifies some part of their layout or whatever isn't bad, but a browser installing addons or changing settings to see if it will crash is bad.
- staticassertion 7y agoWell, it's not like they're hoping it will crash. Presumably they're hoping it won't. Wouldn't it be more irresponsible to deploy to every user without first testing it out?
- chii 7y agothat's what the stable/experimental channels are for - a user who don't want to be tested on (such as a business user) can easily be sure that they are getting a stable release and their settings won't update under them. Experimental/beta channels can have these changes pushed to it, since that's implied by the name, and those people who like living on the edge of new features do so knowingly. Let the users control their destiny, let the users decide when. That's the hallmark of a trustworthy company.
- zlynx 7y agoUsers who intentionally run beta / nightly are not normal users. How many of them run on cheap "desktop" computers they bought on sale at Target, do you think?
- scarejunba 7y ago
- swebs 7y agoIsn't that the entire point of the nightly and beta branches?
- dblohm7 7y agoThose populations are not representative of the release population.