7 ms·

Having a capability for widespread A/B testing is brain-damage?

Nullabillity 7y ago
Yes, running experiments on unknowning and unconsenting users is unethical, especially when the metrics being optimized for usually aren't in the users' best interests. The widespread acceptance for this behaviour could definitely be classified as brain damage.
- staticassertion 7y ago
  So when deploying changes it should just go to every single user with no incremental rollout?
  - benatkin 7y ago
    Yes. The testing should be done on users that opt in (likely less than 0.1% but that should be OK if it's done properly) and it should be rolled out by releasing an update to everyone. These so-called "best practices" often aren't.
    - Vinnl 7y ago
      By relying on opt-in you won't test whether a feature actually works for the average user, so if you don't test it, you might roll something useless/confusing/actively harmful out to all of them instead of just a few.
      - zamalek 7y ago
        Exactly. The recent Windows 10 update fiascos demonstrate this perfectly: they are opt-in (Insider program) and there were too few users experiencing the bugs to overcome the noise of those who weren't. You need a truly random sample.
        benatkin 7y ago
        I think they updated too many parts of Windows at once, not too many computers at once. And I'm not certain that a delayed rollout would have fixed it. Is it fair that a random sampling of people are given the problem?
        hutzlibu 7y ago
        So instead of having enough voluntary or paid testers to detect catastrophic, file deleting bugs in a new version, your solution is to just roll out the new changes to random users expecting a stable system? Something went terrible wrong somewhere, that we got to this.
      - enriquto 7y ago
        And that's OK. Being honest to your users is much more important that having the most efficient deploy system. I, for one, as a matter of principle, opt-in to all user tracking systems that are disabled by default, and try to opt-out whenever they are enabled by default. In the case of firefox, it was never clear to me that the tracking/remote install capability was enabled by default.
    - staticassertion 7y ago
      I don't see the purpose. There seems to be an assumption that A/B testing is inherently bad, therefor should be opt-in. A/B testing is necessary for stable rollouts, that much seems self evident. So what is the evidence that A/B testing is harmful to users?
      - badsectoracula 7y ago
        It isn't A/B testing that is bad, it is running and/or modifying stuff in the user's computer without their knowledge that is bad. If a site modifies some part of their layout or whatever isn't bad, but a browser installing addons or changing settings to see if it will crash is bad.
        staticassertion 7y ago
        Well, it's not like they're hoping it will crash. Presumably they're hoping it won't. Wouldn't it be more irresponsible to deploy to every user without first testing it out?
        chii 7y ago
        that's what the stable/experimental channels are for - a user who don't want to be tested on (such as a business user) can easily be sure that they are getting a stable release and their settings won't update under them. Experimental/beta channels can have these changes pushed to it, since that's implied by the name, and those people who like living on the edge of new features do so knowingly. Let the users control their destiny, let the users decide when. That's the hallmark of a trustworthy company.
        zlynx 7y ago
        Users who intentionally run beta / nightly are not normal users. How many of them run on cheap "desktop" computers they bought on sale at Target, do you think?
        scarejunba 7y ago
  - swebs 7y ago
    Isn't that the entire point of the nightly and beta branches?
    - dblohm7 7y ago
      Those populations are not representative of the release population.
- ehsankia 7y ago
  > the metrics being optimized for usually aren't in the users' best interests Do you have any source for that? If you truly believe that Firefox is optimizing metrics that aren't in the users' best interest in general, why are you even using said software in the first place?
  - loopl 7y ago
    What other browser can you use if you are interested in some sort of privacy? Is Chrome compiled from source safe from tracking?
    - Jerry2 7y ago
      >What other browser can you use if you are interested in some sort of privacy? Waterfox, Safari. Some others too but they're a pain to use (ungoogled-chromium for example). Chrome itself has a bunch of binary blobs so I would stay away from that.
  - hu3 7y ago
    For starters, Mozilla dropped RSS support after shoving proprietary closed source software down our throats in the form of Pocket. I don't see this as having users best interests in mind.
- y0ghur7_xxx 7y ago
  > especially when the metrics being optimized for usually aren't in the users' best interests I don't think so. Mozilla is not google, and apart from the mr. Robot error, that they apologized for, I am sure Mozilla is using it for making the browser better, and not to use you for targeted ads like the other browser maker does.
  - SquareWheel 7y ago
    When has Chrome used A/B testing for ad targeting in the past? Can you link to an example?
    - commoner 7y ago
      Google Chrome uses the RLZ tracking identifier for some installations. "RLZ gives us the ability to accurately measure the success of marketing promotions and distribution partnerships in order to meet our contractual and financial obligations." https://blog.chromium.org/2010/06/in-open-for-rlz.html https://blog.chromium.org/2010/06/in-open-for-rlz.html
      - SquareWheel 7y ago
        While not exactly A/B testing as discussed above, that's still a relevant link. Thank you for sharing.
        commoner 7y ago
        No problem. There's more information in the Google Chrome Privacy Whitepaper: https://www.google.com/intl/en/chrome/privacy/whitepaper.html#measurepromotions https://www.google.com/intl/en/chrome/privacy/whitepaper.htm...
- whamlastxmas 7y ago
  I think this can be compared to McDonald's slightly altering the recipe of something for part of the country. I don't find that unethical at all, and frankly the things we put in our body are a lot more personal than which web browser we use.
  - toss1 7y ago
    Yes, but it isn't (well should not be if they want to stay within the law) adding completely random drugs an garbage to the food. It may be seeking cheaper recipes with no sales losses, better recipes to improve sales, or something else, but it all has to be within the context of proper regulation, i.e.,, FDA. (we can argue elsewhere about the effectiveness of that system). This unlimited testing can easily be seeking to better maximize psychological exploits with no regulation. So, yes, I'm happy to see someone putting in a bit of control.
  - floatingatoll 7y ago
    That is dismissive of how much value people place on their selves and digital aspects thereof. My identity, private conversations, and sexuality are infinitely more personal than the food I consume. I agree that it’s appropriate to A/B test changes, within reasonable bounds. I wish the debate would focus more on what the reasonable bounds are to each objector.
enriquto 7y ago
Having this capability is useful and is certainly not brain-damaged. Adding unsuspecting users to the pool of testers is.
- joecool1029 7y ago
  Mozilla hasn't exactly been known for great decision making with the 'feature' in the past. I disabled across all my clients after this shit: https://www.cnet.com/news/mozilla-backpedals-after-mr-robot-firefox-misstep/ https://www.cnet.com/news/mozilla-backpedals-after-mr-robot-...
kzrdude 7y ago
It's just the usual tug of war between developers and users for control of the computer. A completely careful open source project would normally leave such controversial things off by default, and moz which leans corporate will go part of the way towards enabling such features. I don't mind their compromise, they do good.
- badsectoracula 7y ago
  > It's just the usual tug of war between developers and users for control of the computer. And the annoying part of this being developers who act as if they do not understand (or just brush off) the concerns that the users have. Like, people want control over their own machines. How is that hard to understand?
  - viraptor 7y ago
    Because functionality like that can be used to enable/disable features outside of tests as well. For example gating new things which could become a stability issue. Sometimes it's a choice between control or no control. Sometimes it's between your browser crashing or not. The power users will care about control, but the general population of users not really.
    - admax88q 7y ago
      > For example gating new things which could become a stability issue. And yet we were somehow able to ship stable software to millions of people in the days before telemetry/studies. What telemetry/studies truly provides is offloading QA costs to users.
      - viraptor 7y ago
        The software was also simpler. And we were not really great to shipping stable software in the past.
        7y ago
AnaniasAnanas 7y ago
It certainly is not brain-damage, but it is however very user-hostile. Are you the unlucky one that ended up with the broken A/B testing change? Well, sucks for you and nobody will be able to help you. Do you want to try a specific testing change because you like it/something is broken on your end and that will fix it? Sucks to be you again, you get no choice on this. Do you want to analyze the change to see if they have something bad in it? Nope, not possible unless if you end up being one of the lucky ones. In addition if Mozilla were to get compromised (either as an organization or in software side) could use it as a backdoor on unsuspecting users or even do targeted attacks. None of that would not be an issue if they depended on nightly/beta for changes and error detection, nor would it be if they had a list of possible tests somewhere that the user could manually enable and possibly ask some users explicitly if they want to enable one on the browser startup.
tinus_hn 7y ago
Having a backdoor on by default while you’re pretending to be ‘the good guy’ is brain damage. It’s taking an age to build trust and then throwing it away in a second.

Yes, running experiments on unknowning and unconsenting users is unethical, especially when the metrics being optimized for usually aren't in the users' best interests. The widespread acceptance for this behaviour could definitely be classified as brain damage.

staticassertion 7y ago
So when deploying changes it should just go to every single user with no incremental rollout?
- benatkin 7y ago
  Yes. The testing should be done on users that opt in (likely less than 0.1% but that should be OK if it's done properly) and it should be rolled out by releasing an update to everyone. These so-called "best practices" often aren't.
  - Vinnl 7y ago
    By relying on opt-in you won't test whether a feature actually works for the average user, so if you don't test it, you might roll something useless/confusing/actively harmful out to all of them instead of just a few.
    - zamalek 7y ago
      Exactly. The recent Windows 10 update fiascos demonstrate this perfectly: they are opt-in (Insider program) and there were too few users experiencing the bugs to overcome the noise of those who weren't. You need a truly random sample.
      - benatkin 7y ago
        I think they updated too many parts of Windows at once, not too many computers at once. And I'm not certain that a delayed rollout would have fixed it. Is it fair that a random sampling of people are given the problem?
      - hutzlibu 7y ago
        So instead of having enough voluntary or paid testers to detect catastrophic, file deleting bugs in a new version, your solution is to just roll out the new changes to random users expecting a stable system? Something went terrible wrong somewhere, that we got to this.
    - enriquto 7y ago
      And that's OK. Being honest to your users is much more important that having the most efficient deploy system. I, for one, as a matter of principle, opt-in to all user tracking systems that are disabled by default, and try to opt-out whenever they are enabled by default. In the case of firefox, it was never clear to me that the tracking/remote install capability was enabled by default.
  - staticassertion 7y ago
    I don't see the purpose. There seems to be an assumption that A/B testing is inherently bad, therefor should be opt-in. A/B testing is necessary for stable rollouts, that much seems self evident. So what is the evidence that A/B testing is harmful to users?
    - badsectoracula 7y ago
      It isn't A/B testing that is bad, it is running and/or modifying stuff in the user's computer without their knowledge that is bad. If a site modifies some part of their layout or whatever isn't bad, but a browser installing addons or changing settings to see if it will crash is bad.
      - staticassertion 7y ago
        Well, it's not like they're hoping it will crash. Presumably they're hoping it won't. Wouldn't it be more irresponsible to deploy to every user without first testing it out?
        chii 7y ago
        that's what the stable/experimental channels are for - a user who don't want to be tested on (such as a business user) can easily be sure that they are getting a stable release and their settings won't update under them. Experimental/beta channels can have these changes pushed to it, since that's implied by the name, and those people who like living on the edge of new features do so knowingly. Let the users control their destiny, let the users decide when. That's the hallmark of a trustworthy company.
        zlynx 7y ago
        Users who intentionally run beta / nightly are not normal users. How many of them run on cheap "desktop" computers they bought on sale at Target, do you think?
      - scarejunba 7y ago
- swebs 7y ago
  Isn't that the entire point of the nightly and beta branches?
  - dblohm7 7y ago
    Those populations are not representative of the release population.
ehsankia 7y ago
> the metrics being optimized for usually aren't in the users' best interests Do you have any source for that? If you truly believe that Firefox is optimizing metrics that aren't in the users' best interest in general, why are you even using said software in the first place?
- loopl 7y ago
  What other browser can you use if you are interested in some sort of privacy? Is Chrome compiled from source safe from tracking?
  - Jerry2 7y ago
    >What other browser can you use if you are interested in some sort of privacy? Waterfox, Safari. Some others too but they're a pain to use (ungoogled-chromium for example). Chrome itself has a bunch of binary blobs so I would stay away from that.
- hu3 7y ago
  For starters, Mozilla dropped RSS support after shoving proprietary closed source software down our throats in the form of Pocket. I don't see this as having users best interests in mind.
y0ghur7_xxx 7y ago
> especially when the metrics being optimized for usually aren't in the users' best interests I don't think so. Mozilla is not google, and apart from the mr. Robot error, that they apologized for, I am sure Mozilla is using it for making the browser better, and not to use you for targeted ads like the other browser maker does.
- SquareWheel 7y ago
  When has Chrome used A/B testing for ad targeting in the past? Can you link to an example?
  - commoner 7y ago
    Google Chrome uses the RLZ tracking identifier for some installations. "RLZ gives us the ability to accurately measure the success of marketing promotions and distribution partnerships in order to meet our contractual and financial obligations." https://blog.chromium.org/2010/06/in-open-for-rlz.html https://blog.chromium.org/2010/06/in-open-for-rlz.html
    - SquareWheel 7y ago
      While not exactly A/B testing as discussed above, that's still a relevant link. Thank you for sharing.
      - commoner 7y ago
        No problem. There's more information in the Google Chrome Privacy Whitepaper: https://www.google.com/intl/en/chrome/privacy/whitepaper.html#measurepromotions https://www.google.com/intl/en/chrome/privacy/whitepaper.htm...
whamlastxmas 7y ago
I think this can be compared to McDonald's slightly altering the recipe of something for part of the country. I don't find that unethical at all, and frankly the things we put in our body are a lot more personal than which web browser we use.
- toss1 7y ago
  Yes, but it isn't (well should not be if they want to stay within the law) adding completely random drugs an garbage to the food. It may be seeking cheaper recipes with no sales losses, better recipes to improve sales, or something else, but it all has to be within the context of proper regulation, i.e.,, FDA. (we can argue elsewhere about the effectiveness of that system). This unlimited testing can easily be seeking to better maximize psychological exploits with no regulation. So, yes, I'm happy to see someone putting in a bit of control.
- floatingatoll 7y ago
  That is dismissive of how much value people place on their selves and digital aspects thereof. My identity, private conversations, and sexuality are infinitely more personal than the food I consume. I agree that it’s appropriate to A/B test changes, within reasonable bounds. I wish the debate would focus more on what the reasonable bounds are to each objector.

enriquto 7y ago

Having this capability is useful and is certainly not brain-damaged. Adding unsuspecting users to the pool of testers is.

joecool1029 7y ago
Mozilla hasn't exactly been known for great decision making with the 'feature' in the past. I disabled across all my clients after this shit: https://www.cnet.com/news/mozilla-backpedals-after-mr-robot-firefox-misstep/ https://www.cnet.com/news/mozilla-backpedals-after-mr-robot-...

kzrdude 7y ago

It's just the usual tug of war between developers and users for control of the computer. A completely careful open source project would normally leave such controversial things off by default, and moz which leans corporate will go part of the way towards enabling such features. I don't mind their compromise, they do good.

badsectoracula 7y ago
> It's just the usual tug of war between developers and users for control of the computer. And the annoying part of this being developers who act as if they do not understand (or just brush off) the concerns that the users have. Like, people want control over their own machines. How is that hard to understand?
- viraptor 7y ago
  Because functionality like that can be used to enable/disable features outside of tests as well. For example gating new things which could become a stability issue. Sometimes it's a choice between control or no control. Sometimes it's between your browser crashing or not. The power users will care about control, but the general population of users not really.
  - admax88q 7y ago
    > For example gating new things which could become a stability issue. And yet we were somehow able to ship stable software to millions of people in the days before telemetry/studies. What telemetry/studies truly provides is offloading QA costs to users.
    - viraptor 7y ago
      The software was also simpler. And we were not really great to shipping stable software in the past.
      - 7y ago

AnaniasAnanas 7y ago

It certainly is not brain-damage, but it is however very user-hostile. Are you the unlucky one that ended up with the broken A/B testing change? Well, sucks for you and nobody will be able to help you. Do you want to try a specific testing change because you like it/something is broken on your end and that will fix it? Sucks to be you again, you get no choice on this. Do you want to analyze the change to see if they have something bad in it? Nope, not possible unless if you end up being one of the lucky ones. In addition if Mozilla were to get compromised (either as an organization or in software side) could use it as a backdoor on unsuspecting users or even do targeted attacks. None of that would not be an issue if they depended on nightly/beta for changes and error detection, nor would it be if they had a list of possible tests somewhere that the user could manually enable and possibly ask some users explicitly if they want to enable one on the browser startup.

tinus_hn 7y ago

Having a backdoor on by default while you’re pretending to be ‘the good guy’ is brain damage. It’s taking an age to build trust and then throwing it away in a second.