6 ms·

Yes, running experiments on unknowning and unconsenting users is unethical, especially when the metrics being optimized for usually aren't in the users' best in

Yes, running experiments on unknowning and unconsenting users is unethical, especially when the metrics being optimized for usually aren't in the users' best interests. The widespread acceptance for this behaviour could definitely be classified as brain damage.

staticassertion 8y ago
So when deploying changes it should just go to every single user with no incremental rollout?
- benatkin 8y ago
  Yes. The testing should be done on users that opt in (likely less than 0.1% but that should be OK if it's done properly) and it should be rolled out by releasing an update to everyone. These so-called "best practices" often aren't.
  - Vinnl 8y ago
    By relying on opt-in you won't test whether a feature actually works for the average user, so if you don't test it, you might roll something useless/confusing/actively harmful out to all of them instead of just a few.
    - zamalek 8y ago
      Exactly. The recent Windows 10 update fiascos demonstrate this perfectly: they are opt-in (Insider program) and there were too few users experiencing the bugs to overcome the noise of those who weren't. You need a truly random sample.
      - benatkin 8y ago
        I think they updated too many parts of Windows at once, not too many computers at once. And I'm not certain that a delayed rollout would have fixed it. Is it fair that a random sampling of people are given the problem?
      - hutzlibu 8y ago
        So instead of having enough voluntary or paid testers to detect catastrophic, file deleting bugs in a new version, your solution is to just roll out the new changes to random users expecting a stable system? Something went terrible wrong somewhere, that we got to this.
    - enriquto 8y ago
      And that's OK. Being honest to your users is much more important that having the most efficient deploy system. I, for one, as a matter of principle, opt-in to all user tracking systems that are disabled by default, and try to opt-out whenever they are enabled by default. In the case of firefox, it was never clear to me that the tracking/remote install capability was enabled by default.
  - staticassertion 8y ago
    I don't see the purpose. There seems to be an assumption that A/B testing is inherently bad, therefor should be opt-in. A/B testing is necessary for stable rollouts, that much seems self evident. So what is the evidence that A/B testing is harmful to users?
    - badsectoracula 8y ago
      It isn't A/B testing that is bad, it is running and/or modifying stuff in the user's computer without their knowledge that is bad. If a site modifies some part of their layout or whatever isn't bad, but a browser installing addons or changing settings to see if it will crash is bad.
      - staticassertion 8y ago
        Well, it's not like they're hoping it will crash. Presumably they're hoping it won't. Wouldn't it be more irresponsible to deploy to every user without first testing it out?
        chii 8y ago
        that's what the stable/experimental channels are for - a user who don't want to be tested on (such as a business user) can easily be sure that they are getting a stable release and their settings won't update under them. Experimental/beta channels can have these changes pushed to it, since that's implied by the name, and those people who like living on the edge of new features do so knowingly. Let the users control their destiny, let the users decide when. That's the hallmark of a trustworthy company.
        zlynx 8y ago
        Users who intentionally run beta / nightly are not normal users. How many of them run on cheap "desktop" computers they bought on sale at Target, do you think?
      - scarejunba 8y ago
- swebs 8y ago
  Isn't that the entire point of the nightly and beta branches?
  - dblohm7 8y ago
    Those populations are not representative of the release population.
ehsankia 8y ago
> the metrics being optimized for usually aren't in the users' best interests Do you have any source for that? If you truly believe that Firefox is optimizing metrics that aren't in the users' best interest in general, why are you even using said software in the first place?
- loopl 8y ago
  What other browser can you use if you are interested in some sort of privacy? Is Chrome compiled from source safe from tracking?
  - Jerry2 8y ago
    >What other browser can you use if you are interested in some sort of privacy? Waterfox, Safari. Some others too but they're a pain to use (ungoogled-chromium for example). Chrome itself has a bunch of binary blobs so I would stay away from that.
- hu3 8y ago
  For starters, Mozilla dropped RSS support after shoving proprietary closed source software down our throats in the form of Pocket. I don't see this as having users best interests in mind.
y0ghur7_xxx 8y ago
> especially when the metrics being optimized for usually aren't in the users' best interests I don't think so. Mozilla is not google, and apart from the mr. Robot error, that they apologized for, I am sure Mozilla is using it for making the browser better, and not to use you for targeted ads like the other browser maker does.
- SquareWheel 8y ago
  When has Chrome used A/B testing for ad targeting in the past? Can you link to an example?
  - commoner 8y ago
    Google Chrome uses the RLZ tracking identifier for some installations. "RLZ gives us the ability to accurately measure the success of marketing promotions and distribution partnerships in order to meet our contractual and financial obligations." https://blog.chromium.org/2010/06/in-open-for-rlz.html https://blog.chromium.org/2010/06/in-open-for-rlz.html
    - SquareWheel 8y ago
      While not exactly A/B testing as discussed above, that's still a relevant link. Thank you for sharing.
      - commoner 8y ago
        No problem. There's more information in the Google Chrome Privacy Whitepaper: https://www.google.com/intl/en/chrome/privacy/whitepaper.html#measurepromotions https://www.google.com/intl/en/chrome/privacy/whitepaper.htm...
whamlastxmas 8y ago
I think this can be compared to McDonald's slightly altering the recipe of something for part of the country. I don't find that unethical at all, and frankly the things we put in our body are a lot more personal than which web browser we use.
- toss1 8y ago
  Yes, but it isn't (well should not be if they want to stay within the law) adding completely random drugs an garbage to the food. It may be seeking cheaper recipes with no sales losses, better recipes to improve sales, or something else, but it all has to be within the context of proper regulation, i.e.,, FDA. (we can argue elsewhere about the effectiveness of that system). This unlimited testing can easily be seeking to better maximize psychological exploits with no regulation. So, yes, I'm happy to see someone putting in a bit of control.
- floatingatoll 8y ago
  That is dismissive of how much value people place on their selves and digital aspects thereof. My identity, private conversations, and sexuality are infinitely more personal than the food I consume. I agree that it’s appropriate to A/B test changes, within reasonable bounds. I wish the debate would focus more on what the reasonable bounds are to each objector.

staticassertion 8y ago

So when deploying changes it should just go to every single user with no incremental rollout?

benatkin 8y ago
Yes. The testing should be done on users that opt in (likely less than 0.1% but that should be OK if it's done properly) and it should be rolled out by releasing an update to everyone. These so-called "best practices" often aren't.
- Vinnl 8y ago
  By relying on opt-in you won't test whether a feature actually works for the average user, so if you don't test it, you might roll something useless/confusing/actively harmful out to all of them instead of just a few.
  - zamalek 8y ago
    Exactly. The recent Windows 10 update fiascos demonstrate this perfectly: they are opt-in (Insider program) and there were too few users experiencing the bugs to overcome the noise of those who weren't. You need a truly random sample.
    - benatkin 8y ago
      I think they updated too many parts of Windows at once, not too many computers at once. And I'm not certain that a delayed rollout would have fixed it. Is it fair that a random sampling of people are given the problem?
    - hutzlibu 8y ago
      So instead of having enough voluntary or paid testers to detect catastrophic, file deleting bugs in a new version, your solution is to just roll out the new changes to random users expecting a stable system? Something went terrible wrong somewhere, that we got to this.
  - enriquto 8y ago
    And that's OK. Being honest to your users is much more important that having the most efficient deploy system. I, for one, as a matter of principle, opt-in to all user tracking systems that are disabled by default, and try to opt-out whenever they are enabled by default. In the case of firefox, it was never clear to me that the tracking/remote install capability was enabled by default.
- staticassertion 8y ago
  I don't see the purpose. There seems to be an assumption that A/B testing is inherently bad, therefor should be opt-in. A/B testing is necessary for stable rollouts, that much seems self evident. So what is the evidence that A/B testing is harmful to users?
  - badsectoracula 8y ago
    It isn't A/B testing that is bad, it is running and/or modifying stuff in the user's computer without their knowledge that is bad. If a site modifies some part of their layout or whatever isn't bad, but a browser installing addons or changing settings to see if it will crash is bad.
    - staticassertion 8y ago
      Well, it's not like they're hoping it will crash. Presumably they're hoping it won't. Wouldn't it be more irresponsible to deploy to every user without first testing it out?
      - chii 8y ago
        that's what the stable/experimental channels are for - a user who don't want to be tested on (such as a business user) can easily be sure that they are getting a stable release and their settings won't update under them. Experimental/beta channels can have these changes pushed to it, since that's implied by the name, and those people who like living on the edge of new features do so knowingly. Let the users control their destiny, let the users decide when. That's the hallmark of a trustworthy company.
        zlynx 8y ago
        Users who intentionally run beta / nightly are not normal users. How many of them run on cheap "desktop" computers they bought on sale at Target, do you think?
    - scarejunba 8y ago
swebs 8y ago
Isn't that the entire point of the nightly and beta branches?
- dblohm7 8y ago
  Those populations are not representative of the release population.

ehsankia 8y ago

> the metrics being optimized for usually aren't in the users' best interests Do you have any source for that? If you truly believe that Firefox is optimizing metrics that aren't in the users' best interest in general, why are you even using said software in the first place?

loopl 8y ago
What other browser can you use if you are interested in some sort of privacy? Is Chrome compiled from source safe from tracking?
- Jerry2 8y ago
  >What other browser can you use if you are interested in some sort of privacy? Waterfox, Safari. Some others too but they're a pain to use (ungoogled-chromium for example). Chrome itself has a bunch of binary blobs so I would stay away from that.
hu3 8y ago
For starters, Mozilla dropped RSS support after shoving proprietary closed source software down our throats in the form of Pocket. I don't see this as having users best interests in mind.

y0ghur7_xxx 8y ago

> especially when the metrics being optimized for usually aren't in the users' best interests I don't think so. Mozilla is not google, and apart from the mr. Robot error, that they apologized for, I am sure Mozilla is using it for making the browser better, and not to use you for targeted ads like the other browser maker does.

SquareWheel 8y ago
When has Chrome used A/B testing for ad targeting in the past? Can you link to an example?
- commoner 8y ago
  Google Chrome uses the RLZ tracking identifier for some installations. "RLZ gives us the ability to accurately measure the success of marketing promotions and distribution partnerships in order to meet our contractual and financial obligations." https://blog.chromium.org/2010/06/in-open-for-rlz.html https://blog.chromium.org/2010/06/in-open-for-rlz.html
  - SquareWheel 8y ago
    While not exactly A/B testing as discussed above, that's still a relevant link. Thank you for sharing.
    - commoner 8y ago
      No problem. There's more information in the Google Chrome Privacy Whitepaper: https://www.google.com/intl/en/chrome/privacy/whitepaper.html#measurepromotions https://www.google.com/intl/en/chrome/privacy/whitepaper.htm...

whamlastxmas 8y ago

I think this can be compared to McDonald's slightly altering the recipe of something for part of the country. I don't find that unethical at all, and frankly the things we put in our body are a lot more personal than which web browser we use.

toss1 8y ago
Yes, but it isn't (well should not be if they want to stay within the law) adding completely random drugs an garbage to the food. It may be seeking cheaper recipes with no sales losses, better recipes to improve sales, or something else, but it all has to be within the context of proper regulation, i.e.,, FDA. (we can argue elsewhere about the effectiveness of that system). This unlimited testing can easily be seeking to better maximize psychological exploits with no regulation. So, yes, I'm happy to see someone putting in a bit of control.
floatingatoll 8y ago
That is dismissive of how much value people place on their selves and digital aspects thereof. My identity, private conversations, and sexuality are infinitely more personal than the food I consume. I agree that it’s appropriate to A/B test changes, within reasonable bounds. I wish the debate would focus more on what the reasonable bounds are to each objector.