6 ms·

What does "tolerating experiments" mean? If they can tell it's an experiment, then isn't your change bad? Do you mean "tolerate change"? But then you still eve

by remram 2y ago

What does "tolerating experiments" mean? If they can tell it's an experiment, then isn't your change bad? Do you mean "tolerate change"? But then you still eventually roll out the change to everyone anyway... Or do you mean that users would see a different color for the "buy now" button every day? From a purely statistical point of view, if you select users which "tolerate" your change before you measure how many users "like" your change, you can make up any outcome you want.

j45 2y ago
I suspect it’s because some users will actually be pioneers and early adopters vs believing they are. This kind of threshold adds some flexibility into the subjectivity of finding the best cohort to test a feature with.
- remram 2y ago
  Where the best cohort to test with is the one that agrees with you... You can call this measure "courage" but that is not actually what you are measuring. What you measure is not that different from agreement.
  - j45 2y ago
    I didn’t use the word courage, still I understand what you’re saying.
    - remram 2y ago
      adontz did above, that's what they called this user-tolerance-for-experiments metric. I didn't mean to imply you would too, apologies.
      - j45 2y ago
        Oh, no need to apologize at all. I could have clarified as well that I was leaning more towards the user-tolerance... or as I like to call it user-guess that this feature might be OK with them :) Another thing I like about granular and flexible feature flag management is you can really dial in and learn from which features get used by whom, actually.... instead of building things that will collect dust.
inhumantsar 2y ago
I think you might be mixing things up a bit. the tolerance score wouldn't be tied to a specific change. it's an estimate of how tolerant a person is of changes generally. it's not that different from asking people if they want to be part of a beta testers group or if they would be open to being surveyed by market researchers. targeting like that usually doesn't have a significant impact on the results of individual experiments.
- remram 2y ago
  If only people who like changes like your change, should you really go ahead? Plus you don't know what that correlates to. Maybe being "tolerant of changes" correlates with being particularly computer-savvy, and you're rolling out changes that are difficult to navigate. Maybe it correlates to people who use your site only for a single task, it would appear they don't mind changes across the platform, but they don't see them. Maybe it correlates with people who hate your site now, and are happy you're changing it (but still hate it). You can't use a selected subset that is not obviously uncorrelated from your target variable. This is selection bias as a service.

by remram 2y ago

j45 2y ago

I suspect it’s because some users will actually be pioneers and early adopters vs believing they are. This kind of threshold adds some flexibility into the subjectivity of finding the best cohort to test a feature with.

remram 2y ago
Where the best cohort to test with is the one that agrees with you... You can call this measure "courage" but that is not actually what you are measuring. What you measure is not that different from agreement.
- j45 2y ago
  I didn’t use the word courage, still I understand what you’re saying.
  - remram 2y ago
    adontz did above, that's what they called this user-tolerance-for-experiments metric. I didn't mean to imply you would too, apologies.
    - j45 2y ago
      Oh, no need to apologize at all. I could have clarified as well that I was leaning more towards the user-tolerance... or as I like to call it user-guess that this feature might be OK with them :) Another thing I like about granular and flexible feature flag management is you can really dial in and learn from which features get used by whom, actually.... instead of building things that will collect dust.

inhumantsar 2y ago

I think you might be mixing things up a bit. the tolerance score wouldn't be tied to a specific change. it's an estimate of how tolerant a person is of changes generally. it's not that different from asking people if they want to be part of a beta testers group or if they would be open to being surveyed by market researchers. targeting like that usually doesn't have a significant impact on the results of individual experiments.

remram 2y ago
If only people who like changes like your change, should you really go ahead? Plus you don't know what that correlates to. Maybe being "tolerant of changes" correlates with being particularly computer-savvy, and you're rolling out changes that are difficult to navigate. Maybe it correlates to people who use your site only for a single task, it would appear they don't mind changes across the platform, but they don't see them. Maybe it correlates with people who hate your site now, and are happy you're changing it (but still hate it). You can't use a selected subset that is not obviously uncorrelated from your target variable. This is selection bias as a service.