3 ms·

Solution: don't A/B test your users. A/B testing people without their informed consent is immoral, unethical, and should be illegal.

vehemenz 6mo ago
Depends entirely on the stakes and whether personal data is involved
- inetknght 6mo ago
  > Depends entirely on the stakes and whether personal data is involved Sure. Let me just A/B test whether or not you'll respond positively or negatively to having your news delivered via push notification or delayed by 10 minutes. I'm sure you would appreciate being tested on without your consent, just so that I can make an extra quick buck at your expense. Nothing amoral or unethical about it.
  - pitched 6mo ago
    What do you think about slow rollouts for new features? Like, we think this new push notification system will be loved but let’s ship to only 1% of users in case there’s a horrible unforeseen consequence like occasional 10min delays? Dashboard goes upside down -> revert then work through logs to figure out what the hell went wrong.
    - inetknght 6mo ago
      What do you think of things you purchased changing over time into something you didn't purchase?
      - estomagordo 6mo ago
        That's literally any software subscription ever.
        inetknght 6mo ago
        So you're perfectly okay with repeatedly paying for a shit product, getting shat on by the company in the form of being tested for feedback, and "maybe" getting a better product in the future. Mind you, that "better" isn't necessarily better for you but more explicitly better for the company you're paying. Sounds like someone who doesn't care about being a sheep. Or maybe someone whose salary depends on having sheep.
        nocman 6mo ago
        I think you are making far too wide-sweeping statements. I think most people here probably agree that if Anthropic drops Claude Code from the Pro plan after people have paid with the understanding that it is part of the package, that would be wrong, and they deserve to lose business over it. However, there are plenty of situations where A/B testing is entirely benign, and I would not have any problem with a company doing that testing without getting consent first. Every form of A/B testing is not done just for the gain of the company doing the testing.
        estomagordo 6mo ago
        I am not getting shat on by them working to improve the product. Have you got a screw loose?
        throw-the-towel 6mo ago
        Well, duh, that's precisely what's wrong with subscriptions.
        estomagordo 6mo ago
        That the product might improve?
        throw-the-towel 6mo ago
        In my experience, improvements are so rare compared to regressions, they might as well not exist.
  - vehemenz 6mo ago
    How does your example warrant such moral righteousness? The stakes of a late news notification seem strikingly low.
skeledrew 6mo ago
To play devil's advocate, without A/B testing a lot of decisions would be made with insufficient relevant data, and lead to subpar results that affect the many negatively form the road.
- wat10000 6mo ago
  A lot of decisions made with A/B testing are also made with insufficient relevant data, but it's less obvious since it's easy to think the A/B results cover everything.
- serf 6mo ago
  counter-point : the companies that are most famous for A/B testing routinely are also the ones with the most notoriously non-existent customer service departments globally, facebook/google/amazon/ebay. Groups that harbor dissatisfied customers by essentially being 'the only show in town.'. so, what i'm saying is : I think a lot of companies align themselves with the cash first and then measure whether or not the negative image/user impact is manageable . (in fact I know they operate this way.)
shimman 6mo ago
Agreed and I can't wait until they regulate this stuff out of existence. It's absolutely hostile software technique that is deeply anti-human.
zorobo 6mo ago
It is necessary to have a control group, just as in trials for new drugs.
- greenchair 6mo ago
  no its not.
cycomanic 6mo ago
So you're saying software should never change or you're happy with A testing, but not A/B testing
- inetknght 6mo ago
  > So you're saying software should never change Generally, yes. Make your software better first before releasing it and you won't need to make changes to it. Want a new feature that you didn't have before? That's a new software product. > or you're happy with A testing, but not A/B testing I'm happy with testing when the user has explicitly opted-in for it.

vehemenz 6mo ago

Depends entirely on the stakes and whether personal data is involved

inetknght 6mo ago
> Depends entirely on the stakes and whether personal data is involved Sure. Let me just A/B test whether or not you'll respond positively or negatively to having your news delivered via push notification or delayed by 10 minutes. I'm sure you would appreciate being tested on without your consent, just so that I can make an extra quick buck at your expense. Nothing amoral or unethical about it.
- pitched 6mo ago
  What do you think about slow rollouts for new features? Like, we think this new push notification system will be loved but let’s ship to only 1% of users in case there’s a horrible unforeseen consequence like occasional 10min delays? Dashboard goes upside down -> revert then work through logs to figure out what the hell went wrong.
  - inetknght 6mo ago
    What do you think of things you purchased changing over time into something you didn't purchase?
    - estomagordo 6mo ago
      That's literally any software subscription ever.
      - inetknght 6mo ago
        So you're perfectly okay with repeatedly paying for a shit product, getting shat on by the company in the form of being tested for feedback, and "maybe" getting a better product in the future. Mind you, that "better" isn't necessarily better for you but more explicitly better for the company you're paying. Sounds like someone who doesn't care about being a sheep. Or maybe someone whose salary depends on having sheep.
        nocman 6mo ago
        I think you are making far too wide-sweeping statements. I think most people here probably agree that if Anthropic drops Claude Code from the Pro plan after people have paid with the understanding that it is part of the package, that would be wrong, and they deserve to lose business over it. However, there are plenty of situations where A/B testing is entirely benign, and I would not have any problem with a company doing that testing without getting consent first. Every form of A/B testing is not done just for the gain of the company doing the testing.
        estomagordo 6mo ago
        I am not getting shat on by them working to improve the product. Have you got a screw loose?
      - throw-the-towel 6mo ago
        Well, duh, that's precisely what's wrong with subscriptions.
        estomagordo 6mo ago
        That the product might improve?
        throw-the-towel 6mo ago
        In my experience, improvements are so rare compared to regressions, they might as well not exist.
- vehemenz 6mo ago
  How does your example warrant such moral righteousness? The stakes of a late news notification seem strikingly low.

skeledrew 6mo ago

To play devil's advocate, without A/B testing a lot of decisions would be made with insufficient relevant data, and lead to subpar results that affect the many negatively form the road.

wat10000 6mo ago
A lot of decisions made with A/B testing are also made with insufficient relevant data, but it's less obvious since it's easy to think the A/B results cover everything.
serf 6mo ago
counter-point : the companies that are most famous for A/B testing routinely are also the ones with the most notoriously non-existent customer service departments globally, facebook/google/amazon/ebay. Groups that harbor dissatisfied customers by essentially being 'the only show in town.'. so, what i'm saying is : I think a lot of companies align themselves with the cash first and then measure whether or not the negative image/user impact is manageable . (in fact I know they operate this way.)

shimman 6mo ago

Agreed and I can't wait until they regulate this stuff out of existence. It's absolutely hostile software technique that is deeply anti-human.

zorobo 6mo ago

It is necessary to have a control group, just as in trials for new drugs.

greenchair 6mo ago
no its not.

cycomanic 6mo ago

So you're saying software should never change or you're happy with A testing, but not A/B testing

inetknght 6mo ago
> So you're saying software should never change Generally, yes. Make your software better first before releasing it and you won't need to make changes to it. Want a new feature that you didn't have before? That's a new software product. > or you're happy with A testing, but not A/B testing I'm happy with testing when the user has explicitly opted-in for it.