6 ms·

That is not a problem with A/B testing. It's a problem with the values of the company. I've worked with people who will say, "Oh, this tests well, but we don't

That is not a problem with A/B testing. It's a problem with the values of the company. I've worked with people who will say, "Oh, this tests well, but we don't want to do it because of [long term concerns X and Y]." People who value revenue metrics over all else will still do shitty things for users even if they don't A/B test.

lylejantzi3rd 4y ago
The problem with that approach is you now have evidence that the short term change will show immediate results and no evidence of the long term concerns. Given solid numbers vs somebody's gut, most managers will go with the solid numbers, even if the company has good values.
- eropple 4y ago
  > Given solid numbers vs somebody's gut, most managers will go with the solid numbers, even if the company has good values. If this is true, that indicates that a company's theoretical "good values" are not being passed down to those decision-makers in a way that makes them impact decision-making. Which means the company does not have those "good values" in the first place. They have lip service. Values that are not practiced are not values. You are what you do, both when someone is watching and when someone is not.
  - hectormalot 4y ago
    I think I see where you’re coming from, but how I interpreted it, it’s just really hard to make these decisions, and from a managers perspective its not that black and white: - almost any change will have have people arguing for and against it - if ‘company values’ is a trump card to prevent a change, it will be used by the people against the change - as a manager, to still make decisions in such an environment, you’ll find yourself needing to weigh the upsides and downsides even if there are strong company values (it just puts a higher weight on certain concerns) - as parent said, short term impact supported with numbers is easier to weigh and defend than (possible but unknown) long term detrimental impact. Thus, I think parent is right. Even in corporates with strong company values, it’s easier to prioritize the short term proven impact over long term unproven impact. And therefore, at scale, such decisions will be made.
    - thrashh 4y ago
      Still, that’s what sets a good manager from an average, no? No different with how there are relatively few good engineers but most are average or bad? I mean there are companies that put out good products but most are average at best. I suppose A/B testing is bad in the way that table saws are bad — that they are dangerous and I see a lot of videos of people using them in ways that are gonna chop off their fingers, but I wouldn’t blame the table saw for it or want to take it away.
  - grogenaut 4y ago
    You're assuming bad intent on numbers and good intent on gut. I've seen many places that just go with gut on everything, that's worse. I like working places that get numbers on everything.
  - dahart 4y ago
    You’re making a lot of assumptions. Looking at data and reacting to it absolutely does not imply a lack of values. What if the “good values” are to look at the numbers? This would be the right thing to do for the livelihood of employees, and the right thing to do from a scientific perspective. Some “good values” include never passing dogma down over time but instead questioning assumptions and testing for results.
- projectazorian 4y ago
  This is where an involved founder can make all the difference. They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience.
  - lylejantzi3rd 4y ago
    > They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience. And how do you know the short term changes will degrade the brand and the user experience?
    - mitthrowaway2 4y ago
      Maybe deep down, users love dark patterns! Our judgement is powerless to determine; only the objective data can tell us!
  - chefandy 4y ago
    In my experience founders are often more than happy to toe the line between nudges and shoves.
- MarkSweep 4y ago
  This is why you have long-term holdbacks: a group of users who never sees a set of experiments. That way you can measure over a longer period of time the true impact of a set of experiments.
  - dotancohen 4y ago
    You cannot control for lots of other externalities, such as inflation and competitors coming to market. These parameters will affect long-term users as well.
    - smachiz 4y ago
      isn't that why it's good? You can't control infinite variables, the only ones here are A/B and you're keeping a subset of them as a control group?
      - dotancohen 4y ago
        I suppose that is a way of looking at it.
    - bootsmann 4y ago
      Well presumably these externalities are independent of the variables you are changing so they should affect your hold-out set and your experiment sets equally.
  - rdca 4y ago
    Booking.com does not do it. Once an A/B test shows positive outcome with a high enough confidence level, the experiment goes "full on" and is shown to everybody.
    - esafak 4y ago
      And that's how you get dark patterns. Let them enjoy their engagement lifts until they drive their product over the cliff.
    - 4y ago
- HPsquared 4y ago
  "Seeing like a State" in action.
- wpietri 4y ago
  Solid numbers of what? Here we're talking about numbers on short-term effects only, with zero data on long-term effects. I'd claim that people who are reeds in the wind when there's a short-term gain available do not in fact have good values. "Take the money and run" is a value for sure, and it's common enough. But I'd have a hard time calling it good.
deleted 4y ago
[deleted]
antman 4y ago
There is also a mathematical problem of assigning events and actions to long term effects. The usual IT crowd unfamiliar with the respective literature will try to ab test and grid search out of the actual scientific part of data science. I had also fallen to that trap.
- computerfriend 4y ago
  What would you recommend reading on this?
jiggywiggy 4y ago
There is a grey line. I think only one room left is actually useful info. But yeah they push it too far. But most people in a company will be able to argue for themselves the info is useful and truthful so in their minds it's morally ok.
- toast0 4y ago
  Only one room left would be useful information if it were accurate. Most of the time, booking agents don't really have an accurate picture of inventory though. It's more like only one room available for booking.com to book right now. It could be that the hotel is holding back rooms for other channels or because they like to not be fully booked so far ahead of time or perhaps the hotel has found listing only one room at a time gets them a better look to book ratio (in part because of anxiety inspiring features like this). Without an understanding of the industry though, it's not really useful information.
chambers 4y ago
DHH, founder of Ruby on Rails and Basecamp, drew a line between core values and A/B testing https://world.hey.com/dhh/we-don-t-a-b-test-core-values-91b5f562 https://world.hey.com/dhh/we-don-t-a-b-test-core-values-91b5... > This is the tyranny of easy metrics. It's easy to measure how much money is saved by preventing cancelations, it's much harder to measure how much long-term business is lost by poisoning your reputation with the 99.9% of customers who had to jump hoops and dodge sleazeballs to get out of the subscription. But the latter could well be orders of magnitude money more over the long run.
jvans 4y ago
I agree but a truly nuanced approach to interpretation of A/B tests is rare especially when mixed incentives are involved. Ignoring empirical evidence is bad and taking it as gospel is also bad.
canadianfella 4y ago
[dead]
chefandy 4y ago
Yeah-- I think blaming dark patterns on A/B testing is like blaming thermostats for chilly houses. A/B tests just show if one thing does more of something you're testing for than something else. If you're just testing for "conversions" then you're going to make websites like booking.com. If you're testing to see if users are more stressed out by one situation or another, or testing to see if expert users are stymied by some interface abstraction designed to make things easier for less sophisticated users, then that's totally different.
- tspike 4y ago
  Sure, but it’s a lot harder to test for things like stress or being stymied than it is to test for conversion. Easy implementation + results that make money graphs go up will win every time without C-level involvement.
  - chefandy 4y ago
    It's a lot easier to break something with a hammer than bang in some nails. Using a marker to deface property is a lot easier than using it to draw a picture.
SilasX 4y ago
>I've worked with people who will say, "Oh, this tests well, but we don't want to do it because of [long term concerns X and Y]." Then why were they testing it, if they already knew other concerns would veto that alternative?
- wpietri 4y ago
  I suspect if you were to think a bit, you could come up with some ideas yourself. If you're really stuck, let me know what you've got so far and I'll fill in the rest.
  - SilasX 4y ago
    I think I'll just wait for you to give the substantive reply you have in mind. Accounts being jerks get less leeway from me than people being respectful and/or standing behind their words.
    - wpietri 4y ago
      You'll be waiting quite awhile. If there's anything HN has taught me, it's the meaning of "pearls before swine". And the futility in providing free labor for people who seem to think they're entitled to it.
      - SilasX 4y ago
        It's a discussion forum, amigo. You're supposed to share your insights with those who haven't yet achieved them. If you're not comfortable doing that, you can just not post instead of posting merely to whine and insult. Bookmarking for later.
        wpietri 4y ago
        I share with people who seem like they'll benefit from it. That includes people who have done at least a little work to understanding what's going on. It definitely doesn't include HN's legion of querulous jerks who mainly look for nitpicks, gotchas, and other opportunities to feel smart, even when that comes at the expense of the good discussion this place is nominally for. You get to influence which group people see you as part of.
dhosek 4y ago
When I worked at eHarmony there were two things that we were aware of above all else: 1. Users hated all the advertising even if they were signed in with a paid account and 2. eHarmony had no intention of stopping this because they valued the revenue from the advertising over the user experience. I noticed that Redfin has started adding ads to their site which really annoys me. Folks you’ll get a sizable commission for connecting me with the listing agent. The ad revenue has to be a rounding error compared to that. Why are you damaging your brand by showing me ads for luxury vacations next to houses that I’m looking at buying?
- wpietri 4y ago
  Great examples. To this list, I'd add Amazon's ads. It's providing them with very obvious revenue growth. It's parasitic on their ecommerce business, though, in ways that for me and many others, harm the things that made Amazon popular in the first place. That'll never change, though, because that revenue has enough people attached to it that they'll fight tooth and nail to keep it no matter how much it harms the rest of Amazon.
phendrenad2 4y ago
Companies need to balance between metrics and vision. It's easy to chase metrics and lose track of why your company exists. If you're just chasing metrics, what differentiates you from anyone else? What big bets are you making? Every company I've worked at has fallen into this trap, and it's gotten worse due to the tech slump.
smueller1234 4y ago
I think the problem underneath is quantitative fallacy: https://en.m.wikipedia.org/wiki/McNamara_fallacy https://en.m.wikipedia.org/wiki/McNamara_fallacy

lylejantzi3rd 4y ago

The problem with that approach is you now have evidence that the short term change will show immediate results and no evidence of the long term concerns. Given solid numbers vs somebody's gut, most managers will go with the solid numbers, even if the company has good values.

eropple 4y ago
> Given solid numbers vs somebody's gut, most managers will go with the solid numbers, even if the company has good values. If this is true, that indicates that a company's theoretical "good values" are not being passed down to those decision-makers in a way that makes them impact decision-making. Which means the company does not have those "good values" in the first place. They have lip service. Values that are not practiced are not values. You are what you do, both when someone is watching and when someone is not.
- hectormalot 4y ago
  I think I see where you’re coming from, but how I interpreted it, it’s just really hard to make these decisions, and from a managers perspective its not that black and white: - almost any change will have have people arguing for and against it - if ‘company values’ is a trump card to prevent a change, it will be used by the people against the change - as a manager, to still make decisions in such an environment, you’ll find yourself needing to weigh the upsides and downsides even if there are strong company values (it just puts a higher weight on certain concerns) - as parent said, short term impact supported with numbers is easier to weigh and defend than (possible but unknown) long term detrimental impact. Thus, I think parent is right. Even in corporates with strong company values, it’s easier to prioritize the short term proven impact over long term unproven impact. And therefore, at scale, such decisions will be made.
  - thrashh 4y ago
    Still, that’s what sets a good manager from an average, no? No different with how there are relatively few good engineers but most are average or bad? I mean there are companies that put out good products but most are average at best. I suppose A/B testing is bad in the way that table saws are bad — that they are dangerous and I see a lot of videos of people using them in ways that are gonna chop off their fingers, but I wouldn’t blame the table saw for it or want to take it away.
- grogenaut 4y ago
  You're assuming bad intent on numbers and good intent on gut. I've seen many places that just go with gut on everything, that's worse. I like working places that get numbers on everything.
- dahart 4y ago
  You’re making a lot of assumptions. Looking at data and reacting to it absolutely does not imply a lack of values. What if the “good values” are to look at the numbers? This would be the right thing to do for the livelihood of employees, and the right thing to do from a scientific perspective. Some “good values” include never passing dogma down over time but instead questioning assumptions and testing for results.
projectazorian 4y ago
This is where an involved founder can make all the difference. They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience.
- lylejantzi3rd 4y ago
  > They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience. And how do you know the short term changes will degrade the brand and the user experience?
  - mitthrowaway2 4y ago
    Maybe deep down, users love dark patterns! Our judgement is powerless to determine; only the objective data can tell us!
- chefandy 4y ago
  In my experience founders are often more than happy to toe the line between nudges and shoves.
MarkSweep 4y ago
This is why you have long-term holdbacks: a group of users who never sees a set of experiments. That way you can measure over a longer period of time the true impact of a set of experiments.
- dotancohen 4y ago
  You cannot control for lots of other externalities, such as inflation and competitors coming to market. These parameters will affect long-term users as well.
  - smachiz 4y ago
    isn't that why it's good? You can't control infinite variables, the only ones here are A/B and you're keeping a subset of them as a control group?
    - dotancohen 4y ago
      I suppose that is a way of looking at it.
  - bootsmann 4y ago
    Well presumably these externalities are independent of the variables you are changing so they should affect your hold-out set and your experiment sets equally.
- rdca 4y ago
  Booking.com does not do it. Once an A/B test shows positive outcome with a high enough confidence level, the experiment goes "full on" and is shown to everybody.
  - esafak 4y ago
    And that's how you get dark patterns. Let them enjoy their engagement lifts until they drive their product over the cliff.
  - 4y ago
HPsquared 4y ago
"Seeing like a State" in action.
wpietri 4y ago
Solid numbers of what? Here we're talking about numbers on short-term effects only, with zero data on long-term effects. I'd claim that people who are reeds in the wind when there's a short-term gain available do not in fact have good values. "Take the money and run" is a value for sure, and it's common enough. But I'd have a hard time calling it good.

deleted 4y ago

[deleted]

antman 4y ago

There is also a mathematical problem of assigning events and actions to long term effects. The usual IT crowd unfamiliar with the respective literature will try to ab test and grid search out of the actual scientific part of data science. I had also fallen to that trap.

computerfriend 4y ago
What would you recommend reading on this?

jiggywiggy 4y ago

There is a grey line. I think only one room left is actually useful info. But yeah they push it too far. But most people in a company will be able to argue for themselves the info is useful and truthful so in their minds it's morally ok.

toast0 4y ago
Only one room left would be useful information if it were accurate. Most of the time, booking agents don't really have an accurate picture of inventory though. It's more like only one room available for booking.com to book right now. It could be that the hotel is holding back rooms for other channels or because they like to not be fully booked so far ahead of time or perhaps the hotel has found listing only one room at a time gets them a better look to book ratio (in part because of anxiety inspiring features like this). Without an understanding of the industry though, it's not really useful information.

chambers 4y ago

DHH, founder of Ruby on Rails and Basecamp, drew a line between core values and A/B testing https://world.hey.com/dhh/we-don-t-a-b-test-core-values-91b5f562 https://world.hey.com/dhh/we-don-t-a-b-test-core-values-91b5... > This is the tyranny of easy metrics. It's easy to measure how much money is saved by preventing cancelations, it's much harder to measure how much long-term business is lost by poisoning your reputation with the 99.9% of customers who had to jump hoops and dodge sleazeballs to get out of the subscription. But the latter could well be orders of magnitude money more over the long run.

jvans 4y ago

I agree but a truly nuanced approach to interpretation of A/B tests is rare especially when mixed incentives are involved. Ignoring empirical evidence is bad and taking it as gospel is also bad.

canadianfella 4y ago

[dead]

chefandy 4y ago

Yeah-- I think blaming dark patterns on A/B testing is like blaming thermostats for chilly houses. A/B tests just show if one thing does more of something you're testing for than something else. If you're just testing for "conversions" then you're going to make websites like booking.com. If you're testing to see if users are more stressed out by one situation or another, or testing to see if expert users are stymied by some interface abstraction designed to make things easier for less sophisticated users, then that's totally different.

tspike 4y ago
Sure, but it’s a lot harder to test for things like stress or being stymied than it is to test for conversion. Easy implementation + results that make money graphs go up will win every time without C-level involvement.
- chefandy 4y ago
  It's a lot easier to break something with a hammer than bang in some nails. Using a marker to deface property is a lot easier than using it to draw a picture.

SilasX 4y ago

>I've worked with people who will say, "Oh, this tests well, but we don't want to do it because of [long term concerns X and Y]." Then why were they testing it, if they already knew other concerns would veto that alternative?

wpietri 4y ago
I suspect if you were to think a bit, you could come up with some ideas yourself. If you're really stuck, let me know what you've got so far and I'll fill in the rest.
- SilasX 4y ago
  I think I'll just wait for you to give the substantive reply you have in mind. Accounts being jerks get less leeway from me than people being respectful and/or standing behind their words.
  - wpietri 4y ago
    You'll be waiting quite awhile. If there's anything HN has taught me, it's the meaning of "pearls before swine". And the futility in providing free labor for people who seem to think they're entitled to it.
    - SilasX 4y ago
      It's a discussion forum, amigo. You're supposed to share your insights with those who haven't yet achieved them. If you're not comfortable doing that, you can just not post instead of posting merely to whine and insult. Bookmarking for later.
      - wpietri 4y ago
        I share with people who seem like they'll benefit from it. That includes people who have done at least a little work to understanding what's going on. It definitely doesn't include HN's legion of querulous jerks who mainly look for nitpicks, gotchas, and other opportunities to feel smart, even when that comes at the expense of the good discussion this place is nominally for. You get to influence which group people see you as part of.

dhosek 4y ago

When I worked at eHarmony there were two things that we were aware of above all else: 1. Users hated all the advertising even if they were signed in with a paid account and 2. eHarmony had no intention of stopping this because they valued the revenue from the advertising over the user experience. I noticed that Redfin has started adding ads to their site which really annoys me. Folks you’ll get a sizable commission for connecting me with the listing agent. The ad revenue has to be a rounding error compared to that. Why are you damaging your brand by showing me ads for luxury vacations next to houses that I’m looking at buying?

wpietri 4y ago
Great examples. To this list, I'd add Amazon's ads. It's providing them with very obvious revenue growth. It's parasitic on their ecommerce business, though, in ways that for me and many others, harm the things that made Amazon popular in the first place. That'll never change, though, because that revenue has enough people attached to it that they'll fight tooth and nail to keep it no matter how much it harms the rest of Amazon.

phendrenad2 4y ago

Companies need to balance between metrics and vision. It's easy to chase metrics and lose track of why your company exists. If you're just chasing metrics, what differentiates you from anyone else? What big bets are you making? Every company I've worked at has fallen into this trap, and it's gotten worse due to the tech slump.

smueller1234 4y ago

I think the problem underneath is quantitative fallacy: https://en.m.wikipedia.org/wiki/McNamara_fallacy https://en.m.wikipedia.org/wiki/McNamara_fallacy