4 ms·

The problem with that approach is you now have evidence that the short term change will show immediate results and no evidence of the long term concerns. Given

The problem with that approach is you now have evidence that the short term change will show immediate results and no evidence of the long term concerns. Given solid numbers vs somebody's gut, most managers will go with the solid numbers, even if the company has good values.

eropple 4y ago
> Given solid numbers vs somebody's gut, most managers will go with the solid numbers, even if the company has good values. If this is true, that indicates that a company's theoretical "good values" are not being passed down to those decision-makers in a way that makes them impact decision-making. Which means the company does not have those "good values" in the first place. They have lip service. Values that are not practiced are not values. You are what you do, both when someone is watching and when someone is not.
- hectormalot 4y ago
  I think I see where you’re coming from, but how I interpreted it, it’s just really hard to make these decisions, and from a managers perspective its not that black and white: - almost any change will have have people arguing for and against it - if ‘company values’ is a trump card to prevent a change, it will be used by the people against the change - as a manager, to still make decisions in such an environment, you’ll find yourself needing to weigh the upsides and downsides even if there are strong company values (it just puts a higher weight on certain concerns) - as parent said, short term impact supported with numbers is easier to weigh and defend than (possible but unknown) long term detrimental impact. Thus, I think parent is right. Even in corporates with strong company values, it’s easier to prioritize the short term proven impact over long term unproven impact. And therefore, at scale, such decisions will be made.
  - thrashh 4y ago
    Still, that’s what sets a good manager from an average, no? No different with how there are relatively few good engineers but most are average or bad? I mean there are companies that put out good products but most are average at best. I suppose A/B testing is bad in the way that table saws are bad — that they are dangerous and I see a lot of videos of people using them in ways that are gonna chop off their fingers, but I wouldn’t blame the table saw for it or want to take it away.
- grogenaut 4y ago
  You're assuming bad intent on numbers and good intent on gut. I've seen many places that just go with gut on everything, that's worse. I like working places that get numbers on everything.
- dahart 4y ago
  You’re making a lot of assumptions. Looking at data and reacting to it absolutely does not imply a lack of values. What if the “good values” are to look at the numbers? This would be the right thing to do for the livelihood of employees, and the right thing to do from a scientific perspective. Some “good values” include never passing dogma down over time but instead questioning assumptions and testing for results.
projectazorian 4y ago
This is where an involved founder can make all the difference. They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience.
- lylejantzi3rd 4y ago
  > They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience. And how do you know the short term changes will degrade the brand and the user experience?
  - mitthrowaway2 4y ago
    Maybe deep down, users love dark patterns! Our judgement is powerless to determine; only the objective data can tell us!
- chefandy 4y ago
  In my experience founders are often more than happy to toe the line between nudges and shoves.
MarkSweep 4y ago
This is why you have long-term holdbacks: a group of users who never sees a set of experiments. That way you can measure over a longer period of time the true impact of a set of experiments.
- dotancohen 4y ago
  You cannot control for lots of other externalities, such as inflation and competitors coming to market. These parameters will affect long-term users as well.
  - smachiz 4y ago
    isn't that why it's good? You can't control infinite variables, the only ones here are A/B and you're keeping a subset of them as a control group?
    - dotancohen 4y ago
      I suppose that is a way of looking at it.
  - bootsmann 4y ago
    Well presumably these externalities are independent of the variables you are changing so they should affect your hold-out set and your experiment sets equally.
- rdca 4y ago
  Booking.com does not do it. Once an A/B test shows positive outcome with a high enough confidence level, the experiment goes "full on" and is shown to everybody.
  - esafak 4y ago
    And that's how you get dark patterns. Let them enjoy their engagement lifts until they drive their product over the cliff.
  - 4y ago
HPsquared 4y ago
"Seeing like a State" in action.
wpietri 4y ago
Solid numbers of what? Here we're talking about numbers on short-term effects only, with zero data on long-term effects. I'd claim that people who are reeds in the wind when there's a short-term gain available do not in fact have good values. "Take the money and run" is a value for sure, and it's common enough. But I'd have a hard time calling it good.

eropple 4y ago

> Given solid numbers vs somebody's gut, most managers will go with the solid numbers, even if the company has good values. If this is true, that indicates that a company's theoretical "good values" are not being passed down to those decision-makers in a way that makes them impact decision-making. Which means the company does not have those "good values" in the first place. They have lip service. Values that are not practiced are not values. You are what you do, both when someone is watching and when someone is not.

hectormalot 4y ago
I think I see where you’re coming from, but how I interpreted it, it’s just really hard to make these decisions, and from a managers perspective its not that black and white: - almost any change will have have people arguing for and against it - if ‘company values’ is a trump card to prevent a change, it will be used by the people against the change - as a manager, to still make decisions in such an environment, you’ll find yourself needing to weigh the upsides and downsides even if there are strong company values (it just puts a higher weight on certain concerns) - as parent said, short term impact supported with numbers is easier to weigh and defend than (possible but unknown) long term detrimental impact. Thus, I think parent is right. Even in corporates with strong company values, it’s easier to prioritize the short term proven impact over long term unproven impact. And therefore, at scale, such decisions will be made.
- thrashh 4y ago
  Still, that’s what sets a good manager from an average, no? No different with how there are relatively few good engineers but most are average or bad? I mean there are companies that put out good products but most are average at best. I suppose A/B testing is bad in the way that table saws are bad — that they are dangerous and I see a lot of videos of people using them in ways that are gonna chop off their fingers, but I wouldn’t blame the table saw for it or want to take it away.
grogenaut 4y ago
You're assuming bad intent on numbers and good intent on gut. I've seen many places that just go with gut on everything, that's worse. I like working places that get numbers on everything.
dahart 4y ago
You’re making a lot of assumptions. Looking at data and reacting to it absolutely does not imply a lack of values. What if the “good values” are to look at the numbers? This would be the right thing to do for the livelihood of employees, and the right thing to do from a scientific perspective. Some “good values” include never passing dogma down over time but instead questioning assumptions and testing for results.

projectazorian 4y ago

This is where an involved founder can make all the difference. They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience.

lylejantzi3rd 4y ago
> They’re often the only ones with the authority and incentive to say no to short-sighted cash grabs that degrade the brand and the user experience. And how do you know the short term changes will degrade the brand and the user experience?
- mitthrowaway2 4y ago
  Maybe deep down, users love dark patterns! Our judgement is powerless to determine; only the objective data can tell us!
chefandy 4y ago
In my experience founders are often more than happy to toe the line between nudges and shoves.

MarkSweep 4y ago

This is why you have long-term holdbacks: a group of users who never sees a set of experiments. That way you can measure over a longer period of time the true impact of a set of experiments.

dotancohen 4y ago
You cannot control for lots of other externalities, such as inflation and competitors coming to market. These parameters will affect long-term users as well.
- smachiz 4y ago
  isn't that why it's good? You can't control infinite variables, the only ones here are A/B and you're keeping a subset of them as a control group?
  - dotancohen 4y ago
    I suppose that is a way of looking at it.
- bootsmann 4y ago
  Well presumably these externalities are independent of the variables you are changing so they should affect your hold-out set and your experiment sets equally.
rdca 4y ago
Booking.com does not do it. Once an A/B test shows positive outcome with a high enough confidence level, the experiment goes "full on" and is shown to everybody.
- esafak 4y ago
  And that's how you get dark patterns. Let them enjoy their engagement lifts until they drive their product over the cliff.
- 4y ago

HPsquared 4y ago

"Seeing like a State" in action.

wpietri 4y ago

Solid numbers of what? Here we're talking about numbers on short-term effects only, with zero data on long-term effects. I'd claim that people who are reeds in the wind when there's a short-term gain available do not in fact have good values. "Take the money and run" is a value for sure, and it's common enough. But I'd have a hard time calling it good.