3 ms·

Someone once pointed out to me how "engagement" is often misread by companies. The product team makes a change to their website or app and suddenly engagement

Someone once pointed out to me how "engagement" is often misread by companies. The product team makes a change to their website or app and suddenly engagement skyrockets. They think, "Wow! That change really got people to use our product. Let's do more!" when, in reality, the change they made is initially so confusing that their users spend more time than they would otherwise just trying to find their way around all the new changes. This, coupled with new product owners, product managers, and executives who feel like they need to justify their positions and the easiest way to do that is to "improve" the product and we have a negative feedback loop that is perceived by everyone within the company as a virtuous cycle. All the while they're just pissing their customers off.

AJ007 7y ago
The experienced managers know that the difference, but others could either be easily fooled or gaming the statistics to help boost their internal politics standing. The bigger problem is when these engagement bumps are valid but destroy long term value in the cohort. The auto binge optimization makes viewers more cautious about opening the Netflix app altogether. This happens with things like fast food restaurants. Subway expanded rapidly, and then tried holding up their sales with discounts. How long can someone eat at Subway weekly or daily before they never want to go there again? The bar is so high, you are way betting off starting a new franchise brand than trying things like removing yoga mat chemicals from your bread or hiring a new mascot who isn’t a child rapist.
- thaumasiotes 7y ago
  > How long can someone eat at Subway weekly or daily before they never want to go there again? Bread is one of only a very few foods that seem to be immune to "food exhaustion", the tendency to want to stop eating something you've been eating. (Milk is another, but I have a feeling that only applies to lactose-tolerant people.) So if you varied the fillings, probably indefinitely. How long can you have cereal for breakfast before you never want to eat cereal again?
  - rawTruthHurts 7y ago
    About a couple of weeks, twenty days max.
  - asdff 7y ago
    It's not just bread, though. It's subway. I believe the veggie option is still yesterdays hummus crudely spread around a bun with bell peppers, black olives, an raw onions. The meats are cafeteria tier, in fact I think the meatball sub recipe is the exact slop served in my old dining hall. They undermined their masterful subliminal marketing on the entire millennial generation by raising the $5 foot long sub to near chipotle bowl price points, and in my city at least cheap excellent street food is already on every corner. I only ever settle for subway, it's not a destination.
    - thaumasiotes 7y ago
      I happen to agree about subway, but that's a personal taste that immediately implies it shouldn't be able to run a store in the first place. "This is terrible food" just isn't the same argument as "it's good, but you get tired of it after a month". High sales that taper off aren't compatible with the first.
the-pigeon 7y ago
Absolutely. A/B testing causes way more bad decisions than good ones because people run the tests wrong and they interpret the data wrong. But I've found explaining that to people to be an uphill battle. They get sold on an "objective" way to optimize the UI and then don't want to hear that the method is actually very subjective when not done correctly. And it's hard to execute and interpret correctly so every A/B test I've seen ran has been very subjective.
- pvorb 7y ago
  Haha. So I'll do better than the average company doing A/B testing by simply avoiding the methodology altogether? That's nice to know.
  - the-pigeon 7y ago
    I'm a huge believer in the "expert product manager" route. Meaning have one person who talks to all the stakeholders and every type of user. Who just knows your product backwards and forwards. And thus can drive what changes you need to make to push things forward. What exactly the means depends on the nature of the product.
asdff 7y ago
I wonder if autoplay was just a scheme to bump up these numbers for shareholders? Frequently, I'd used to leave my xbox running netflix, because the xbox takes 5 mins to start up, and just shut off the TV. Not anymore now that it would watch an entire made for netflix movie in the background and start polluting the feed with innumerable potemkin netflix series I'd never even heard of. Engagement while I'm sleeping in bed.

AJ007 7y ago

The experienced managers know that the difference, but others could either be easily fooled or gaming the statistics to help boost their internal politics standing. The bigger problem is when these engagement bumps are valid but destroy long term value in the cohort. The auto binge optimization makes viewers more cautious about opening the Netflix app altogether. This happens with things like fast food restaurants. Subway expanded rapidly, and then tried holding up their sales with discounts. How long can someone eat at Subway weekly or daily before they never want to go there again? The bar is so high, you are way betting off starting a new franchise brand than trying things like removing yoga mat chemicals from your bread or hiring a new mascot who isn’t a child rapist.

thaumasiotes 7y ago
> How long can someone eat at Subway weekly or daily before they never want to go there again? Bread is one of only a very few foods that seem to be immune to "food exhaustion", the tendency to want to stop eating something you've been eating. (Milk is another, but I have a feeling that only applies to lactose-tolerant people.) So if you varied the fillings, probably indefinitely. How long can you have cereal for breakfast before you never want to eat cereal again?
- rawTruthHurts 7y ago
  About a couple of weeks, twenty days max.
- asdff 7y ago
  It's not just bread, though. It's subway. I believe the veggie option is still yesterdays hummus crudely spread around a bun with bell peppers, black olives, an raw onions. The meats are cafeteria tier, in fact I think the meatball sub recipe is the exact slop served in my old dining hall. They undermined their masterful subliminal marketing on the entire millennial generation by raising the $5 foot long sub to near chipotle bowl price points, and in my city at least cheap excellent street food is already on every corner. I only ever settle for subway, it's not a destination.
  - thaumasiotes 7y ago
    I happen to agree about subway, but that's a personal taste that immediately implies it shouldn't be able to run a store in the first place. "This is terrible food" just isn't the same argument as "it's good, but you get tired of it after a month". High sales that taper off aren't compatible with the first.

the-pigeon 7y ago

Absolutely. A/B testing causes way more bad decisions than good ones because people run the tests wrong and they interpret the data wrong. But I've found explaining that to people to be an uphill battle. They get sold on an "objective" way to optimize the UI and then don't want to hear that the method is actually very subjective when not done correctly. And it's hard to execute and interpret correctly so every A/B test I've seen ran has been very subjective.

pvorb 7y ago
Haha. So I'll do better than the average company doing A/B testing by simply avoiding the methodology altogether? That's nice to know.
- the-pigeon 7y ago
  I'm a huge believer in the "expert product manager" route. Meaning have one person who talks to all the stakeholders and every type of user. Who just knows your product backwards and forwards. And thus can drive what changes you need to make to push things forward. What exactly the means depends on the nature of the product.

asdff 7y ago

I wonder if autoplay was just a scheme to bump up these numbers for shareholders? Frequently, I'd used to leave my xbox running netflix, because the xbox takes 5 mins to start up, and just shut off the TV. Not anymore now that it would watch an entire made for netflix movie in the background and start polluting the feed with innumerable potemkin netflix series I'd never even heard of. Engagement while I'm sleeping in bed.