6 ms·

They've probably done testing and found that for some significant percentage of married couples, seeing their spouse's posts caused them to reduce their Faceboo

by TipVFL 9y ago

They've probably done testing and found that for some significant percentage of married couples, seeing their spouse's posts caused them to reduce their Facebook usage. So unhappy marriages are ruining it for everyone. I feel like the general trend of UI simplification and a/b testing everything has companies chasing after anything that will push their general metrics up in the short term, at the expense of slowly disenfranchising a large portion of their users. I interviewed at Netflix recently for a position on their UI team, and I questioned the interviewer about a lot of annoying aspects of the Netflix interface. He was aware of every complaint, and his explanation for every one of them was he wishes he could change it but the a/b testing shows they get higher engagement with it the way it is. So, for example, they show you movies you have already said you don't like, because enough people give a thumbs down and then later watch the movie. In the short term very few people are going to cancel Netflix because of this, but enough of these decisions in succession is erosive to customer goodwill. Personally, these kind of decisions wouldn't bother me that much if apps and websites made it easy to customize your experience. Facebook makes it difficult by opaquely shaping your newsfeed and giving you what seems like hundreds of options across a half dozen settings pages. Netflix makes it difficult by just giving you practically no options, but there's enough demand that's there a tiny industry in just making websites and plugins to make Netflix usable.

throwaway2048 9y ago
yes, a/b testing optimizes for an average user that dosent exist, and everyone is unhappy with.
- hawktheslayer 9y ago
  Reminds me of the old tale of the man who drowned in a lake with an average depth of 3 feet.
- aikinai 9y ago
  A/B doesn’t design for an average user; it can be all, a tiny niche, or something in between depending on what you’re measuring. For example, DAU and engagement will be affected by drastically different distributions of users.
nojvek 9y ago
I guess A/B testing will tell you if the horse will become faster if you change from iron shoes to carbon fiber ones. A/B testing won’t ever get you a Tesla from a horse.
- gatmne 9y ago
  It will also tell you that pumping horses full of steroids will make them go faster. Tech's absurd obsession with metrics and statistics will be their undoing.
- mygo 9y ago
  If you do enough binary switching, you eventually will get a Tesla. Assuming a Tesla is optimal. Let’s test to find out. How do you think the eyeball arose from unicellular eukaryotes?
  - throwaway2048 9y ago
    local maxima trap hill climbing style optimizing, many (most?) times improvements require jumping large sub optimal chasms that can never be crossed by gradual improvement.
  - Franciscouzo 9y ago
    You won't get out of local minima/maxima with hill climbing though. Eyeballs are nowhere near optimal either.
  - nora4 9y ago
    I don't think so. A/B testing is like gradient descent which is a greedy algorithm. You move in the direction that locally looks best. Evolution on the other hand allows for suboptimal species to persist for enough time to let them develop their advantage. (In the language of optimization evolution allows you to go past the local optimal and reach global optima by allowing you to move in non-optimal direction -- as long as the move is not catastrophic.)
    - mygo 9y ago
rhizome 9y ago
Did you ask what their definition of engagement is, and why it's the primary measure?
dpark 9y ago
> In the short term very few people are going to cancel Netflix because of this, but enough of these decisions in succession is erosive to customer goodwill. The “auto preview” is so unpleasant that I’ve essentially stopped using Netflix entirely. I’ve no doubt that their testing shows it increases engagement, but I find it so jarring and abrasive that I rarely open the app any more. I’m sure they still have content I want to watch, but I can’t stand the experience of trying to find it. I would love to have more insight into their A/B engagement metrics. I’m sure they capture if immediate engagement goes up when a feature is enabled, and probably the same over time periods of days or weeks. Do they also capture if total app usage declines 75% over 6 months after a feature is enabled?
- garyclarke27 9y ago
  That’s funny - I Like the auto preview, in fact I would like them to go further and advertise their content to me, as live TV does. When I watch Netflix at the end of a hard day I’m in lazy consumer mode, I want content to be suggested to me. I hate wading through static dull pictures which give me very few clues as to the content they represent. Netflix are unstoppable - they offer incredible value compared to extortionate SKY, i cancelled their £120 per month package 6 months ago and I have never regretted it.
  - Ntrails 9y ago
    I'm listening to music or half-watching twitch whilst scrolling through Netflix to decide what I'll watch with supper. It's incredibly frustrating to have them autoplay video content, to the extent that I've started going back to browsing torrent sites instead. It doesn't help that their website is a laggy piece of shit with miserable discover-ability. But, you know. Ugh.
    - dom96 9y ago
      Surely the video is muted by default, why is that a problem?
      - 9y ago
pjc50 9y ago
> he wishes he could change it but the a/b testing shows they get higher engagement with it the way it is The upshot of this is that if Netflix or Facebook _do_ see a slow drift of users away from the platform, they are institutionally incapable of understanding why. Engagement is not the same thing as satisfaction, which is far harder to measure. (I'm also now wondering how static the results of A/B testing really are - if an idea fails AB even once does the company abandon it forever?)
- candiodari 9y ago
  But engagement is what advertisers pay for, isn't it ? Clicks ! Clicks ! Clicks ! So I'm pretty sure that they are institutionally incapable of understanding why for a very good reason.

by TipVFL 9y ago

throwaway2048 9y ago

yes, a/b testing optimizes for an average user that dosent exist, and everyone is unhappy with.

hawktheslayer 9y ago
Reminds me of the old tale of the man who drowned in a lake with an average depth of 3 feet.
aikinai 9y ago
A/B doesn’t design for an average user; it can be all, a tiny niche, or something in between depending on what you’re measuring. For example, DAU and engagement will be affected by drastically different distributions of users.

nojvek 9y ago

I guess A/B testing will tell you if the horse will become faster if you change from iron shoes to carbon fiber ones. A/B testing won’t ever get you a Tesla from a horse.

gatmne 9y ago
It will also tell you that pumping horses full of steroids will make them go faster. Tech's absurd obsession with metrics and statistics will be their undoing.
mygo 9y ago
If you do enough binary switching, you eventually will get a Tesla. Assuming a Tesla is optimal. Let’s test to find out. How do you think the eyeball arose from unicellular eukaryotes?
- throwaway2048 9y ago
  local maxima trap hill climbing style optimizing, many (most?) times improvements require jumping large sub optimal chasms that can never be crossed by gradual improvement.
- Franciscouzo 9y ago
  You won't get out of local minima/maxima with hill climbing though. Eyeballs are nowhere near optimal either.
- nora4 9y ago
  I don't think so. A/B testing is like gradient descent which is a greedy algorithm. You move in the direction that locally looks best. Evolution on the other hand allows for suboptimal species to persist for enough time to let them develop their advantage. (In the language of optimization evolution allows you to go past the local optimal and reach global optima by allowing you to move in non-optimal direction -- as long as the move is not catastrophic.)
  - mygo 9y ago

rhizome 9y ago

Did you ask what their definition of engagement is, and why it's the primary measure?

dpark 9y ago

> In the short term very few people are going to cancel Netflix because of this, but enough of these decisions in succession is erosive to customer goodwill. The “auto preview” is so unpleasant that I’ve essentially stopped using Netflix entirely. I’ve no doubt that their testing shows it increases engagement, but I find it so jarring and abrasive that I rarely open the app any more. I’m sure they still have content I want to watch, but I can’t stand the experience of trying to find it. I would love to have more insight into their A/B engagement metrics. I’m sure they capture if immediate engagement goes up when a feature is enabled, and probably the same over time periods of days or weeks. Do they also capture if total app usage declines 75% over 6 months after a feature is enabled?

garyclarke27 9y ago
That’s funny - I Like the auto preview, in fact I would like them to go further and advertise their content to me, as live TV does. When I watch Netflix at the end of a hard day I’m in lazy consumer mode, I want content to be suggested to me. I hate wading through static dull pictures which give me very few clues as to the content they represent. Netflix are unstoppable - they offer incredible value compared to extortionate SKY, i cancelled their £120 per month package 6 months ago and I have never regretted it.
- Ntrails 9y ago
  I'm listening to music or half-watching twitch whilst scrolling through Netflix to decide what I'll watch with supper. It's incredibly frustrating to have them autoplay video content, to the extent that I've started going back to browsing torrent sites instead. It doesn't help that their website is a laggy piece of shit with miserable discover-ability. But, you know. Ugh.
  - dom96 9y ago
    Surely the video is muted by default, why is that a problem?
    - 9y ago

pjc50 9y ago

> he wishes he could change it but the a/b testing shows they get higher engagement with it the way it is The upshot of this is that if Netflix or Facebook _do_ see a slow drift of users away from the platform, they are institutionally incapable of understanding why. Engagement is not the same thing as satisfaction, which is far harder to measure. (I'm also now wondering how static the results of A/B testing really are - if an idea fails AB even once does the company abandon it forever?)

candiodari 9y ago
But engagement is what advertisers pay for, isn't it ? Clicks ! Clicks ! Clicks ! So I'm pretty sure that they are institutionally incapable of understanding why for a very good reason.