4 ms·

Lots of folks here are very negative on what seems to be a clear-cut example of an A/B test. If Netflix sees improvement of their objectives, they will roll it

Lots of folks here are very negative on what seems to be a clear-cut example of an A/B test. If Netflix sees improvement of their objectives, they will roll it out. Maybe they will even use predictive modeling based on your usage as to whether or not they show interstitial ads to you or not.

ebbv 8y ago
You’re showing a lot of faith that they will interpret the results fairly and act on them. The auto play video while browsing is pretty universally reviled and they haven’t turned that off.
- meesterdude 8y ago
  Nor provided any means by preference to disable. Which means they aren't even willing to acknowledge a significant portion of users don't like the behavior and allowing them any recourse. Which means the autoplay is more for them than for us.
- jondwillis 8y ago
  Yes, I am. I like the auto playing feature. I like the idea of interstitial promotional ads. I am Netflix’s target user and I consider myself a fairly nuanced consumer of their media.
  - josho 8y ago
    What value do you get from ads running on Netflix?
blitmap 8y ago
Not to sound obtuse, but I subscribed to their service to watch shows/movies and not be A/B-tested. I know I agreed to it in the skimmed ToS, but the main appeal was to not have ads of any kind. Oh well ->cancel();
true_tuna 8y ago
Sounds right. If you don’t want ads on Netflix do this. The first time you see an ad, stop watching Netflix immediately. Then cancel your subscription, destroy your television, drop out of society, go live in a cabin in the woods.
- charlesdm 8y ago
  Well, they'll likely increase prices again for a "no ad" experience.
troydavis 8y ago
How does a short-term A/B test measure long-term negative sentiment? Measuring minutes played or cancellations over a few months is pretty easy. Measuring contribution to a gradual loss of trust, fewer or weaker recommendations to others, and perhaps cancellations months or years later is all but impossible. It's really hard to construct a split test that's deep enough to fully measure negative reactions. Usually that's not an issue because the thing being tested either has a positive or neutral impact; other than by accident, it's not downright offensive. Fully measuring something that a meaningful percentage of users consider offensive would be really tough. It's even possible for a decrease in trust that doesn't manifest in user behavior for years. Maybe they stay customers and continue playing media, but start treating Netflix's future decisions with skepticism. The closest to an indicator is probably NPS, but NPS has its own issues. (The fact that Netflix isn't testing a user-facing "Show a preroll trailer from Netflix's best recommendation" setting says a lot about their willingness to ignore long-term, hard-to-measure impacts…)
daveFNbuck 8y ago
Most of the complaints I'm reading are about how this is the sort of thing that harms user experience while improving the kinds of metrics that are measured in an A/B test. Maybe they'll put in a lot of effort to undo it for me, but chances are this is just a way my life will be slightly worse from now on. Should I be positive about that?

ebbv 8y ago

You’re showing a lot of faith that they will interpret the results fairly and act on them. The auto play video while browsing is pretty universally reviled and they haven’t turned that off.

meesterdude 8y ago
Nor provided any means by preference to disable. Which means they aren't even willing to acknowledge a significant portion of users don't like the behavior and allowing them any recourse. Which means the autoplay is more for them than for us.
jondwillis 8y ago
Yes, I am. I like the auto playing feature. I like the idea of interstitial promotional ads. I am Netflix’s target user and I consider myself a fairly nuanced consumer of their media.
- josho 8y ago
  What value do you get from ads running on Netflix?

blitmap 8y ago

Not to sound obtuse, but I subscribed to their service to watch shows/movies and not be A/B-tested. I know I agreed to it in the skimmed ToS, but the main appeal was to not have ads of any kind. Oh well ->cancel();

true_tuna 8y ago

Sounds right. If you don’t want ads on Netflix do this. The first time you see an ad, stop watching Netflix immediately. Then cancel your subscription, destroy your television, drop out of society, go live in a cabin in the woods.

charlesdm 8y ago
Well, they'll likely increase prices again for a "no ad" experience.

troydavis 8y ago

How does a short-term A/B test measure long-term negative sentiment? Measuring minutes played or cancellations over a few months is pretty easy. Measuring contribution to a gradual loss of trust, fewer or weaker recommendations to others, and perhaps cancellations months or years later is all but impossible. It's really hard to construct a split test that's deep enough to fully measure negative reactions. Usually that's not an issue because the thing being tested either has a positive or neutral impact; other than by accident, it's not downright offensive. Fully measuring something that a meaningful percentage of users consider offensive would be really tough. It's even possible for a decrease in trust that doesn't manifest in user behavior for years. Maybe they stay customers and continue playing media, but start treating Netflix's future decisions with skepticism. The closest to an indicator is probably NPS, but NPS has its own issues. (The fact that Netflix isn't testing a user-facing "Show a preroll trailer from Netflix's best recommendation" setting says a lot about their willingness to ignore long-term, hard-to-measure impacts…)

daveFNbuck 8y ago

Most of the complaints I'm reading are about how this is the sort of thing that harms user experience while improving the kinds of metrics that are measured in an A/B test. Maybe they'll put in a lot of effort to undo it for me, but chances are this is just a way my life will be slightly worse from now on. Should I be positive about that?