3 ms·
Netflix is not very good at seeing things that don't show up in the data. If it can't be A/B tested, they are fairly blind to it. There is no good way to A/B
by jedberg 6y ago
Netflix is not very good at seeing things that don't show up in the data. If it can't be A/B tested, they are fairly blind to it.
There is no good way to A/B test how cancellations affect signups. You can't release new seasons of a show to just some customers in a region. And if you only release new seasons in some regions, there is no way to know if the effect was because of local cultural reasons.
Until there is a large movement of people who make a lot of noise about not signing up for Netflix specifically because of their reputation for cancellation, this probably won't change.
And until then, ironically, this reputation will only make the problem worse. People will not watch a new show until they know it isn't cancelled, leading to even more cancellations because of low viewership.
- dragonwriter 6y ago> There is no good way to A/B test how cancellations affect signups. Not really an A/B test, but you can certainly from internal data see how individual viewership of cancelled shows (or even break it down to shows cancelled with and without closure, if you take the effort to have someone code shows for that) correlates with downstream retention. For the reputational stuff, I'd be very surprised if they are not using one or more of the consumer survey services to track that and the role decisions like cancelling series in general or in particular play into it.
- magicalhippo 6y ago> Netflix is not very good at seeing things that don't show up in the data. This reminds me of Battlefield 3. The designers of Battlefield 3 used data mining a lot to drive their decisions, particularly around multiplayer weapon balance. For a long time the competitive players said there was a balance issue with the M16, but the designers were adamant there was no issue because they didn't see any particular issue in the data. The M16 was used a bit more than the others, but it was also the weapon everyone started with and had unlocked. So they thought everything was fine. Then, after quite a while, someone made a video clearly highlighting the issue. And then the designers finally took a closer look and found that gosh darn, there was a bug in the weapon stats. The M16 did not get less inaccurate when you got hit, unlike all the other weapons. So if you equipped a M16 you were significantly more likely to successfully return fire, compared to the other weapons. Like an order of magnitude. Which significantly increased the chance of surviving the firefight if you were a moderately competent player. Of course the competitive folks had figured this out quickly and predominantly used the M16, but they hardly made a blip as a percentage of the total player base so it didn't really show up in the stats.