13 ms·

> When I worked there the product team at Netflix had two KPIs all new features were tested against: hours watched and retention. So, I totally understand rete

by saurik 7y ago

> When I worked there the product team at Netflix had two KPIs all new features were tested against: hours watched and retention. So, I totally understand retention: that one makes sense to me, as that's revenue for Netflix; if I'm more likely to renew my account, then Netflix makes more money. But... I honestly don't understand why Netflix is optimizing for hours watched, and in fact I had assumed it would be the opposite (leading to Netflix actually making it kind of annoyingly difficult to watch the full credits on movies, for example). > But...the product team at Doritos does the same thing. This makes sense to me: the more Doritos I eat, the more money Doritos makes. This isn't true of Netflix: every time I watch something, it costs Netflix money. I would, naively, expect that what Netflix would want to optimize for is some weird balance of retention (at high priority) and least hours watched (at a much lower priority). > So yes, Netflix wants you to spend more hours watching Netflix and the product team is scientifically engineering the product to make it more addictive. I also appreciate that the more hours I watch of Netflix, the more I might be "addicted" to doing so... but the reason I'd expect Netflix to care about that is because it means higher retention, which is already being directly measured. Any other aspects of my addiction don't seem to benefit Netflix: it costs Netflix money every time I stream something from them... ...and, it would seem to me crucially, it means that Netflix is "on the hook" for more and more content. If, on a relatively short-term basis (such as during an experiment), my measured retention is the same, but my hours watched is higher, then it means I'm burning through content I like to watch at a faster rate, and so Netflix needs to spend more money to rotate their catalog and create more originals. It just seems so weird that Netflix would be optimizing for "hours watched", and I'd love to learn from someone more on the inside who was involved in that why they would do that. I understand why YouTube and Facebook optimize for "engagement" (even though I also think it is abhorrent) as 1) their revenue is based on advertisements and 2) they spread using network effects; neither of these are true for Netflix.

rsweeney21 7y ago
Hours watched is a leading indicator of retention. The more people watch, the more likely they are to remain a customer.
- mgreg 7y ago
  This makes sense but I'm guessing there are diminishing returns that may actually turn negative at some point. I.e. How much more likely is someone who watches 21 hours vs 20 hours to be retained? Then how much more likely is someone who watches 41 hours vs 40 hours to be retained? I'm guessing that on average there is a much smaller difference between 41 and 40 than between 21 and 20. I'm also guessing it may turn negative as some people have shown they will subscribe for a month, binge on their favorite/new shows, and then cancel. Perhaps renewing in a few months time to do it again. Would also be curious about other metrics such as search time : watch time. I'd imaging people spending more time searching and less time watching get frustrated and are more likely to cancel. With autoplay it eliminates searching for some. I'm also curious why there isn't a setting to enable/disable autoplay.
gist 7y ago
There are probably so many angles to this but I will give one. Netflix is more likely to be used and loved if it gives someone something similar to the thing they just finished watching (like the next episode) then if they lose them to something they might not like by giving them the opportunity to find other content. Also even if the content is not the next episode I feel I would be more likely to go with a suggested (new) video or content rather than something that takes me time to find (and therefore in theory happier with the service). For example there are so many choices on Netflix that I find I often might decide to not even take a look because of what I think the content is. [1] [1] An example of this is the most excellent (now in S3) 'Money Heist'. The Dali masks were an instant turnoff for me that for whatever reason (rightly or wrongly) made me think it was some kind of fantasy rather than a crime drama (not saying my thoughts were right but that is what I thought).
refurb 7y ago
I would assume that Netflix has data that shows hours watched is directly correlated with retention. Think of hours watched as a leading indicator for retention. You don't want to only track retention because by the time it drops, it's too late.
jressey 7y ago
Why hours watched? Because Netflix wants as much of your time as you're willing to give. The more ingrained they are in your life, the less elastic your demand. We've seen them raise prices and remove just about all of the worthwhile content, and they have more subscribers than ever. Stay tuned for more of this.
- scarface74 7y ago
  They didn’t “remove” content. It was taken from them by the owners.
  - jressey 7y ago
    That is an incredibly pedantic statement. They chose not to pay the cost that the owners charged them. Netflix is an equal actor.
    - scarface74 7y ago
      There was no “cost” they could have paid Disney (and now Fox), Warner, or NBC Universal to keep their content when they wanted it back to start their own streaming service with exclusive content.
- giggles_giggles 7y ago
  I wonder how many of those extra hours watched are because the stream just didn't stop after the person fell asleep. I can't tell you the number of times I've fallen asleep while Netflix played (or walked away) and it went right into some other show I didn't care to watch (and never actually did). If hours streamed is your metric for engagement.. of course having the stream continue on its own would make that number rise. It doesn't actually say anything at all about real engagement.
  - asark 7y ago
    Ours has definitely ended up playing to an empty room for a good long while, a few times.
dragonwriter 7y ago
> So, I totally understand retention: that one makes sense to me, as that's revenue for Netflix; if I'm more likely to renew my account, then Netflix makes more money. But... I honestly don't understand why Netflix is optimizing for hours watched, and in fact I had assumed it would be the opposite Hours watched differences may predict retention differences farther in the future better than short-term measurements of retention, which may measure reaction to changes more than it measures the steady-state effect of the change. If you want to act on A/B tests faster than directly measuring long-term retention would allow, having a proxy measure of long-term retention that mitigates the risk of optimizing for transitional rather than steady-state retention has value.
- satya71 7y ago
  At least for my case, these optimizations directly led me to stop watching Netflix.
  - asark 7y ago
    Unstoppable play-while-browsing plus the price hikes has me poised to drop it as soon as I finish two shows I'm near the end of. Probably in the next month. Either problem alone wouldn't get me over the "bother to cancel" speedbump, but I dread opening Netflix unless I'm browsing straight to something I already know is on there, these days. If it were still super cheap, whatever, but I can get a couple cheap services for its price now. Out it goes. I'll just start picking it up a month a year or so to binge whatever I've missed. Playing-on-the-menu and autoplay-after-episode would keep me from recommending it to older relatives. The last thing they need is their devices doing shit without their telling them to, especially with audio and video involved for extra confusing-the-hell out of them, doubly so when they're trying to direct it to do something (browsing the media selection). Probably direct them to Hulu, I guess, if they asked.
    - wolco 7y ago
      You could just turn it off in the settings.
neura 7y ago
Just taking a guess that the more you watch, the more value you feel that you're getting for your monthly subscription fee.
logfromblammo 7y ago
I'm not on the inside, but I have a possible explanation. Netflix has competitors. If we assume that leisure-time hours are fixed in the short term, the more hours you spend watching Netflix, the less you have available for Hulu, Amazon Prime, CBS, HBO, MLB, or Disney streaming services. Thus, if all those have comparable prices, more hours watching Netflix translates to greater subscriber-perceived entertainment value for money spent on watching Netflix than on the other services. Netflix does not have to be number one, but it does have to be high enough on people's ranked lists to be above their individual fragmentation threshold. If a customer's hours watched is low and dropping, Netflix is in danger of losing them as a customer, especially when Disney comes online later. Perhaps if CBS All Access tracked that number, they would give up, and contract with Netflix or Hulu to stream their content. It seems to me that it is the service that would be least likely to get subscribers that are already at or near their fragmentation tolerance.
- pmontra 7y ago
  > Netflix has competitors. If we assume that leisure-time hours are fixed in the short term, the more hours you spend watching Netflix, the less you have available for Hulu, Amazon Prime, CBS, HBO, MLB, or Disney streaming services. Not only those competitors but any other activity, from reading HN to working to going out with friends, etc.
  - logfromblammo 7y ago
    We could lump those together as screen-time versus non-screen-time. I hypothesize that converting an hour from one subcategory of screen-time to another is easier than converting it from non-screen-time to screen-time. I.e. it's easier to convince someone to watch Netflix instead of watching Hulu, than it is to convince them to watch Netflix or Hulu instead of hosting/attending a summer outdoor grilling party. Netflix would show greater gains relative to competitors by converting more hours spent watching their streams. Any attempt to bring in more hours from the outside would also tend to help competitors, since they would then have an easier time poaching those hours from Netflix later. That might be beneficial in a nascent industry, but I think we're past that now. Nobody is going to be launching an explicit "stay in, and watch more TV" campaign any time soon, but apparently they all will whisper "now that you're already watching TV, why not just keep doing it?" via autoplay mechanisms. And time spent working is definitely not leisure time, by definition.
ckarmann 7y ago
While you watch Netflix, you don't watch a competitor. In a way, this autoplay feature removes a friction for the customer that doesn't have to decide what to watch next. This removes the "danger" that the customer think about that cool show that is not provided by Netflix. At the end of the month, the customer may think about how much they use a TV provider and not the other, and decide to keep Netflix and ditch any other subscription they may have. It's true that it burns through Netflix content but they made 1,500 hours of original content in 2018, content that they have to create anyway for other reasons, so a lot of your free time is already covered.
- clinta 7y ago
  But isn't that just a proxy for retention then? Why optimize for hours watched which correlates with retention when you can just measure and optimize for retention directly?
  - ckarmann 7y ago
    Ah, I understand what you mean. Well, if you know variable X is a proxy for Y, but Y is a lot longer to measure than X, then I guess it makes sense to optimize X knowing that it will increase Y (if you are sure X will bring Y).
  - rkeene2 7y ago
    It's not exactly a proxy, but a leading/lagging indicator, which has its own value.
wolco 7y ago
The cost produce something is sunk already. If everyone watches (birdbox) and talks about it can get more people to subscribe. The more you watch the more likely you are satified. They won't make new content for you because you watch more. They know you will watch a variety of shows and will watch some less known content. Plus view numbers are important for awards and privately they share with producers to encourage them to sell new content/partner up. No matter how much you pay artists still want the world to see what they created.

by saurik 7y ago

rsweeney21 7y ago

Hours watched is a leading indicator of retention. The more people watch, the more likely they are to remain a customer.

mgreg 7y ago
This makes sense but I'm guessing there are diminishing returns that may actually turn negative at some point. I.e. How much more likely is someone who watches 21 hours vs 20 hours to be retained? Then how much more likely is someone who watches 41 hours vs 40 hours to be retained? I'm guessing that on average there is a much smaller difference between 41 and 40 than between 21 and 20. I'm also guessing it may turn negative as some people have shown they will subscribe for a month, binge on their favorite/new shows, and then cancel. Perhaps renewing in a few months time to do it again. Would also be curious about other metrics such as search time : watch time. I'd imaging people spending more time searching and less time watching get frustrated and are more likely to cancel. With autoplay it eliminates searching for some. I'm also curious why there isn't a setting to enable/disable autoplay.

gist 7y ago

There are probably so many angles to this but I will give one. Netflix is more likely to be used and loved if it gives someone something similar to the thing they just finished watching (like the next episode) then if they lose them to something they might not like by giving them the opportunity to find other content. Also even if the content is not the next episode I feel I would be more likely to go with a suggested (new) video or content rather than something that takes me time to find (and therefore in theory happier with the service). For example there are so many choices on Netflix that I find I often might decide to not even take a look because of what I think the content is. [1] [1] An example of this is the most excellent (now in S3) 'Money Heist'. The Dali masks were an instant turnoff for me that for whatever reason (rightly or wrongly) made me think it was some kind of fantasy rather than a crime drama (not saying my thoughts were right but that is what I thought).

refurb 7y ago

I would assume that Netflix has data that shows hours watched is directly correlated with retention. Think of hours watched as a leading indicator for retention. You don't want to only track retention because by the time it drops, it's too late.

jressey 7y ago

Why hours watched? Because Netflix wants as much of your time as you're willing to give. The more ingrained they are in your life, the less elastic your demand. We've seen them raise prices and remove just about all of the worthwhile content, and they have more subscribers than ever. Stay tuned for more of this.

scarface74 7y ago
They didn’t “remove” content. It was taken from them by the owners.
- jressey 7y ago
  That is an incredibly pedantic statement. They chose not to pay the cost that the owners charged them. Netflix is an equal actor.
  - scarface74 7y ago
    There was no “cost” they could have paid Disney (and now Fox), Warner, or NBC Universal to keep their content when they wanted it back to start their own streaming service with exclusive content.
giggles_giggles 7y ago
I wonder how many of those extra hours watched are because the stream just didn't stop after the person fell asleep. I can't tell you the number of times I've fallen asleep while Netflix played (or walked away) and it went right into some other show I didn't care to watch (and never actually did). If hours streamed is your metric for engagement.. of course having the stream continue on its own would make that number rise. It doesn't actually say anything at all about real engagement.
- asark 7y ago
  Ours has definitely ended up playing to an empty room for a good long while, a few times.

dragonwriter 7y ago

> So, I totally understand retention: that one makes sense to me, as that's revenue for Netflix; if I'm more likely to renew my account, then Netflix makes more money. But... I honestly don't understand why Netflix is optimizing for hours watched, and in fact I had assumed it would be the opposite Hours watched differences may predict retention differences farther in the future better than short-term measurements of retention, which may measure reaction to changes more than it measures the steady-state effect of the change. If you want to act on A/B tests faster than directly measuring long-term retention would allow, having a proxy measure of long-term retention that mitigates the risk of optimizing for transitional rather than steady-state retention has value.

satya71 7y ago
At least for my case, these optimizations directly led me to stop watching Netflix.
- asark 7y ago
  Unstoppable play-while-browsing plus the price hikes has me poised to drop it as soon as I finish two shows I'm near the end of. Probably in the next month. Either problem alone wouldn't get me over the "bother to cancel" speedbump, but I dread opening Netflix unless I'm browsing straight to something I already know is on there, these days. If it were still super cheap, whatever, but I can get a couple cheap services for its price now. Out it goes. I'll just start picking it up a month a year or so to binge whatever I've missed. Playing-on-the-menu and autoplay-after-episode would keep me from recommending it to older relatives. The last thing they need is their devices doing shit without their telling them to, especially with audio and video involved for extra confusing-the-hell out of them, doubly so when they're trying to direct it to do something (browsing the media selection). Probably direct them to Hulu, I guess, if they asked.
  - wolco 7y ago
    You could just turn it off in the settings.

neura 7y ago

Just taking a guess that the more you watch, the more value you feel that you're getting for your monthly subscription fee.

logfromblammo 7y ago

I'm not on the inside, but I have a possible explanation. Netflix has competitors. If we assume that leisure-time hours are fixed in the short term, the more hours you spend watching Netflix, the less you have available for Hulu, Amazon Prime, CBS, HBO, MLB, or Disney streaming services. Thus, if all those have comparable prices, more hours watching Netflix translates to greater subscriber-perceived entertainment value for money spent on watching Netflix than on the other services. Netflix does not have to be number one, but it does have to be high enough on people's ranked lists to be above their individual fragmentation threshold. If a customer's hours watched is low and dropping, Netflix is in danger of losing them as a customer, especially when Disney comes online later. Perhaps if CBS All Access tracked that number, they would give up, and contract with Netflix or Hulu to stream their content. It seems to me that it is the service that would be least likely to get subscribers that are already at or near their fragmentation tolerance.

pmontra 7y ago
> Netflix has competitors. If we assume that leisure-time hours are fixed in the short term, the more hours you spend watching Netflix, the less you have available for Hulu, Amazon Prime, CBS, HBO, MLB, or Disney streaming services. Not only those competitors but any other activity, from reading HN to working to going out with friends, etc.
- logfromblammo 7y ago
  We could lump those together as screen-time versus non-screen-time. I hypothesize that converting an hour from one subcategory of screen-time to another is easier than converting it from non-screen-time to screen-time. I.e. it's easier to convince someone to watch Netflix instead of watching Hulu, than it is to convince them to watch Netflix or Hulu instead of hosting/attending a summer outdoor grilling party. Netflix would show greater gains relative to competitors by converting more hours spent watching their streams. Any attempt to bring in more hours from the outside would also tend to help competitors, since they would then have an easier time poaching those hours from Netflix later. That might be beneficial in a nascent industry, but I think we're past that now. Nobody is going to be launching an explicit "stay in, and watch more TV" campaign any time soon, but apparently they all will whisper "now that you're already watching TV, why not just keep doing it?" via autoplay mechanisms. And time spent working is definitely not leisure time, by definition.

ckarmann 7y ago

While you watch Netflix, you don't watch a competitor. In a way, this autoplay feature removes a friction for the customer that doesn't have to decide what to watch next. This removes the "danger" that the customer think about that cool show that is not provided by Netflix. At the end of the month, the customer may think about how much they use a TV provider and not the other, and decide to keep Netflix and ditch any other subscription they may have. It's true that it burns through Netflix content but they made 1,500 hours of original content in 2018, content that they have to create anyway for other reasons, so a lot of your free time is already covered.

clinta 7y ago
But isn't that just a proxy for retention then? Why optimize for hours watched which correlates with retention when you can just measure and optimize for retention directly?
- ckarmann 7y ago
  Ah, I understand what you mean. Well, if you know variable X is a proxy for Y, but Y is a lot longer to measure than X, then I guess it makes sense to optimize X knowing that it will increase Y (if you are sure X will bring Y).
- rkeene2 7y ago
  It's not exactly a proxy, but a leading/lagging indicator, which has its own value.

wolco 7y ago

The cost produce something is sunk already. If everyone watches (birdbox) and talks about it can get more people to subscribe. The more you watch the more likely you are satified. They won't make new content for you because you watch more. They know you will watch a variety of shows and will watch some less known content. Plus view numbers are important for awards and privately they share with producers to encourage them to sell new content/partner up. No matter how much you pay artists still want the world to see what they created.