3 ms·

I'm not sure you need to. Not picking on you at all, just going on a tangent because I've been thinking recently about the good ideas that end up on the cuttin

by gen220 3y ago

I'm not sure you need to. Not picking on you at all, just going on a tangent because I've been thinking recently about the good ideas that end up on the cutting floor because their impact, while positive, is intractable or completely impractical to quantify. Some ideas (like this Netflix one!) are "obvious" winners, because they have a diffused, positive impact in many important dimensions – each of which is almost impossible to measure, but the integral of which is almost certainly greater than the idea's cost, probably by one or two OOM. If there's high conviction in an idea being a 10x idea, and attempting it costs quite little, it's better to do it now and maybe consider measuring it later. Who cares if it's 9x or 12x ROI, the point is it has a big margin of safety and large expected returns on capital. But in a "we can't greenlight this project unless we can directly attribute it to positive motion in our KPI" org, these flavors of idea are dead on arrival. The double-kicker is that the cost of trying to measure these ideas is often greater than the cost of the idea. For some ideas, especially when it's a rounding error on the company's annual budget, it should be OK if we don't invest much effort into quantifying the results. It's one of those rare set of ideas where it's anathema to the professional/investor culture of SV of the last decade, while simultaneously being how plucky Seed/Series A companies can punch above their weight.

taeric 3y ago
I would push back on this specific case, though. What makes this an obvious winner? Netflix was already clearly on the upward swing. At the time of this contest, I'm not even sure I remember who their competition was. Now, I think it is fully fair to say that this was very cheap for them to do. In which case, why not do it? But I think you would be hard pressed to give me a counter factual world that is believable where not doing this had a meaningful impact on Netflix's future.
- travisjungroth 3y ago
  Hiring. They got a lot of attention for this.
  - taeric 3y ago
    As I said in another reply, I don't remember them having trouble hiring in the first place? I can certainly mostly agree with this. I just also think them producing their own shows was likely more impactful, and had basically nothing to do with this. Don't get me wrong, plenty of moves in life are "win more." Such that I could see this one fitting that description. But a lot of "win more" choices are surprisingly low on the ROI side, when you control for everything else in that situation.
    - travisjungroth 3y ago
      Coca-Cola doesn't have trouble selling soda, but they still have commercials. Of course making their own shows was more impactful in a bunch of ways. But, the investment is also so much more. A season of a show is in the tens to hundreds of millions.
      - taeric 3y ago
        I think that is a bit of an apples to oranges comparison. Coke does brand management. They are not necessarily educating folks to their existence. But, the real difference is that Coke is, in fact, selling their own brand and products. For Netflix, they are largely selling access to other people's products. In a very real way, Netflix found an arbitrage opportunity for viewing movies. With how poorly the content owners calculated things, they got good deals at the beginning so that they could do well on their investment in the CDN infrastructure so that they could deliver a good experience to users with minimal unit costs. This is, in a large sense, why Netflix is having to raise prices now. The content owners are starting to demand more money for that access, driving up their unit costs. Combined with a draught of original content, expect that they will lose access to their own productions in time. You can already see this with the likes of Daredevil.
        travisjungroth 3y ago
        I'm talking about hiring. I believe this project had positive ROI for Netflix (where I used to work) because of how it helped bring attention to Netflix's ML problems. Maybe you disagree with this, totally fine. My comment about Coca-Cola was to draw the analogy that just because they don't have trouble hiring (which you mentioned) doesn't mean it can't be helped.
- sosborn 3y ago
  Their competition was the old outdated business model. As easy as it was to say that streaming was the future, they still had to convince millions of people to learn about and sign up for it.
  - taeric 3y ago
    This implies that a large number of subscribers are here because of this. I would hazard most subscribers don't know this even happened. Much less did it have any impact on a choice to sign up. That was driven much more directly by high speed internet access.
    - sosborn 3y ago
      I would argue that any PR helps drive subscriptions.
      - taeric 3y ago
        I wouldn't fully disagree, but I would point out that they had a ton of press at the time for virtue of who they were. Indeed, a large part of why we still talk of them doing this contest, is because of who they were and are. More directly, though, if you were to plot subscriptions to Netflix over time across an unlabeled chart, would you be able to see this contest?
        sosborn 3y ago
        I guarantee you they got much more than $1M worth of media placement for this stunt.
        taeric 3y ago
        If we are only referring to the cost of any media placement, I can't really disagree. My assertion is fully on ROI of this. Take my challenge. Plot their subscribers and revenue over time, hiding the year/month. See if you can spot any sort of inflection where this happened. It /could/ be that they would have seen declining numbers absent this. Such that this prevented a bad inflection on the chart. I don't know that I find that compelling.
- gen220 3y ago
  > I think you would be hard pressed to give me a counter factual world that is believable where not doing this had a meaningful impact on Netflix's future. Some scale for perspective: if this had a 10x ROI, they'd get $10mm for $1mm. At EOY 2009 (same year this competition was paid out), Netflix had a twelve-month-trailing Revenue of $1.671B. In 2006 they took in ~$1Bn. So it's 0.06% of annual revenue at payout time, 0.1% of annual revenue at announcement/green-light time. The idea had a large margin of safety: in the worst case, nobody pays attention and you've generated some IP which would have cost ~$1mm to generate anyways, and which would theoretically increase retention (a top 3 KPI) by X%. And it had big upside potential, in the form of additional marginal hires, press/exposure of the core product in the news cycle leading to marginal subscribers. Re hiring: I think it's totally plausible that it contributed to ~5% of 500 peoples' employment decisions since 2009. Re subscribers: it's plausible to think it contributed ~1% to 10k peoples' decisions to subscribe, most of whom have retained to-date. It's classic "Brand Advertising" – and although it's difficult to quantify, it works. Those numbers could be way higher, but it doesn't really matter if they were or weren't, the margin of safety, uncapped upside of the brand advertising angle, and low relative cost made it a bona fide good idea. Edit: FWIW, that we're still discussing this idea in 2024 is more evidence that it was a good idea. This whole thread is full of Netflix's target demographic. We're all talking about how well-compensated and smart Netflix engineers are... You would never be able to purchase that kind of long-burn, compounding, high-expected-value attention for $1mm from any advertising network, in 2006-2009 or now.
  - taeric 3y ago
    Right, I think this is a good analysis. But, I challenge that they actually got these returns. Specifically from this effort. It is popular among the type of crowd that posts here in HN. I would be surprised if any of my non-tech family even knows it happened. I think I would push on the ~5% of 500 people's employment decisions. With the sizes of the teams that competed, I'd not be too surprised that the teams involved were largely the extent of who was directly impacted by this contest. Everyone else was as happy as those of us that read any of the blogs/articles that they would publish about their infrastructure. Which is not nothing, but probably hard to spot the difference. (Ironically, I would argue that the raised morale of the people that publish the articles are probably more measurable? Which, I confess I don't think is a consistent position from me.) And I stress that I agree it was a safe thing to do. I am even glad that they did it. I just don't know that you could really convince me they would not have had the revenue that they had without doing this contest.
chmod775 3y ago
Yup. $1m is nothing. We live in a world where movies have $100m marketing budgets, game companies invest hundreds of millions into esports events and other material, and tiny companies are bought out "just in case" for similar amounts. $1m to generate some publicity and attract talent? Sure. They're probably paying headhunters similar amounts in comissions each year.
- taeric 3y ago
  Movies have very real returns that are trivial to see from box office results and merchandizing. Not to mention they have famously perverse incentives to spend as much of the profit from a movie as they can, due to how a lot of the royalties are structured. More to the point, though, this is a bit of a tangent/deflection from my question. How do you assert the return on this $1m? This line simply says that no return was needed for it to be spent. It shows nothing to say how you calculate what is made from it.

by gen220 3y ago

taeric 3y ago

I would push back on this specific case, though. What makes this an obvious winner? Netflix was already clearly on the upward swing. At the time of this contest, I'm not even sure I remember who their competition was. Now, I think it is fully fair to say that this was very cheap for them to do. In which case, why not do it? But I think you would be hard pressed to give me a counter factual world that is believable where not doing this had a meaningful impact on Netflix's future.

travisjungroth 3y ago
Hiring. They got a lot of attention for this.
- taeric 3y ago
  As I said in another reply, I don't remember them having trouble hiring in the first place? I can certainly mostly agree with this. I just also think them producing their own shows was likely more impactful, and had basically nothing to do with this. Don't get me wrong, plenty of moves in life are "win more." Such that I could see this one fitting that description. But a lot of "win more" choices are surprisingly low on the ROI side, when you control for everything else in that situation.
  - travisjungroth 3y ago
    Coca-Cola doesn't have trouble selling soda, but they still have commercials. Of course making their own shows was more impactful in a bunch of ways. But, the investment is also so much more. A season of a show is in the tens to hundreds of millions.
    - taeric 3y ago
      I think that is a bit of an apples to oranges comparison. Coke does brand management. They are not necessarily educating folks to their existence. But, the real difference is that Coke is, in fact, selling their own brand and products. For Netflix, they are largely selling access to other people's products. In a very real way, Netflix found an arbitrage opportunity for viewing movies. With how poorly the content owners calculated things, they got good deals at the beginning so that they could do well on their investment in the CDN infrastructure so that they could deliver a good experience to users with minimal unit costs. This is, in a large sense, why Netflix is having to raise prices now. The content owners are starting to demand more money for that access, driving up their unit costs. Combined with a draught of original content, expect that they will lose access to their own productions in time. You can already see this with the likes of Daredevil.
      - travisjungroth 3y ago
        I'm talking about hiring. I believe this project had positive ROI for Netflix (where I used to work) because of how it helped bring attention to Netflix's ML problems. Maybe you disagree with this, totally fine. My comment about Coca-Cola was to draw the analogy that just because they don't have trouble hiring (which you mentioned) doesn't mean it can't be helped.
sosborn 3y ago
Their competition was the old outdated business model. As easy as it was to say that streaming was the future, they still had to convince millions of people to learn about and sign up for it.
- taeric 3y ago
  This implies that a large number of subscribers are here because of this. I would hazard most subscribers don't know this even happened. Much less did it have any impact on a choice to sign up. That was driven much more directly by high speed internet access.
  - sosborn 3y ago
    I would argue that any PR helps drive subscriptions.
    - taeric 3y ago
      I wouldn't fully disagree, but I would point out that they had a ton of press at the time for virtue of who they were. Indeed, a large part of why we still talk of them doing this contest, is because of who they were and are. More directly, though, if you were to plot subscriptions to Netflix over time across an unlabeled chart, would you be able to see this contest?
      - sosborn 3y ago
        I guarantee you they got much more than $1M worth of media placement for this stunt.
        taeric 3y ago
        If we are only referring to the cost of any media placement, I can't really disagree. My assertion is fully on ROI of this. Take my challenge. Plot their subscribers and revenue over time, hiding the year/month. See if you can spot any sort of inflection where this happened. It /could/ be that they would have seen declining numbers absent this. Such that this prevented a bad inflection on the chart. I don't know that I find that compelling.
gen220 3y ago
> I think you would be hard pressed to give me a counter factual world that is believable where not doing this had a meaningful impact on Netflix's future. Some scale for perspective: if this had a 10x ROI, they'd get $10mm for $1mm. At EOY 2009 (same year this competition was paid out), Netflix had a twelve-month-trailing Revenue of $1.671B. In 2006 they took in ~$1Bn. So it's 0.06% of annual revenue at payout time, 0.1% of annual revenue at announcement/green-light time. The idea had a large margin of safety: in the worst case, nobody pays attention and you've generated some IP which would have cost ~$1mm to generate anyways, and which would theoretically increase retention (a top 3 KPI) by X%. And it had big upside potential, in the form of additional marginal hires, press/exposure of the core product in the news cycle leading to marginal subscribers. Re hiring: I think it's totally plausible that it contributed to ~5% of 500 peoples' employment decisions since 2009. Re subscribers: it's plausible to think it contributed ~1% to 10k peoples' decisions to subscribe, most of whom have retained to-date. It's classic "Brand Advertising" – and although it's difficult to quantify, it works. Those numbers could be way higher, but it doesn't really matter if they were or weren't, the margin of safety, uncapped upside of the brand advertising angle, and low relative cost made it a bona fide good idea. Edit: FWIW, that we're still discussing this idea in 2024 is more evidence that it was a good idea. This whole thread is full of Netflix's target demographic. We're all talking about how well-compensated and smart Netflix engineers are... You would never be able to purchase that kind of long-burn, compounding, high-expected-value attention for $1mm from any advertising network, in 2006-2009 or now.
- taeric 3y ago
  Right, I think this is a good analysis. But, I challenge that they actually got these returns. Specifically from this effort. It is popular among the type of crowd that posts here in HN. I would be surprised if any of my non-tech family even knows it happened. I think I would push on the ~5% of 500 people's employment decisions. With the sizes of the teams that competed, I'd not be too surprised that the teams involved were largely the extent of who was directly impacted by this contest. Everyone else was as happy as those of us that read any of the blogs/articles that they would publish about their infrastructure. Which is not nothing, but probably hard to spot the difference. (Ironically, I would argue that the raised morale of the people that publish the articles are probably more measurable? Which, I confess I don't think is a consistent position from me.) And I stress that I agree it was a safe thing to do. I am even glad that they did it. I just don't know that you could really convince me they would not have had the revenue that they had without doing this contest.

chmod775 3y ago

Yup. $1m is nothing. We live in a world where movies have $100m marketing budgets, game companies invest hundreds of millions into esports events and other material, and tiny companies are bought out "just in case" for similar amounts. $1m to generate some publicity and attract talent? Sure. They're probably paying headhunters similar amounts in comissions each year.

taeric 3y ago
Movies have very real returns that are trivial to see from box office results and merchandizing. Not to mention they have famously perverse incentives to spend as much of the profit from a movie as they can, due to how a lot of the royalties are structured. More to the point, though, this is a bit of a tangent/deflection from my question. How do you assert the return on this $1m? This line simply says that no return was needed for it to be spent. It shows nothing to say how you calculate what is made from it.