6 ms·

They got far more than $1M in marketing from it. It would have been worth it for twice as much.

by teruakohatu 3y ago

taeric 3y ago
I'm curious how you quantify that?
- kamikaz1k 3y ago
  at least for the internal goal of hiring top ML talent, you can probably just calculate how long it took to hiring the good ones, and then how much money that team made.
  - taeric 3y ago
    Are they objectively better at ML for movie recommendations than they were beforehand? I confess I've never been too impacted by their recommendations. Outside of new releases, but that doesn't really use any ML.
- kamkazemoose 3y ago
  You can think, how much would their marketing team have to spend to get the same results that the algorithm contract gave. I'm not a marketing expert, but I'm sure they have metrics like consumer sentiment, name recognition, number of users visiting the site, google search trends, etc. There could also be benefits in recruitment, and that can be estimated based on how much you'd have to pay an external recruiter to bring in candidates the applied, or other things like that. It was in the news a lot, and was discussed on a lot of tech sites. Plus it gets people talking about their recomendation algorithm, and makes people thing Netflix subscription is more valuable becasue it recommends good shows. It wouldn't be cheap to get the amount of media that they got through more traditional marketing.
  - taeric 3y ago
    Right, but that is a lot of what I'm asking. What makes you think those are all better due to this contest? If I recall, a lot of why it was making the news was because Netflix was already popular in the industry. They certainly weren't that new of a name. The main one I don't think I would have doubts about is the recruitment. But, I don't recall them being a place that needed recruitment help, even at that time. To be clear, I found the thing fun to consider. I certainly am not upset that they did it. I do harbor a gut feeling that its ROI is greatly overstated.
- gen220 3y ago
  I'm not sure you need to. Not picking on you at all, just going on a tangent because I've been thinking recently about the good ideas that end up on the cutting floor because their impact, while positive, is intractable or completely impractical to quantify. Some ideas (like this Netflix one!) are "obvious" winners, because they have a diffused, positive impact in many important dimensions – each of which is almost impossible to measure, but the integral of which is almost certainly greater than the idea's cost, probably by one or two OOM. If there's high conviction in an idea being a 10x idea, and attempting it costs quite little, it's better to do it now and maybe consider measuring it later. Who cares if it's 9x or 12x ROI, the point is it has a big margin of safety and large expected returns on capital. But in a "we can't greenlight this project unless we can directly attribute it to positive motion in our KPI" org, these flavors of idea are dead on arrival. The double-kicker is that the cost of trying to measure these ideas is often greater than the cost of the idea. For some ideas, especially when it's a rounding error on the company's annual budget, it should be OK if we don't invest much effort into quantifying the results. It's one of those rare set of ideas where it's anathema to the professional/investor culture of SV of the last decade, while simultaneously being how plucky Seed/Series A companies can punch above their weight.
  - taeric 3y ago
    I would push back on this specific case, though. What makes this an obvious winner? Netflix was already clearly on the upward swing. At the time of this contest, I'm not even sure I remember who their competition was. Now, I think it is fully fair to say that this was very cheap for them to do. In which case, why not do it? But I think you would be hard pressed to give me a counter factual world that is believable where not doing this had a meaningful impact on Netflix's future.
    - travisjungroth 3y ago
      Hiring. They got a lot of attention for this.
      - taeric 3y ago

They got far more than $1M in marketing from it. It would have been worth it for twice as much.

taeric 3y ago

I'm curious how you quantify that?

kamikaz1k 3y ago
at least for the internal goal of hiring top ML talent, you can probably just calculate how long it took to hiring the good ones, and then how much money that team made.
- taeric 3y ago
  Are they objectively better at ML for movie recommendations than they were beforehand? I confess I've never been too impacted by their recommendations. Outside of new releases, but that doesn't really use any ML.
kamkazemoose 3y ago
You can think, how much would their marketing team have to spend to get the same results that the algorithm contract gave. I'm not a marketing expert, but I'm sure they have metrics like consumer sentiment, name recognition, number of users visiting the site, google search trends, etc. There could also be benefits in recruitment, and that can be estimated based on how much you'd have to pay an external recruiter to bring in candidates the applied, or other things like that. It was in the news a lot, and was discussed on a lot of tech sites. Plus it gets people talking about their recomendation algorithm, and makes people thing Netflix subscription is more valuable becasue it recommends good shows. It wouldn't be cheap to get the amount of media that they got through more traditional marketing.
- taeric 3y ago
  Right, but that is a lot of what I'm asking. What makes you think those are all better due to this contest? If I recall, a lot of why it was making the news was because Netflix was already popular in the industry. They certainly weren't that new of a name. The main one I don't think I would have doubts about is the recruitment. But, I don't recall them being a place that needed recruitment help, even at that time. To be clear, I found the thing fun to consider. I certainly am not upset that they did it. I do harbor a gut feeling that its ROI is greatly overstated.
gen220 3y ago
I'm not sure you need to. Not picking on you at all, just going on a tangent because I've been thinking recently about the good ideas that end up on the cutting floor because their impact, while positive, is intractable or completely impractical to quantify. Some ideas (like this Netflix one!) are "obvious" winners, because they have a diffused, positive impact in many important dimensions – each of which is almost impossible to measure, but the integral of which is almost certainly greater than the idea's cost, probably by one or two OOM. If there's high conviction in an idea being a 10x idea, and attempting it costs quite little, it's better to do it now and maybe consider measuring it later. Who cares if it's 9x or 12x ROI, the point is it has a big margin of safety and large expected returns on capital. But in a "we can't greenlight this project unless we can directly attribute it to positive motion in our KPI" org, these flavors of idea are dead on arrival. The double-kicker is that the cost of trying to measure these ideas is often greater than the cost of the idea. For some ideas, especially when it's a rounding error on the company's annual budget, it should be OK if we don't invest much effort into quantifying the results. It's one of those rare set of ideas where it's anathema to the professional/investor culture of SV of the last decade, while simultaneously being how plucky Seed/Series A companies can punch above their weight.
- taeric 3y ago
  I would push back on this specific case, though. What makes this an obvious winner? Netflix was already clearly on the upward swing. At the time of this contest, I'm not even sure I remember who their competition was. Now, I think it is fully fair to say that this was very cheap for them to do. In which case, why not do it? But I think you would be hard pressed to give me a counter factual world that is believable where not doing this had a meaningful impact on Netflix's future.
  - travisjungroth 3y ago
    Hiring. They got a lot of attention for this.
    - taeric 3y ago