14 ms·
How useful was the Netflix Prize challenge for Netflix?
- alberth 5y agoWhat’s odd is that the person responding in Quora who’s speaking as if he has authority on this matter started at Netflix in 2011, 2 years after the contest ended. The contest started in 2006 and was awarded in 2009. https://en.m.wikipedia.org/wiki/Netflix_Prize https://en.m.wikipedia.org/wiki/Netflix_Prize
- nextaccountic 5y agoHe worked 10 years there, and was former research director, so.. maybe he knows a bit or two about this prize? He did work on productionize the 2007 winner, after all.
- alberth 5y agoHis LinkedIn profile says otherwise. 2011-2017 was his time at Netflix. https://www.linkedin.com/in/xamatriain https://www.linkedin.com/in/xamatriain
- dimitrios1 5y agoEveryone knows you can't lie on your LinkedIn profile, too.
- alberth 5y agoWhy would the person lie about working at another company (Telefonica) from 2007-2011.
- dimitrios1 5y agoIt was a joke.
- tomnipotent 5y agoThe 2007 winner was still in production in 2011. I'm not sure that qualifies as "productionizing" as that word hints of 1st pass work, but he certainly has production experience with it.
- mmahemoff 5y ago"Research/Engineering Director at Netflix (2011-2014)" would probably know a thing or two about how useful the contest was. (He also might have worked there before that time in a different role.)
- godelski 5y agoNot far in he writes > Ten years ago, when I was leading Algorithms at Netflix So... I think he does know a thing or two about specifically this issue.
- tyingq 5y agoHe was the head of the area (Research/Engineering) that managed the recommendation engine. Part of that would be understanding the history.
- Ozzie_osman 5y agoHe's also a reputable source on Machine Learning in general and pretty well known in the community (I worked under him).
- squarecog 5y agoWhy is this odd? If he started two years later and there was not a trace of the Prize work at the company, that would be an indicator that the competition was not important. If he started and could still see knock-on effects from the competition, that's an indicator that it was important. Plus, he didn't just start at Netflix. He "took over the small team that was working and maintaining the rating prediction algorithm that included the first year Progress Prize solution." Yeah, that sounds like he has some authority on the matter.
- alberth 5y agoCould it not be something more simple like, Netflix didn’t originally have profiles. So my child watching kid shows and me watching action shows were all feeding into the same recommendation system resulting in subpar results. That and they originally had a 5 star rating system which they then dropped. Its very possible Netflix realized they needed to course correct the UX and as a result the winners algorithm was solving for a problem that no longer applied because it was using assumptions (rating system & no existence of different profiles) that were no longer relevant.
- trutannus 5y agoQuora's culture is odd in general. Writing "authoritative" (even they're just LARPing as an expert) is basically why people go there in the first place. It's not uncommon to read something egregiously inaccurate written as indisputable fact. Good examples can be found in anything to do with history. On a funny note, Jordan Peterson used to be a well known Quora answer writer prior to his current career as an internet celebrity. Source: https://www.quora.com/profile/Jordan-B-Peterson https://www.quora.com/profile/Jordan-B-Peterson
- mikeyouse 5y agoHa - nice find. All sorts of the typical sad self-promotion you find on Quora.
- getlawgdon 5y agoJordan Peterson is a total fraud.
- deleted 5y ago[deleted]
- dmitryminkovsky 5y agoI've always wondered about this. I first read about this challenge in "Programming Collective Intelligence,"[0] the O'Reilly book from 2007 that drove me to become a professional programmer. It starts like this: > Netflix is an online DVD rental company that lets people choose movies to be sent to their homes, and makes recommendations based on the movies that customers have previously rented. It was a different and exciting time back then! I never finished that book but hope to some day... :) [0] https://www.oreilly.com/library/view/programming-collective-intelligence/9780596529321/ch01.html https://www.oreilly.com/library/view/programming-collective-...
- Avalaxy 5y agoIt's a good book! I read it before we started calling it "artificial intelligence" or "machine learning". It was just data mining back then. I think the book and the algorithms in it are still very relevant today.
- dmitryminkovsky 5y agoI’ve wondered whether that’s the case :)
- plasma 5y agoI’d enjoy seeing a more creative approach to recommendations for something as big as Netflix. A few suggestions: 1. Review channels by genre 2. Trailer TV - let me leave a “comedy” trailer channel running that shows the trailer and movie rating and details at the bottom, let me easily skip to the next trailer (or let it play out)
- smnrchrds 5y agoAnything can beat their current algorithm of "push people towards our originals, preferences be damned". EDIT: I should add in terms of customer satisfaction, not revenue. I am sure forcing their originals down people's throats is great for their revenues.
- echelon 5y agoLicensed content costs them more. Users that consume licensed content are worth less if that's all they consume.
- rurp 5y agoWell yeah, that's what is in Netflix's interest, but there's a reason many companies succeeded by focus on improving the user's experience. I guess Netflix has a a different approach.
- prepend 5y agoI think Netflix doesn’t have enough content to do this. They have their own originals and just a thin layer of other stuff. So I think any truly personalized content channel would get exhausted quickly. What I’d like to have is just a channel of curated or semi curated movie content that I can leave running or forward through to watch. I recently stayed in a hotel with 6 channels of hbo. It’s kind of refreshing to have “hbo comedy” with random stuff like Beverly Hills cop and billy Madison on at 2pm in the afternoon. Netflix doesn’t have enough content to do this, so they keep recommending the same crap originals to me over and over, knowing that I don’t watch them.
- nodesocket 5y agoThat’s some great guerrilla marketing (or a business hack). At best, they actually get a better (production) algorithm. At worse, they get thought leadership among the ML/AI community and insane publicity still paying dividends today. Reed Hastings is a business savant.
- deleted 5y ago[deleted]
- DantesKite 5y agoIt was useful, just not in the way it was intended.
- mmahemoff 5y agoTLDR: Very useful. It's true they didn't use the winning algorithm, because it was only a small improvement and they were already moving to streaming, where predicting consumption matters more than predicting ratings. However, they did put an earlier submission into production. Moreover, the competition was a powerful recruiting tool.
- BB212 5y agoI think the discussion misses the most important part: The goal of the Netflix prize wasn't to come up with the best algorithm - it was to make the Netflix brand exciting and legitimate to engineers. At the time, Netflix wasn't super high-tech and I'm sure it was hard for them to get the top talent they needed. It seems silly in retrospect now, but I'm certain the reason this was approved was because they wanted the free advertising this would provide within graduate classes and academia in general.
- Supermancho 5y ago> At the time, Netflix wasn't super high-tech I will admit that it was interesting to see what algorithms were poised to be cutting edge in media recommendation. The result was rather disappointing to me. Netflix STILL isn't that exciting from anything but a compensation standpoint. The problems at netflix are about programming, while the technical challenges are droll at best.
- setr 5y agoIMO the recommendations are no good because they fundamentally take the wrong approach — rather than ask the user what they like, they try to guess what you like based on usage (which really doesn't correlate well — I watch a lot of garbage because I can’t find things I like, and I don’t have anything better to do.) And they don’t ask because users don’t provide useful answers. But users don’t provide useful answers, because rating things doesn’t do anyone any good. I’m of the belief that if you can make ratings useful (catalogue all movies, including not on Netflix; give useful ways to view/update your lists; have direct relationships to recommendations), you would have dramatically better recommendations for dramatically less effort/complexity. I don’t think you’ll ever get to “good” recommendations based on usage. The data is fundamentally garbage. Of course, the other side is that Netflix isn’t interested in recommending things I like; their goal is to recommend things I’ll put up with. They just need 1 show worth watching and subscribing for every now and then, and N shows to keep me mildly amused to stop me from dropping it between good ones
- rrrrrrrrrrrryan 5y ago
- spicyramen 5y agoBack in 2010 my wife was doing her master's and Netflix recruited MBA students to help them with their business model for streaming. The task was to understand if DVD made sense and if streaming was too risky. It panned out pretty well and the rest is history
- IncRnd 5y agoI was a solver on a different team for the netflix challenge. Our team didn't win the grand prize. I would have expected that a blog post would discuss how this was structured. Netflix contracted with innocentive.com, which is a website for solvers, and contracting to that website expanded Netflix's reach to a greater available pool of solvers. As far as I recall, all the allowed solvers for the netflix challenge _had_ to go through innocentive. I'm not sure if they would have been able to get the same level of improvement if they had not contracted with a set of potential solver teams like that. The original challenge listing for Netflix is no longer listed at innocentive.com, but an industrious person may be able to find it on archive.org or somewhere similar.
- tomnipotent 5y agoI don't remember InnoCentive being involved with the Netflix Prize in any official capacity, but I do remember the PR storm they launched after the prize was awarded to make sure they were mentioned in news articles about it.
- IncRnd 5y agoI signed up for the challenge, through innocentive, in July of 2008. It's entirely possible that as a new solver at the time I fell for innocentive's PR. The netflix challenge was actually the first I ever signed-up to work.
- rocqua 5y agoNote, not just the original listing has been taking down, but also the data used in the challenge. I believe because a few people in that set actually got de-anonimized. But certainly because of privacy concerns. I think the data is still kicking around somewhere in torrent land. I also still have my own copy somewhere, I think.
- Delk 5y agoIIRC it had something to do with researchers being able to statistically identify Netflix users by using IMDB for comparison. A Wired article from 2010 [1] suggests it lead to some legal liability risks for Netflix. [1] https://www.wired.com/2010/03/netflix-cancels-contest/ https://www.wired.com/2010/03/netflix-cancels-contest/
- ausumm 5y agohttps://ausum.io/s/lnz2iFXJAJ11QJYUYOQBoWYHW8ThYENN5o8Ms7AjqQQ https://ausum.io/s/lnz2iFXJAJ11QJYUYOQBoWYHW8ThYENN5o8Ms7Ajq... - summarized this article into an audio clip: 1 minute and 40 seconds. Listen or download the audio file.
- ausumm 5y agoAnd the other article linked in the thread (you could download and listen to this post whenever you'd like.) Chapter 1. Introduction to Collective Intelligence - OReilly https://ausum.io/s/lPSp1k9-y11o1rjXI7E8EvBx4tOe5bxVjKzxtbJ2KRk https://ausum.io/s/lPSp1k9-y11o1rjXI7E8EvBx4tOe5bxVjKzxtbJ2K...
- redis_mlc 5y agoI did some work for the Recommendation team (the oversized checks were still arrayed along the wall when I was there.) The later winning entries were too compute-intensive to implement, or not enough of an advantage over the existing engine to justify more compute. So I would say it was a real competition, not just PR, even though a particular solution wasn't used. (Factoid: It was the the last team to start the migration to Cassandra.)
- uyt 5y ago> are open algorithmic contests useful and valuable? Kaggle has been around for a long time now. If it works, I would expect them to be pumping out tons of interesting results from winners but I don't think I've heard many stories like that. It seems to be mostly useful for recruiting purposes?
- _pastel 5y agoKaggle competitions rarely produce interesting algorithmic results. But I highly encourage you to read the winners' solutions. They are full of clever data insight, augmentations, regularizations, feature engineering, and preprocessing and postprocessing tricks. But above all, compared to the academic literature, it's shocking how much time and creativity they spend on validation. Maybe I'm reading the wrong papers, but the flashy new neural architectures rarely even mention their validation setup; Kaggle winners sometimes devote half of their explanation to it. It's part of their secret sauce. Two personal favorites: (1) https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/114254 https://www.kaggle.com/c/severstal-steel-defect-detection/di.... The "random defect blackout" was a really clever data augmentation. (2) https://www.kaggle.com/c/ieee-fraud-detection/discussion/111284 https://www.kaggle.com/c/ieee-fraud-detection/discussion/111.... Particularly how they reduced overfitting with adverserial validation. They trained a separate model to distinguish between train and test sets, and then dropped features that ranked highly in feature importance on that model. That's probably a well-known technique in some circles, but I had never seen anything like it before.
- raverbashing 5y ago> But I highly encourage you to read the winners' solutions. They are full of clever data insight, augmentations, regularizations, feature engineering, and preprocessing and postprocessing tricks. > But above all, compared to the academic literature, it's shocking how much time and creativity they spend on validation. Maybe I'm reading the wrong papers, but the flashy new neural architectures rarely even mention their validation setup; Kaggle winners sometimes devote half of their explanation to it I agree, but in the end it is a competition, and the solution that scores the most is not always the solution that is "the most interesting" (or practical, or best in real world cases) Though the details you mention are interesting, and can definitely apply at real-life solutions.
- bcrosby95 5y agoIIRC they announced pretty quickly that they wouldn't be using the winning solution due to complexity for little gain. A common thought at the time was that it was worth it because it showed how far they would have to go to make their algorithm better and that it wasn't worth it.
- danielharan 5y agoI learned about AI by participating in the challenge, and ended up co-founding an AI company as a result. Netflix got more than 1 million dollar in free advertising from it, and are still getting brand value out of it today. They implemented some of the algorithms, and probably got a 10X ROI through retention alone. As mentioned in the Quora answer, they were also able to recruit top talent - and that's much harder to put an ROI figure on.
- kqr 5y agoSimilar story here. The company I work at founded when a couple of very bright people realised why the Netflix stuff won't generalise to many other areas, and sought to find something that did instead. So far, very successful!
- calderwoodra 5y agoI had the pleasure of working closely with Yehuda (one of the earlier prize winners) at Google where he works on TV and Movie recommendations (Search "what to watch" to see his team's work). He's extremely intelligent and passionate about this space, and every time we spoke I felt like I was learning something new. You can listen to him give an in depth talk about the Netflix problem and solution here [1]. [1] https://youtu.be/YWMzgCsFIFY https://youtu.be/YWMzgCsFIFY
- __vim__ 5y agoThanks. Will watch later
- thom 5y agoDo people feel that Netflix has a large enough catalog that its recommendation system really matters? The only useful feature it ever had for me was the ‘new this week’ category that seems to have been retired.
- doublescoop 5y agoThe company reportedly had 100,000 DVD titles in their catalog, so yeah. Remember how old this contest was.
- qbasic_forever 5y agoTheir online selection was enormous ~2010 when they still had the DVD business, and crucially all of the movie studies still thought online was a fad and were happy to cut deals to stream their entire catalogs for pennies. Netflix was unbelievable back then and felt like it had every movie in existence. Over the years all of that back catalog has been clawed back and Netflix morphed into more of a showcase for their own content. But for a few golden years it was amazing to go in and like/dislike a bunch of stuff to see it just start recommending and streaming tons of classic films I always wanted to see.
- alexilliamson 5y agoDVD Netflix exists still! I got it recently somewhat as a funny bit.
- jmilloy 5y agoAs of last year when I finally quit, the DVD catalogue was still great, but service was so slow that it costs less to rent movies to stream individually.
- dalbasal 5y agoReally matters could be interpreted different ways. Expressed in startup-metric terms, it might be "% of users that watch something" or "time spent browsing before watching something." Any real improvement on such is probably important. Think of it like Google's famous obsession with speed. Did returning search results 17ms faster really matter? It's hard to say for sure, but I suspect it did. That said, I agree personally. I don't like Netflix' UI. I suspect you could hand code a browsing/ranking UI of similar value, from a casual users' perspective. >> large enough catalog I think this is a case where Netflix didn't end up where they expected the. I think they expected to have a vast catalogue... a "spotify of movies." It just didn't go that way. You could also reverse the question. Does netflix have a big enough dataset to make a great recommendation system? I think this might be the more pertinent question. Google & FB have their vast ad-centric datasets. I suspect these could be used to make a recommendation engine that's a lot better. They haven't really done this for youtube though. The priority is to match ads to users. For this, they're willing to push the envelope on how they use user data. For youtube recommendations, it doesn't seem that youtube gets access to much data from outside of youtube.
- koonsolo 5y agoI don't get why they wanted to go all out on AI for this. I use Netflix, and a simple algorithm would already be an improvement: - If I watched a movie and gave it a thumbs down, don't recommend it - If I watched a movie < 1 month ago, don't recommend it - If I browsed over a movie 50 times, read the info, and still didn't play it, stop recommending it. - If I watched the last episode, remove the "new episodes" banner. WTF
- vletal 5y ago- Do you really get recommenditions for movies you already watched or down voted? I'm not aware of that happening to me. It's not it the case that you use multiple different profiles within a single account? I sometimes see that for shows we finished, yet people tend to rewatch episodes... - Removing a movie based on "not payed X times" would remove all popular movies for all users in a multiple of X steps. -I agree with the new episodes banner.
- teichmann 5y agoI'm not sure about down voted ones, but I can confirm that I sometimes get already watched movies recommended again (even in recommendation mails).
- klausjensen 5y agoCompletely agree. I am constantly baffled at the bullshit Netflix suggests me. I just went to the frontpage of my Netflix and...: - "My list" recommends 3 series where I have alread watched the last episode - "Only on Netflix" recommends another two series, I have already wathed. - A section displayed is "Watch together for older kids" - I dont have kids, never watched any kids stuff on my account. - "Documentaries" contains six suggestions, 3 of which I have already watched on netflix". - I am suggested several shows, which are good - but I have already watched outside Netflix earlier - and there is no way to tell Netflix (none that I know of, anyway) 9 out of 10 times when I go to Netflix, I intend to continue watching a series - but Netflix makes me scroll past SEVEN sections of recommendations to get to "Continue watching.." before showing me the series I have watched 1-2 episodes of most days for the past two weeks. Maybe they are just too busy making sure all new series are woke-i-fied to care about how this simple stuff works?
- paxys 5y agoThis prize was the biggest deal in the tech world back in ~2006-2010. They made way more than $1 million back in advertising and engineering recruitment alone, even if they didn't use the winning algorithm at all.
- the_biot 5y agoI haven't been a Netflix customer in some years, but at the time their algorithm was pretty obvious: recommend only Netflix-produced content. It's the elephant in the room here.
- studentrob 5y agoOh man this was huge. You have to remember, machine learning was nothing back then. It was on nobody's radar. Then comes this flashy $1 million prize. Tons of universities had teams. So it really helped their recruitment. It also likely contributed to the idea of creating Kaggle which has itself greatly contributed to data-science education by giving everyone an open forum in which to compete. Then there were other signficant projects around this time like ImageNet which became a competition too. That open dataset led to tons of research and applications.
- dalbasal 5y ago"ROI" is probably the wrong frame for this question. "Worth it" is better. "Better than X" might be better still, since it frames the question such that "X" needs to be defined and quantified. The benefits of such a competition are pretty nebulous, and there's no way to convince an ardent skeptic. OTOH, many business decisions are like this and skepticism isn't a viable frame in many cases. Netflix got visibility with investors and potential employees. Netflix's recommendation engine became famous, even though it doesn't seem impressive as a user. The exercise created a structured way of thinking about their recommendation algorithm. They cemented its importance. Even though they didn't implement the winning solution, they did get a useful benchmark. This was potentially very useful in further decisions in R&Ding the recommendation engine in-house. All that for $1m?
- rocqua 5y agoI used the same dataset for a small project in my CS master. It was a really fun challenge, and it taught me a lot. Most notably, it taught me that it was incredibly hard to make significant progress past the most simplest and naive approach. That approach was "Take average rating a user gives, take the average rating a movie gets, multiply". (Ratings normalized to be between 0 and 1). Just using this method would give us 95% of the accuracy of our final method. I think I calculated, and compared to the prize winning result, our method got ~90% as accurate a result.
- kbelder 5y agoThis is an important point about a lot of sophisticated models; you're really fighting for a few percent improvement over simple approaches. Sometimes a basic linear regression will get you 70% there, while a trained neural net will bring that up to... 75%. A few percent can make a difference, especially in competitive areas; but the biggest win is just getting something in where there was nothing before. It's a bit like optimizing code.
- sytelus 5y agoRecommandation algorithms are massive sales driver if you depend on long tail. This was true for back in the days and long term subscriber retention would almost entirely depend on great recommandations. Unfortuately, Netflix figured out that they need not depend on long tail to retain their market value. Instead the new idea is same as HBO model. Now the recommandations are basically garbedge.
- samplenoise 5y agoThe author is still underselling the significance of the progress made during the first years IMO. The simple idea that is still behind most practical recommender systems (using gradient descent to do SVD to complete the rating matrix) was first described in 2006 by Simon Funk [1]. Koren, who ended up taking home a big part of the prize, recently wrote another paper about how that basic idea still outperforms most “AI” (deep neural) recommenders today [2]. [1] https://sifter.org/~simon/journal/20061211.html https://sifter.org/~simon/journal/20061211.html [2] https://arxiv.org/abs/1905.01395 https://arxiv.org/abs/1905.01395
- gameswithgo 5y agoNetflix suggestions used to be amazing, but sometime after their contest they totally changed how it works and they became useless. This article maybe hints at why with a short “predicting consumption was more important than ratings anyway” why? what does he mean? netflix had a killer advantage with the old rating system and then dumped it why?
- blowski 5y agoThere's a subtle but valuable (for Netflix) difference between "you watched x so you'll probably like y" vs "if we can get you to start watching this show you'll probably pay another month's subscription".
- chanmad29 5y agoOnly Netflix content is worse than their algorithms. I've completely stopped surfing the platform and stick to my own watchlist or search for specific movies based on the recommendation lists out there.
- mydpy 5y agoThe article doesn’t emphasize the impact made by new technologies invented in pursuit of this prize. For example, this contest was a motivating factor for the creators of Apache Spark.