32 ms·

Netflix never used its $1M algorithm (2012)

294 points by reqo 3y ago

deleted 3y ago
[deleted]
teruakohatu 3y ago
They got far more than $1M in marketing from it. It would have been worth it for twice as much.
- taeric 3y ago
  I'm curious how you quantify that?
  - kamikaz1k 3y ago
    at least for the internal goal of hiring top ML talent, you can probably just calculate how long it took to hiring the good ones, and then how much money that team made.
    - taeric 3y ago
      Are they objectively better at ML for movie recommendations than they were beforehand? I confess I've never been too impacted by their recommendations. Outside of new releases, but that doesn't really use any ML.
  - kamkazemoose 3y ago
    You can think, how much would their marketing team have to spend to get the same results that the algorithm contract gave. I'm not a marketing expert, but I'm sure they have metrics like consumer sentiment, name recognition, number of users visiting the site, google search trends, etc. There could also be benefits in recruitment, and that can be estimated based on how much you'd have to pay an external recruiter to bring in candidates the applied, or other things like that. It was in the news a lot, and was discussed on a lot of tech sites. Plus it gets people talking about their recomendation algorithm, and makes people thing Netflix subscription is more valuable becasue it recommends good shows. It wouldn't be cheap to get the amount of media that they got through more traditional marketing.
    - taeric 3y ago
      Right, but that is a lot of what I'm asking. What makes you think those are all better due to this contest? If I recall, a lot of why it was making the news was because Netflix was already popular in the industry. They certainly weren't that new of a name. The main one I don't think I would have doubts about is the recruitment. But, I don't recall them being a place that needed recruitment help, even at that time. To be clear, I found the thing fun to consider. I certainly am not upset that they did it. I do harbor a gut feeling that its ROI is greatly overstated.
  - gen220 3y ago
    I'm not sure you need to. Not picking on you at all, just going on a tangent because I've been thinking recently about the good ideas that end up on the cutting floor because their impact, while positive, is intractable or completely impractical to quantify. Some ideas (like this Netflix one!) are "obvious" winners, because they have a diffused, positive impact in many important dimensions – each of which is almost impossible to measure, but the integral of which is almost certainly greater than the idea's cost, probably by one or two OOM. If there's high conviction in an idea being a 10x idea, and attempting it costs quite little, it's better to do it now and maybe consider measuring it later. Who cares if it's 9x or 12x ROI, the point is it has a big margin of safety and large expected returns on capital. But in a "we can't greenlight this project unless we can directly attribute it to positive motion in our KPI" org, these flavors of idea are dead on arrival. The double-kicker is that the cost of trying to measure these ideas is often greater than the cost of the idea. For some ideas, especially when it's a rounding error on the company's annual budget, it should be OK if we don't invest much effort into quantifying the results. It's one of those rare set of ideas where it's anathema to the professional/investor culture of SV of the last decade, while simultaneously being how plucky Seed/Series A companies can punch above their weight.
    - taeric 3y ago
      I would push back on this specific case, though. What makes this an obvious winner? Netflix was already clearly on the upward swing. At the time of this contest, I'm not even sure I remember who their competition was. Now, I think it is fully fair to say that this was very cheap for them to do. In which case, why not do it? But I think you would be hard pressed to give me a counter factual world that is believable where not doing this had a meaningful impact on Netflix's future.
      - travisjungroth 3y ago
        Hiring. They got a lot of attention for this.
        taeric 3y ago
gathersnow 3y ago
Netflix became so user-hostile and it is truly baffling to me. They act as if they are paid by amount of time you spend watching movies and have all of these dark patterns shoving the user all over the place and disorienting them. I remember when they had tools to see what your friends were watching and to discover hidden gems which was always such a fun experience. Does Netflix have such content nowadays? I really wouldn't know. They optimize the experience for binge-watching and nothing else. I wish so badly that they'd do a letterboxd thing and allow for curation of their catalog but it all goes back to the ephemeral nature of their content and how they have to hide that fact. They care so little about that approach now it's no wonder they never used this algorithm.
- rocky1138 3y ago
  I agree. Netflix died the day they switched to thumbs-up and thumbs-down versus 5-star rating content. My recommended list was so solid for so long and all of that went down the drain.
  - gathersnow 3y ago
    Can you even vote now? I think they do a lot of it by inferring from what you watch.
  - iwontberude 3y ago
    Their death was presaged when their executives decided to "become HBO before HBO can become us" and when they spun out Roku for lack of appetite to be a proper tech company. Reed Hastings wasn't capable of running that business and so their vision shrank to match. Netflix sold a lot of people a lie, their customers, their employees and their investors. We jumped on board thinking Netflix was serious about the living room experience in a wholistic way.
    - anthonypasq 3y ago
      how was netflix ever going to continue being a "tech company." Video streaming is a commodity at this point, they are not longer doing anything technologically revolutionary. Once every other media company could do what they were doing they realized they have to become an entertainment company. It was shrewd foresight on their end, otherwise they wouldnt exist anymore. Unless they tried to license out their infrastructure or something
avelis 3y ago
I remember seeing the leaderboard rankings before the prize was awarded. If I recall correctly the top two teams were very close to the coveted goal. They ended merging to become one team and spit the prize once they achieved the goal.
bilsbie 3y ago
I wonder if it paid off for recruiting though?
n2d4 3y ago
Those contests are never about the actual code/algorithms they produce, it's always about recruiting the people who write them. For the price of $1mil (plus a bit of engineering time) they got a list of talented engineers who have too much time, and also everyone now knows that they are the company with the really challenging problems. Forward the list to recruiting, and you got a bunch of new hires, much more effective than paying for job ads.
- deleted 3y ago
  [deleted]
- deleted 3y ago
  [deleted]
ffhhj 3y ago
> It’s now extended beyond the US and in to Canada, 43 Latin-American countries There are about 21 LatAm countries.
- bryanrasmussen 3y ago
  probably there's an advanced algorithm at work here that says otherwise.
mushufasa 3y ago
in Reed's "No Rules Rules" book, they discuss how this contest was really a way to recruit top-quality engineering talent, which is a key assumption in how they ran their culture (highly paid small teams with lots of freedom, trusted to know what's best from the ground-up rather than top-down). They didn't really need an algorithm in the first place, they just brainstormed what would sound the coolest to developers.
- shermantanktop 3y ago
  Everything that Netflix says or does externally wrt to tech appears to be for this purpose, including open-sourcing some things. It is all catnip for the HN crowd to come work for them.
  - mrguyorama 3y ago
    Netflix pays in the same ballpark as other FAANG companies but the product is so much clearer (stream people movies and shows, maybe recommend them something to watch) and the complete lack of grey ethics like in AdTech or a Monopoly like Microsoft is the real catnip.
    - cybrox 3y ago
      They use a lot of the same concepts as AdTech but only for their own content without shilling for everyone who is willing to pay.
      - Cyphase 3y ago
        I think that context makes a reasonable difference for some people. (The following is a tangent.) To put it another way, using a slightly different kind of situation, I don't want BigCorp's AI watching everything I do to "serve me better", but I'm very interested in the idea of a local AI that watches everything I do and helps me be N times more efficient, while I maintain full control. I don't care if the local AI uses the same data-gathering, data analysis, etc. concepts as BigCorp's AI to accomplish the goals I want it to accomplish. Caveats include: - The local AI is Skinner-boxing you or similar. This isn't about the AI having conscious malicious intent; it could be a design defect. Also, a local, offline, open source video game can still be addictive. - The risk of the system being compromised / manipulated, and all that data on you being exfiltrated.
slt2021 3y ago
This prize/competition has become a meme in itself and inspired hordes and hordes of people to dabble with recommendation systems. The advance of recsys is in big part thanks for Netflix because it attracted a lot of people to the field, all companies started developing inhouse RecSys just like today you hear about inhouse LLM/chatgpt wrappers etc
minimaxir 3y ago
The Netflix Prize competition (2006, completed in 2009) was a Kaggle competition before Kaggle competitions (2010), with the same business-side incentives. Given the meteoric rise of ML/AI in the past few years, I'm surprised that Kaggle doesn't come up more often. It was all the rage 2013-2018...then most I heard about it was that it allowed free access to TPUs.
- htrp 3y ago
  I feel like kaggle is a copycat wasteland for most projects now. You don't quite get top talent, rather you get a bunch of people xgboosting their way up your leaderboard.
  - erhserhdfd 3y ago
    "...rather you get a bunch of people xgboosting their way up your leaderboard" --> I don't see that as a universally bad thing. I feel this is actually very representative of the majority of ML projects that most organizations encounter. Most organizations don't have petabytes of data and a huge compute budget to train a DNN. They typically have megabytes to gigabytes of somewhat crappy data and need something that can be developed and deployed relatively quickly for low cost.
- hcarlens 3y ago
  There have been quite a few interesting Kaggle competitions in recent years, as well as other interesting ML/data science competitions on other platforms. Platforms like Kaggle, DrivenData, Zindi, AIcrowd, CodaLab and others are running dozens-hundreds of competitions a year in total, including ones linked to top academic conferences. One interesting recent one is this one on LLM efficiency - trying to see to what extent people can fine-tune an LLM with just 1GPU and 24h: https://llm-efficiency-challenge.github.io/ https://llm-efficiency-challenge.github.io/ Or the Makridakis series of challenges, running since the 80s, which are a great testbed for time-series models (the 6th one finished just last year): https://mofc.unic.ac.cy/the-m6-competition/ https://mofc.unic.ac.cy/the-m6-competition/
AeroNotix 3y ago
Most companies spend a ridiculous amount of money for not much at all. The crazy amount of waste in boom times is, frankly, disgusting. You all have witnessed it, some of you may have even tried to stem the tide of horrific spending, some of you are to blame. Go to the billing page of whatever third-party thing you use and just recoil in horror how much you spend.
resters 3y ago
Not surprising. I'd been using movielens for recommendations prior to the prize, and even today Netflix still does not provide high quality content recommendations. Worse yet it seems to think I want to watch the three things that are most popular on the platform. This indicates that the Netflix UX business is being significantly mismanaged.
- mvdtnz 3y ago
  I suspect the Netflix recommendations system would be excellent if they had a catalog to back it. But the Netflix catalog is of such poor quality that even a perfect recommendation system ends up recommending garbage because it's all they have.
  - zoover2020 3y ago
    Isn't their algorithm also very intendedly only catering to Netflix originals by default? I find its algo absolutely rubbish and most of my initial excitement to go watch something with my wife on the couch instantly fades as soon as we face the doomscrolling of finding content to watch. It's like zapping but worse
    - onli 3y ago
      For me it's the missing quality markers. Netflix provides no way at all to see whether a show or movie is good. So when one starts scrolling and does not find immediately something one likes, there is no reason to ever find anything good - because nothing seems good, and all alike. Since I added a userscript to see IMDB ratings directly next to the netflix shows it's way easier to find something promising.
  - cortesoft 3y ago
    I always hear people say this about the Netflix catalog, but I never understand what metric or criteria people are using. I find there is a ton of great content on Netflix, and I have way more things I want to watch on there than time to watch it. I always wonder if people who say this either have very particular interests, or are trying to find specific tv shows/movies and not finding them on Netflix. If you are not set on a particular thing, it seems to me that there is a ton of great content.
bazil376 3y ago
Isn’t $1m just like 2 eng salaries for Netflix?
- dev-tacular 3y ago
  If levels.fyi is to be trusted, it like one L7's total compensation: https://www.levels.fyi/companies/netflix/salaries https://www.levels.fyi/companies/netflix/salaries
rappatic 3y ago
Ironically, Netflix's recommendation algorithm is now notoriously bad. In my experience, it seems to heavily push whatever "original" they just dumped $100 million into as well as what's most popular on the platform at the time. But it makes sense, since people increasingly rely on social media for deciding what to watch. A dollar Netflix spends improving their recommendation algorithm simply won't stack up to the dollar TikTok/Instagram/etc. spends improving theirs, because social media apps have so much more data to work with. It's probably more economical from their POV to let broad social media trends dictate what people watch.
- kevincox 3y ago
  I suspect their algorithm has other success metrics than "did user enjoy movie". For example if the movie is a Netfix Original or exclusive movie then it is more valuable to Netflix because they may discuss or recommend it, which leads to more signups. Similarly they may prefer newer shows that are more likely to be discussed and raise hype than older movies that even if the user loves them may be less likely to advertise to others.
- 972811 3y ago
  in my experience from the outside it's difficult to separate their rec algorithm quality vs. the quality of their inventory. you're making the assumption they have a ton of great stuff that they're just not showing you, but they may not.
- VikingCoder 3y ago
  If any streaming service wanted to give my family good recommendations, they would let me select ALL of the people who are watching right now. Husband. Wife. Husband and Wife. Husband and 4th grader. Wife and Pre-K. Husband, Wife, 4th grader, and Pre-K. 4th grader. Pre-K. 4th grader and Pre-K. Each of those has a totally different viewing pattern and preferences.
  - evanmoran 3y ago
    This 100%. I promise you that I’m not secretly very interested in Dragon Rider episodes without my kids present. They have their own accounts, but it is hard to share family movies or when they just use the wrong account. I also think half of my viewing is with my spouse. We are pretty aligned in some shows but we are different enough where “My List” really means when I’m alone, and we are missing an “Our List” for when we are together.
zzbzq 3y ago
Using a recommendation algorithm doesn't make any sense for Netflix' new business model where they produce their own content. It should be easy. They greenlight or buy the rights certain content for different demographics to maximize the coverage of their user base that has their needs satisfied enough to stay subscribed. If you're a 18-35 male, they've surely got people there working daily to make sure there's a content pipeline coming just for you. They shouldn't need AI to tell me that, they just need to identify when I'm in the demographic of one of their shows coming, and tell me about it. Maybe like a list. Still, they seem like they're doing a bad job at it, as I never really know what new shows Netflix is making for me. Sometimes I find out about them years later. For example, compare this to Disney+. I vaguely know about every Marvel and Star Wars thing 2 years in advance, and usually vaguely know what order they're coming out, and I don't even know how I know this, somehow I just know. Okay, that's easy because it's basically 2 IPs. But ultimately Netflix is doing something similar behind the scenes, why haven't they succeeded at making me aware of what content I'm supposed to be hyped for?
- cqqxo4zV46cp 3y ago
  This feels like a disingenuous argument. “Young to middle-aged man likes Marvel and Star Wars” is such an obvious pick that it just about invites parody. As someone with at this point zero interest in Star Wars / Marvel content I a) wouldn’t retain the information that you did and b) wouldn’t find it to be that impressive that D+ was shoving it into my face. To not understand the usefulness of recommendation algorithms almost feels intentionally contrarian.
- cybrox 3y ago
  To be honest, I would be happy if their 20+ people UX team would create an experience where I can easily find what I was watching instead of shoving stuff down my throat that I have no interest in. Then again, as long as people pay for that experience, it will continue to be as unbearable as it is.
  - lubesGordi 3y ago
    This is the problem across the board, regardless of service almost. Everything is some crappy recommendation system, instead of enabling me to find what I want. In addition to sucking in general, recommendation systems have this great property of radicalizing people politically.
dangerboysteve 3y ago
Personally, I never thought they would have in the first place. I was thinking this was more for recruiting.
eachro 3y ago
incentive problem
chris-orgmenta 3y ago
> Well, $1m may sound like a lot of money for Netflix to pay for something that wasn’t really used The title to me sounds analogous to 'I didn't use that piece of code I wrote on the weekend (but I probably learned something from it)'. If it were 10m, maybe relevant? But hardly to their bottom line. https://www.macrotrends.net/stocks/charts/NFLX/netflix/revenue https://www.macrotrends.net/stocks/charts/NFLX/netflix/reven... ... Most companies that size (in evolving industries at least) should have 10-100 of those moonshot/R&D/marketing projects on the go. And I'm speaking to the choir here - exactly: surely most readers of thenextweb would be thinking the same here.
liendolucas 3y ago
I don't get the obsession of some companies to recommend things. I'm happy not only for not paying Netflix but also for not wasting time watching crap. I have my own manual recommender: browse IMDB or Rotten Tomatoes of past directors, cast, etc that I've enjoyed. Also from a good old friend that from time to time recommends me films as we have very similar preferences. Randomly browsing genres worked very well too. That has become a near perfect movie recommender over many years.
- slig 3y ago
  They push very aggressively for whatever new TV series / movie they just released that is "top 1 in your country". Spotify is the same with whatever crappy playlists and podcasts they want you to listen. YouTube doesn't even bother showing results that you searched past the 10 or so results, it's "content you might like" which is crap and unrelated to anything I've watched, 100% of the time.
  - liendolucas 3y ago
    We live in a crap-stream world LOL.
- cqqxo4zV46cp 3y ago
  Is this a serious comment? Netflix does not and should not base its strategy on how you specifically choose to consume media. Surely you understand that most Netflix users don’t use IMDb/RT at all, let alone as a recommendation ‘engine’. Netflix, especially Netflix back then, is highly reliant upon in-product recommendations because if people sought out specific pieces of media they’d often find that Netflix didn’t have it.
- deleted 3y ago
  [deleted]
- jakearmitage 3y ago
  Exactly. Give me proper filters in the search tool, I'll find my own movies.
- 3y ago
codelikeawolf 3y ago
Hang on a sec, it looks like BelKor's Pragmatic Chaos and The Ensemble had the exact same Best Test Score and % Improvement. Did BelKor win because they submitted their solution 20 minutes earlier than The Ensemble? Or is that Best Submit Time something else?
- Irishsteve 3y ago
  Yes - from what I remember the team that came second lost out due to something like 6 decimal places and the time diff in submission. It was pretty close!
  - codelikeawolf 3y ago
    Wow! That's bananas! Imagine being 20 minutes and 6 decimal places away from a million bucks.
    - thaumasiotes 3y ago
      Being six decimal places away from winning means the winner was one million times better than you.
      - bagels 3y ago
        They're talking about 0.999999 vs 0.999998, or ~1 part in 1 million different
    - Irishsteve 3y ago
      I might be mis-remembering this part - but I believe napoleon dynamite had its own algorithm in the ensemable because it was a tricky one to predict the rating for.
gyudin 3y ago
Netflix just needs decent movies in it’s library :)
deleted 3y ago
[deleted]
tunesmith 3y ago
Interesting that the answer is basically "because of akrasia". I wish more of these recommendation algorithms were based more off of who we'd like to be rather than how we behave.
softwaredoug 3y ago
As someone who works in search relevance, having just a great algorithm isn't worth much. You need all the team and know-how that has the maturity to maintain such an algorithm. Not just the ML skills. But all the bazillion ops, data quality, and many other things that go around it. I've worked with a lot of teams that have one smart person building stuff off to the side, in R or a Notebook, and then nobody knows how to productionize it. They try to throw the algorithm over the fence. Even if the team somehow succeeds in getting it into an A/B test, it eventually falls by the wayside, unless they can build the team and workflow around that person / algorithm / methodology
- whimsicalism 3y ago
  > I've worked with a lot of teams that have one smart person building stuff off to the side, in R or a Notebook, and then nobody knows how to productionize it. They try to throw the algorithm over the fence. Even if the team somehow succeeds in getting it into an A/B test, it eventually falls by the wayside, unless they can build a team and workflow around that person / algorithm / methodology. In my view, this marks a cultural failure - and certainly not on the part of the 'one smart person.'
  - softwaredoug 3y ago
    100%. I totally agree. I've both been the person and the team here. And it suucks when you see how things could be done better.
- geoduck14 3y ago
  You are definitely right. The problem to solve quickly becomes process and not technical
LudwigNagasena 3y ago
Anyone who has experience with ML wouldn’t be surprised by that. Oftentimes ML competitions are about combining dozens of models together to juice the extra 0.01%—something that isn’t viable in a production environment as the quote in the article confirms. AFAIK, modern ML challenges try to combat that by moving from answer-only submissions to code submissions and putting constraints on compute.
bob_theslob646 3y ago
Dr. David Belanger (RIP) was a part of advising this team (BelKor)and knew all about this, but never ever bragged about until one day I stumbled upon it and asked him about his involvement. https://www.thrillist.com/entertainment/nation/the-netflix-prize https://www.thrillist.com/entertainment/nation/the-netflix-p... Also, they definitely used code from the first two competitions in the company. "The first year of the competition, in 2006 and 2007, the technical advancements that were made by us and the other big teams I think were really significant in the field of recommender systems," says Volinsky. He thinks the idea that Netflix didn't use the results is a misconception. "We gave them our code. They definitely did implement and use those breakthroughs that we made in the first year." For more that are interested , I can speak about it to the best of my ability. https://ieeexplore.ieee.org/author/38180399800 https://ieeexplore.ieee.org/author/38180399800 ( that is some things about him) Essentially he would say that they initially shelved it over time as it was not needed, but they definitely used. Belanger tragically died November 18, 2022.
- gwern 3y ago
  Their 2015 paper on how they were doing recommenders also mentions that they used bits and pieces of the prize work: https://gwern.net/doc/reinforcement-learning/exploration/2015-gomezuribe.pdf https://gwern.net/doc/reinforcement-learning/exploration/201...
nashashmi 3y ago
Engineers called it "Not worth my valuable time" kind of engineering effort. An early version of the 10x engineer (who refuses to do lots of meaningless work).
aaron695 3y ago
[dead]
gwbas1c 3y ago
I remember meeting someone at a networking event who believed they could win the competition and get > 90% accuracy. I tried to explain that you can't read minds; you can't account for the fact that I'm going to watch a movie that I saw in a random internet post that I didn't know that I liked... Which was my real beef with the contest: Computers can't read minds. They can't analyze the content of the movie and have the emotional experience that a person can.
- robrenaud 3y ago
  Computers are learning to read minds. https://amp.theguardian.com/technology/2023/may/01/ai-makes-non-invasive-mind-reading-possible-by-turning-thoughts-into-text https://amp.theguardian.com/technology/2023/may/01/ai-makes-...
  - stonogo 3y ago
    This article has no bearing on the topic at hand.
- anjc 3y ago
  > Computers can't read minds. They can't analyze the content of the movie and have the emotional experience that a person can No but you can read minds by proxy, via ratings, which is what the dataset consisted of.
- mr_toad 3y ago
  > They can't analyze the content of the movie They probably could. Apple photos is already capable of judging the quality of your photos, so it’s not too much of a leap to think that a model could be trained to recognise good from bad movies. On the other hand there are far fewer movies than photos for training on so it’s hard to say how good the results would be.
jfengel 3y ago
Streaming gives them much better information than a user's voluntary stars. Did you actually watch it? All of it? How soon? All at once? User ratings are fraught. The things a user actually does are more likely to be sincere. And it doesn't put any burden on the user. I have a feeling that even the simple thumb up/down means less than whether you actually finished watching the movie.
- theshackleford 3y ago
  To bad they did'nt use any of that information they have on me to recommend me things I actually wanted to watch. They just kept pushing the same shit I never once displayed intention in watching over, and over and over again. Sounds like everybody is winning other than me. Though given I cancelled and never looked back, maybe I am winning afterall.
  - jfengel 3y ago
    The problem is that they just don't have enough content you want. I think they'd give it to you if they had it. They're not hiding it. They just have so much catalog and when you've watched the good parts, cancel.
gverrilla 3y ago
Netflix feels like Blockbuster did some years ago.
- Infinitesimus 3y ago
  How so?
  - gverrilla 3y ago
    swallowed about-to-die off-hype (my own opinion of course)
- deleted 3y ago
  [deleted]
deleted 3y ago
[deleted]
fsckboy 3y ago
does anybody here know a lot about recommendation algorithms? The one I'm interested in seems obvious to me, but it's never implemented so maybe it doesn't work, or it serves outliers and not the mainstream so why bother. The data is there, but I don't get to mine it. (heheh "mine" it) I don't want to know what's popular, I want to know what other people who share my past tastes are watching currently. Doesn't matter the size of my niche, why can't i find my niche, and be shown my cohort? I just don't feel it's an impossible task to find, for example, a set of people who enjoyed Deadpool 1, Guardians of the Galaxy 1, but think the entire rest of the superhero cavalcade is utter garbage. ok, I'll allow Kickass also, and a grudging nod to Iron Man 1. (as an illustration, I just searched to look up the name of Iron Man, I could only remember Tony Stark. Top search results were "a fictional character", "played by R Downey Jr", "Marvel cinematic universe"... You see, none of those is Iron Man 1. I don't care about the horde of people who care about a supposed cinematic universe. We (in my cohort) liked the first one, and what we liked about it wasn't continued in the 2nd. How about preferences that show I like a few early films with say George Clooney, but after that, no. Julia Roberts, same. I remember when Meryl Streep was a new actress, but now she's a red flag. I'm sure my tastes aren't shared by everybody, but I'm also sure, since I can articulate reasons, there must be others who in general share them, people who are willing to watch what's new and trendy in some way, but then not beat the dead horse. Oh, Spielberg, a few winners, but overall "nope nope nope", the way he aims at sentiment is a zero for me. I originally had this idea back in the 80's when Consumer Reports had their monthly mail-in bingo card for what current movies you liked, then they would tabulate what was popular among their readers. That type of "best of" list is entirely missing co-variance, which I think is where I live.
- anjc 3y ago
  That's called collaborative filtering (https://en.wikipedia.org/wiki/Collaborative_filtering https://en.wikipedia.org/wiki/Collaborative_filtering) and is perhaps the most battle-hardened and most effective approaches in recommender systems. Even now, novel deep learning approaches implement the concept, but simple naive approaches are still as/more effective. The first paper published on it specifically in the field was in the 90s but the seeds of it go back to the 70s. It would have made a good thesis in the 80s :) Part of why it's so effective are for the reasons you outline, in that it can find items that you'll probably like, that aren't similar to items you already like, based on people that have similar tastes to you.
jameshart 3y ago
[flagged]
- deleted 3y ago
  [deleted]
mav3ri3k 3y ago
> “We have adapted our personalization algorithms to this new scenario in such a way that now 75% of what people watch is from some sort of recommendation,” says Netflix I am always sceptical of these claims. Today algorithms have become ai models. Sorting algorithms are recommendation engines.
j2kun 3y ago
This article directly contradicts an article written by Netflix engineers Xavier Amatriain and Justin Basilico, who were in charge of putting the BellKor algorithm in production: https://netflixtechblog.com/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429 https://netflixtechblog.com/netflix-recommendations-beyond-t... > But once we overcame those challenges, we put the two algorithms into production, where they are still used as part of our recommendation engine.
- thefringthing 3y ago
  This article links to that blog post and quotes from it.

294 points by reqo 3y ago

deleted 3y ago

[deleted]

teruakohatu 3y ago

They got far more than $1M in marketing from it. It would have been worth it for twice as much.

taeric 3y ago
I'm curious how you quantify that?
- kamikaz1k 3y ago
  at least for the internal goal of hiring top ML talent, you can probably just calculate how long it took to hiring the good ones, and then how much money that team made.
  - taeric 3y ago
    Are they objectively better at ML for movie recommendations than they were beforehand? I confess I've never been too impacted by their recommendations. Outside of new releases, but that doesn't really use any ML.
- kamkazemoose 3y ago
  You can think, how much would their marketing team have to spend to get the same results that the algorithm contract gave. I'm not a marketing expert, but I'm sure they have metrics like consumer sentiment, name recognition, number of users visiting the site, google search trends, etc. There could also be benefits in recruitment, and that can be estimated based on how much you'd have to pay an external recruiter to bring in candidates the applied, or other things like that. It was in the news a lot, and was discussed on a lot of tech sites. Plus it gets people talking about their recomendation algorithm, and makes people thing Netflix subscription is more valuable becasue it recommends good shows. It wouldn't be cheap to get the amount of media that they got through more traditional marketing.
  - taeric 3y ago
    Right, but that is a lot of what I'm asking. What makes you think those are all better due to this contest? If I recall, a lot of why it was making the news was because Netflix was already popular in the industry. They certainly weren't that new of a name. The main one I don't think I would have doubts about is the recruitment. But, I don't recall them being a place that needed recruitment help, even at that time. To be clear, I found the thing fun to consider. I certainly am not upset that they did it. I do harbor a gut feeling that its ROI is greatly overstated.
- gen220 3y ago
  I'm not sure you need to. Not picking on you at all, just going on a tangent because I've been thinking recently about the good ideas that end up on the cutting floor because their impact, while positive, is intractable or completely impractical to quantify. Some ideas (like this Netflix one!) are "obvious" winners, because they have a diffused, positive impact in many important dimensions – each of which is almost impossible to measure, but the integral of which is almost certainly greater than the idea's cost, probably by one or two OOM. If there's high conviction in an idea being a 10x idea, and attempting it costs quite little, it's better to do it now and maybe consider measuring it later. Who cares if it's 9x or 12x ROI, the point is it has a big margin of safety and large expected returns on capital. But in a "we can't greenlight this project unless we can directly attribute it to positive motion in our KPI" org, these flavors of idea are dead on arrival. The double-kicker is that the cost of trying to measure these ideas is often greater than the cost of the idea. For some ideas, especially when it's a rounding error on the company's annual budget, it should be OK if we don't invest much effort into quantifying the results. It's one of those rare set of ideas where it's anathema to the professional/investor culture of SV of the last decade, while simultaneously being how plucky Seed/Series A companies can punch above their weight.
  - taeric 3y ago
    I would push back on this specific case, though. What makes this an obvious winner? Netflix was already clearly on the upward swing. At the time of this contest, I'm not even sure I remember who their competition was. Now, I think it is fully fair to say that this was very cheap for them to do. In which case, why not do it? But I think you would be hard pressed to give me a counter factual world that is believable where not doing this had a meaningful impact on Netflix's future.
    - travisjungroth 3y ago
      Hiring. They got a lot of attention for this.
      - taeric 3y ago

gathersnow 3y ago

Netflix became so user-hostile and it is truly baffling to me. They act as if they are paid by amount of time you spend watching movies and have all of these dark patterns shoving the user all over the place and disorienting them. I remember when they had tools to see what your friends were watching and to discover hidden gems which was always such a fun experience. Does Netflix have such content nowadays? I really wouldn't know. They optimize the experience for binge-watching and nothing else. I wish so badly that they'd do a letterboxd thing and allow for curation of their catalog but it all goes back to the ephemeral nature of their content and how they have to hide that fact. They care so little about that approach now it's no wonder they never used this algorithm.

rocky1138 3y ago
I agree. Netflix died the day they switched to thumbs-up and thumbs-down versus 5-star rating content. My recommended list was so solid for so long and all of that went down the drain.
- gathersnow 3y ago
  Can you even vote now? I think they do a lot of it by inferring from what you watch.
- iwontberude 3y ago
  Their death was presaged when their executives decided to "become HBO before HBO can become us" and when they spun out Roku for lack of appetite to be a proper tech company. Reed Hastings wasn't capable of running that business and so their vision shrank to match. Netflix sold a lot of people a lie, their customers, their employees and their investors. We jumped on board thinking Netflix was serious about the living room experience in a wholistic way.
  - anthonypasq 3y ago
    how was netflix ever going to continue being a "tech company." Video streaming is a commodity at this point, they are not longer doing anything technologically revolutionary. Once every other media company could do what they were doing they realized they have to become an entertainment company. It was shrewd foresight on their end, otherwise they wouldnt exist anymore. Unless they tried to license out their infrastructure or something

avelis 3y ago

I remember seeing the leaderboard rankings before the prize was awarded. If I recall correctly the top two teams were very close to the coveted goal. They ended merging to become one team and spit the prize once they achieved the goal.

bilsbie 3y ago

I wonder if it paid off for recruiting though?

n2d4 3y ago

Those contests are never about the actual code/algorithms they produce, it's always about recruiting the people who write them. For the price of $1mil (plus a bit of engineering time) they got a list of talented engineers who have too much time, and also everyone now knows that they are the company with the really challenging problems. Forward the list to recruiting, and you got a bunch of new hires, much more effective than paying for job ads.

deleted 3y ago
[deleted]
deleted 3y ago
[deleted]

ffhhj 3y ago

> It’s now extended beyond the US and in to Canada, 43 Latin-American countries There are about 21 LatAm countries.

bryanrasmussen 3y ago
probably there's an advanced algorithm at work here that says otherwise.

mushufasa 3y ago

in Reed's "No Rules Rules" book, they discuss how this contest was really a way to recruit top-quality engineering talent, which is a key assumption in how they ran their culture (highly paid small teams with lots of freedom, trusted to know what's best from the ground-up rather than top-down). They didn't really need an algorithm in the first place, they just brainstormed what would sound the coolest to developers.

shermantanktop 3y ago
Everything that Netflix says or does externally wrt to tech appears to be for this purpose, including open-sourcing some things. It is all catnip for the HN crowd to come work for them.
- mrguyorama 3y ago
  Netflix pays in the same ballpark as other FAANG companies but the product is so much clearer (stream people movies and shows, maybe recommend them something to watch) and the complete lack of grey ethics like in AdTech or a Monopoly like Microsoft is the real catnip.
  - cybrox 3y ago
    They use a lot of the same concepts as AdTech but only for their own content without shilling for everyone who is willing to pay.
    - Cyphase 3y ago
      I think that context makes a reasonable difference for some people. (The following is a tangent.) To put it another way, using a slightly different kind of situation, I don't want BigCorp's AI watching everything I do to "serve me better", but I'm very interested in the idea of a local AI that watches everything I do and helps me be N times more efficient, while I maintain full control. I don't care if the local AI uses the same data-gathering, data analysis, etc. concepts as BigCorp's AI to accomplish the goals I want it to accomplish. Caveats include: - The local AI is Skinner-boxing you or similar. This isn't about the AI having conscious malicious intent; it could be a design defect. Also, a local, offline, open source video game can still be addictive. - The risk of the system being compromised / manipulated, and all that data on you being exfiltrated.

slt2021 3y ago

This prize/competition has become a meme in itself and inspired hordes and hordes of people to dabble with recommendation systems. The advance of recsys is in big part thanks for Netflix because it attracted a lot of people to the field, all companies started developing inhouse RecSys just like today you hear about inhouse LLM/chatgpt wrappers etc

minimaxir 3y ago

The Netflix Prize competition (2006, completed in 2009) was a Kaggle competition before Kaggle competitions (2010), with the same business-side incentives. Given the meteoric rise of ML/AI in the past few years, I'm surprised that Kaggle doesn't come up more often. It was all the rage 2013-2018...then most I heard about it was that it allowed free access to TPUs.

htrp 3y ago
I feel like kaggle is a copycat wasteland for most projects now. You don't quite get top talent, rather you get a bunch of people xgboosting their way up your leaderboard.
- erhserhdfd 3y ago
  "...rather you get a bunch of people xgboosting their way up your leaderboard" --> I don't see that as a universally bad thing. I feel this is actually very representative of the majority of ML projects that most organizations encounter. Most organizations don't have petabytes of data and a huge compute budget to train a DNN. They typically have megabytes to gigabytes of somewhat crappy data and need something that can be developed and deployed relatively quickly for low cost.
hcarlens 3y ago
There have been quite a few interesting Kaggle competitions in recent years, as well as other interesting ML/data science competitions on other platforms. Platforms like Kaggle, DrivenData, Zindi, AIcrowd, CodaLab and others are running dozens-hundreds of competitions a year in total, including ones linked to top academic conferences. One interesting recent one is this one on LLM efficiency - trying to see to what extent people can fine-tune an LLM with just 1GPU and 24h: https://llm-efficiency-challenge.github.io/ https://llm-efficiency-challenge.github.io/ Or the Makridakis series of challenges, running since the 80s, which are a great testbed for time-series models (the 6th one finished just last year): https://mofc.unic.ac.cy/the-m6-competition/ https://mofc.unic.ac.cy/the-m6-competition/

AeroNotix 3y ago

Most companies spend a ridiculous amount of money for not much at all. The crazy amount of waste in boom times is, frankly, disgusting. You all have witnessed it, some of you may have even tried to stem the tide of horrific spending, some of you are to blame. Go to the billing page of whatever third-party thing you use and just recoil in horror how much you spend.

resters 3y ago

Not surprising. I'd been using movielens for recommendations prior to the prize, and even today Netflix still does not provide high quality content recommendations. Worse yet it seems to think I want to watch the three things that are most popular on the platform. This indicates that the Netflix UX business is being significantly mismanaged.

mvdtnz 3y ago
I suspect the Netflix recommendations system would be excellent if they had a catalog to back it. But the Netflix catalog is of such poor quality that even a perfect recommendation system ends up recommending garbage because it's all they have.
- zoover2020 3y ago
  Isn't their algorithm also very intendedly only catering to Netflix originals by default? I find its algo absolutely rubbish and most of my initial excitement to go watch something with my wife on the couch instantly fades as soon as we face the doomscrolling of finding content to watch. It's like zapping but worse
  - onli 3y ago
    For me it's the missing quality markers. Netflix provides no way at all to see whether a show or movie is good. So when one starts scrolling and does not find immediately something one likes, there is no reason to ever find anything good - because nothing seems good, and all alike. Since I added a userscript to see IMDB ratings directly next to the netflix shows it's way easier to find something promising.
- cortesoft 3y ago
  I always hear people say this about the Netflix catalog, but I never understand what metric or criteria people are using. I find there is a ton of great content on Netflix, and I have way more things I want to watch on there than time to watch it. I always wonder if people who say this either have very particular interests, or are trying to find specific tv shows/movies and not finding them on Netflix. If you are not set on a particular thing, it seems to me that there is a ton of great content.

bazil376 3y ago

Isn’t $1m just like 2 eng salaries for Netflix?

dev-tacular 3y ago
If levels.fyi is to be trusted, it like one L7's total compensation: https://www.levels.fyi/companies/netflix/salaries https://www.levels.fyi/companies/netflix/salaries

rappatic 3y ago

Ironically, Netflix's recommendation algorithm is now notoriously bad. In my experience, it seems to heavily push whatever "original" they just dumped $100 million into as well as what's most popular on the platform at the time. But it makes sense, since people increasingly rely on social media for deciding what to watch. A dollar Netflix spends improving their recommendation algorithm simply won't stack up to the dollar TikTok/Instagram/etc. spends improving theirs, because social media apps have so much more data to work with. It's probably more economical from their POV to let broad social media trends dictate what people watch.

kevincox 3y ago
I suspect their algorithm has other success metrics than "did user enjoy movie". For example if the movie is a Netfix Original or exclusive movie then it is more valuable to Netflix because they may discuss or recommend it, which leads to more signups. Similarly they may prefer newer shows that are more likely to be discussed and raise hype than older movies that even if the user loves them may be less likely to advertise to others.
972811 3y ago
in my experience from the outside it's difficult to separate their rec algorithm quality vs. the quality of their inventory. you're making the assumption they have a ton of great stuff that they're just not showing you, but they may not.
VikingCoder 3y ago
If any streaming service wanted to give my family good recommendations, they would let me select ALL of the people who are watching right now. Husband. Wife. Husband and Wife. Husband and 4th grader. Wife and Pre-K. Husband, Wife, 4th grader, and Pre-K. 4th grader. Pre-K. 4th grader and Pre-K. Each of those has a totally different viewing pattern and preferences.
- evanmoran 3y ago
  This 100%. I promise you that I’m not secretly very interested in Dragon Rider episodes without my kids present. They have their own accounts, but it is hard to share family movies or when they just use the wrong account. I also think half of my viewing is with my spouse. We are pretty aligned in some shows but we are different enough where “My List” really means when I’m alone, and we are missing an “Our List” for when we are together.

zzbzq 3y ago

Using a recommendation algorithm doesn't make any sense for Netflix' new business model where they produce their own content. It should be easy. They greenlight or buy the rights certain content for different demographics to maximize the coverage of their user base that has their needs satisfied enough to stay subscribed. If you're a 18-35 male, they've surely got people there working daily to make sure there's a content pipeline coming just for you. They shouldn't need AI to tell me that, they just need to identify when I'm in the demographic of one of their shows coming, and tell me about it. Maybe like a list. Still, they seem like they're doing a bad job at it, as I never really know what new shows Netflix is making for me. Sometimes I find out about them years later. For example, compare this to Disney+. I vaguely know about every Marvel and Star Wars thing 2 years in advance, and usually vaguely know what order they're coming out, and I don't even know how I know this, somehow I just know. Okay, that's easy because it's basically 2 IPs. But ultimately Netflix is doing something similar behind the scenes, why haven't they succeeded at making me aware of what content I'm supposed to be hyped for?

cqqxo4zV46cp 3y ago
This feels like a disingenuous argument. “Young to middle-aged man likes Marvel and Star Wars” is such an obvious pick that it just about invites parody. As someone with at this point zero interest in Star Wars / Marvel content I a) wouldn’t retain the information that you did and b) wouldn’t find it to be that impressive that D+ was shoving it into my face. To not understand the usefulness of recommendation algorithms almost feels intentionally contrarian.
cybrox 3y ago
To be honest, I would be happy if their 20+ people UX team would create an experience where I can easily find what I was watching instead of shoving stuff down my throat that I have no interest in. Then again, as long as people pay for that experience, it will continue to be as unbearable as it is.
- lubesGordi 3y ago
  This is the problem across the board, regardless of service almost. Everything is some crappy recommendation system, instead of enabling me to find what I want. In addition to sucking in general, recommendation systems have this great property of radicalizing people politically.

dangerboysteve 3y ago

Personally, I never thought they would have in the first place. I was thinking this was more for recruiting.

eachro 3y ago

incentive problem

chris-orgmenta 3y ago

> Well, $1m may sound like a lot of money for Netflix to pay for something that wasn’t really used The title to me sounds analogous to 'I didn't use that piece of code I wrote on the weekend (but I probably learned something from it)'. If it were 10m, maybe relevant? But hardly to their bottom line. https://www.macrotrends.net/stocks/charts/NFLX/netflix/revenue https://www.macrotrends.net/stocks/charts/NFLX/netflix/reven... ... Most companies that size (in evolving industries at least) should have 10-100 of those moonshot/R&D/marketing projects on the go. And I'm speaking to the choir here - exactly: surely most readers of thenextweb would be thinking the same here.

liendolucas 3y ago

I don't get the obsession of some companies to recommend things. I'm happy not only for not paying Netflix but also for not wasting time watching crap. I have my own manual recommender: browse IMDB or Rotten Tomatoes of past directors, cast, etc that I've enjoyed. Also from a good old friend that from time to time recommends me films as we have very similar preferences. Randomly browsing genres worked very well too. That has become a near perfect movie recommender over many years.

slig 3y ago
They push very aggressively for whatever new TV series / movie they just released that is "top 1 in your country". Spotify is the same with whatever crappy playlists and podcasts they want you to listen. YouTube doesn't even bother showing results that you searched past the 10 or so results, it's "content you might like" which is crap and unrelated to anything I've watched, 100% of the time.
- liendolucas 3y ago
  We live in a crap-stream world LOL.
cqqxo4zV46cp 3y ago
Is this a serious comment? Netflix does not and should not base its strategy on how you specifically choose to consume media. Surely you understand that most Netflix users don’t use IMDb/RT at all, let alone as a recommendation ‘engine’. Netflix, especially Netflix back then, is highly reliant upon in-product recommendations because if people sought out specific pieces of media they’d often find that Netflix didn’t have it.
deleted 3y ago
[deleted]
jakearmitage 3y ago
Exactly. Give me proper filters in the search tool, I'll find my own movies.
3y ago

codelikeawolf 3y ago

Hang on a sec, it looks like BelKor's Pragmatic Chaos and The Ensemble had the exact same Best Test Score and % Improvement. Did BelKor win because they submitted their solution 20 minutes earlier than The Ensemble? Or is that Best Submit Time something else?

Irishsteve 3y ago
Yes - from what I remember the team that came second lost out due to something like 6 decimal places and the time diff in submission. It was pretty close!
- codelikeawolf 3y ago
  Wow! That's bananas! Imagine being 20 minutes and 6 decimal places away from a million bucks.
  - thaumasiotes 3y ago
    Being six decimal places away from winning means the winner was one million times better than you.
    - bagels 3y ago
      They're talking about 0.999999 vs 0.999998, or ~1 part in 1 million different
  - Irishsteve 3y ago
    I might be mis-remembering this part - but I believe napoleon dynamite had its own algorithm in the ensemable because it was a tricky one to predict the rating for.

gyudin 3y ago

Netflix just needs decent movies in it’s library :)

deleted 3y ago

[deleted]

tunesmith 3y ago

Interesting that the answer is basically "because of akrasia". I wish more of these recommendation algorithms were based more off of who we'd like to be rather than how we behave.

softwaredoug 3y ago

As someone who works in search relevance, having just a great algorithm isn't worth much. You need all the team and know-how that has the maturity to maintain such an algorithm. Not just the ML skills. But all the bazillion ops, data quality, and many other things that go around it. I've worked with a lot of teams that have one smart person building stuff off to the side, in R or a Notebook, and then nobody knows how to productionize it. They try to throw the algorithm over the fence. Even if the team somehow succeeds in getting it into an A/B test, it eventually falls by the wayside, unless they can build the team and workflow around that person / algorithm / methodology

whimsicalism 3y ago
> I've worked with a lot of teams that have one smart person building stuff off to the side, in R or a Notebook, and then nobody knows how to productionize it. They try to throw the algorithm over the fence. Even if the team somehow succeeds in getting it into an A/B test, it eventually falls by the wayside, unless they can build a team and workflow around that person / algorithm / methodology. In my view, this marks a cultural failure - and certainly not on the part of the 'one smart person.'
- softwaredoug 3y ago
  100%. I totally agree. I've both been the person and the team here. And it suucks when you see how things could be done better.
geoduck14 3y ago
You are definitely right. The problem to solve quickly becomes process and not technical

LudwigNagasena 3y ago

Anyone who has experience with ML wouldn’t be surprised by that. Oftentimes ML competitions are about combining dozens of models together to juice the extra 0.01%—something that isn’t viable in a production environment as the quote in the article confirms. AFAIK, modern ML challenges try to combat that by moving from answer-only submissions to code submissions and putting constraints on compute.

bob_theslob646 3y ago

Dr. David Belanger (RIP) was a part of advising this team (BelKor)and knew all about this, but never ever bragged about until one day I stumbled upon it and asked him about his involvement. https://www.thrillist.com/entertainment/nation/the-netflix-prize https://www.thrillist.com/entertainment/nation/the-netflix-p... Also, they definitely used code from the first two competitions in the company. "The first year of the competition, in 2006 and 2007, the technical advancements that were made by us and the other big teams I think were really significant in the field of recommender systems," says Volinsky. He thinks the idea that Netflix didn't use the results is a misconception. "We gave them our code. They definitely did implement and use those breakthroughs that we made in the first year." For more that are interested , I can speak about it to the best of my ability. https://ieeexplore.ieee.org/author/38180399800 https://ieeexplore.ieee.org/author/38180399800 ( that is some things about him) Essentially he would say that they initially shelved it over time as it was not needed, but they definitely used. Belanger tragically died November 18, 2022.

gwern 3y ago
Their 2015 paper on how they were doing recommenders also mentions that they used bits and pieces of the prize work: https://gwern.net/doc/reinforcement-learning/exploration/2015-gomezuribe.pdf https://gwern.net/doc/reinforcement-learning/exploration/201...

nashashmi 3y ago

Engineers called it "Not worth my valuable time" kind of engineering effort. An early version of the 10x engineer (who refuses to do lots of meaningless work).

aaron695 3y ago

[dead]

gwbas1c 3y ago

I remember meeting someone at a networking event who believed they could win the competition and get > 90% accuracy. I tried to explain that you can't read minds; you can't account for the fact that I'm going to watch a movie that I saw in a random internet post that I didn't know that I liked... Which was my real beef with the contest: Computers can't read minds. They can't analyze the content of the movie and have the emotional experience that a person can.

robrenaud 3y ago
Computers are learning to read minds. https://amp.theguardian.com/technology/2023/may/01/ai-makes-non-invasive-mind-reading-possible-by-turning-thoughts-into-text https://amp.theguardian.com/technology/2023/may/01/ai-makes-...
- stonogo 3y ago
  This article has no bearing on the topic at hand.
anjc 3y ago
> Computers can't read minds. They can't analyze the content of the movie and have the emotional experience that a person can No but you can read minds by proxy, via ratings, which is what the dataset consisted of.
mr_toad 3y ago
> They can't analyze the content of the movie They probably could. Apple photos is already capable of judging the quality of your photos, so it’s not too much of a leap to think that a model could be trained to recognise good from bad movies. On the other hand there are far fewer movies than photos for training on so it’s hard to say how good the results would be.

jfengel 3y ago

Streaming gives them much better information than a user's voluntary stars. Did you actually watch it? All of it? How soon? All at once? User ratings are fraught. The things a user actually does are more likely to be sincere. And it doesn't put any burden on the user. I have a feeling that even the simple thumb up/down means less than whether you actually finished watching the movie.

theshackleford 3y ago
To bad they did'nt use any of that information they have on me to recommend me things I actually wanted to watch. They just kept pushing the same shit I never once displayed intention in watching over, and over and over again. Sounds like everybody is winning other than me. Though given I cancelled and never looked back, maybe I am winning afterall.
- jfengel 3y ago
  The problem is that they just don't have enough content you want. I think they'd give it to you if they had it. They're not hiding it. They just have so much catalog and when you've watched the good parts, cancel.

gverrilla 3y ago

Netflix feels like Blockbuster did some years ago.

Infinitesimus 3y ago
How so?
- gverrilla 3y ago
  swallowed about-to-die off-hype (my own opinion of course)
deleted 3y ago
[deleted]

deleted 3y ago

[deleted]

fsckboy 3y ago

does anybody here know a lot about recommendation algorithms? The one I'm interested in seems obvious to me, but it's never implemented so maybe it doesn't work, or it serves outliers and not the mainstream so why bother. The data is there, but I don't get to mine it. (heheh "mine" it) I don't want to know what's popular, I want to know what other people who share my past tastes are watching currently. Doesn't matter the size of my niche, why can't i find my niche, and be shown my cohort? I just don't feel it's an impossible task to find, for example, a set of people who enjoyed Deadpool 1, Guardians of the Galaxy 1, but think the entire rest of the superhero cavalcade is utter garbage. ok, I'll allow Kickass also, and a grudging nod to Iron Man 1. (as an illustration, I just searched to look up the name of Iron Man, I could only remember Tony Stark. Top search results were "a fictional character", "played by R Downey Jr", "Marvel cinematic universe"... You see, none of those is Iron Man 1. I don't care about the horde of people who care about a supposed cinematic universe. We (in my cohort) liked the first one, and what we liked about it wasn't continued in the 2nd. How about preferences that show I like a few early films with say George Clooney, but after that, no. Julia Roberts, same. I remember when Meryl Streep was a new actress, but now she's a red flag. I'm sure my tastes aren't shared by everybody, but I'm also sure, since I can articulate reasons, there must be others who in general share them, people who are willing to watch what's new and trendy in some way, but then not beat the dead horse. Oh, Spielberg, a few winners, but overall "nope nope nope", the way he aims at sentiment is a zero for me. I originally had this idea back in the 80's when Consumer Reports had their monthly mail-in bingo card for what current movies you liked, then they would tabulate what was popular among their readers. That type of "best of" list is entirely missing co-variance, which I think is where I live.

anjc 3y ago
That's called collaborative filtering (https://en.wikipedia.org/wiki/Collaborative_filtering https://en.wikipedia.org/wiki/Collaborative_filtering) and is perhaps the most battle-hardened and most effective approaches in recommender systems. Even now, novel deep learning approaches implement the concept, but simple naive approaches are still as/more effective. The first paper published on it specifically in the field was in the 90s but the seeds of it go back to the 70s. It would have made a good thesis in the 80s :) Part of why it's so effective are for the reasons you outline, in that it can find items that you'll probably like, that aren't similar to items you already like, based on people that have similar tastes to you.

jameshart 3y ago

[flagged]

deleted 3y ago
[deleted]

mav3ri3k 3y ago

> “We have adapted our personalization algorithms to this new scenario in such a way that now 75% of what people watch is from some sort of recommendation,” says Netflix I am always sceptical of these claims. Today algorithms have become ai models. Sorting algorithms are recommendation engines.

j2kun 3y ago

This article directly contradicts an article written by Netflix engineers Xavier Amatriain and Justin Basilico, who were in charge of putting the BellKor algorithm in production: https://netflixtechblog.com/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429 https://netflixtechblog.com/netflix-recommendations-beyond-t... > But once we overcame those challenges, we put the two algorithms into production, where they are still used as part of our recommendation engine.

thefringthing 3y ago
This article links to that blog post and quotes from it.