12 ms·
Netflix's Metaflow: Reproducible machine learning pipelines
- toxik 6y agoEdit: this is a somewhat OT rant. Netflix’s recommender system is hands down the worst I have ever seen. Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. Even if I watch the first of a trilogy (LotR, for example). How can they be so terrible at this? The categories in the main browsing view are also hysterically arbitrary. It kind of looks like a topic model with back-constructed titles for the topics. Finally, they replaced the ratings with “% matching”. I guess so they can recommend their subpar productions even if they get low ratings.
- ed25519FUUU 6y agoDo you have a counter example of a streaming service that does recommendations better?
- taeric 6y agoI read this as a condemnation of recommender systems in general. That or a ceding of recommendations to marketing. That feels too cynical, but more accurate.
- xiphias2 6y agoRecommendations maybe not, but I love that in HBO I can sort by IMDB score to find great movies/series that I haven't watched yet. I just watched Broadwalk Empire as an example, and loved it. Netflix doesn't show IMDB scores, so I have to check it always independently...it sucks.
- DoofusOfDeath 6y agoBack when I used Netflix primarily for DVDs, the recommender system worked pretty well for me. Much later, when they switched to simple thumbs up/down, the recommender system was entirely useless to me. (Not merely because of the dumbed-down rating system; the recommendations were genuinely bad.) For the time in between, I'm not sure if the degradation was gradual, sporadic, or not degraded at all.
- treis 6y agoIt's pretty clear that their recommendation system and broader UX is designed, at least in part, to obfuscate how much content they have and how good it is. Back when it was DVDs and they had basically everything it was more about finding the next best thing for you. Now it's finding the next best thing they have and, preferably, something they own the rights to.
- ramraj07 6y agoIf you add dvd subscriptions you can still get the original Netflix recommendations back, including sorting by top predicted rating (which to me is eerily amazing). I used to keep the dvd subscription mainly for the recommender, with the delivered blurays considered an extra bonus for very rare movies you couldn't stream even if you wanted to pay.
- qw3rty01 6y agoNot a streaming service, but a lot of movie and TV databases tend to have decent recommendations for movies and shows similar to the one you're viewing. Although in this case, one could argue that the processing is offloaded to users who provide the recommendations.
- Reubend 6y agoThe MovieLens recommender system works pretty well.
- jointpdf 6y agoIt’s not a home run for me, but I think Spotify recommendations are quite good. They clearly use some form of content based recommendation (extracting features from the music itself) blended with other methods. It seems to make an honest attempt at serendipity (songs/artists you may like but would otherwise be unlikely to discover). I still think recommender engines should always enable some form of user tuning. If it doesn’t, then the recommender is a tool for services to control your behavior rather than the other way around.
- octostone 6y agoUnless I’m missing this functionality somewhere (entirely possible), this lack of user-tuning has ruined Spotify recs for me, since I listen to entirely different playlists when working or meditating. I don’t check out recommendations to get the latest binaural beats or nature sounds, you know? Although even before I started listening to Spotify while working etc, it seemed to have run out of things to recommend. My weekly discover playlist would be half things I’d already liked. So...who knows. But I miss the discovery functionality quite a bit.
- jointpdf 6y agoI personally have a similar problem with the Spotify generated playlists, but I have persnickety preferences in electronic music so it often whiffs. But on the spectrum of recommendation engines it is on the side of honest effort (whereas Netflix is not). And for most users and musical palates I think it works great. Just yesterday my mom complimented me on my Christmas music DJ skills, but it was just the generated continuation of her own playlist. Nothing really beats the recommendations of a human curator with exquisite taste, and just listening to new music nonstop and plucking out the gems as you go.
- nwsm 6y agoI like Spotify's recommendations and have found lots of great artists from it. But now I feel I'm in a kind of Spotify-created rut of listening to static groups of artists. I have mostly stopped listening to my daily mixes because of this. It's also easy to theorize about "big media" controlling what I listen to, so I still feel the need to do my own exploring, even when I'm getting recommended good fresh stuff.
- hogFeast 6y agoSpotify. Discover Weekly is very good, particularly considering they have only one chance to recommend with 30-40 songs, and quite a large universe to match on. Not a streaming service but Google's Discover news is also very good (probably the best recommendations I have come across).
- toxik 6y agoThe basic problem is understanding WHY I liked something. If I watch Tintin because it’s a cozy throwback to my childhood, that does not mean I would like every single Studio Ghibli movie in my recommendations. Similarly, if I play Blacklist in the background as basically noise, I don’t want to see a bunch of related shows. I guess I could give it a thumbs down but I only do that for actually terrible movies. Also Spotify and Apple music seem to have okay recommendations
- kyawzazaw 6y agoMusic is different from movies and television though. There are beats and rhythms that are easy to identify, lyrics easy to analyze and artists that are roughly categorized.
- toxik 6y agoI feel like that’s not true, music is hard to analyze. Movies have synopses, and are generally categorized by their cast alone. I could figure out LotR belongs together from these datapoints, easily. Not to mention the fact that people watch trilogies in sequence like 99% of the time.
- JohnJamesRambo 6y agoSpotify's Discover Weekly has a pipeline straight into my brain and not only what I like, but what I will like in the future.
- Hard_Space 6y agoI long ago ceased to believe that the Netflix recommender system serves any other purpose than to fulfill the company's internal obligations to push favored content, depending on what it cost. Sadly, the same is now true for Amazon Prime, which is an even hotter mess.
- umvi 6y agoI turned on "Super Wings" for my kid to watch on Prime Video which at first glance seemed to be a fairly decent Paw Patrol knock off, but then as I listened to the episodes in the background, I realized that the entire show is basically an advertisement for Amazon Prime in disguise. Seriously look it up, the entire premise of the show is people ordering packages and the "Super Wings" delivering the packages to the consumer as quickly as possible... .
- alisonkisk 6y agoSeems like you got what you expected... Super Wings is the corporate side of the Government-Corporate complex.
- borroka 6y agoThat's exactly what's happening. Source: you can imagine.
- visarga 6y agoNot just the recommender, but the design is horrible for me. I can't rest my mouse anywhere, it auto-starts something or enlarges something, it's all too twitchy. When you're watching an episode, there's no navigation link from the play screen to the main page of the series. As if they don't want us to navigate the site, instead be led on their happy path.
- kyawzazaw 6y agoI think so. This way, you don't easily arrive to the conclusion the items I want to look for all don't exist here.
- freetonik 6y agoGood news is that you can disable autoplay of previews [0]. Bad news is that it takes weeks to propogate this setting change to all clients. I had to wait about two weeks for my apply tv client to stop auto previews. I suspect the queue service is powered by snail mail. [0] https://help.netflix.com/en/node/2102 https://help.netflix.com/en/node/2102
- emidln 6y agoTheir recommender system likely has multiple inputs that are under specific constraints and weightings. How well their recommendation algorithms work will probably never be known by the general public as we are force-fed a steady diet of whitelisted staff picks and promotional items.
- typon 6y agoSo what's the point of making tools like Metaflow? Justifying salaries for their engineers I suppose?
- dessant 6y agoNetflix often promotes new content by overzealously recommending it, the constant suggestion you see for The Queen’s Gambit is probably an ad.
- drstewart 6y ago>Finally, they replaced the ratings with “% matching”. I guess so they can recommend their subpar productions even if they get low ratings. That's always what ratings were. People didn't understand that (as you can see), so they changed it to make it more transparent. https://www.businessinsider.com/why-netflix-replaced-its-5-star-rating-system-2017-4 https://www.businessinsider.com/why-netflix-replaced-its-5-s... >Netflix’s star ratings were personalized, and had been from the start. That means when you saw a movie on Netflix rated 4 stars, that didn’t mean the average of all ratings was 4 stars. Instead, it meant that Netflix thought you’d rate the movie 4 stars, based on your habits (and other people's ratings). But many people didn’t get that.
- alisonkisk 6y agoIt's not just that (obviously, since It's not even possible for me to rate a movie more granular than thumbs up/down) What happened is that the notion changed from "predict a scalar rating" to "predict a binary satisfaction." As parent poster noted, the effect of this is to push "3 star" and "4 star" acceptable shows to the user, instead of "5 star" great (in the user's view) shows. Also, in Netflix's defense, users are horribly inconsistent in their expressed ratings (they'll rate a movie based on their personal mood at the time, and they'll binge shows they claim aren't 5 stars while ignoring their 5 star movies)
- mikeyjk 6y agoThis was a sad turning point for me. We used to have a single streaming platform with an awesome library, a granular review system, and user reviews. You could easily take a quick look to read other users thoughts on a film. Now I have to lookup metacritic / reviews myself, and the NF recommendation is based on whether I've said something is 'palatable enough to watch, isn't terrible, but I'd never watch it again' (thumbs up). I've taken to only thumbs upping stuff that I particularly like to see if that's any better. It all seems to be the same from an uninformed end user perspective. I remember they stated the reviews would still be available in some form for export. It's no longer the Netflix of old imo.
- 6y ago
- hateful 6y agoI really think that if they gave up all the complicated algorithms and went with a simple algorithm out of the 90s we'd be much happier with the recommendations.
- alisonkisk 6y agoDo you have such an algorithm? Netflix doesn't make money by showing people stuff they don't like.
- toxik 6y agoNot for a lack of trying though...
- danielscrubs 6y agoThey actually might... think of it like gym memberships. Sure you need a hook, but as Mandalorian showed, that can be a single exclusive tv show that doesn’t require much broadband cost. Maybe not the best solution long term but since when has that been the shareholders goal?
- deleted 6y ago[deleted]
- AnssiH 6y agoI see this complaint about poor recommendations very often. But the recommendations seem to work perfectly for me - I wonder whether that is the case for the silent majority? The match % is usually spot on for me, and I've never seen it recommend any titles I've given a thumbs-down to.
- 1123581321 6y agoFor many of the complainers, Netflix is showing them something they would actually enjoy. How to communicate that is challenging. I remember Pandora’s generated stations having a similar issue years ago, where it would catch similarities between on-trend bands and bands that had fallen out of favor. It was right, but it could sometimes be hard to submit to the insight of the algorithm. Spotify is one that I think does this well as they understand how not to give a recommendation that insults or offends, even if it’s technically an accurate response.
- ska 6y ago> recommender system is hands down the worst I have ever seen. In my limited experience, they are all dismal and best ignored entirely. When it was easier to see something plausibly like a real user review, that helped I guess.
- dddbbb 6y agoMaybe it's different in other countries (I am in the UK), but I feel as if there isn't enough content for a recommender system to even be useful. I feel like after browsing through the catalogue a few times, I have a rough idea of most things I would ever possibly be interested in. There's just not that much there. Either that, or the recommender system is working too well and I never see anything beyond what Netflix wants me to.
- phillc73 6y agoI have to agree. As far as I recall the Netflix search engine has never found the exact title I've been searching for. Even the recommendations it then shows me are so far off piste that they're not really even close to what I'm looking for. At least with Amazon Prime there's a high chance they'll at least find the title one searches for and if I am really motivated I can pay a few bucks to watch it. Netflix just draws a blank. That's not to say I haven't watched some entertaining things on Netflix, but they seem much better suited to TV series than movies, and I almost feel like I found decent things to watch inspite of their recommendation system, not because of it. An ML tool from Netflix? It feels like the last thing I'm likely to use.
- johncena33 6y agoCan we please take these off-topic rants somewhere else? Lately, it's hard to find any insightful comments in the disucssion section of engineering or technical articles. Because the whole conversation is derailed by something unrelated off-topic rants. There are a lot of social media, forums and even HN submissions where everyone can rant. Please keep the technical submissions clean.
- tsjq 6y agoFully agree with this. Yuck, Netflix recos are ultra bad.
- Havoc 6y ago>Netflix’s recommender system is hands down the worst Until you log into prime video. Can’t manage to give me a “continue watching last thing button”. That’s literally the most likely thing I want to watch. Also routinely suggest starting with S02 even though I’ve not watched S01. Never mind machine learning some common sense would be greatly appreciated
- bra-ket 6y agoNetflix actually organized a major machine learning competition more than a decade ago[1] with thousands of the best researchers trying to beat some internal benchmark by a few percent, for $1M prize, I'm wondering where did all that "learning" go. My guess is that whatever rankings they currently produce maximize some internal revenue target and that target' user base is not me or you. [1] https://en.wikipedia.org/wiki/Netflix_Prize https://en.wikipedia.org/wiki/Netflix_Prize
- Havoc 6y agoYup. It just ends up irritating users though if the suggestions break basic common sense. If I watched S01E04 yesterday I want to watch S01E05 today. The interface should be suggesting that PLUS whatever else the ML comes up with in addition, not instead of. Prime is full of minor irritations like that make me wonder whether Amazon engineers dogfood enough
- mr_toad 6y ago> Every single thing I watch, it suggests The Queen’s Gambit and two other random Netflix productions. I always assumed those three were human selections, not part of the recommendations. Like the paid ads on top of your search results.
- jerzyt 6y agoNetflix's biggest problem is not the recommendation engine, but their lack of content. If they allowed you to filter out the junk you don't want to see and what you've already seen, there would be hardly anything left. They've been losing rights to stream left and right. That's why there's such a rush to produce their own content in other countries. Some of that is halfway decent, but there's a lot of formulaic, repetitive stuff. And while saying that, I appreciate their high level technical staff. These decisions are made by bean counters.
- weego 6y agoI don't think it's the worst, it's just more that you've made the assumption that Netflix wants a pure set of recommendations. Marketing and business drivers will always trump algo results, so what you're seeing are mostly artificial boosts given to globally 'hot' properties.
- vtuulos 6y agoIf you are curious about how Netflix uses Metaflow to power behind-the-scenes machine learning, take a look at this recent blog article https://netflixtechblog.com/supporting-content-decision-makers-with-machine-learning-995b7b76006f https://netflixtechblog.com/supporting-content-decision-make... Also I'm happy to answer any questions (I lead the Metaflow team at Netflix).
- itamarst 6y agoHey, been meaning to reach out. There's a bit in the Metaflow docs that talks about choosing resources, like RAM: "as a good measure, don't request more resources than what your workflow actually needs. On the other hand, never optimize resources prematurely." The problem is that for memory, too little means out-of-memory crashes, so the tendency I've seen is to over-provision memory, which ends up getting very expensive at scale. This choice between "my process crashes" and "I am incentivized to make my process organizationally expensive" isn't ideal. Do you have any ways you deal with this at Netflix, or have you seen ways other Metaflow users deal with it? I have some ideas on how this could be made better (some combination of being able to catch OOM situations deterministically, memory profiling, and sizing RAM by input size for repeating batch jobs), based in part on some tooling I've been working on for memory profiling: https://pythonspeed.com/fil https://pythonspeed.com/fil, so would love to talk about it if you're interested.
- deleted 6y ago[deleted]
- vtuulos 6y agoI'd love to hear more what you have in mind! Feel free to drop by at our chat at https://gitter.im/metaflow_org/community https://gitter.im/metaflow_org/community While it is true that auto-sizing resources is hard and the easiest approach is to oversize @resources, the situation isn't as bad as it sounds: 1) In Metaflow, @resource requests are specific to a function/step, so you end up using resources only for a short while typically. It would be expensive to keep big boxes idling 24/7 but that's not necessary. 2) You can use spot instances to lower costs, sometimes dramatically. 3) It is pretty easy to see the actual resource consumption on any monitoring system, e.g. CloudWatch, so you can adjust manually if needed. 4) A core value proposition of Metaflow is to make both prototyping and production easy. While optimizing resource consumption may be important for large-scale production workloads, it is rarely the first concern when prototyping. In practice at Netflix, we start with overprovisioning and then focus on optimizing only the workflows that mature to serious production and end up being too expensive if left unoptimized. It turns out that this is a small % of all workflows.
- pknerd 6y agoI am trying to figure out Kubflow. Surprisingly I found this one easier to write. Haven't run it or used it yet.
- savin-goyal 6y agoTake a look at metaflow.org/sandbox if you want to test drive Metaflow.
- manojlds 6y agoIt takes me into a verification and waiting flow. Useless.
- savin-goyal 6y agoGive it a few minutes :)
- the_duke 6y agoSetting up a decent, comprehensive, self-hosted (!) ML environment is still extremely, frustratingly difficult. What I really want is a single solution, or a set of pluggable, integrated components that offer: * training data and model storage (on top of a blob store like S3, minio, ...) * interactive dev environments (Notebooks, dev containers, ...) * training (with history, comparisons, parameters, ...) with experiments for parameter tuning * serving/deploying for production * a permission system so researchers and developers can only access what they are supposed to * software heritage, probably via Docker images or Nix packages, combined with source code references * (cherry on top: some kind of integrated labeling system and UI) Right now you have to cobble this together from different tools that are all pretty suboptimal. The big players can set up sophisticated systems, but I'm curious to hear how other startups are currently solving this.
- mmq 6y agoHi @the_duke, disclaimer I am one of the authors of an open-source solution (https://github.com/polyaxon/polyaxon https://github.com/polyaxon/polyaxon) that specializes in the experimentation and automation phase of the data-science lifecycle. Our tool provides exactly the kind of abstraction you mentioned: * Training, data operations, and interactive workspaces (https://polyaxon.com/docs/experimentation/ https://polyaxon.com/docs/experimentation/) * A scalable history and comparison table (https://polyaxon.com/docs/management/runs-dashboard/comparison/ https://polyaxon.com/docs/management/runs-dashboard/comparis...) * Currently pipelines and concurrency management is on the commercial version (https://polyaxon.com/docs/automation/ https://polyaxon.com/docs/automation/) but several companies use Polyaxon with other tools like Kubeflow (https://medium.com/mercari-engineering/continuous-delivery-and-automation-pipelines-in-machine-learning-with-polyaxon-and-kubeflow-d6a3668715de https://medium.com/mercari-engineering/continuous-delivery-a...) or it can be used with MetaFlow for the pipelines part. I would really like to hear your thoughts and feedback.
- manojlds 6y ago> but several companies use Polyaxon with other tools like Kubeflow or it can be used with MetaFlow for the pipelines part. Isn't that what the parent you are replying to is talking about with "Right now you have to cobble this together from different tools that are all pretty suboptimal."
- smurf_t 6y agoWhat are the main advantages compared to Airflow. We use Airflow to orchestrate ML jobs/tasks, and I found it to be more flexible comparing other tools we tested.
- vtuulos 6y agoMetaflow is largely complementary to a job scheduler like Airflow. Technically you could export Metaflow workflows to Airflow, although the specific integration doesn't exist yet. For more details, see this blog article https://netflixtechblog.com/unbundling-data-science-workflows-with-metaflow-and-aws-step-functions-d454780c6280 https://netflixtechblog.com/unbundling-data-science-workflow...
- dustinhopkins 6y agoHow does this compare against TensorFlow Extended (TFX)? https://www.tensorflow.org/tfx https://www.tensorflow.org/tfx
- savin-goyal 6y agoMetaflow was built to assist in both developing ML models and deploying/managing them in production. AFAIK, TFX is focused on the deployment story of ML pipelines. https://docs.metaflow.org/introduction/what-is-metaflow#should-i-use-metaflow https://docs.metaflow.org/introduction/what-is-metaflow#shou...
- dustinhopkins 6y agoIt's focused on building ML pipelines (similar to what Cortex aims to be). In addition, it also conveniently supports integration with Orchestrators like Airflow, Kubeflow, Beam etc. This book "Building Machine Learning Pipelines: Automating Model Life Cycles with TensorFlow" (https://www.amazon.com/dp/1492053198/ref=cm_sw_em_r_mt_dp_Igq4Fb9BK0Q3X https://www.amazon.com/dp/1492053198/ref=cm_sw_em_r_mt_dp_Ig...) goes into great details. I was curious to see what advantage Metaflow offered over TFX.
- Peteris 6y agoIf you are feeling overwhelmed with yet another machine learning pipeline automation framework, you should check out Kedro (https://github.com/quantumblacklabs/kedro https://github.com/quantumblacklabs/kedro). Kedro has the simplest, leanest, functional-programming inspired pipeline definition and also spits out AirFlow and other formats readily + comes in with an integrated visualisation framework which is stunning & effective.
- sysprogs 6y agoThis looks similar to Google's Mediapipe [0] [0] https://github.com/google/mediapipe https://github.com/google/mediapipe
- spicyramen 6y agoIs there a comparison with TFX, KubeFlow pipelines and AirFlow?
- sarak12070 6y agoMan stacks 79 poker chips, breaks 47th Guinness record of 2020 https://interestingnewsclub.blogspot.com/2020/12/man-stacks-79-poker-chips-breaks-47th.html https://interestingnewsclub.blogspot.com/2020/12/man-stacks-...
- sandGorgon 6y agohas anyone done a comparison of ML pipelines from a devops centric perspective ? For example, Metaflow doesnt support kubernetes today - https://github.com/Netflix/metaflow/issues/16 https://github.com/Netflix/metaflow/issues/16 so ultimately the scale up story in most of these management tools is iffy. I previously asked about kubeflow here - https://news.ycombinator.com/item?id=24808090 https://news.ycombinator.com/item?id=24808090 . Seems people think its pretty "horrendous". It seems most of these tools assume a very specialised devops team who will work around the ml tool...rather than the ml tool making this easy.
- vtuulos 6y agoAs of today, Metaflow supports AWS Batch for scaling out and up. While supporting K8S is convenient in the operations point of view (assuming you have a K8S cluster already), it doesn't make the scalability story any better.
- sandGorgon 6y agosure - im not talking about scalability from a science perspective, but really from practical applicability perspective. Most people already are running k8s of some kind. I beginning to see k8s increasingly as an invariant. Plus not all of us are on AWS. Secondly, AWS Batch is only applicable for metaflow for ml training.
- nicktorba 6y agoThis would be super useful. Based on this thread, the comparison should include * metaflow (model training on AWS Batch) * polyaxon (model training on kubernetes) * pachyderm (experimentation) * hopsworks (model training/serving/ and more, mostly on kubernetes) * cortex (model serving on kubernetes) * seldon-core (model serving and monitoring on kubernetes) and likely more that I missed. I can see why it would be so hard to put together this comparison. Even with all these tools, there is still a lot of manual work for data scientists or DevOps engineers the data scientists pass their models off to. It also seems there is yet to be a fully open source DevOps stack. Most companies still build custom software to glue together manual processes (like integrations between different tools for training, deploying, monitoring, etc). This could be one factor why more comparisons of these tools and stack discussions have not been more popular - they can't share them yet.
- nicktorba 6y agoI really like the combination of these two tools. I've played with cortex before, and it is easy to use, but I am still questionable if automating kubernetes deployments through an easy code interface, without much kubernetes know-how, is safe. In my experience, even when you have a tool automating a lot of kubernetes for you, you will still run into trouble that will be best handled if you are familiar with kubernetes. I'm not sure what debugging utilities cortex has, but I think the ultimate solution to this problem will be a tool that truly allows users to not think about the fact their deployments are running on kubernetes at all. I'm also interested in the similarities of Cortex and Seldon-core. Of course, seldon-core does not automate infra provisioning, but based on my previous point, I think many teams are better off being more hands on with this infra. Lastly, there is a third tool missing from the mix - monitoring. I think cortex offers some tools in this area, but I wish they would make a part two showing how the monitoring functionality they offer can integrate into a retraining pipeline within metaflow. This post shows you how to get started, but it doesn't show you how to maintain applications long term.