26 ms·
Python at Netflix
- 1ba9115454 8y agoMost of these use cases would be better served with a type safe language.
- vturner 8y agoDown voted you will be, the dynamic typing sheriffs are on the watch ;)
- chadash 8y agoI didn't downvote the parent, but I think that doing so is justified because the comment doesn't add much to the conversation, whether you agree with it or not. To be a substantive comment, it should explain _why_ typed languages would be a better choice for these particular use cases.
- tsss 8y agoAt this point it should be common sense, there are arguments about this in practically every thread that mentions Python or JS. As much as I hate dynamic typing, there are use-cases where I don't care about it much. What makes Python truly an awful experience in any case is the completely asinine dependency management.
- mirceal 8y agohaha. if you think dependency management is bad in python you should take a look at go :O
- coldtea 8y agoDownvoted grantparent because it's a non-argument comment, that offers nothing to the discussion, and based on nothing.
- ahoka 8y agoJust like yours.
- sametmax 8y agoI read a lot of data analysis, prototyping, configuration, monitoring and deployment, for which Python is very, very well suited. So I don't follow how you got there.
- bouncing 8y agoI'm going to start a drinking game every time someone says that.
- tclancy 8y agoYou'll want to get on a liver donor list now.
- cuddlecake 8y agoWill you start the game everytime someone says that, or will your game be to drink everytime someone says that? Honestly curious what your methodology is.
- bouncing 8y agoDrink every time someone says that. See you at rehab!
- luord 7y agoIt's been going on for sixty years (something the defenders of both paradigms often seem to ignore) and will never end, you'll die.
- skywhopper 8y agoThis isn't a very useful comment in this form. What aspects of these specific use cases would be better served by a type-safe language? Why is type safety the most important attribute of language choice in this scenario?
- throwaway55554 8y agoQuestion: If they would be, don't you think Netflix would go that route? "Better" is contextually dependent. There's no silver bullet to anything we do in tech.
- jniedrauer 8y ago> don't you think Netflix would go that route? Not necessarily. Python has a few huge advantages that aren't (directly) related to the quality or utility of the language. For example, hiring and training people to use Python is going to be easier and cheaper than hiring and training people to use C++. Additionally, as mentioned in the article, Python has a massive and healthy ecosystem of high quality packages. And then there's the cost of porting an existing codebase to a new language...
- throwaway55554 8y agoThen one could argue Netflix is using the best tool for their purpose, so a type safe language wouldn't be 'better'.
- 781 8y agoNetflix are famous for paying the top salaries in the industry, you will take a paycut leaving them. Programmer cost is absolutely not a reason for anything done there. https://www.salaryproject.com/salaries/company/netflix/senior-software-engineer https://www.salaryproject.com/salaries/company/netflix/senio...
- weberc2 8y agoYou're supporting the parent's point that it's contextually dependant.
- inglor 8y agoYou think you're being downvoted because people like dynamic typing very much. You're being downvoted because: A) You left a comment with a strong opinion without expressing it while dismissing other peoples' experience B) Python is in fact a type safe language for a while now https://docs.python.org/3/library/typing.html https://docs.python.org/3/library/typing.html - you have pluggable "a la bracha" types.
- enitihas 8y agoI think you meant dynamic typing and not dynamic programming.
- 0815test 8y ago> Python is in fact a type safe language for a while now Well it has a typing mechanism, which is better than nothing, but has the mechanism been shown to be safe? C has types too, but I'm not sure that it would be described as type safe.
- inglor 8y agoShowing a typing mechanism to be safe is a very hard thing to do. There are a few methodologies I'm familiar with. What most people think about - formal proof - doesn't work for semi-obvious reasons (as you can easily reduce "is this program safe" to the halting problem). The research I've seen is basically: - Take a class room of independently selected subjects (read: CS undergrad students). - Give them the same coding problem in multiple languages or in the same language with or without compile time type checking. - Measure the number of defects said programs have and hopefully draw a conclusion about the safety of the language. All the research I've read on the topic has been pretty poor (though maybe I'm just bad at finding it). Indeed there is no such research for typed Python (although the pluggalbe types research in general is great) but there is also no such research (that I've found) for Idris, ReasonML, Swift, Haskell, Rust or any of the usual "suspects" for "good types". Please enlighten me :]
- mirceal 8y agoyou may be the exception, but most people i’ve met that make this claim have no idea what they are talking about. how would the cases be better suited for a type safe language? what is type safety, in your opinion?
- sara12318 8y agoWorld's biggest cat Even you can't believe it's size See her video https://howto105.blogspot.com/2019/04/worlds-biggest-cat.html https://howto105.blogspot.com/2019/04/worlds-biggest-cat.htm... See the video a dog shopping and lots of other acts like humans http://bit.ly/2DCpo9I http://bit.ly/2DCpo9I Fish walking with legs like humans See the video on link below http://bit.ly/2XRGyYD http://bit.ly/2XRGyYD This couple is different Girls are both twins and boys are also twin brothers see how they look in video http://bit.ly/2VwoZQd http://bit.ly/2VwoZQd A cute dog is dancing in video Just watch this video and try to control your laugh http://bit.ly/2ZK1XVs http://bit.ly/2ZK1XVs
- bredren 8y agoInteresting to see a touch of Flask used for some internal APIs.
- alexpotato 8y agoPeople talk a lot about complicated stacks but Python Flask + some basic HTML/CSS/Vanilla JS can solve a LOT of problems. This is especially true inside big orgs that have lots of silos and need "Rosetta Stones" that translate between the silos.
- kgraves 8y ago> "Python is the industry standard for all of the major applications we use to create Animated and VFX content, so it goes without saying that we are using it very heavily..." Interesting... very curious to see Netflix using tools in the VFX industry, Shotgun & Nuke to name a few, I wish they can expand more on this.
- erlangNewb 8y agoNetflix produces content too
- mixmastamyk 8y agoA lot of it, which needs compositing, post-production, encoding, etc. Python is great for pipeline code to script the compiled libraries that do the number crunching. Some of it is licensed/farmed out to places like DreamWorks however (new Voltron/She-ra), which is mostly a Java/Spring shop, to my knowledge.
- coffeedoughnuts 8y agoI used to work for a major vfx house who did much work on Netflix original shows there will be a certain amount of sharing needed between vendors, and so Netflix will need to have some vfx expertise in-house to coordinate that (similar to how a film will have a vfx supervisor that works for the film production company, who liases with supervisors at each vendor company) but it would be very sensible for them to have an in-house asset library (which may included work written in python, not just images or textures) so that if they want to switch vfx vendor between seasons of a show it's easy to do so in general, python rules in the vfx world. there's some c++ too. Sadly, because it's been this way for so long, many houses are still stuck in a python 2 world with huge legacy codebases
- sametmax 8y agoDo you know what the Netflix culture is regarding remote work ? I've been looking for a job for the first time in 10 years of Python, and Netflix is one of the rare big companies that I still have respect for. https://jobs.netflix.com/ https://jobs.netflix.com/ has several offers I would be a good fit for, but I'm not ready to relocate to the US.
- dudul 8y agoThey don't do it.
- drewg123 8y agoWe do have an office in Paris: https://jobs.netflix.com/locations/paris-france https://jobs.netflix.com/locations/paris-france Netflix is generally opposed to remote work, but some teams may make case by case exceptions (especially in Open Connect, the organization which manages the CDN). I'd encourage you to speak to a recruiter.
- sametmax 8y agoThank you. Will do.
- 2trill2spill 8y ago
- xtreak29 8y agoI am wondering if PyPy is used somewhere for workloads that need performance.
- mixmastamyk 8y agoProbably not, more likely to call out to their numerous Java projects and C/C++ video tools.
- z3t4 8y agoWasn't Netflix a Node.JS shop two years ago when Node was popular and now when Python is the most popular they are a Python shop ? :P (sarcasm) It's good to use many languages as it favors a micro-service architecture.
- tombert 8y agoIs Python the most popular language now? I'm not trying to argue, I'm trying to see how out-of-the-loop I am nowadays.
- yifanl 8y agoDepends on how you measure most popular. I think I saw a post here saying that the most new questions on SO were for python over js now, which might be a good enough proxy?
- tombert 8y agoI suppose Python is so popular in the ML and data-science spaces now that it makes some sense. I don't know a lot about those spaces, and the little bit I've played with neural networks I've done with Clojure because I'm hipster. Amazing how much of a pure-functional bubble I live in now. I should probably branch out more.
- adrianhel 8y agoThe university I went to have now switched from Java to Python for introductory computer science. So this type of change may influence this increase in popularity as well. I remember being a bit baffled that they used Java and not Python as an introductory language. It's a very obvious choice for beginners. Lol, I remember the first lesson. class HelloWorld { public static void main(String[] args) { System.out.println("Hello, world"); } }
- istjohn 7y ago
- sara27362 8y agoEven you can't believe there is a hen in world's biggest animals See the video how they looks and what they do https://cutt.ly/1rOU7Q https://cutt.ly/1rOU7Q Fish walking with legs like humans See the video on link below http://bit.ly/2XRGyYD http://bit.ly/2XRGyYD This couple is different Girls are both twins and boys are also twin brothers see how they look in video http://bit.ly/2VwoZQd http://bit.ly/2VwoZQd A cute dog is dancing in video Just watch this video and try to control your laugh http://bit.ly/2ZK1XVs http://bit.ly/2ZK1XVs
- bouncing 8y agoI had actually looked at Flask-RESTPlus for a relatively recent project before deciding to use FastAPI.
- randomsearch 8y agoSurprised they use Flask. I'm working on my first larger, industrial-strength REST API in Python and I've found the Django Rest Framework to be more suited once you get to that level of complexity.
- bouncing 8y agoProbably whoever made it just preferred Flask. django-rest-framework is nice because it's maintained well and it's consistent with the rest of Django, but plenty of people just prefer the Flask way of doing things, even if it's a scrappier set of tools in some ways.
- swah 8y agoI'm using DRF in a project and don't really like the code/magic I end up with. IIRC I was happier with Flask and SQL... (But of course, for the bits that maps properly into its model, DRF its awesome)
- bouncing 8y agoYeah. You can always get into what you do/don't like about a framework, but the same can be said of Flask. FWIW, I don't like DRF's reliance on serializers that do more than serialization.
- ajford 8y agoDjango works great if you're buying into the Django ecosystem. But if you're standing up a simple REST API, I've found Flask is much more flexible. Either approach will get you to where you're going, but I find Flask gets out of your way and lets you do your thing, while you have to find the Django Way of doing what you need to do. This may be biased (I've been a Flask user since 0.7), and perhaps I'm stuck in my ways, but I've found I either need to find a supported and blessed Django plugin for the thing I need, or build something much more complex than I'd need in Flask.
- ilmahkids 8y agoDo you recognize what the Netflix culture is in regards to remote work ? I've been searching for an occupation without precedent for a long time of Python, and Netflix is one of the uncommon huge organizations that regardless. ilmahkids.co.uk
- glckr 8y agoInteresting. I had it in my head that Netflix was a pure Java shop. Was I wrong in the first place, or has something changed?
- 2StepsOutOfLine 8y agoFrom what I understand from their talk on how they (don't) do devops they adopt a polygot architecture and teams use different languages.
- linuxdude314 8y agoDifferent teams using different languages doesn't mean they don't do DevOps.
- monsieurbanana 8y agoI read that as "in a talk on how they don't do devops they also happened to talk about how their teams are polyglots".
- vbsteven 8y agoI believe their is a fair bit of Kotlin used there too (based on some @netflix.com addresses in Kotlin libraries like Strikt)
- wesleytodd 8y agoI mainly write js but our team maintains a scala app as well. It is much more of a polyglot company with a history of java (I have only been here 1.5y so this is from hearsay and my expirence, not a definitive source).
- tanilama 8y agoMainly a Java shop. They published Node and even some C stuff back then in their blog posts.
- deleted 7y ago[deleted]
- butterpeanut 8y agoId like to contribute to the netflix code base by offering what is obviously a missing but much needed piece of code: def enableAutoPlay( flag ): annoyingAutoPlay = flag return
- fokinsean 8y agoI don't think that's valid syntax, so it likely won't make it in their repo
- terryschiavo22 8y agoDrats! We were so close!
- hyeomans 8y agoAt least as an option hidden somewhere.
- 781 8y agoTV's should also have this feature. When you switch to a new channel, it should be displayed in a paused state and you should press a button on the remote to start playing. Auto-play doesn't make sense anywhere, streaming apps, youtube, or TVs. Only crazy people expect video to auto-play in a video app.
- icedchai 7y ago
- ptah 8y agonow to get the apple tv app to use more than the bottom third of screen for letterbox sized navigation by making the massive "thumbnail"/trailer at the top a third of the screen, so i can actually find something good to watch...
- gvd 8y agoI fail to understand the use of python in a distributed environment while the language has such poor concurrency support (on top of the lack of a type system). You can make your application HA, but they are obviously not trying to squeeze out every CPU cycle.
- gshulegaard 8y ago> I fail to understand the use of python in a distributed environment while the language has such poor concurrency support Because it's a distributed environment probably is exactly why. Python has (arguably) great concurrency support apart from Multi-threading. https://www.youtube.com/watch?v=MCs5OvhV9S4 https://www.youtube.com/watch?v=MCs5OvhV9S4 So if you need concurrency in the context of a single thread, then Python's GIL is a non-starter. But a distributed environment is not likely one of those. Edit: I should amend concurrency in a single thread to: concurrency in a single thread that is compute gated...since coroutines can give you pseudo concurrency in a single thread provided you're workload has blocking steps like IO or TCP calls.
- wongarsu 8y agoIn fact it's precisely Python's deficiency in multithreading that lead to it having one of the best ecosystems for every other form of concurrency, like green threads and multiprocess applications.
- pletnes 8y agoIf you’re doing (data analysis|simulations|Image processing) you can offload computation to numpy, which releases the GIL. This allows nice multicore speedups with python and threading. The same holds for various CPU intensive standard library functions implemented in C. The GIL issue is real, but posts like this one confused me for years. Please, don’t exaggerate GIL issues.
- weberc2 8y agoNot everything is amenable to numpy and it's pretty easy to make performance worse by throwing numpy at every problem. For example, if your array contains Python objects that are part of an operation, you've likely just introduced a significant performance regression. Worse, there's no way to detect these regressions except to have performance tests. Please, don't understate GIL issues.
- peterwwillis 8y agoReason #8,201 not to model your org after FAANG: writing custom software just to operate your site is like building your own tools to till your farm. I suppose if you hire world-class toolmakers you'll have some very good tools, but that's not quite the point of the farm, is it?
- fossuser 8y agoIf you hire world class tool makers to make the best tools for your farm then your farm has a strategic advantage. Really good tools can make a big difference. You really do need world class though - the trade off of custom means your tools really do need to be a lot better than the standard.
- prdonahue 8y agoAt Cloudflare we use their Python-based Lemur (https://github.com/Netflix/lemur https://github.com/Netflix/lemur) application to issue (on some days) 1M SSL certificates.
- prolepunk 8y ago"You've read 11 stories this months, lets make things official". Medium is trying to look like NY Times, minus any kind of effort to actually write the stories.
- lma21 8y agoWhy can't I read their articles through Pocket?
- Tycho 8y agoDoes anyone have more info about this or know of similar projects? We lean on the many of the statistical and mathematical libraries (numpy, scipy, ruptures, pandas) to help automate the analysis of 1000s of related signals when our alerting systems indicate problems. We’ve developed a time series correlation system...
- jamespo 8y agoThere's https://prometheus.io/ https://prometheus.io/ which you're probably familiar with
- aaronblohowiak 8y agoWhat would you like to know?
- Tycho 7y agoThe correlations part specifically, what does it do, what’s the modelling approach, and is there more info about it anywhere?
- aaronblohowiak 7y agoCool, I’ll ping the folks responsible and have them reply here soon.
- relix42 7y agoInitially we wrote the library to help us answer the question about what change may be causing impact to our customers' experiences. Out of the billions of time series metrics we have we knew there were about 10,000 or so likely (for this initial use case), possible candidates and we wanted to reduce that set as far as possible and either (1) find /the/ candidate for problem or (2) produce a short list of things for the humans to look into. In order to get to the point where we could ensure high likelihood of apropos correlations, we needed to do some work on the signals first. First we detect if any of the possible candidate signals were born or died in the interesting time period. We use these time points to reduce the window we'll use to pass to the correlation functions. We can also detect any changepoints in the time series and apply similar logic. Once we've determined the best window bounds for each candidate signal, we use pearson and spearman correlation functions to get a score for the pair of signals -- the initial signal that started the inquiry and the candidate signal using the determined time window. The code is about 98% data preparation, signal analysis, and window determination and about 2% correlation work. (I've tried to summarize quite a bit, let me know if you'd like clarifications or have other questions.)
- sandGorgon 8y ago>The ability to drop into a bpython shell and improvise has saved the day more than once. what do you really do here - connect to the flask instance and route requests manually ?
- kabacha 8y agoAlso who uses anything but ptpython!
- aaronblohowiak 7y agoOur python code does not handle user requests directly. Our team uses python to control the control plane of traffic — we flip dns records, control cross-region proxying, re-steer cdn-based reverse proxying and scaling of the hundreds of micro services that power the Netflix experience. We’ve improvised various custom traffic distribution patterns, operated outside of our normal traffic-shifting workflows, and written/run quick scripts to modify scaling fleetwide.
- sandGorgon 7y agohi - so im actually curious about the infrastructure that allows you to connect bpython to it. Is it a rq process that you connect to, etc ? I would love to setup an architecture like this that lets me connect to stuff through a CLI. Also, I'm assuming you are on kubernetes (or something). How does this bpython business work through all those layers ?
- aaronblohowiak 7y agoWe ssh into a box in production, run bpython and import the libraries ( some environmental stuff, boto3, our own code, etc). At that point, there isn’t much difference from a bpython session and the regular application code... Ssh’ing into a production box does involve connecting through a bastion, and knowing which box you want to connect to, but that isn’t too hard with spinnaker.
- 7y ago
- whoevercares 7y agoMetaflow looks like a superhero solution for ML/DL. Hope they open source it one day