7 ms·
Python at Netflix
- jedberg 14y agoAs one of the main drivers of Python at Netflix, this one is near and dear to my heart. Let me know if you have any questions, or come visit our booth!
- arpineh 14y agoHi. There was many interesting things listed at the blog post. I have been planning a system similar in idea to Sting, your OLAP server (?). I'm planning a light RPC style system on top of Pandas dataframe objects, transferring HTTP requests to Pandas method calls. Memory usage of such a system with our data is too expensive if all data is in use. So, I'm investigating possible caching and database solutions. Could you perhaps elaborate on the implementation of Sting's in memory database? [posted from yet another HN account]
- jedberg 14y agoI just posted this above: "I don't have any info about Sting, but we're working on some blog posts that go deeper into each of these topics. I'll make sure that the interest in Sting is accounted for in the prioritization." So now that's +2. :)
- daemon13 14y ago+3 please :-) The idea of marrying "slow" Python and OLAP is disturbing my sleep!
- pwang 14y agoHave you looked at the Cubes project? http://databrewery.org/cubes.html http://databrewery.org/cubes.html
- yellow 14y agoI'm pushing for more python use at my company as well. It's tough as everyone is drinking the ruby kool-aid. P.S. I really enjoyed your talk at QCon SF last year!
- irahul 14y ago> It's tough as everyone is drinking the ruby kool-aid. Ruby and Python are similar in terms of execution speed and developer productivity. If your work requires something that Python has and Ruby doesn't(say numpy), then it might make sense; otherwise what would you accomplish by switching from Ruby to Python?
- jonascopenhagen 14y agowhat would you accomplish by switching from Ruby to Python? Simpler syntax?
- mixmastamyk 14y agoA larger and more varied developer community, resulting in applicability to a wider range of problems.
- deleted 14y ago[deleted]
- e1ven 14y agoKeep it up! Posts like these are great on a tech level, but they're also good recruitment - Seeing Python, along with hackers that I respect (like you) at Netflix make me at least the littlest bit tempted to go there ;)
- jedberg 14y agoYou've caught on to our ploy! But in all seriousness, we like to share what we do because sharing is cool and it attracts other people who like to share cool technology, which are the kind of people we like to work with.
- rbanffy 14y agoI strongly suspect preventing me from visiting PyCon is one of the reasons my CTO scheduled so many product launches for the next two weeks...
- pwang 14y agoYou should also come by our booth (Continuum Analytics) and we can talk shop!
- DoggettCK 14y agoWe were actually talking about your company in our sprint planning meeting a few minutes ago, since we're doing in-house Python analytics. You might be getting a call.
- pwang 14y agoAwesomeness. We are making great headway on Blaze and the new persistence format to supplant HDF5/PyTables (Although, Blaze will always work great with carray, NumPy, and Pytables), and the Numba compiler project is working out great. Wakari (http://wakari.io http://wakari.io) is nearing 1.0 and we're about to turn on paid monthly plans for beefier nodes and GPU nodes, and we have also been in talks with several companies about in-house private deployments. Give us a shout or shoot me an email at pwang (at) continuum.io! :-)
- bsimpson 14y agoYou know how Google claims that by saving fractions of a second for every search that billions of people do, they've saved many man-lives of productivity? As a leading engineer at both Reddit and Netflix, you've undone all of that. Well done.
- qxf2 14y agoUnfair tone. I hope it was in jest. My life is not all about productivity. There is a leisure aspect as well: http://en.chessbase.com/home/TabId/211/PostId/4008728 http://en.chessbase.com/home/TabId/211/PostId/4008728
- jedberg 14y agoI always tell people that I've spent the last six years of my life as a digital crack dealer.
- BrianEatWorld 14y agoWhere can I find more info about Sting? I didn't see a link in the article and some quick Google searches haven't proved fruitful.
- jedberg 14y agoI don't have any info about Sting, but we're working on some blog posts that go deeper into each of these topics. I'll make sure that the interest in Sting is accounted for in the prioritization.
- pwang 14y agoKurt Brown had some slides about data & Python at the NYC Python meetup last October. He did a Sting demo there and some of the slides talk about it (I think... it might be a different thing, though. October was a long time ago..) http://www.meetup.com/nycpython/files/ http://www.meetup.com/nycpython/files/
- tootie 14y agoI have to say that after doing a python project I totally don't get why people love it so much. The syntax is slightly cleaner, but the lack of threading, first-class functions and dependency management tools make it waste at least as much time as it saves. I honestly don't believe in the concept of a programming language being able to make code more or less readable. A Java project with a POM file can be up and running on any OS in about 5 minutes.
- brown9-2 14y agoPython doesn't have first class functions?
- maxerickson 14y agoI often get the idea that people confuse anonymous and first class functions (apparently because they discover the latter expressed in terms of the former).
- nightpool 14y agoa) python-pip combined with virtualenv gives very good requirement support. b) python does have first class functions? c) threading still happens, you just can't take advantage of multiple cores. This isn't as big of a problem as you'd think though, you can still write very fast code.
- frankwiles 14y agoUmmm, not sure where you're getting your information but Python has threads (http://docs.python.org/2/library/threading.html http://docs.python.org/2/library/threading.html), first-class functions (http://en.wikipedia.org/wiki/First-class_function http://en.wikipedia.org/wiki/First-class_function), and dependency management tools (google for pip, easy_install, etc.)
- herge 14y agoI think he was talking about threads that run on multiple processors. Also, packaging is a whole different can of worms (but hopefully will be sorted by distutils2!).
- bryanwb 14y agowould you be interested to talk about these tools on the http://foodfightshow.org http://foodfightshow.org devops podcast? we would love to discuss them with you!
- jedberg 14y agoI would, but I could probably find you people who know more about the tools, like the folks who wrote them. :) Hit me up on email (it's in my profile).
- deleted 14y ago[deleted]
- cpenner461 14y agoHey jedberg - any plans/timeline for open sourcing aminator? It sounds pretty cool...
- deleted 14y ago[deleted]
- voidlogic 14y agoAs a Java/Python shop I think Netflix should evaluate Go after 1.1 RC comes out in April. Go is very popular with people who have heavily used both Java and Python (as well as C).
- robertfw 14y agoJust curious, why would this be downvoted?
- Afforess 14y agoIf I had to guess, I would say because GP didn't actually give any real reasons why adding complexity (a new language) would improve or solve a problem. Advocating languages is fine, we all have our pet languages, but it is important to remember they each are just one tool in the toolbox and that what is more important is how the language solves the problem well, instead of just blindly advocating one language over another.
- voidlogic 14y agoLanguage comparison are so ridiculously multi-dimensional and my knowledge of Netflix's internal applications are shallow enough I thought specifics would be a waste of time. I see Go in general as a reaction to the overcomplexity of Java/C++ and the performance issues of Python/Ruby. Since Go seems to be popular with people who use Java/Python, and Netflix uses Java/Python, I was curious if Go was on the table. I would hardly call my original post blind advocacy...
- sigzero 14y agoSome folks would consider this as an attempt to hijack the discussion. I am not saying that is what you are trying to do but what some folks would think. It gets tedious sometimes when the post is specifically about "Python at whatever" and people post "Hey you should use this language instead." Maybe that is why you were down voted. (FWIW, I did not down vote you.)
- netcraft 14y agoRe: Web applications in python - I am curious about the overall feeling for them - Do you write web apps in python because rapid development makes up for any performance problems and if you run into problems you would rewrite it in something faster, or do you feel like a python web app can be just as good as alternatives when it comes to speed? Put another way - when you think of python for web applications / services - do you think rapid development and speed, or just rapid development and you would go somewhere else for speed?
- irahul 14y agoI don't think anybody equates Python with execution speed. Writing web applications in Python facilitates rapid development, and since most of the web applications are io bound, slow execution speed doesn't make noticeable difference.
- netcraft 14y agoSo when you think of execution speed in web applications / services - what do you think of?
- irahul 14y ago> when you think of execution speed in web applications The whole web application is not CPU bound. The parts which are CPU intensive are rewritten in performant language. For eg, if you are twitter, you still render templates in Rails, and write CPU bound services in Java/Scala. Replacing the whole app with Scala/Java will be pointless. That doesn't mean that every time you need a service, you can't use Ruby/Python. If all your API server does is check cache for data, load data if there is a cache miss, encode json and return it, Python/Ruby will do just fine.
- netcraft 14y agoUnderstood - was just curious when it comes to CPU intensive portions what you would characterize as performant vs python. Scala/Lift is on my list to check out - So my question now is - if you got to the point of needing to write CPU bound services in Scala, why wouldn't you go ahead and write everything in Scala? Does python offer enough advantages to outweigh having to support multiple languages/environments/ecosystems?
- jkldotio 14y agoPython really does hit a sweet-spot for many types of operations. Devops capacity in system scripting, crawling and scraping with excellent web frameworks (Django through to Bottle). On top of all that there are great machine learning libraries as well as good APIs to almost anything not written in Python. My site gets huge amounts of NLP, machine learning and information extraction done in just a few lines because of libraries available in python that aren't in the languages that it's usually compared to. I've also pretty much replaced bash with the ipython shell at this point, as you can use all of the bash commands anyway and even seamlessly mix them with python, file_list = !ls, it's a excellent environment for data analysis and manipulation.
- jedberg 14y agoYou know, it never occurred to me to use iPython as my shell. I think that would make me more productive. I'll have to try that, thanks!
- jkldotio 14y agoI have 'ipython --profile pysh' in my terminal's custom command, on servers I'm a bit more cautious and have a 'pu' (power up, or python up) command which is basically the first thing I type. You have to watch out for the shell history not working quite the same and that newly installed programs won't be seen until you do a '%rehashx'. Other than that it's pretty much all gravy. Now I just need to get around to transitioning to that python tiling window manager. While I've got you, and seeing you are not longer at Reddit, what do you think of jkl.io (launched today)?
- pwang 14y agoI think someone asked the question at a Scipy conference a few years back, "Who here uses IPython as their primary shell", and IIRC about 30% of the people raised their hands.
- yen223 14y agoSomeone once said that Python is the second-best language for everything. I found that to be true.
- mixmastamyk 14y agoAny remote jobs or must you live near Los Gatos?
- jedberg 14y agoGenerally we want people to be in Los Gatos, but there are exceptions, mostly in the CDN team.
- donutdan4114 14y agoWhy does Netflix only allow 4-10 character passwords? My weak password is 14 characters. There is no reason to have a length limit.
- belandrew 14y agoMy Netflix password is 20 characters. Where did you get that it only allows 4-10?
- donutdan4114 14y agoTrying to update my password: http://i.imgur.com/wLNpR0C.png http://i.imgur.com/wLNpR0C.png