6 ms·

I wonder what the thought process was behind moving their web service stack (partially?) to node.js in the first place. For a company with the scale and resourc

by remon 12y ago

I wonder what the thought process was behind moving their web service stack (partially?) to node.js in the first place. For a company with the scale and resources of Netflix it's not exactly an obvious choice.

joshuacc 12y ago
They went into this a bit on the NodeUp podcast: http://nodeup.com/seventyone http://nodeup.com/seventyone
- StevePerkins 12y ago
  Are there some salient points that can be summarized here? That's an hour-and-twenty-minute podcast episode.
  - remon 12y ago
    The motivation part starts around 8:30. It's hard to claim complete objectivity when it comes to things like this but it sounds particularly unconvincing to me. Node.js' sweetspot is providing an easy route to server development for front-end developers and it seems this is roughly what happened. It's not a huge step for Netflix that already has an API approach that is very UI centric as outlined here http://techblog.netflix.com/2013/01/optimizing-netflix-api.html http://techblog.netflix.com/2013/01/optimizing-netflix-api.h.... To me personally this doesn't seem like a healthy step forward but I hope someone can provide some reasons why you'd want to move to Node.js from, in the case of Netflix, Java.
    - bcoates 12y ago
      I'm a non-frontend dev who uses node.js for server/systems/embedded work. My preference for node is based primarily on its sane concurrency model. I basically use it as a handy scripting language for doing rapid prototyping of libuv programs. My on-paper plan is to fall back to libuv and C if I find myself really stuck for CPU, but this is something that's never actually happened to me. (I have had to abandon or modify libraries with performance pathologies, though) There are other libraries with in other languages the same sane model but they tend to be ecosystems within the language community that aren't compatible with the rest of the language. Doing concurrency both correctly and performantly in Java, in particular, is a pain in the ass.
      - remon 12y ago
        I'm not disagreeing and that's a perfectly valid reason to go for node for smaller projects or for prototyping. What many people fail to realise is that the "concurrency" model (it is solved by simply not having concurrency) isn't a magic bullet. It comes with large drawbacks. Not a lot of things are complicated because people like it that way. Some things are as complicated as they need to be to be used effectively. I agree that doing concurrency in Java can be a pain in the ass (although some microservice libraries can simplify some cases) but for the most part it's a pain in the ass because concurrency is hard, not because Java is bad. Doing it the way node.js does is as easy to do in Java as it is in node.
        bcoates 12y ago
        An example of basic concurrency I have great difficulty doing correctly in Java: n = 0; avg = 0; function twothings() { thing1(function(ret) { ++n; avg += (ret - avg)/n; }); thing2(function(ret) { ++n; avg += (ret - avg)/n; }); } I want to dispatch two I/O operations to run concurrently and have their completions modify some shared state in a serializable fashion, which can be read at any stage in the process and have a valid result. I shouldn't need to give up on nondeterministic simultaneous execution or faff about with locking mechanisms to get this.
        deleted 12y ago
        [deleted]
        necubi 12y ago
        What I think you're missing is that in JS, this is not being run concurrently. Existing JS runtimes are single-threaded—if you actually want to take advantage of multiple CPUs you must run more than one process. It's very easy not to have concurrency bugs when there's no concurrency. In Java, you have access to real threads and can actually run two pieces of code concurrently. This introduces a lot of potential issues, but also allows you to take advantage of multiple cores. If all you want is non-blocking IO (what node gives you), there are a handful of great libraries that provide that on the JVM.
        12y ago
personZ 12y ago
Netflix seems to operate like a tech start-up that is trying to glue together a ragtag collections of often unsuitable solutions because of limited funding. It is a deeply perplexing company. Similar is LinkedIn, as an aside -- despite being fairly formidable now, I regularly have entire feeds disappear, their caching is abhorrent, they can't markup text properly, and so on. It seems very amateur hour, yet they regularly publish "how it's done" documents that see wide applause despite often completely contradicting their prior missives.
- general_failure 12y ago
  Netflix overhires quite a bit. They also pay their (very good) engineers very high salaries (possibly highest in the valley) which results is very low attrition. The net result is a lot of engineers having a lot of time and all these engineers experiment away on technology. In fact, they have quite a bit of NIH syndrome internally because the engineers have nothing to do.
  - guiomie 12y ago
    NIH ?
    - general_failure 12y ago
      Not invented here. Redoing stuff which already exists.
      - surganov 12y ago
        Sometimes it's okay to eliminate dependencies: http://www.joelonsoftware.com/articles/fog0000000007.html http://www.joelonsoftware.com/articles/fog0000000007.html
  - eikenberry 12y ago
    I don't think you have those reasons in proper order. I think the ability of the engineers to experiment and use new tools is why they have the high level of retention. I've seen plenty of places that pay much higher than the competition yet have very high turnover due to the a conservative culture which dictates the tools and doesn't encourage experimentation.
yourad_io 12y ago
What are the arguments against node.js in their use case? Not looking to start any wars, but I was under the impression that if you know what you're doing* node.js is pretty awesome. This particular bug had to do with a misunderstanding regarding the express API. * for the most part: understand async and closures/memory leaks.
- yourad_io 12y ago
  To give some arguments in favour: - "Here we see our latencies drop down to 1 ms and remain there after we deployed our fix." <- Performance (from TFA) - They can hire from a vastly larger pool of developers. Since they've got both budget and brand recognition, they should have no problems hiring high end JS guys & gals.
  - remon 12y ago
    Why are these arguments in favor of Node.js over their current Java based stack? Java is both considerably more speedy for this sort of workload and there's a significantly larger pool of high end Java server developers compared to high end JavaScript server developers. Budget and brand recognition are not necessarily in favor of node.js over other stacks. That simply helps with getting the better engineers in general.
    - yourad_io 12y ago
      They are arguments in favor of node.js being a reasonable tool for the job. > Java is both considerably more speedy for this sort of workload They're quoting 1ms latencies, that's pretty speedy. Maybe Java could do <1ms and/or support slightly more concurrent requests. How much difference does it actually make when you're already responding after 1 millisecond? > there's a significantly larger pool of high end Java server developers compared to high end JavaScript server developers Oh, I would have thought there are far more JS* folks out there than Java. > Budget and brand recognition are not necessarily in favor of node.js over other stacks. That simply helps with getting the better engineers in general. That's what I meant - them being in favour of Netflix as an employer - they aren't "Bob's Digital Agency" so they can have their pick of the litter of node devs. * I'm not sure if the "server" distinction is meaningful here - if you know JS, you know node.js - simply a matter of familiarizing yourself with node-specific APIs.
tjholowaychuk 12y ago
I share this thought, I'm not trolling, I really believe node is a bad solution for something like Netflix. Node has its perks but for a money making machine that relies solely on being available and providing a good customer experience, not so much. I can't imagine the ops nightmares at that size, one buggy code path and the entire cluster could be down. These are issues that drove me away from Node to Go, in my opinion Node has way too many issues to run in money-making scenarios.
- nmjohn 12y ago
  > in my opinion Node has way too many issues to run in money-making scenarios. You can say that about a lot of languages other than node. How many billion dollar companies have their software written in PHP - a language that many people would agree has far more glaring issues than node. My point, is that I believe your comments miss the larger picture. There is far more involved with deciding which language to build a product with than "which language is best." I don't disagree with you about Go being a great language - but for most companies, it is not even remotely practical to use. Hiring talented Go developers is hard because there are so few. Their current employees may or may not know Go, what do you do about that? Etc.
  - tjholowaychuk 12y ago
    For sure, but jumping into Node from a clean slate is probably not the best idea, legacy is different. If anything the comment on hiring highlights the use of Node being problematic. It's difficult to write robust systems in Node even as a seasoned Node developer.
- nthtran 12y ago
  Could you expand more on the ops issues you have encountered in your systems?
- deleted 12y ago
  [deleted]

by remon 12y ago

joshuacc 12y ago

They went into this a bit on the NodeUp podcast: http://nodeup.com/seventyone http://nodeup.com/seventyone

StevePerkins 12y ago
Are there some salient points that can be summarized here? That's an hour-and-twenty-minute podcast episode.
- remon 12y ago
  The motivation part starts around 8:30. It's hard to claim complete objectivity when it comes to things like this but it sounds particularly unconvincing to me. Node.js' sweetspot is providing an easy route to server development for front-end developers and it seems this is roughly what happened. It's not a huge step for Netflix that already has an API approach that is very UI centric as outlined here http://techblog.netflix.com/2013/01/optimizing-netflix-api.html http://techblog.netflix.com/2013/01/optimizing-netflix-api.h.... To me personally this doesn't seem like a healthy step forward but I hope someone can provide some reasons why you'd want to move to Node.js from, in the case of Netflix, Java.
  - bcoates 12y ago
    I'm a non-frontend dev who uses node.js for server/systems/embedded work. My preference for node is based primarily on its sane concurrency model. I basically use it as a handy scripting language for doing rapid prototyping of libuv programs. My on-paper plan is to fall back to libuv and C if I find myself really stuck for CPU, but this is something that's never actually happened to me. (I have had to abandon or modify libraries with performance pathologies, though) There are other libraries with in other languages the same sane model but they tend to be ecosystems within the language community that aren't compatible with the rest of the language. Doing concurrency both correctly and performantly in Java, in particular, is a pain in the ass.
    - remon 12y ago
      I'm not disagreeing and that's a perfectly valid reason to go for node for smaller projects or for prototyping. What many people fail to realise is that the "concurrency" model (it is solved by simply not having concurrency) isn't a magic bullet. It comes with large drawbacks. Not a lot of things are complicated because people like it that way. Some things are as complicated as they need to be to be used effectively. I agree that doing concurrency in Java can be a pain in the ass (although some microservice libraries can simplify some cases) but for the most part it's a pain in the ass because concurrency is hard, not because Java is bad. Doing it the way node.js does is as easy to do in Java as it is in node.
      - bcoates 12y ago
        An example of basic concurrency I have great difficulty doing correctly in Java: n = 0; avg = 0; function twothings() { thing1(function(ret) { ++n; avg += (ret - avg)/n; }); thing2(function(ret) { ++n; avg += (ret - avg)/n; }); } I want to dispatch two I/O operations to run concurrently and have their completions modify some shared state in a serializable fashion, which can be read at any stage in the process and have a valid result. I shouldn't need to give up on nondeterministic simultaneous execution or faff about with locking mechanisms to get this.
        deleted 12y ago
        [deleted]
        necubi 12y ago
        What I think you're missing is that in JS, this is not being run concurrently. Existing JS runtimes are single-threaded—if you actually want to take advantage of multiple CPUs you must run more than one process. It's very easy not to have concurrency bugs when there's no concurrency. In Java, you have access to real threads and can actually run two pieces of code concurrently. This introduces a lot of potential issues, but also allows you to take advantage of multiple cores. If all you want is non-blocking IO (what node gives you), there are a handful of great libraries that provide that on the JVM.
        12y ago

personZ 12y ago

Netflix seems to operate like a tech start-up that is trying to glue together a ragtag collections of often unsuitable solutions because of limited funding. It is a deeply perplexing company. Similar is LinkedIn, as an aside -- despite being fairly formidable now, I regularly have entire feeds disappear, their caching is abhorrent, they can't markup text properly, and so on. It seems very amateur hour, yet they regularly publish "how it's done" documents that see wide applause despite often completely contradicting their prior missives.

general_failure 12y ago
Netflix overhires quite a bit. They also pay their (very good) engineers very high salaries (possibly highest in the valley) which results is very low attrition. The net result is a lot of engineers having a lot of time and all these engineers experiment away on technology. In fact, they have quite a bit of NIH syndrome internally because the engineers have nothing to do.
- guiomie 12y ago
  NIH ?
  - general_failure 12y ago
    Not invented here. Redoing stuff which already exists.
    - surganov 12y ago
      Sometimes it's okay to eliminate dependencies: http://www.joelonsoftware.com/articles/fog0000000007.html http://www.joelonsoftware.com/articles/fog0000000007.html
- eikenberry 12y ago
  I don't think you have those reasons in proper order. I think the ability of the engineers to experiment and use new tools is why they have the high level of retention. I've seen plenty of places that pay much higher than the competition yet have very high turnover due to the a conservative culture which dictates the tools and doesn't encourage experimentation.

yourad_io 12y ago

What are the arguments against node.js in their use case? Not looking to start any wars, but I was under the impression that if you know what you're doing* node.js is pretty awesome. This particular bug had to do with a misunderstanding regarding the express API. * for the most part: understand async and closures/memory leaks.

yourad_io 12y ago
To give some arguments in favour: - "Here we see our latencies drop down to 1 ms and remain there after we deployed our fix." <- Performance (from TFA) - They can hire from a vastly larger pool of developers. Since they've got both budget and brand recognition, they should have no problems hiring high end JS guys & gals.
- remon 12y ago
  Why are these arguments in favor of Node.js over their current Java based stack? Java is both considerably more speedy for this sort of workload and there's a significantly larger pool of high end Java server developers compared to high end JavaScript server developers. Budget and brand recognition are not necessarily in favor of node.js over other stacks. That simply helps with getting the better engineers in general.
  - yourad_io 12y ago
    They are arguments in favor of node.js being a reasonable tool for the job. > Java is both considerably more speedy for this sort of workload They're quoting 1ms latencies, that's pretty speedy. Maybe Java could do <1ms and/or support slightly more concurrent requests. How much difference does it actually make when you're already responding after 1 millisecond? > there's a significantly larger pool of high end Java server developers compared to high end JavaScript server developers Oh, I would have thought there are far more JS* folks out there than Java. > Budget and brand recognition are not necessarily in favor of node.js over other stacks. That simply helps with getting the better engineers in general. That's what I meant - them being in favour of Netflix as an employer - they aren't "Bob's Digital Agency" so they can have their pick of the litter of node devs. * I'm not sure if the "server" distinction is meaningful here - if you know JS, you know node.js - simply a matter of familiarizing yourself with node-specific APIs.

tjholowaychuk 12y ago

I share this thought, I'm not trolling, I really believe node is a bad solution for something like Netflix. Node has its perks but for a money making machine that relies solely on being available and providing a good customer experience, not so much. I can't imagine the ops nightmares at that size, one buggy code path and the entire cluster could be down. These are issues that drove me away from Node to Go, in my opinion Node has way too many issues to run in money-making scenarios.

nmjohn 12y ago
> in my opinion Node has way too many issues to run in money-making scenarios. You can say that about a lot of languages other than node. How many billion dollar companies have their software written in PHP - a language that many people would agree has far more glaring issues than node. My point, is that I believe your comments miss the larger picture. There is far more involved with deciding which language to build a product with than "which language is best." I don't disagree with you about Go being a great language - but for most companies, it is not even remotely practical to use. Hiring talented Go developers is hard because there are so few. Their current employees may or may not know Go, what do you do about that? Etc.
- tjholowaychuk 12y ago
  For sure, but jumping into Node from a clean slate is probably not the best idea, legacy is different. If anything the comment on hiring highlights the use of Node being problematic. It's difficult to write robust systems in Node even as a seasoned Node developer.
nthtran 12y ago
Could you expand more on the ops issues you have encountered in your systems?
deleted 12y ago
[deleted]