19 ms·

Node.js in Flame Graphs

• gadders 12y ago
  OFFTOPIC: "Today, I want to share some recent learnings from performance tuning this new application stack." The word you want is "lessons".
  • quarterto 12y ago
    "learnings" is a perfectly cromulent word: https://books.google.com/ngrams/graph?content=learnings&year_start=1700&year_end=2000&corpus=15&smoothing=3&share=&direct_url=t1%3B%2Clearnings%3B%2Cc0 https://books.google.com/ngrams/graph?content=learnings&year...
    • gadders 12y ago
      Nope. You see an option for a plural here? http://www.merriam-webster.com/dictionary/learning http://www.merriam-webster.com/dictionary/learning
      • from_elsewhere 12y ago
        Dictionaries ain't arbiters of the English language. And quite fortunately so, I should note.
      • M2Ys4U 12y ago
        Dictionaries are descriptive, not prescriptive. They can only tell you if something is a word, not whether something isn't a word.
        • gadders 12y ago
          Well certainly there is no law against using made-up words. If that's what you want to do, have at it. At least it will help people get their buzzword bingo cards filled up sooner.
      • not_kurt_godel 12y ago
        https://books.google.com/ngrams/graph?content=learnings&year_start=1800&year_end=2000&corpus=15&smoothing=3&share=&direct_url=t1%3B%2Clearnings%3B%2Cc0 https://books.google.com/ngrams/graph?content=learnings&year...
    • mcguire 12y ago
      If you are Sacha Baron Cohen, yes.
      • quarterto 12y ago
        Use of this word predates Sacha Baron Cohen's entire existence.
  • mikeryan 12y ago
    OFFTOPIC: "Today, I want to share some recent learnings from performance tuning this new application stack." The word you want is "lessons". Jeez, not only is your comment offtopic and pedantic, it's also wrong.
    • swasheck 12y ago
      "it's also wrong." I believe the word you want is "incorrect." /sarcasm
    • gadders 12y ago
      I flagged it as Offtopic. But it is correct. "Learnings" is at best a pointless affectation, like saying "utilised" instead of "used". At worst, according to the OED, it's not even a word. http://english.stackexchange.com/questions/19227/plural-of-learning http://english.stackexchange.com/questions/19227/plural-of-l... Also: http://www.merriam-webster.com/dictionary/learning http://www.merriam-webster.com/dictionary/learning No plural option.
      • cdelsolar 12y ago
        I play tournament Scrabble and learnings is a valid word in it.
      • Scramblejams 12y ago
        Continuing the OT and heedless of the downvotes: I appreciate your efforts, but they can't help themselves -- so many people love using recently invented (and redundant) words because they think it makes them sound smart and hip. Sorta like people who think JS is an amazing language and love to sell that point of view.
        • gadders 12y ago
          I think their must be a voting ring of marketers that are upset their latest buzzword-bingo entry has been called out :-)
      • chocolateboy 12y ago
  • jwhitlark 12y ago
    I didn't like it when people started using "learnings", but I'm beginning to appreciate some subtle shadings it seems to have over lessons. It seems more discovery oriented, and less sure it's correct, or at least, less widely correct. Sort of like "hacky, working hypotheses, that we think are worth sharing."
• ajsharma 12y ago
  This is the first I've heard of restify, but it seems like a useful framework for the main focus of most Node developers I know, which is to replace an API rather than a web application.
  • ecaron 12y ago
    You're going to love restify. We use it at TrackIf and can't imagine our API running on anything else. Couple it with Swagger and you won't be looking back :-)
    • deleted 12y ago
      [deleted]
• vkjv 12y ago
  > ...as well as increasing the Node.js heap size to 32Gb. > ...also saw that the process’s heap size stayed fairly constant at around 1.2 Gb. This is because 1.2 GB is the max allowed heap size in v8. Increasing beyond this value has no effect. > ...It’s unclear why Express.js chose not to use a constant time data structure like a map to store its handlers. It it is non-trivial (not possible?) to do this in O(1) for routes that use matching / wildcards, etc. This optimization would only be possible for simple routes.
  • quarterto 12y ago
    There are ways around that: http://blog.caustik.com/2012/04/11/escape-the-1-4gb-v8-heap-limit-in-node-js/ http://blog.caustik.com/2012/04/11/escape-the-1-4gb-v8-heap-...
  • herge 12y ago
    > It it is non-trivial (not possible?) to do this in O(1) for routes that use matching / wildcards I'd be impressed if they did it consistently in O(1) for static routes. I think they were looking for O(log(number of different routes)) instead of O(n).
    • azinman2 12y ago
      If the paths have some kind of non crazy Regex leading up to what gets munched for path variables (eg /path1 versus /path2) then you could at least build a tree of maps at each level, which would be roughly constant time for the most common cases.
      • herge 12y ago
        > roughly constant time for the most common cases. For small values of n ;) A tree of maps would be consistently log(n), like any map. Even a hashtable would hash to buckets eventually, and are log(n).
        • TheLoneWolfling 12y ago
• thedufer 12y ago
  > It’s unclear why Express.js chose not to use a constant time data structure like a map to store its handlers. Its actually quite clear - most routes are defined by a regex rather than a string, so there is no built-in structure (if there's a way at all) to do O(1) lookups in the routing table. A router that only allowed string route definitions would be faster but far less useful. I can't explain away the recursion, though. That seems wholly unnecessary. Edit: Actually, I figured that out, too. You can put middleware in a router so it only runs on certain URL patterns. The only difference between a normal route handler and a middleware function is that a middleware function uses the third argument (an optional callback) and calls it when done to allow the route matcher to continue through the routes array. This can be asynchronous (thus the callback), so the router has to recurse through the routes array instead of looping.
  • cataphract 12y ago
    On the other hand, without the recursion I don't see how the flame graph would have been helpful.
    • msmillie 12y ago
      In flame graphs heavy iteration tends to give "wider" structures. Recall the stacks aren't ordered on the X axis by time, but by contents.
  • quotemstr 12y ago
    Of course there's a faster way! Combine all the routes into a DFA, then run the DFA over the URL. It's guaranteed to run in constant space and O(n) (n=URL length) time! The union of any set of regular languages is itself a regular language. You can use Ragel[1] to build your automaton. [1] http://www.colm.net/open-source/ragel/ http://www.colm.net/open-source/ragel/
    • thedufer 12y ago
      Yeah, but what's n there? Isn't that going to be something like the sum of each route's length? That doesn't buy us anything (well, probably a smaller constant). Unless you go the NFA route, but I'm pretty sure that costs non-constant space.
• debacle 12y ago
  Doesn't this seem like a bug in the express router? All of the additional routes in the array are dead (can't be routed to).
  • dugmartin 12y ago
    No, because the route list is also used for middleware. The recursive search is because middleware routes have a third next parameter that allows the search to run asynchronously.
    • deleted 12y ago
      [deleted]
• ecaron 12y ago
  My biggest takeaway from this article is that Netflix is moving from Express to Restify, and I look forward to watching the massive uptick this has on https://github.com/mcavage/node-restify/graphs/contributors https://github.com/mcavage/node-restify/graphs/contributors
  • sadkingbilly 12y ago
    Yes, but their original bug was from dynamically loading routes from an external source. I don't see how Express is to blame for this. Moving to Restify is not a solution, but they state having different reasons for moving (support for bunyan logging? But Express already supports this too).
    • ecnahc515 12y ago
      the bug was related to dynamically loading routes, but the true cause was that express allowed duplicate handlers. They were loading routes dynamically correctly, that wasn't a problem, it was that when doing that express let them duplicate routes.
      • jonny_eh 12y ago
        If they were loading duplicate routes, how was it they were "loading routes dynamically correctly"?
      • sisk 12y ago
        That's a feature. From the API docs: > Multiple callbacks may be given; all are treated equally, and behave just like middleware. The only exception is that these callbacks may invoke next('route') to bypass the remaining route callback(s). This mechanism can be used to perform pre-conditions on a route, then pass control to subsequent routes if there's no reason to proceed with the current route.
  • gaving 12y ago
    not exactly highly endorsed https://twitter.com/tjholowaychuk/status/382084838179614721 https://twitter.com/tjholowaychuk/status/382084838179614721
• TheLoneWolfling 12y ago
  > benchmarking revealed merely iterating through each of these handler instances cost about 1 ms of CPU time 1ms / entry? What is it doing that it's spending 3 million cycles on a single path check?
  • jdmichal 12y ago
    Running (uncompiled?) regular expressions, it seems.
    • baudehlo 12y ago
      V8 compiles regular expressions to machine code.
      • simi_ 12y ago
        You're the author of Haraka, a tool I'm prepared to try in production. This makes me _really_ worried that I'm making the right choice. (To explain: obviously C++ churns out machine code, the parent was talking about compiled vs uncompiled regexes – if I'm not terribly wrong, the compilation step is turning the regex into a finite automaton.)
        • baudehlo 12y ago
          V8 literally compiles regexps to X86 machine code the first time they are executed. They are not compiled into an FSA that gets walked in the traditional sense. Hopefully that lowers your concern level.
          • simi_ 12y ago
            Interesting, thanks for the info!
          • TheLoneWolfling 12y ago
            Doesn't that mean that you have exponential worst-case complexity?
• forrestthewoods 12y ago
  If I had to pick one line to highlight (not to criticize, but was a wise lesson worth sharing) it would be this one: "First, we need to fully understand our dependencies before putting them into production."
  • gdulli 12y ago
    In my experience developers constantly overestimate the gain of using a new dependency and underestimate the amount of effort it will take to sufficiently understand it. (Or fail to make the effort, not understanding the risks.) This is why developers without significant experience should not be making decisions about the tech stack.
  • _RPM 12y ago
    [Subjective] Not to criticize the Express.js code base, but have you tried reading it? It is very complicated and there are a bunch of clever things going on. I think it could have been written simpler and easier to understand. A problem with frameworks is that they are written by people who want to show off how clever they are.
• drderidder 12y ago
  > our misuse of the Express.js API was the > ultimate root cause of our performance issue That's unfortunate. Restify is a nice framework too, but mistakes can be made with any of them. Strongloop has a post comparing Express, Restify, hapi and LoopBack for building REST API's for anyone interested. http://strongloop.com/strongblog/compare-express-restify-hapi-loopback/ http://strongloop.com/strongblog/compare-express-restify-hap...
• BradRuderman 12y ago
  Why are they loading in routes from an external source? Is that normal, I have never seen that before.
  • mjr578 12y ago
    We like the option of dynamically loading new routes, that point to new endpoints. We also have the ability to release new versions of our UI without redeploying (or restarting) our servers.
    • deleted 12y ago
      [deleted]
    • deleted 12y ago
      [deleted]
    • BradRuderman 12y ago
      Ok you add a new route but how do you reference what code should be executed when that route is hit?
      • mjr578 12y ago
        We have something that loads up, via requires, the action (or route) that should be run when a URL is encountered.
        • BradRuderman 12y ago
          Second dmak, would love to understand more. I can following dynamically loading routes but can't follow how that would be implemented end to end. Some things I would be interested in: - Where do the keep the code that gets executed for new routes? Is that deployed dynamically as well? - If you are changing routes dynamically how do you test in non-prod, are you constantly syncing non prod with prod? - How do you control what you deploy dynamically vs what you migrate through the environments?
    • deleted 12y ago
• remon 12y ago
  I wonder what the thought process was behind moving their web service stack (partially?) to node.js in the first place. For a company with the scale and resources of Netflix it's not exactly an obvious choice.
  • joshuacc 12y ago
    They went into this a bit on the NodeUp podcast: http://nodeup.com/seventyone http://nodeup.com/seventyone
    • StevePerkins 12y ago
      Are there some salient points that can be summarized here? That's an hour-and-twenty-minute podcast episode.
      • remon 12y ago
        The motivation part starts around 8:30. It's hard to claim complete objectivity when it comes to things like this but it sounds particularly unconvincing to me. Node.js' sweetspot is providing an easy route to server development for front-end developers and it seems this is roughly what happened. It's not a huge step for Netflix that already has an API approach that is very UI centric as outlined here http://techblog.netflix.com/2013/01/optimizing-netflix-api.html http://techblog.netflix.com/2013/01/optimizing-netflix-api.h.... To me personally this doesn't seem like a healthy step forward but I hope someone can provide some reasons why you'd want to move to Node.js from, in the case of Netflix, Java.
        • bcoates 12y ago
          I'm a non-frontend dev who uses node.js for server/systems/embedded work. My preference for node is based primarily on its sane concurrency model. I basically use it as a handy scripting language for doing rapid prototyping of libuv programs. My on-paper plan is to fall back to libuv and C if I find myself really stuck for CPU, but this is something that's never actually happened to me. (I have had to abandon or modify libraries with performance pathologies, though) There are other libraries with in other languages the same sane model but they tend to be ecosystems within the language community that aren't compatible with the rest of the language. Doing concurrency both correctly and performantly in Java, in particular, is a pain in the ass.
• bentcorner 12y ago
  Interesting article. I have a lot of experience dealing with ETLs in WPA on the Windows side - it's an awesome tool that gives you similar insights. I haven't used it for looking at javascript stacks before though, so I don't know if it'll do that.
• qodeninja 12y ago
  wow. I love that Netflix us using Node and even more curious that they would use express.
• elwell 12y ago
  TIL, SVG's can display labels on element hover: http://cdn.nflximg.com/ffe/siteui/blog/yunong/200mins.svg http://cdn.nflximg.com/ffe/siteui/blog/yunong/200mins.svg Nice, contained way to show data like this.
  • TazeTSchnitzel 12y ago
    SVGs can contain ECMAScript, video, canvases, animation, all sorts of things. They're the replacement for Flash that nobody seems to use.
    • wmil 12y ago
      They've been around since 2001, but browser support only got good recently. Initially Adobe and Corel made browser plugins but Corel scaled back and Adobe bought Macromedia. Also SVG GUI tools haven't been great. Inkscape is strongly print focussed, or at least was the last time I used it. There's no reason to set a page size on a web svg... Someone needs to build an SVG equivalent of Flash Pro.
      • TazeTSchnitzel 12y ago
        You do need to set a page size, it's the reference point everything is measured relative to. It's only a problem if you can't zoom in/out infinitely.
    • escape_goat 12y ago
      Well, there was a good reason for that, for a long time: IE didn't even begin to support .svg until version 9, which means that it isn't a realistic option for deployment for anyone who needs to support pre-evergreen browsers. I investigated them a year or two or so ago, because I found that AngularJS could work quite nicely within an .svg document, which opened up some exciting possibilities. My recollection is that at the time, there were some critical cross-browser problems with font rendering that made the topic very kludgy and complicated. I except that matters have improved since, but I do not know to what extent.
• general_failure 12y ago
  A very good reason to go with express is TJ. He was the initial author of express and he is quite brilliant when it comes to code quality. Of course, TJ is no more part of the community but his legacy lives :-)
• Pharohbot 12y ago
  I wonder how Netflix would perform with using Dart with the DartVM. I reckon it would be faster than Node based on benchmarks I've seen. Chrome DartVM support is right around the corner ;)
• codelucas 12y ago
  > This turned out be caused by a periodic (10/hour) function in our code. The main purpose of this was to refresh our route handlers from an external source. This was implemented by deleting old handlers and adding new ones to the array. Unfortunately, it was also inadvertently adding a static route handler with the same path each time it ran. I don't understand the need of refreshing route handlers. Could someone explain they needed to do this, and also why from an external source?
  • Terr_ 12y ago
    It sounds to me like some kind of configuration/IT-infrastructure-control issue. I wouldn't be surprised if it relates to some sort of "we want a way to toggle this live without an actual code-push" narrative.
  • untog 12y ago
    I assume this is some kind of continuous delivery-style thing, where they can push up and amend endpoints without bringing down the entire system.
  • mjr578 12y ago
    We refresh periodically as we dynamically deploy new UI code, which can be accessed at new routes. (/home and /homeV2 for example) This allows us to not have to restart our servers or push out new server code just to serve a new UI at a different (or the same) route.
• pcl 12y ago
  Second, given a performance problem, observability is of the utmost importance I couldn't agree with this more. Understanding where time is being spent and where pools etc. are being consumed is critical in these sorts of exercises.
• coldcode 12y ago
  I must admit I could enjoy just doing this type of analysis all day long. Yet I hate non computing puzzles.
• hit8run 12y ago
  I would have written my apis in golang and not nodejs. Go is way faster in my experience and it feels leaner to create something because creating a web service can be productively doneout of box. Node apps tend to depend on thousands of 3rd party dependencies which makes the whole thing feel fragile to me.
• augustl 12y ago
  A surprising amount of path recognizers are O(n). Paths/routes are a great fit for radix trees, since there's typically repetitions, like /projects, /projects/1, and /projects/1/todos. The performance is O(log n). I built one for Java: https://github.com/augustl/path-travel-agent https://github.com/augustl/path-travel-agent
  • kyllo 12y ago
    How does your radix trie implementation handle variables in the URL paths, in a nutshell?
    • augustl 12y ago
      This is one of the reasons for why I call it a "bastardized radix tree" in the README :) The routes are stored as "nodes". There's a root node. It has a hash map of child nodes, by name. It also has a list of "parameterized" nodes. When a node gets a path segment, it will first look in its hash map. If nothing is there, it'll call the parameterized nodes in sequence. Typically there's just one parameterized node. For the following paths: /projects /projects/new /projects/special /projects/:project-id The root node will have a single item in its hash map, "projects". No items in the parameterized node. The node for "projects" will have to items in its hash map, "new" and "special". It will have a single item in it's parameterized node, for :project-id. I updated the README just now with a slightly more detailed explanation :)
      • kyllo 12y ago
        Very cool, thanks for taking the time to explain! I recently learned how tries work so it was cool to see a real-world implementation like yours.
• dmitrygr 12y ago
  So the lesson is to actually know the code you deploy to prod? Is that not obvious?
  • kin 12y ago
    Not everyone digs into the code behind their dependencies.
  • SixSigma 12y ago
    Your webserver, your dns resolver, your database, your operating system, the compilers they used, how about the bios, or the northbridge
  • serve_yay 12y ago
    Everyone should read the terms & agreements too.
• talkingtab 12y ago
  an unfortunate title. Ha ha "flames" ha ha "Node.js" but the article is really about express. Not so "ha ha"
  • snlacks 12y ago
    They renamed it "in flame graphs." For the audience, this was originally titled: "Node.JS in Flames"
• wpietri 12y ago
  From the article: > What did we learn from this harrowing experience? First, we need to fully understand our dependencies before putting them into production. Is that the lesson to learn? That scares me, because a) it's impossible, and b) it lengthens the feedback loop, decreasing systemic ability to learn. The lesson I'd learn from that would be something like "Roll new code out gradually and heavily monitor changes in the performance envelope." Basically, I think the approach of trying to reduce mean time between failure is self-limiting, because failure is how you learn. I think the right way forward for software is to focus on reducing incident impact and mean time to recovery.
  • quaunaut 12y ago
    > I think the right way forward for software is to focus on reducing incident impact and mean time to recovery. So in this case, guarantee you have a strong means of evaluating performance and maybe even include it by default just to be sure.
  • akkartik 12y ago
    Without over-training on this one incident, and without guidance on how to get from here to there (I'm still working on that): 1. Don't get suckered by interfaces, share code. If you create code for others to share ("libraries"), stop trying to hide its workings. 2. You don't have to learn how everything works before you do anything. But you should expect to learn about internals proportional to the time you spend on a subsystem. Current software is too "lumpy" -- it requires days or months of effort before yielding large rewards. The first hour of investigation should yield an hour's reward. 3. "Production" is not a real construct. There will always be things that break so gradually that you won't notice until they've gone through all your processes. Give up on up-front prevention, focus instead on practicing online forensics. And that starts with building up experience on your dependencies. More elaboration: http://akkartik.name/post/libraries2 http://akkartik.name/post/libraries2 My attempt at a solution: http://akkartik.name/about http://akkartik.name/about My motto: reward curiosity.
• rwaldin 12y ago
  I'm surprised nobody has mentioned that express has a built in mechanism for sublinear matching against the entire list of application routes. All you have to do is nest Routers (http://expressjs.com/4x/api.html#router http://expressjs.com/4x/api.html#router) based on URL path steps and you will reduce the overall complexity of matching a particular route from O(n) to near O(log n).
• pm90 12y ago
  I love these kinds of investigations into problems in production. I mean, you really have to admire their determination in getting to the root of the problem. In some ways, these engineers are not that different from academic researchers, in that they are devising experiments, verifying techniques, all in the pursuit of the question: why?
• exratione 12y ago
  The express router array is pretty easy to abuse, it's true. For example, as something you probably shouldn't ever do: https://www.exratione.com/2013/03/nodejs-abusing-express-3-to-enable-late-addition-of-middleware/ https://www.exratione.com/2013/03/nodejs-abusing-express-3-t... I guess the Netflix situation is one of those that doesn't occur in most common usage; certainly dynamically updating the routes in live processes versus just redeploying the process containers hadn't occurred to me as a way to go.
• sysk 12y ago
  > We also saw that the process’s heap size stayed fairly constant at around 1.2 Gb. > Something was adding the same Express.js provided static route handler 10 times an hour. Why didn't it increase the heap size? Maybe it was too small to be noticeable?
• ChuckMcM 12y ago
  The moneyquote: "We made incorrect assumptions about the Express.js API without digging further into its code base. As a result, our misuse of the Express.js API was the ultimate root cause of our performance issue." This situation is my biggest challenge with software these days. The advice to "just use FooMumbleAPI!" is rampant and yet the quality of the implemented APIs and the amount of review they have had varies all over the map. Consequently any decision to use such an API seems to require one first read and review the entire implementation of the API, otherwise you get the experience that NetFlix had. That is made worse by good APIs where you spend all that time reviewing them only to note they are well written, but each version which could have not so clued in people committing changes might need another review. So you can't just leave it there. And when you find the 'bad' ones, you can send a note to the project (which can respond anywhere from "great, thanks for the review!" to "if you don't like it why not send us a pull request with what you think is a better version.") What this means in practice is that companies that use open source extensively in their operation, become slower and slower to innovate as they are carrying the weight of a thousand different systems of checks on code quality and robustness, which people using closed source will start delivering faster and faster as they effectively partition the review/quality question to the person selling them the software and they focus on their product innovation. There was an interesting, if unwitting, simulation of this going on inside Google when I left, where people could check-in changes to the code base that would have huge impacts across the company causing other projects to slow to a halt (in terms of their own goals) while they ported to the new way of doing things. In this future world changes, like the recently hotly debated systemd change, will incur costs while the users of the systems stop to re-implement in the new context, and there isn't anything to prevent them from paying this cost again and again. A particularly Machievellan proprietary source vendor might fund programmers to create disruptive changes to expressly inflict such costs on their non-customers. I know, too tin hat, but it is what I see coming.
  • darkandbrooding 12y ago
    Please forgive me if I'm misinterpreting you, but the lament you make about open source software seems to me to be more about distributed systems. It is an interesting observation, and now that you've pointed it out I can observe the pattern at past and present employers. But I have seen that pattern on internal software, written by the company for the company. This makes me think the problem is an architectural one. I hesitate to comment about the relative velocities of open source vs proprietary software because I do not have enough experience with commercial, third party software. My sample size is too small, but I'm inclined to agree with you. I don't disagree with your Machiavellian conspiracy, either, but I've worked in marketing and advertising so I know that some of the villains are on the payroll. Maybe there needs to be a third category? There's open source software, written by someone who has no particular relationship to you. There's commercial software, written by someone who has a positive economic relationship with you. And then there is ... corporate?... software, written by someone who might think they have a zero sum relationship with you.
• hardwaresofton 12y ago
  Responses are already firing in: https://news.ycombinator.com/item?id=8632220 https://news.ycombinator.com/item?id=8632220
• degobah 12y ago
  tl;dr: * Netflix had a bug in their code. * But Express.js should throw an error when multiple route handlers are given identical paths. * Also, Express.js should use a different data structure to store route handlers. EDIT: HN commentors disagree. * node.js CPU Flame Graphs (http://www.brendangregg.com/blog/2014-09-17/node-flame-graphs-on-linux.html http://www.brendangregg.com/blog/2014-09-17/node-flame-graph...) are awesome!
• drinchev 12y ago
  NodeJS Project has already a similar issue about recursive route matching. https://github.com/strongloop/express/issues/2412 https://github.com/strongloop/express/issues/2412
• notastartup 12y ago
  this is why you stick to tried and true methods folks. this is such a typical node.js fanboy mentality. "reinventing the wheels is justified because asynchronous". or "i want this trendy way to do things just because everyone else is jumping on the bandwagon". Give me flask + uwsgi + nginx anyday.
  • CmonDev 12y ago
    In the end of the day Node.js is just a Reactor Pattern implementation in a form of bunch of scripts.
• _Marak_ 12y ago
  I read: "This turned out be caused by a periodic (10/hour) function in our code. The main purpose of this was to refresh our route handlers from an external source. This was implemented by deleting old handlers and adding new ones to the array" refresh our route handlers from an external source This is not something that should be done in live process. If you are updating the state of the node, you should be creating a new node and killing the old one. Aside from hitting a somewhat obvious behavior for messing with the state of express in running process, once you have introduced the idea of programmatically putting state into your running node you have seriously impeded the abiltity to create a stateless fault tolerant distributed system.
  • emeraldd 12y ago
    When I concluded what they had to be doing and then read the actual confirmation of what they were doing I was somewhat shocked. Why on Earth would you want to programatically recreate the routes in an express app?!?!? It would be really interesting to see a write up on what/why they think this kind of behavior is needed in the first place ....
• bcoates 12y ago
  It's not just the extra lookups -- static in express is deceptively dog-slow. For every request it processes, it stats every filename that might satisfy the URL. This results in an enormous amount of useless syscall/IO overhead. This bit me pretty hard on a high-throughput webservice endpoint with an unnoticed extra static middleware. I wound up catching it with the excellent NodeTime service. Now that I look at it, there's a TOCTOU bug on the fstat/open callback, too: https://github.com/tj/send/blob/master/index.js#L570-L605 https://github.com/tj/send/blob/master/index.js#L570-L605 This should be doing open-then-fstat, not stat-then-open.
• revelation 12y ago
  Crazy talk. In 1ms, I can perspective transform a moderately big image. NodeJS cant iterate through a list. We really need a 60 fps equivalent for web stuff. You have 16ms, thats it.
  • nostrademons 12y ago
    FWIW, a lot of the stuff the Chrome+Polymer team is working on explicitly has a 60fps goal and 16ms frame budgets. I remember in my last project at Google, I spent a lot of time working with the Chrome team to get various parts of websearch rendering under the 16ms budget. (I couldn't manage it given the amount of legacy code we had to work with, but various other people have continued the work since I left, so hopefully they've had more luck.)
• Fishrock123 12y ago
  I would like to mention that Netflix could have consulted the express maintainers (us) but didn't. Source: myself - https://github.com/strongloop/express/pull/2237#issuecomment-59681175 https://github.com/strongloop/express/pull/2237#issuecomment...
• deleted 12y ago
  [deleted]
• tjholowaychuk 12y ago
  Sounds like a documentation issue, or lack of a staging environment. I've written and maintained countless large Express applications and routing was never even remotely a bottleneck, thus the simple & flexible linear lookup. I believe we had an issue or two open for quite a while in case anyone wanted to report real use-cases that performed poorly. Possibly worth mentioning, but there's really nothing stopping people from adding dtrace support to Express, it could easily be done with middleware. Switching frameworks seems a little heavy-handed for something that could have been a 20 minute npm module.
• jaytaylor 12y ago
  I am upset that the title has been changed from "Node.js in Flames". Which is not only the real title of the article, but also a reasonable description of what they've been facing with Node. #moderationfail
  • dang 12y ago
    I can see why you would feel that way, but the title, while clever and (I'll take your word for it) fitting, was arguably misleading and unarguably baity. The HN guidelines call for changing such titles, so the moderators were just doing their job. There likely would have been more complaints about the title if we hadn't changed it.
    • jaytaylor 12y ago
      Understood. And thank you for following up, I really do appreciate it.
• clebio 12y ago
  > I can’t imagine how we would have solved this problem without being able to sample Node.js stacks and visualize them with flame graphs. This has me scratching my head. The diagrams are pretty, maybe, but I can't read the process calls from them (the words are truncated because the graphs are too narrow). And I can't see, visually, which calls are repeated. They're stacked, not grouped, and the color palette is quite narrow (color brewer might help here?). At least, I _can_ imagine how you could characterize this problem without novel eye-candy. Use histograms. Count repeated calls to each method and sort descending. Sampling is only necessary if you've got -- really, truly, got -- big data (which Netflix probably does), but I don't think the author means 'sample' in a statistical sense. It sounds more like 'instrumentation', decorating the function calls to produce additional debugging information. Either way, once you have that, there are various common ways to isolate performance bottlenecks. Few of which probably require visual graphs. There's also various lesser inefficiencies in the flame graphs: is it useful (non-obvious) that every call is a child of `node`, `node::Start`, `uv_run`, etc.? Vertical real-estate might be put to better use with a log-scale? Etcetera, etc.
  • donavanm 12y ago
    > The diagrams are pretty, maybe, but I can't read the process calls from them (the words are truncated because the graphs are too narrow). Flame Graphs provide SVGs by default. You should be able to zoom if your broser supports it. The current version also supports "zooming" in to any frame in the stack, resetting that frame as the base of the display. Also WRT the base frames of 'node' et al its because Flame Graphs are a general use tool for stack visualization, it might be 'main' for a c program or the scheduler looking at a system. > They're stacked, not grouped, and the color palette is quite narrow (color brewer might help here?). Colors by default have no meaning and the palette is configurable. The current lib can also assign colors by instruction count/ipc and width by call count, if you have access to that. > Sampling is only necessary if you've got -- really, truly, got -- big data (which Netflix probably does), but I don't think the author means 'sample' in a statistical sense. It is sampling. Flame graphs re typically used with something like perf/dtrace/oprofile which dumps stacks at a couple hundred to thousand hertz. Actual call tracing is (typically) not feasible for large/prod stacks.
  • 12y ago
• MichaelGG 12y ago
  Would someone explain what I'm missing about the flame graphs? Why are they indispensable here? In a normal profiler, you'd just expand the hot path and see what had the most samples. Apart from making recursion very explicit, what special aspect do flame graphs expose?