16 ms·
Walmart Node.js Memory Leak
- batbomb 13y agoCan anyone tell me if there is reason for this in bash? DEST=~~/public/walmart.graphs
- stewars 13y agoNot bash. '~~' gets replaced by the MANTA_USER environment variable by the manta command line tool mput.
- ambirex 13y agoThank you, I really enjoy detailed write-ups like this. It is fascinating to see how an engineer approaches an elusive problem.
- ryanseys 13y agoAnd a one-line fix. Damn that must be satisfying.
- yen223 13y agoReminds me of that old joke: The office photocopier broke down, so the manager called in a repairman. The repairman takes one look at the machine, draws an 'X' at the problem part, and hands the manager a bill for $500. The manager was shocked at the price, and demanded an itemized bill. The repairman simply wrote: Marking the 'X' - $1 Knowing where to put the 'X' - $499
- lstamour 13y agoI started Googling the Picasso "principle" about it being a lifetime to know how to do it, but it turned into Googling this one instead. Found a snippet, "Karl Steinmetz (German-born, U.S citizen), the well known electrical engineer who worked out many details of a.c. theory and was responsible largely for the adoption of a.c. for commercial use, was once called in by the General Electric Company to examine a poorly performing transformer. After a few minutes, Steinmetz marked an x on the transformer core and said, “It will work if you take off the turns from this x to the end.” The prescription worked well, and Steinmetz later sent G.E. a bill for his service of $10,000. The company official thought the bill excessive and asked for the itemization. Steinmetz then sent them a more detailed bill: For putting x on transformer core : $1; for knowing where to put the x: $9999." It's funny that in today's world, both Picasso and Steinmetz take "minutes" to do this, but in perhaps earlier tellings, it took hours for Picasso to do his work and days for Steinmetz: http://edisontechcenter.org/CharlesProteusSteinmetz.html http://edisontechcenter.org/CharlesProteusSteinmetz.html
- yen223 13y agoWow, thanks for the link! I would never have thought that the joke was based on an actual incident. For those who are wondering about the Picasso principle, it's based on a story about a woman who asked Picasso why he charged $5000 for a painting, when it only took him seconds to paint it. He replied, "Madam, it took my entire life!".
- zb 13y agoNever happened. http://www.snopes.com/business/genius/where.asp http://www.snopes.com/business/genius/where.asp That page does suggest a possible origin for the (equally apocryphal) Picasso fable, though, in a quote from James McNeill Whistler.
- lstamour 13y agoDid I read the same page you did? It seemed quite generic and gave the following rating. Which means it hasn't been explicitly disproven. In fact, I'd suggest that these things might indeed have happened if the skilled worker, needing to prove his worth, used such a glib line because he'd heard it somewhere else. But I'm skeptical too, that's why I was Googling. You'd think someone, somewhere would know how long each took and was more consistent in the retelling. Or that they would have kept the itemized receipt for the joke. LEGEND: Hollow yellow bullets are the ones most commonly associated with "pure" urban legends — entries that describe plausible events so general that they could have happened to someone, somewhere, at some time, and are therefore essentially unprovable. Some legends that describe events known to have occurred in real life are also put into this category if there is no evidence that the events occurred before the origination of the legends.
- gruseom 13y agoSnopes provides no citation for the Whistler story. That prompted me to look it up: http://en.wikipedia.org/wiki/James_Abbott_McNeill_Whistler#Ruskin_trial http://en.wikipedia.org/wiki/James_Abbott_McNeill_Whistler#R... I had no idea that this quote has such a delightful and well-documented origin. I'd only heard the story told about Picasso (and various mechanics and engineers). A great example of how these things morph over time. The story is delightful because it pitted two great Victorian aesthetes against one another. Ruskin had said this about Whistler: I have seen, and heard, much of Cockney impudence before now; but never expected to hear a coxcomb ask two hundred guineas for flinging a pot of paint in the public's face. So Whistler sued for defamation and was examined by Ruskin's lawyer: Holker: Did it take you much time to paint the Nocturne in Black and Gold? How soon did you knock it off? Whistler: Oh, I 'knock one off' possibly in a couple of days – one day to do the work and another to finish it. Holker: The labour of two days is that for which you ask two hundred guineas? Whistler: No, I ask it for the knowledge I have gained in the work of a lifetime. The insinuation in the lawyer's question ("how soon did you knock it off?") is hilarious! Whistler, by the way, was a great wit and had a famous skirmish with Oscar Wilde: http://quoteinvestigator.com/2013/09/05/oscar-will/ http://quoteinvestigator.com/2013/09/05/oscar-will/ ... which inspired this Monty Python classic: http://www.youtube.com/watch?v=UxXW6tfl2Y0 http://www.youtube.com/watch?v=UxXW6tfl2Y0
- Scottopherson 13y agoMan I'd be shocked too if the repairman only drew an 'X' on the problem part instead of repairing the problem part.
- rcthompson 13y agoIronically, this page hangs Chrome indefinitely when I try to load it. Luckily it only hangs the tab so I can still close it. I guess I'll fire up Firefox to see if I can actually read the article. Edit: Actually, it loads fine in a private browsing tab, so it must be a bad interaction with some extension. Oh well.
- tfb 13y agoIt loads instantly for me. I'm using Chrome 31.0.1650.57 on Windows 7.
- rcthompson 13y ago31.0.1650.57 on OSX 10.9. Firefox handles it just fine. Could be some interaction with an extension I have installed.
- dfc 13y agoI am curious why you find this ironic? What is your definition of irony?
- pritambaral 13y agoChrome uses V8. Chrome is the primary user of V8. Not supporting OP's definition of irony, whatever it is, just speculating how OP could've thought of it.
- patrickg_zill 13y agoThat is pretty impressive - I love how they could use DTrace to scope out what was going on.
- charlieflowers 13y agoFYI, a typo -- "illusive" -> "elusive". (haven't read further yet, just wanted to let you know).
- ilaksh 13y agoI think there are still quite a few C and C++ programmers out there. To me this is a great example of why it is better software engineering to write a server in something like Node.js. Because rather than having a million code bases with potential memory leaks like this one, there is just the Node code. In ordinary JavaScript code its impossible to cause a problem just that.
- sbov 13y agoIt is fairly easy to create a long running server in a GC'd language that will continually consume more memory. Some don't like to call it a memory leak, which is why I put it the way I did, but the effect is the same. At the end of the day, the more that you think this is impossible the more likely your programs will experience it. So please don't think that your program is immune to this because you use Javascript.
- tantalor 13y agoGood example might be a server process which never releases memory, so the longer it runs the more memory it "consumes". That is, the maximum memory required to handle any previous request. This might be a well known solved problem, but I have heard it mentioned before.
- tehwalrus 13y agoI have an apache box that runs a bunch of PHP and flat HTML sites. I have to set it to only use 10 processes, and to kill them every half hour, because they all gradually swell up to 35MB each (which I imagine is where they've loaded pretty much all the PHP on my server, independently of each other). Without the number limit, or the kill policy, the server runs out of RAM and crashes. (it's only a cheap one, with 512MB RAM.) Luckily it's a very low traffic set of sites, so these limits don't break the experience. I'm glad I didn't have to solve this problem any deeper!
- driverdan 13y ago
- city41 13y agoI've been running an extremely simple Node application on 0.10.18 for a while now and it has a very gradual memory leak. My code is just a few dozen lines, and it all seems pretty innocent. I am also using Hapi, so I thought maybe Hapi has a leak in it somewhere. Now I wonder if I have the same leak as Walmart here. I just now upgraded to 0.10.22 and am curious to see where I end up. If the leak goes away then hot damn, I got lucky :)
- retr0h 13y agoI've always loved the debugging tools in solaris (smartos or whatever now).
- diminoten 13y agoI'm actually looking into a segfault issue deep in the bowels of a C++ addon we have in node.js (anyone in #node.js will have seen me over the past few weeks ask about it), but what reading this makes me realize is how woefully underequipped I am to hunt for problems of this nature. My problem is likely in one of our addons, but this kind of debugging, this whole genre of problem solving is entirely beyond me. How do I get to this level? What do I need to learn? To study? It's just a little depressing to read something like this and see how far the road ahead goes, despite how far I've already traveled...
- ryanobjc 13y agoI find you need two things: - the "troubleshooter" mentality/thinking pattern - extensive system knowledge I haven't figured out how to teach #1, except maybe for "don't have anyone to ask for help" and #2 is self explanatory.
- barrkel 13y agoYou need determination and experience, and some knowledge of how code is compiled at a low level. Tools like those described in the article are handy, but aren't absolutely necessary. They save a lot of time, but the same effects can usually be gotten by more laborious means. You have a segfault. You should know where in the code it's occurring already; it's either an access to bad memory with the instruction pointer (IP) at the point of access, or it's an attempt to execute code with the IP pointing at the bad memory, in which case the top of the stack (or, depending on calling convention, one of the registers) normally contains the place where it came from (necessarily, since the code expected to be returned to). There are ways to turn an instruction pointer into line number offset when you have appropriate debug info, if you can't get the program running under a debugger. Given the line number, segfaults can typically be split into three categories: plain bad logic, use after free, and memory corruption. The last is hardest to find IME, most easily done using a debugger and hardware breakpoints on memory address modifications, but you need a stable repro and a consistent memory allocator that gives predictable addresses for every rerun. If any of the above is meaningless to you, it should give you some clues as to where you need to research.
- stiff 13y ago
- jt884 13y agoNode.js is still cancer.
- jzwinck 13y agoI'd like to read more about how we can prevent this class of error going forward. Could stronger typing or RAII or some other feature or trick have made the bug apparent at compile time? I made a very basic Node.js module in C++ with V8 and it was surprisingly difficult to make a good (idiomatic JS behaviour, believably bug-free) wrapper for a straightforward class and factory method. I say this coming from Boost Python and Luabind, where there are some tricky parts to bind complex classes, but simple ones are easy enough, and once written, obviously correct.
- aaronbrethorst 13y agoWonderful blog post; major props for the engineering time expenditure. But, why do you have an Olark chat widget that says "Contact Sales". I don't want to have anything to do with those schlubs! If anything, I want to talk to serious engineers like you! Perhaps a better call to action would be: * Talk to us about how we can solve your problems * Chat with us * We can help you too * What's up?
- davidw 13y agoI looked at node.js for a system I'm involved with creating, but ultimately we went with Erlang just because it's been around a lot longer and is more stable in terms of things like this. We're working on a semi-embedded system that will not always be on-line or accessible for debugging. We also considered Go, which probably would have been more familiar to C++ guys, but it was also deemed a bit immature even if it seems like a very pleasant language to work with. Cool writeup though!
- andreypopp 13y ago2 or 3 years ago I hit memory leak in Erlang's stdlib's httpc... just saying.
- gcb1 13y agois inet part of stdlib?
- andreypopp 13y agoWell, it's a part of OTP, which is, I believe, "the true" stdlib of Erlang.
- rcb 13y agoThis is impressive work by the Joyent team! I've seen two sources of memory leaks in Erlang based systems: 1) unbounded process message queues, and 2) passing binaries across process (pid) boundaries. Many beginning erlangers run into these, and they're relatively easy to identify and correct. With a little practice, these become easy patterns to recognize and avoid. As far as httpc, I'm unaware of that bug -- but I can say that I recently worked on a commercial product that leveraged httpc as a core component of the service, and it worked fine.
- ZephyrP 13y agoDon't forget non tail-call methods and funky parse_transforms!
- jokoon 13y agowe know that node.js is a bad piece of software, you don't need to remind us about it all the time (down vote me)
- jt884 13y agoI agree. Here, have an upvote from a fellow curmudgeon.
- joeblau 13y agoExcellent details on the sleuthing that went on to find this error. I think it's great that there are great tools available to debug errors like this and your write up helps me in learning more about how to go about properly debugging my Node apps.
- atomical 13y agoI assume that they can restart the server at intervals or use load balancing. A few months of developer timer for something like this seems excessive unless he was working on something else as well.
- spyc3r 13y agoAs a former software engineer at Walmart I can tell you that a few months for something like that is nothing to them. They employ several thousand devs at the home office. Having one of them focus on a bug like this isn't an issue in terms of time or money. In their minds its worth it given the scale of the enterprise.
- jnazario 13y agocool writeup. while not a node.js user, i love these sorts of tours of system internals - i always learn a lot, both specific tools and also processes of using them. thanks for the details, very articulate and useful stuff.