8 ms·
How I spent two weeks hunting a memory leak in Ruby (2015)
- busterarm 10y agoWhat a timely post. I had a similar issue that I was tracing last week that did end up being my Ruby code...and it turns out I was modifying a constant like in the example. What a fun read! I've been learning more about Ruby's GC since 2.1 and this got me looking even deeper -- definitely picked up a couple of new tricks/tools from this. Thank you be9!
- mwpmaybe 10y agoFreeze your constants!
- busterarm 10y agoI used to do this with strings (now you don't have to), but this is great advice!
- mwpmaybe 10y agoYeah, it's one of those rare Ruby gotchas. If you assign some kind of a data structure to a constant, you'll get a warning if you try to reassign the constant, but the data structure itself will silently mutable. To disallow this behavior, simply freeze the data structure. If you have nested data structures, you must freeze deeply. ENUM = [0, 1, 2].freeze LOOKUP = { :A => [1, 2, 3].freeze, :B => [4, 5, 6].freeze }.freeze # or: LOOKUP = { :A => [1, 2, 3], :B => [4, 5, 6] }.tap { |s| s.values.each(&:freeze) }.freeze
- rgtk 10y agoThere is no reason for using native extensions in scripting languages like Ruby when program load is pretty low (especially when they are used for performance reason, not binding). Great insight, though. Author described this experience as it was a great venture... in hindsight I suppose :-)
- mnarayan01 10y ago> when program load is pretty low The leak was in a gem.
- rgtk 10y agoGem is still library albeit its name.
- mnarayan01 10y ago1. AFAIK it's the most popular Markdown gem around (ref. https://www.ruby-toolbox.com/categories/markup_processors https://www.ruby-toolbox.com/categories/markup_processors) so using it in a project is more "default" than it is "optimization". 2. With over 10M downloads, I assume at least some of them spend large percentages of time using it. 3. It's Markdown; replicating the idiosyncrasies of SmartyPants (https://daringfireball.net/projects/smartypants/ https://daringfireball.net/projects/smartypants/) in a different language is not something many people are going to want to do. I'm not saying that people should or should not use Redcarpet; just that it's unlikely that anyone engaged in "premature optimization" here.
- teej 10y agoI'd say it's appropriate for anything in the critical path - talking to databases being the only major thing that comes to mind. I agree that a native extension for markdown parsing is way overkill. Stick with pure-language extensions (they're so much easier to work with!) until you can measure and quantify the cost and benefit of switching.
- yxhuvud 10y agoThe reason for the binding can be that no-one can be assed to reimplement the functionality. It is not as if anyone expect the scripting languages to implement their own versions of openssl.
- scott_s 10y agoYour criticism really applies to the gem authors, which to me, does not make sense. If I, as a library developer, want my library to be used on the critical path in heavy-load situations, and the library is for a language like Ruby, I'm going to figure out which part of my library are the performance critical parts, and implement them in a native language.
- pjleonhardt 10y agoThat is fantastic dedication to finding the source of the memory leak. I'm not sure I'd have quite the devotion or expertise to track that one down.
- bru 10y agoAbout the patch at https://github.com/vmg/redcarpet/pull/516/files https://github.com/vmg/redcarpet/pull/516/files, couldn't the update simply have been the following? - return Data_Wrap_Struct(klass, rb_redcarpet_rbase_mark, NULL, rndr); + return Data_Wrap_Struct(klass, rb_redcarpet_rbase_mark, xfree, rndr);
- busterarm 10y agoC is not really my area of expertise, but doesn't xfree have alternatives? like free, freeif, etc? I'd much rather have to change that later in one place than several (if there end up being more calls to xfree in the code) if at some point the implementation needs to change again.
- simcop2387 10y agoSo a from a quick googling it looks like xfree()[1] is a ruby specific thing, and should be used internally for everything allocated by ruby. It also looks like it's got the same signature as the function created for this so that should work there too. [2] [1] http://clalance.blogspot.com/2011/01/writing-ruby-extensions-in-c-part-12.html http://clalance.blogspot.com/2011/01/writing-ruby-extensions... [2] http://inferior-products.com/docs/userdocs/ruby19/html/d8/d16/gc_8c.html#a0bffec5b2cc004adcebb6802e7620387 http://inferior-products.com/docs/userdocs/ruby19/html/d8/d1...
- wyldfire 10y agoThese are always interesting bug hunts. I tried to add ASan to the travis config for this project but I couldn't quite figure out how to change CFLAGS and/or CC. Never used ruby but interwebs hinted that the bundle config/install commands might accept "--cc" and "--with-cflags" commands. It's ignored when I tried it though [1]. [1] https://travis-ci.org/androm3da/redcarpet/jobs/159711520 https://travis-ci.org/androm3da/redcarpet/jobs/159711520
- darkhorn 10y agoDoes PHP have similar issues? I think it shouldn't.
- jmcgough 10y agoPHP has garbage collection now, so it's a lot harder to cause a memory leak. Before PHP 5.3.0, cyclical references were one of the biggest causes of memory leaks. However, you can still create memory leaks just like you could in Ruby - dynamic allocation in a c extension that isn't freed up (like in this story), or creating objects that can't be garbage collected (like a slowing expanding array in a global variable).
- lotyrin 10y agoPHP extensions in C are pretty rare since it's so hard to distribute them (outside a few exceptions that exist in common package management systems), so in that sense, I guess? But no, a PHP extension in C can have a memory leak like any other C code. It's not at all uncommon actually, since there are fewer consequences; CGI mode PHP cleans up after every request, FastCGI mode PHP by default restarts workers after a number of requests because of the class of effects (unwanted state) that includes memory leaks.
- vemv 10y agoWhile the described debug process deserves nothing but my applause, I wonder - could the issue have been more easily diagnosed? Ideally much less tooling would have been needed.
- samsk 10y agoshameless plug: I've made malloc hooking preload library for hunting such bugs, that automatizes most of the described work - https://github.com/samsk/log-malloc2 https://github.com/samsk/log-malloc2
- nateberkopec 10y agoThis looks really, really cool. You should download the bugged version of this gem and make a screencast showing how your tool makes this process easier.
- deleted 10y ago[deleted]
- kyledrake 10y agoInteresting post, but I did want to chime in quickly and say that it's pretty absurd to only have 1GB for a web application in 2016, even for a small one. This is why I've been dismissive of using Heroku for my projects, even though I run a lean stack. They have higher RAM options, but they are incredibly expensive. For the cost of a 14GB Heroku Dyno, I can buy a dedicated server off ebay with 32GB ECC every single month. I don't use all of that RAM, but it's nice to have some extra RAM to toss around when you need to in a pinch. I get that it means you have to "run a server", and insert arguments for expensive cloud providers vs DIY servers here but I don't think it's any less crazy than being forced to chase a GC white whale for two weeks on a tiny memory leak to avoid a huge rate hike on your hosting bill. On an aside, I've only had one problematic memory leak with ruby ever (the infamously leaky RMagick), I threw this in every time I used the lib and it solved it for me: GC.start(full_mark: true, immediate_sweep: true) Bit of a performance hit, but not enough to cause a problem for my use case.
- foobarian 10y agoMy favorite Ruby memory leak was in a text processing app that was extracting via regex short strings (< 50 chars) out of 10MB+ text files. The short strings were apparently substring-like objects that held on the original 10MB text and indexed into it at a particular offset. Solution? Return the short_string + "" and then the GC did its thing.
- mikekchar 10y ago> it's pretty absurd to only have 1GB for a web application in 2016 While I don't for a minute doubt what you say, as an old guy, the absurdity of 1GB being insufficient for serving up web pages hits me pretty hard.
- oldmanjay 10y agoWeb applications don't just serve up web pages, which puts you in the unenviable position of being dismissive or ignorant about the topic.
- 10y ago
- ajsharma 10y agoGreat read. What's more, I found out that my company's codebase was still using the broken 3.3.2 version (code fix in the works). Love it when reading HN pays off immediately like that :)
- andrewchambers 10y agoOne you identified the leaked address I think a reverse debugger could have helped track the alloc down.
- stretchwithme 10y agoWouldn't it be useful if the community tested EVERY gem to see if it leaked memory, perhaps each developer stepping up to test one, in an app that just exercised the gem in question? To test all of the ones we use individually is prohibitive. But thousands of people could do test one.
- andrewvijay 10y agoAmazing stuff. I took time to understand this article by going through each part in detail. Learnt quite some stuff. Thank you for this incredible story!
- allendoerfer 10y agoIs there a simple way to notice your own subtle mistakes, when writing in a foreign language? The missing articles in texts from Slavic people bug me more than they should and I wonder what kind of (stylistic) errors I myself make. The English grammar is mostly just simplified German, but there are some constructs, which I know of, but do not naturally use (e.g. "your every …", "to name but two") - they somehow seem twisted to me. Another one are compound words, I have given up on distinguishing between spaces, hypens and actual compounds and just use spaces almost everywhere.
- co_dh 10y agoI had similar experience of finding memory leak in Python. It's a 64 G server, but we load everything from database to memory. I didn't success. This experience made me question myself: is it wrong to create a lot of objects in memory and let them reference each other, and hope Garbage Collector to magically work? If instead of object, we just put data into memory more organized as a (column) database, then we don't need GC anymore. OOP made programming easier, by modeling real world as object. At the same time, it made memory management harder, since it's not nature to the hardware's memory model: a linear array of memory cells. I wish in future, there is a place for this paradigm of in memory database model of programming.
- firebones 10y agoIt's called "data-oriented design". Look into the gaming industry, where there's a lot of anti-OOP sentiment. They're primarily coming at it from the standpoint of optimizing memory access for cache lines, but there are side benefits that come from it around reducing memory allocation and usage, and avoiding the cost of abstractions that come with hidden inefficiencies.
- ktRolster 10y agoA little worrisome that the fix hasn't been merged. (edit: it's been fixed)
- richardwhiuk 10y agoDid you get all the way to the end of the blog post? "Update on 2015/09/29. Redcarpet fix has been released."
- ktRolster 10y agooh, thanks. Disregard my previous comment.