6 ms·

>GitLab has memory leaks. These memory leaks manifest themselves in long-running processes, such as Unicorn workers. (The Unicorn master process is not known to

by parfe 8y ago

>GitLab has memory leaks. These memory leaks manifest themselves in long-running processes, such as Unicorn workers. (The Unicorn master process is not known to leak memory, probably because it does not handle user requests.) >To make these memory leaks manageable, GitLab comes with the unicorn-worker-killer gem. This gem monkey-patches the Unicorn workers to do a memory self-check after every 16 requests. If the memory of the Unicorn worker exceeds a pre-set limit then the worker process exits. * I know this is a Rails issue, but I cannot believe this is still a problem. I remember when the RoR folks blamed the Rails memory leaks on Zed, and Mongrel, more than a decade ago. They were obviously incorrect. Did the community simply give up on fixing the problem? * https://docs.gitlab.com/ee/administration/operations/unicorn.html#unicorn-worker-killer https://docs.gitlab.com/ee/administration/operations/unicorn...

obl 8y ago
I have no idea about either ruby or rails but since ruby is a managed language this kind of issues seem like they should be pretty easy to debug ? After all, if the gc is not freeing the memory then it must know a reachable path to it so the blame should be easy to assign. Does ruby not have any good heap profiling tool ?
- skrebbel 8y ago
  I don't know anything about Rails but most memory leaks in managed languages are stuff like lists growing infinitely large, parts of them pointing to data that is no longer relevant. Think badly implemented caches, stuff like that. If you have enough not-perfectly-cleanly-separated layers of data management interacting, then this gets hard to debug pretty fast.
  - pjmlp 8y ago
    Not when you have GC graphical debuggers like Java,.NET and JavaScript do.
qop 8y ago
Were you under the impression that Rails has a functioning dev community? I am very familiar with the rubyverse, and boy do I got news for you. Rails is to Ruby as the Big Dig is to Boston. But it's hard to get Rails devs to admit it. They know it, of course. They live it. They breathe bad code, they exist as extensions of a philosophy that has failed it's way to the top. I used to think Zed was a dramatic weenie. Then I got a bit deeper in Rails to learn how to pursue better exploits. And deeper. And deeper. And finally I just hit a point where I just wasn't willing to work on RoR-related gigs. Convention over configuration is one of those ideas that goes great on paper. And then you start in on some buzzwords and fancy lingo, it's the bread and butter of selling to project managers. But there is a point where CoC breaks down at a technical level, and that point is literally the first moment you come across something difficult to express Rails is weak in doing things ANY other way than the Rails way, and what developers do (and it's not their fault) is using their creativity and ingenuity to solve the problem. I believe this is why metaprogramming in Ruby has such a vocal backing, because there are millions of shims holding up millions of Rails' shitty choices that depend on method_missing fuckery or string-based define_method gadgets. In the spirit of what it means to solve hacker-ish problems, it's great fun to program like that for a while, and then a year or two down the line you look at that shit and it can really break you down. Good developers mistake themselves for bad developers because of this, to be honest. Zed's probably had the idea to write an entire book of how insanely FUCKED Rails actually is, but to make the book plausibly thorough would mean a human being being exposed to far more stupidity than modern science can safely treat for. It'd be dangerous to have that much stupid intake just for a book. That unicorn-worker-killer gem is a perfect caricature of the kinda problem-solving you run into with Rails. Let it die a horrible death, restart that shit with some logic to let it keep dying predictably. That technique, multiplied by every component in the system, multiplied by every developer on the team, multiplied by MVC requiring developers to have to know too broadly the system they're working on, multiplied by however many years and Rails versions the project has somehow survived, you approach something I can only imagine is quite similar to what hell must be.
- artursapek 8y ago
  I spent 3 years working on a Rails app and it was enough to make me swear off Rails for the rest of my life. I think the Rails dev community has good intentions, and the design of Rails is actually pretty nice. The problem in my opinion is more with Ruby, and human laziness. Ruby lets programmers do absolutely anything they can think of no matter how unexpected or un-intuitive it will be to their colleagues. Combined with the rigid conventions of Rails' design, this inevitably leads to a maintenance disaster with people, like you say, getting too creative. Refactoring a large Ruby on Rails codebase is basically impossible. There's no compiler to help you along; hell the code is evaluated lazily. So instead of refactoring, people monkey-patch their problems away. It's horrible, really really horrible. I'm now getting flashbacks to trying to understand how files get loaded in that codebase... I need a drink
briandear 8y ago
Why are they still using unicorn over Puma? That’s one question. Using unicorn killer is the sign of some bad engineering. It’s the equivalent of restarting your server every hour.
- sytse 8y ago
  We are planning to start using multi threading and Puma so we can reduce the memory requirements of GitLab. The last time we shipped that we had to revert due to hard to diagnose errors. We're taking a more incremental approach now.

by parfe 8y ago

obl 8y ago

I have no idea about either ruby or rails but since ruby is a managed language this kind of issues seem like they should be pretty easy to debug ? After all, if the gc is not freeing the memory then it must know a reachable path to it so the blame should be easy to assign. Does ruby not have any good heap profiling tool ?

skrebbel 8y ago
I don't know anything about Rails but most memory leaks in managed languages are stuff like lists growing infinitely large, parts of them pointing to data that is no longer relevant. Think badly implemented caches, stuff like that. If you have enough not-perfectly-cleanly-separated layers of data management interacting, then this gets hard to debug pretty fast.
- pjmlp 8y ago
  Not when you have GC graphical debuggers like Java,.NET and JavaScript do.

qop 8y ago

Were you under the impression that Rails has a functioning dev community? I am very familiar with the rubyverse, and boy do I got news for you. Rails is to Ruby as the Big Dig is to Boston. But it's hard to get Rails devs to admit it. They know it, of course. They live it. They breathe bad code, they exist as extensions of a philosophy that has failed it's way to the top. I used to think Zed was a dramatic weenie. Then I got a bit deeper in Rails to learn how to pursue better exploits. And deeper. And deeper. And finally I just hit a point where I just wasn't willing to work on RoR-related gigs. Convention over configuration is one of those ideas that goes great on paper. And then you start in on some buzzwords and fancy lingo, it's the bread and butter of selling to project managers. But there is a point where CoC breaks down at a technical level, and that point is literally the first moment you come across something difficult to express Rails is weak in doing things ANY other way than the Rails way, and what developers do (and it's not their fault) is using their creativity and ingenuity to solve the problem. I believe this is why metaprogramming in Ruby has such a vocal backing, because there are millions of shims holding up millions of Rails' shitty choices that depend on method_missing fuckery or string-based define_method gadgets. In the spirit of what it means to solve hacker-ish problems, it's great fun to program like that for a while, and then a year or two down the line you look at that shit and it can really break you down. Good developers mistake themselves for bad developers because of this, to be honest. Zed's probably had the idea to write an entire book of how insanely FUCKED Rails actually is, but to make the book plausibly thorough would mean a human being being exposed to far more stupidity than modern science can safely treat for. It'd be dangerous to have that much stupid intake just for a book. That unicorn-worker-killer gem is a perfect caricature of the kinda problem-solving you run into with Rails. Let it die a horrible death, restart that shit with some logic to let it keep dying predictably. That technique, multiplied by every component in the system, multiplied by every developer on the team, multiplied by MVC requiring developers to have to know too broadly the system they're working on, multiplied by however many years and Rails versions the project has somehow survived, you approach something I can only imagine is quite similar to what hell must be.

artursapek 8y ago
I spent 3 years working on a Rails app and it was enough to make me swear off Rails for the rest of my life. I think the Rails dev community has good intentions, and the design of Rails is actually pretty nice. The problem in my opinion is more with Ruby, and human laziness. Ruby lets programmers do absolutely anything they can think of no matter how unexpected or un-intuitive it will be to their colleagues. Combined with the rigid conventions of Rails' design, this inevitably leads to a maintenance disaster with people, like you say, getting too creative. Refactoring a large Ruby on Rails codebase is basically impossible. There's no compiler to help you along; hell the code is evaluated lazily. So instead of refactoring, people monkey-patch their problems away. It's horrible, really really horrible. I'm now getting flashbacks to trying to understand how files get loaded in that codebase... I need a drink

briandear 8y ago

Why are they still using unicorn over Puma? That’s one question. Using unicorn killer is the sign of some bad engineering. It’s the equivalent of restarting your server every hour.

sytse 8y ago
We are planning to start using multi threading and Puma so we can reduce the memory requirements of GitLab. The last time we shipped that we had to revert due to hard to diagnose errors. We're taking a more incremental approach now.