6 ms·

Maybe I’m missing something, but it seems the only changes here over upstream are jamlloc and/or malloc_trim. I’m not as familiar with malloc_trim, but jemalloc

Maybe I’m missing something, but it seems the only changes here over upstream are jamlloc and/or malloc_trim. I’m not as familiar with malloc_trim, but jemalloc is fairly trivial to setup without recompiling Ruby (using LD_PRELOAD), and there are even buildpacks for environments like Heroku. It seems like more time was spent on the marketing page than on the actual optimizations.

FooBarWidget 5y ago
Here I explain why it can quickly become not so trivial: https://news.ycombinator.com/item?id=27872696 https://news.ycombinator.com/item?id=27872696 The goal of Fullstaq Ruby is to democratize the fight against Ruby memory bloat. Democratization means that as many people should be able to reap the benefits as possible. It's 2021 now, and expecting users to compile Ruby or Jemalloc from source is no longer realistic. Compiling anything is no longer a non-scary, low-friction thing to do.
- wgjordan 5y ago
  The parent comment noted that using jemalloc for Ruby without compiling is already trivially easy, so the argument that 'compiling is hard' is irrelevant even if true. The real argument is that installing jemalloc separately and running Ruby with an environment variable is too hard.
  - FooBarWidget 5y ago
    In my comment, as well as in the FAQ, I explain why even using Jemalloc _without_ compiling Ruby has its own caveats. The Jemalloc version matters a lot. For reasons that are not yet clear, significant memory savings are only achieved with Jemalloc 3, not with Jemalloc 5. Your distribution only ships one Jemalloc version. So likely you need to compile Jemalloc 3 yourself. Here you are already entering compilation land. But Jemalloc 3 no longer compiles by default on some modern distributions, such as Debian 10. Fullstaq Ruby fixes this by patching Jemalloc for you. Furthermore, which Ruby binaries are you using? The ones provided by the Linux distribution are perpetually outdated. Another of Fullstaq Ruby's value proposition is that we supply binaries for the latest Ruby version, quickly. We packaged Ruby 3.0 on the same day it came out. "Fullstaq Ruby vs LD_PRELOADing Jemalloc yourself": https://github.com/fullstaq-labs/fullstaq-ruby-server-edition#vs-ld_preloading-jemalloc-yourself https://github.com/fullstaq-labs/fullstaq-ruby-server-editio...
    - wgjordan 5y ago
      Thanks, these extra details are helpful and seem like the more significant motivations underlying this distribution. > For reasons that are not yet clear, significant memory savings are only achieved with Jemalloc 3, not with Jemalloc 5. So the real advantage of this package is that it bundles a 6+ year-old, unsupported version of Jemalloc, because the more recent versions found in current OS distributions don't yield memory savings in practice- for unknown reasons. This doesn't instill very much confidence. I would be much more excited by efforts to investigate the jemalloc > 3.x changes so Ruby can work optimally with current releases packaged in modern Linux distributions, rather than double-down on a workaround that requires bundling an increasingly-ancient version of the software. I should also add - as mentioned by the jemalloc author [1], the addition of the time-based purging feature is likely responsible for memory-usage differences between jemalloc 3.x and 5.x, so you can reduce `dirty_decay_ms` and `muzzy_decay_ms` to get 3.x-like memory usage. I have been using this configuration in production since 2018 for significant memory savings in Ruby using jemalloc 5.x. [1] https://bugs.ruby-lang.org/issues/14718#note-86 https://bugs.ruby-lang.org/issues/14718#note-86
      - FooBarWidget 5y ago
        I think that is an overly cautious take on things. I'll explain why. First, "for unknown reasons" deserves more nuance. The vague, high-level reason is clear: Jemalloc 3 behaves differently from Jemalloc 5, having different algorithms and data structures. What I mean by unknown is not so much an indication of incomprehensible arcane magic, and that things can collapse at any time. What I mean is that it's not known in what way the algorithms and data structures are different. Consider that before I did my 2019 research on why Ruby memory bloating occurs[1], Ruby apps suffered from memory bloat "for unknown reasons". That didn't mean that before 2019, all Ruby apps were houses of cards waiting to fall over. It's like saying "I don't understand why this Linux kernel upgrade made things faster" -- the kernel developers know but they have better things to do than to answer your questions. And the fact that knowledge about a new optimization in the Linux kernel is not widespread, does not mean that that kernel version is unstable. Nobody truly understands every single detail about all parts of the stack. Yet I can build reliable, high-available web apps just fine without understanding how for example how 5G works and why users on 5G can access my app faster than on 4G. The differences between Jemalloc 3 and 5 are not explicitly documented anywhere, and to find out requires research. I intend on doing that some time in the future, but not now. Jemalloc 3 is proven to work, it's proven to be stable. The combination of Ruby + Jemalloc is proven to work well, not only because we've had several years of user feedback now, but also because Github has tested this combination for years now even before Fullstaq Ruby. The pragmatic thing to do is not to prioritize figuring out exactly how Jemalloc 5 works. It's to continue the packaging work to make Ruby + Jemalloc 3 available to the public. Jemalloc 5 can wait. [1] https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html https://www.joyfulbikeshedding.com/blog/2019-03-14-what-caus...
- jbotdev 5y ago
  It seems like the main value-add of this project is building/packaging pretty much standard Ruby in a consistent way across many Linux distributions, which is something that’s been hard to come by. I would lean into that more than anything else.
- IfOnlyYouKnew 5y ago
  I don't know if 2021 is much different than, say, 2012, in that I don't mind compiling something as long as it works? I'm sure there are intricacies with ruby's configuration, specifically. And that you invested time and have a good understanding of it. But I'm not convinced it's a product and not a service, or a task for someone on staff for larger organisations. As but one thing: to be a product it needs to have some universal value for a significant chunk of scenarios (possibly with some configuration options). But at that point, it's not clear why it shouldn't be a contribution to the upstream project. Is there, possibly, an "Enterprise Edition" or "Cloud Deployment Pro Plan" in the works? I'm also (and that's neither here or there but I can't help myself) not entirely sure how democracy got involved here. While the rule of law and a participatory citizenry do tend to lead to obvious benefits, "reaping" evokes a one-way process that is sorta antithetical to the cooperative nature of democracies? Dunno, maybe it's just me. It's also on some buzzword bingo cards, although I can't quite say in what specific context...
  - FooBarWidget 5y ago
    > But I'm not convinced it's a product and not a service, or a task for someone on staff for larger organisations. From my experience with users and the industry at large, the amount of people who are comfortable with compilation and sysadmin-y tasks are rapidly declining, proportion-wise. The industry is heading towards ever-more specialization. Many many backend developers nowadays don't want to think about infrastructure at all, they just want to focus on business logic. There are a huge amount of backend developers who have never seen './configure && make install'. In an organization with sufficiently advanced human capital, yes there is someone who can take care of that. But the existance of such a person can be taken less and less for granted nowadays even in very large organizations. There is also the factor of: should we do this work ourselves? Lots and lots of developer tooling nowadays are extremely slick. We've been spoiled. I have been spoiled. I can still write C++ but I don't want to bother with './configure && patch && make install' anymore. The standard nowadays is higher. Why should I spend a day installing a custom-patched Ruby when someone else can do that for me and all I have to do is to add an APT repo? Especially when I always have better things to do? > it's not clear why it shouldn't be a contribution to the upstream project. This is explained in the FAQ: https://github.com/fullstaq-labs/fullstaq-ruby-server-edition#why-a-new-distribution-why-not-contribute-to-ruby-core https://github.com/fullstaq-labs/fullstaq-ruby-server-editio... There is also a discussion here: https://news.ycombinator.com/item?id=27870740 https://news.ycombinator.com/item?id=27870740 > Is there, possibly, an "Enterprise Edition" or "Cloud Deployment Pro Plan" in the works? There is not. There are absolutely no paid features, nor plans to monetize. There are even enough reasons to not monetize. See the FAQ: https://github.com/fullstaq-labs/fullstaq-ruby-server-edition/blob/main/README.md#will-fullstaq-ruby-become-paid-in-the-future https://github.com/fullstaq-labs/fullstaq-ruby-server-editio... And the project vision, which is community-based: https://www.joyfulbikeshedding.com/blog/2020-05-15-why-fullstaq-ruby.html https://www.joyfulbikeshedding.com/blog/2020-05-15-why-fulls... > I'm also (and that's neither here or there but I can't help myself) not entirely sure how democracy got involved here. Democratization, not democracy. Democratization is about making something available to as many people as possible. Which is a different concept from democracy. Given that so many people are uncomfortable with compiling, or with LD_PRELOAD, or with anything outside of "bundle install", asking people to "just compile Jemalloc 3, make sure to apply this patch, then modify your systemd init script to include LD_PRELOAD" is too much to ask. It shuts down an entire range of people from benefiting from Jemalloc. With democratization, I seek to combat this.
taf2 5y ago
At least 10 years ago it was normal or at least in my circles to always compile your own Ruby with all the production settings needed… jemalloc … we just build our own rpms and it’s pretty easy then to roll out upgrades too …
sprite 5y ago
Thanks I didn’t know this was possible. I’m currently running Ruby compiled with jemalloc in production on elastic beanstalk with a custom AMI. If I can swap back to their AMI and just use LD_PRELOAD it will make upgrades a lot easier.

FooBarWidget 5y ago

Here I explain why it can quickly become not so trivial: https://news.ycombinator.com/item?id=27872696 https://news.ycombinator.com/item?id=27872696 The goal of Fullstaq Ruby is to democratize the fight against Ruby memory bloat. Democratization means that as many people should be able to reap the benefits as possible. It's 2021 now, and expecting users to compile Ruby or Jemalloc from source is no longer realistic. Compiling anything is no longer a non-scary, low-friction thing to do.

wgjordan 5y ago
The parent comment noted that using jemalloc for Ruby without compiling is already trivially easy, so the argument that 'compiling is hard' is irrelevant even if true. The real argument is that installing jemalloc separately and running Ruby with an environment variable is too hard.
- FooBarWidget 5y ago
  In my comment, as well as in the FAQ, I explain why even using Jemalloc _without_ compiling Ruby has its own caveats. The Jemalloc version matters a lot. For reasons that are not yet clear, significant memory savings are only achieved with Jemalloc 3, not with Jemalloc 5. Your distribution only ships one Jemalloc version. So likely you need to compile Jemalloc 3 yourself. Here you are already entering compilation land. But Jemalloc 3 no longer compiles by default on some modern distributions, such as Debian 10. Fullstaq Ruby fixes this by patching Jemalloc for you. Furthermore, which Ruby binaries are you using? The ones provided by the Linux distribution are perpetually outdated. Another of Fullstaq Ruby's value proposition is that we supply binaries for the latest Ruby version, quickly. We packaged Ruby 3.0 on the same day it came out. "Fullstaq Ruby vs LD_PRELOADing Jemalloc yourself": https://github.com/fullstaq-labs/fullstaq-ruby-server-edition#vs-ld_preloading-jemalloc-yourself https://github.com/fullstaq-labs/fullstaq-ruby-server-editio...
  - wgjordan 5y ago
    Thanks, these extra details are helpful and seem like the more significant motivations underlying this distribution. > For reasons that are not yet clear, significant memory savings are only achieved with Jemalloc 3, not with Jemalloc 5. So the real advantage of this package is that it bundles a 6+ year-old, unsupported version of Jemalloc, because the more recent versions found in current OS distributions don't yield memory savings in practice- for unknown reasons. This doesn't instill very much confidence. I would be much more excited by efforts to investigate the jemalloc > 3.x changes so Ruby can work optimally with current releases packaged in modern Linux distributions, rather than double-down on a workaround that requires bundling an increasingly-ancient version of the software. I should also add - as mentioned by the jemalloc author [1], the addition of the time-based purging feature is likely responsible for memory-usage differences between jemalloc 3.x and 5.x, so you can reduce `dirty_decay_ms` and `muzzy_decay_ms` to get 3.x-like memory usage. I have been using this configuration in production since 2018 for significant memory savings in Ruby using jemalloc 5.x. [1] https://bugs.ruby-lang.org/issues/14718#note-86 https://bugs.ruby-lang.org/issues/14718#note-86
    - FooBarWidget 5y ago
      I think that is an overly cautious take on things. I'll explain why. First, "for unknown reasons" deserves more nuance. The vague, high-level reason is clear: Jemalloc 3 behaves differently from Jemalloc 5, having different algorithms and data structures. What I mean by unknown is not so much an indication of incomprehensible arcane magic, and that things can collapse at any time. What I mean is that it's not known in what way the algorithms and data structures are different. Consider that before I did my 2019 research on why Ruby memory bloating occurs[1], Ruby apps suffered from memory bloat "for unknown reasons". That didn't mean that before 2019, all Ruby apps were houses of cards waiting to fall over. It's like saying "I don't understand why this Linux kernel upgrade made things faster" -- the kernel developers know but they have better things to do than to answer your questions. And the fact that knowledge about a new optimization in the Linux kernel is not widespread, does not mean that that kernel version is unstable. Nobody truly understands every single detail about all parts of the stack. Yet I can build reliable, high-available web apps just fine without understanding how for example how 5G works and why users on 5G can access my app faster than on 4G. The differences between Jemalloc 3 and 5 are not explicitly documented anywhere, and to find out requires research. I intend on doing that some time in the future, but not now. Jemalloc 3 is proven to work, it's proven to be stable. The combination of Ruby + Jemalloc is proven to work well, not only because we've had several years of user feedback now, but also because Github has tested this combination for years now even before Fullstaq Ruby. The pragmatic thing to do is not to prioritize figuring out exactly how Jemalloc 5 works. It's to continue the packaging work to make Ruby + Jemalloc 3 available to the public. Jemalloc 5 can wait. [1] https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html https://www.joyfulbikeshedding.com/blog/2019-03-14-what-caus...
jbotdev 5y ago
It seems like the main value-add of this project is building/packaging pretty much standard Ruby in a consistent way across many Linux distributions, which is something that’s been hard to come by. I would lean into that more than anything else.
IfOnlyYouKnew 5y ago
I don't know if 2021 is much different than, say, 2012, in that I don't mind compiling something as long as it works? I'm sure there are intricacies with ruby's configuration, specifically. And that you invested time and have a good understanding of it. But I'm not convinced it's a product and not a service, or a task for someone on staff for larger organisations. As but one thing: to be a product it needs to have some universal value for a significant chunk of scenarios (possibly with some configuration options). But at that point, it's not clear why it shouldn't be a contribution to the upstream project. Is there, possibly, an "Enterprise Edition" or "Cloud Deployment Pro Plan" in the works? I'm also (and that's neither here or there but I can't help myself) not entirely sure how democracy got involved here. While the rule of law and a participatory citizenry do tend to lead to obvious benefits, "reaping" evokes a one-way process that is sorta antithetical to the cooperative nature of democracies? Dunno, maybe it's just me. It's also on some buzzword bingo cards, although I can't quite say in what specific context...
- FooBarWidget 5y ago
  > But I'm not convinced it's a product and not a service, or a task for someone on staff for larger organisations. From my experience with users and the industry at large, the amount of people who are comfortable with compilation and sysadmin-y tasks are rapidly declining, proportion-wise. The industry is heading towards ever-more specialization. Many many backend developers nowadays don't want to think about infrastructure at all, they just want to focus on business logic. There are a huge amount of backend developers who have never seen './configure && make install'. In an organization with sufficiently advanced human capital, yes there is someone who can take care of that. But the existance of such a person can be taken less and less for granted nowadays even in very large organizations. There is also the factor of: should we do this work ourselves? Lots and lots of developer tooling nowadays are extremely slick. We've been spoiled. I have been spoiled. I can still write C++ but I don't want to bother with './configure && patch && make install' anymore. The standard nowadays is higher. Why should I spend a day installing a custom-patched Ruby when someone else can do that for me and all I have to do is to add an APT repo? Especially when I always have better things to do? > it's not clear why it shouldn't be a contribution to the upstream project. This is explained in the FAQ: https://github.com/fullstaq-labs/fullstaq-ruby-server-edition#why-a-new-distribution-why-not-contribute-to-ruby-core https://github.com/fullstaq-labs/fullstaq-ruby-server-editio... There is also a discussion here: https://news.ycombinator.com/item?id=27870740 https://news.ycombinator.com/item?id=27870740 > Is there, possibly, an "Enterprise Edition" or "Cloud Deployment Pro Plan" in the works? There is not. There are absolutely no paid features, nor plans to monetize. There are even enough reasons to not monetize. See the FAQ: https://github.com/fullstaq-labs/fullstaq-ruby-server-edition/blob/main/README.md#will-fullstaq-ruby-become-paid-in-the-future https://github.com/fullstaq-labs/fullstaq-ruby-server-editio... And the project vision, which is community-based: https://www.joyfulbikeshedding.com/blog/2020-05-15-why-fullstaq-ruby.html https://www.joyfulbikeshedding.com/blog/2020-05-15-why-fulls... > I'm also (and that's neither here or there but I can't help myself) not entirely sure how democracy got involved here. Democratization, not democracy. Democratization is about making something available to as many people as possible. Which is a different concept from democracy. Given that so many people are uncomfortable with compiling, or with LD_PRELOAD, or with anything outside of "bundle install", asking people to "just compile Jemalloc 3, make sure to apply this patch, then modify your systemd init script to include LD_PRELOAD" is too much to ask. It shuts down an entire range of people from benefiting from Jemalloc. With democratization, I seek to combat this.

taf2 5y ago

At least 10 years ago it was normal or at least in my circles to always compile your own Ruby with all the production settings needed… jemalloc … we just build our own rpms and it’s pretty easy then to roll out upgrades too …

sprite 5y ago

Thanks I didn’t know this was possible. I’m currently running Ruby compiled with jemalloc in production on elastic beanstalk with a custom AMI. If I can swap back to their AMI and just use LD_PRELOAD it will make upgrades a lot easier.