6 ms·

As much as I appreciate all the components that Gitlab utilizes behind the scenes running an instance of this on a 4GB droplet consumes the entire memory of tha

As much as I appreciate all the components that Gitlab utilizes behind the scenes running an instance of this on a 4GB droplet consumes the entire memory of that unit until things come to a crawl. Does Gitlab actually need to utilize all those components and when I disable certain things like Prometheus and etc. it seems to only marginally make things better. What is consumes so much RAM and why?

rstat1 4y ago
I self-hosted a gitlab instance for a while on a local VM with 4GB of RAM and it (being the VM) would crash randomly because the OOM-killer killed too many important processes. I no longer self-host or use GitLab for this precise reason.
- 411111111111111 4y ago
  I think if you're going to self host a git server on a small server you'd be better off using gitea/gogs
  - rstat1 4y ago
    I agree. I didn't know about those at the time though, and ended up with a custom solution for no reason other than because I could :P
imran-iq 4y ago
> What is consumes so much RAM and why? That's just a complex rails application for you, at $DAYJOB we have to provision 11gb for just a single container running a single instance of the app
- danielheath 4y ago
  I’m running a high traffic rails app with 15 years of source history; our instances have no trouble at 4gb for both unicorn (8 processes) and sidekiq (5 processes). Except for image processing, which uses a ton of ram and readily OOMs.
  - imran-iq 4y ago
    I've worked at two very large rails shops (think >10 billion dollar companies) both of which have had stupidly high ram requirements for their code without any image processing. Gitlab which also happens to be another billion dollar rails shop, also requires a stupid amount of ram as per the parent comment. I wonder what the common thread is...
    - onion2k 4y ago
      You seem to be suggesting Rails memory usage scales with market cap, which would be incredible if true. "Why does the app keep crashing?" "Because the share price went up again!"
    - chucke 4y ago
      It definitely correlates. Ruby is not the culprit though. It uses and abuses memory as fairly as other similarly GC backed dynamic languages. The issue may be rails and the monolithic pattern, which favours rapid feature development and postpones optimization for when necessary, or one can no longer realise that snippet you wrote 2y ago fetching data from the db scales well for 5 objects, but not 500 objects, but you forgot about that already. Ruby has a generational GC which already supports compaction, but not in a production ready continuous way, which means most usages have it turned off, and memory pages tend to overcoming and stay fragmented for long running processes. Python has the same issue and node AFAIK.
    - mekster 4y ago
      It's apparent that a bigger company has more money to throw at the performance problem if that can solve it. Smaller companies want better optimization of their spending with better coding standards.
    - ascar 4y ago
      It has been a few years since, but I wrote a standalone small Rails app for an internal service for a company that also runs a giant Rails app for their main business. I ended up heavily tuning the ruby GC parameters, which reduced the final resource usage of the app especially over longer continued use nearly tenfold. There is a lot to gain if you know the actual object and memory limits your app needs for it's core usage rather than letting Ruby allocate more and more memory. Caveat is that was a long time ago with Ruby ~2.3 and things have certainly changed and probably improved by now. That service has been as stable as one could hope. It just runs in a docker container on a dedicated machine that's also used for other development purposes and consumes a couple of hundred MB of ram. The number of times it was down in the past ~6 years can be counted on one hand.
- mekster 4y ago
  Please stop blaming rails... It's the developer who just doesn't know how things work under the hood and code as they like and memory usage explodes.
Helmut10001 4y ago
There're special instructions to reduce Gitlab Hardware requirements for small appliances [1]. [1]: https://docs.gitlab.com/omnibus/settings/memory_constrained_envs.html https://docs.gitlab.com/omnibus/settings/memory_constrained_...
- pixel_tracing 4y ago
  Thing is I’ve tried this, it only marginally improves things as mentioned in original post
chrisfosterelli 4y ago
This is something I've noticed a lot of open core projects suffer from. They make architecture decisions that make sense for their huge, multi-tenant production system but correspondingly make self-hosting on a single node feel like administering a rube goldberg machine.
- vbezhenar 4y ago
  My opinion is that they optimize for huge deployments. Tiny companies are supposed to use cloud. And self-hosting is supposed to be used by big companies with thousands of people. I guess that's where their money is, anyway. That said, they're doing good job of documenting things required for smaller installations.
  - totetsu 4y ago
    So true. I was in charge of setting up self hosted GitLab at tiny computer company. Was very proud of the cicd setup untill the boss asked for review apps to be launched for each commit. Gitlab had no solution for this but to use Kubernetes.
  - mekster 4y ago
    Bullshit, the inability to properly optimize doesn't mean it's optimized well for huge deployments. It will need much more resource on huge deployments and still quite some RAM on small deployments too. At this point, the code base is too big for them to fix the performance and they want to use their time on new features and bug fixes as that's what customers are asking more while customers may solve the performance problem by spending more money on the server. It's just that the initial technical decisions sucked and it's no longer possible to fix it at this point.
    - deleted 4y ago
      [deleted]
- mr-karan 4y ago
  This is so true. Sentry's architecture[1] is insanely complex. It runs some ~30 containers on the instance and all you've to pray is that their tooling[2] for `docker-compose` _just works_. [1]: https://develop.sentry.dev/architecture/ https://develop.sentry.dev/architecture/ [2]: https://github.com/getsentry/self-hosted https://github.com/getsentry/self-hosted
rubin55 4y ago
I remember back in the day we were evaluating Puppet vs CFengine. During our testing we noted how memory intensive Puppet was, which we assumed at the time was due to it being based on Ruby. Afaik Ruby can quite easily consume a bunch of memory, it doesn't surprise me that an app like GitLab, which is based on Ruby on Rails I think, consumes somewhat largesse amounts.
runemadsen 4y ago
My experience with GitLab is that they accelerated very quickly in the beginning with a focus on new features. This made the whole ecosystem incredibly slow, and it's unbearable for me to run as a service on my own hardware / cloud accounts. I guess that's the problem with these open source services that also rely on paid platform income. I have tested out https://gogs.io/ https://gogs.io/ and the difference in speed is just incredible.
- jorams 4y ago
  My experience with Gogs is now a few years out of date, but after using it for a few months we ended up switching to GitLab. Gogs was amazingly fast and required few resources, but with moderately-sized repositories or even slightly large pull requests it slowed to a crawl. At some point it screwed up some on-disk repository state causing pull request merges to trigger server errors, which required manual cleanup. Meanwhile GitLab (omnibus), with a few users on a VPS with 4GB memory, remains consistently fast and entirely problem-free.

rstat1 4y ago

I self-hosted a gitlab instance for a while on a local VM with 4GB of RAM and it (being the VM) would crash randomly because the OOM-killer killed too many important processes. I no longer self-host or use GitLab for this precise reason.

411111111111111 4y ago
I think if you're going to self host a git server on a small server you'd be better off using gitea/gogs
- rstat1 4y ago
  I agree. I didn't know about those at the time though, and ended up with a custom solution for no reason other than because I could :P

imran-iq 4y ago

> What is consumes so much RAM and why? That's just a complex rails application for you, at $DAYJOB we have to provision 11gb for just a single container running a single instance of the app

danielheath 4y ago
I’m running a high traffic rails app with 15 years of source history; our instances have no trouble at 4gb for both unicorn (8 processes) and sidekiq (5 processes). Except for image processing, which uses a ton of ram and readily OOMs.
- imran-iq 4y ago
  I've worked at two very large rails shops (think >10 billion dollar companies) both of which have had stupidly high ram requirements for their code without any image processing. Gitlab which also happens to be another billion dollar rails shop, also requires a stupid amount of ram as per the parent comment. I wonder what the common thread is...
  - onion2k 4y ago
    You seem to be suggesting Rails memory usage scales with market cap, which would be incredible if true. "Why does the app keep crashing?" "Because the share price went up again!"
  - chucke 4y ago
    It definitely correlates. Ruby is not the culprit though. It uses and abuses memory as fairly as other similarly GC backed dynamic languages. The issue may be rails and the monolithic pattern, which favours rapid feature development and postpones optimization for when necessary, or one can no longer realise that snippet you wrote 2y ago fetching data from the db scales well for 5 objects, but not 500 objects, but you forgot about that already. Ruby has a generational GC which already supports compaction, but not in a production ready continuous way, which means most usages have it turned off, and memory pages tend to overcoming and stay fragmented for long running processes. Python has the same issue and node AFAIK.
  - mekster 4y ago
    It's apparent that a bigger company has more money to throw at the performance problem if that can solve it. Smaller companies want better optimization of their spending with better coding standards.
  - ascar 4y ago
    It has been a few years since, but I wrote a standalone small Rails app for an internal service for a company that also runs a giant Rails app for their main business. I ended up heavily tuning the ruby GC parameters, which reduced the final resource usage of the app especially over longer continued use nearly tenfold. There is a lot to gain if you know the actual object and memory limits your app needs for it's core usage rather than letting Ruby allocate more and more memory. Caveat is that was a long time ago with Ruby ~2.3 and things have certainly changed and probably improved by now. That service has been as stable as one could hope. It just runs in a docker container on a dedicated machine that's also used for other development purposes and consumes a couple of hundred MB of ram. The number of times it was down in the past ~6 years can be counted on one hand.
mekster 4y ago
Please stop blaming rails... It's the developer who just doesn't know how things work under the hood and code as they like and memory usage explodes.

Helmut10001 4y ago

There're special instructions to reduce Gitlab Hardware requirements for small appliances [1]. [1]: https://docs.gitlab.com/omnibus/settings/memory_constrained_envs.html https://docs.gitlab.com/omnibus/settings/memory_constrained_...

pixel_tracing 4y ago
Thing is I’ve tried this, it only marginally improves things as mentioned in original post

chrisfosterelli 4y ago

This is something I've noticed a lot of open core projects suffer from. They make architecture decisions that make sense for their huge, multi-tenant production system but correspondingly make self-hosting on a single node feel like administering a rube goldberg machine.

vbezhenar 4y ago
My opinion is that they optimize for huge deployments. Tiny companies are supposed to use cloud. And self-hosting is supposed to be used by big companies with thousands of people. I guess that's where their money is, anyway. That said, they're doing good job of documenting things required for smaller installations.
- totetsu 4y ago
  So true. I was in charge of setting up self hosted GitLab at tiny computer company. Was very proud of the cicd setup untill the boss asked for review apps to be launched for each commit. Gitlab had no solution for this but to use Kubernetes.
- mekster 4y ago
  Bullshit, the inability to properly optimize doesn't mean it's optimized well for huge deployments. It will need much more resource on huge deployments and still quite some RAM on small deployments too. At this point, the code base is too big for them to fix the performance and they want to use their time on new features and bug fixes as that's what customers are asking more while customers may solve the performance problem by spending more money on the server. It's just that the initial technical decisions sucked and it's no longer possible to fix it at this point.
  - deleted 4y ago
    [deleted]
mr-karan 4y ago
This is so true. Sentry's architecture[1] is insanely complex. It runs some ~30 containers on the instance and all you've to pray is that their tooling[2] for `docker-compose` _just works_. [1]: https://develop.sentry.dev/architecture/ https://develop.sentry.dev/architecture/ [2]: https://github.com/getsentry/self-hosted https://github.com/getsentry/self-hosted

rubin55 4y ago

I remember back in the day we were evaluating Puppet vs CFengine. During our testing we noted how memory intensive Puppet was, which we assumed at the time was due to it being based on Ruby. Afaik Ruby can quite easily consume a bunch of memory, it doesn't surprise me that an app like GitLab, which is based on Ruby on Rails I think, consumes somewhat largesse amounts.

runemadsen 4y ago

My experience with GitLab is that they accelerated very quickly in the beginning with a focus on new features. This made the whole ecosystem incredibly slow, and it's unbearable for me to run as a service on my own hardware / cloud accounts. I guess that's the problem with these open source services that also rely on paid platform income. I have tested out https://gogs.io/ https://gogs.io/ and the difference in speed is just incredible.

jorams 4y ago
My experience with Gogs is now a few years out of date, but after using it for a few months we ended up switching to GitLab. Gogs was amazingly fast and required few resources, but with moderately-sized repositories or even slightly large pull requests it slowed to a crawl. At some point it screwed up some on-disk repository state causing pull request merges to trigger server errors, which required manual cleanup. Meanwhile GitLab (omnibus), with a few users on a VPS with 4GB memory, remains consistently fast and entirely problem-free.