4 ms·

My new hobby is installing vanilla Gitlab CE on a vanilla server and waiting until it crashes the whole thing. Takes something between 8 to 36 hours. The serve

My new hobby is installing vanilla Gitlab CE on a vanilla server and waiting until it crashes the whole thing. Takes something between 8 to 36 hours. The server is completely gone, no ping, no ssh. It takes frantic remote power button pushes to even turn it off. Afterwards I reconfigure various config lines and repeat.

mattl 6y ago
How much RAM? Are you installing from Omnibus packages? I ask because I have two servers and have near perfect uptime.
- ianbooker 6y ago
  64 GB on a 16 core machine that has nothing else to do. If any, I think this is maybe too much, every worker just spawns more and more workers. I restricted Postgres from taking all RAM, Sidekiq and Puma to low numbers. But there are more.. I follow the instructions straight from Gitlab.
  - brodock 6y ago
    Consider having databases in a separate instance. 2x32GB is better than one 64GB. It's specially important at this size to have git in a different machine than the database as both are I/O constrained.
Jugurtha 6y ago
I have somewhat similar experience. I have 16GB of RAM, 12 CPU. You can reduce the number of unicorn workers in `config/gitlab.rb`.
- lbotos 6y ago
  what are you seeing in the logs? OOM-killer? Also, newer versions use Puma which lower RAM needs if you are seeing Unicorn OOM Killer
  - Jugurtha 6y ago
    Ah, my bad. I have `12.10.6 (833223f2a7f)`. Time to pull, as it really hogged my RAM. The pattern is that it will periodically consume much more resources. Thanks for the heads-up.
- ianbooker 6y ago
  That was a theory I had: That Unicorn leaks and that Unicorn Killer is not killing any of the workers, since it does so all 160 requests and there are no request happening at the moment. Then I saw the new installation uses Puma ;)
  - joshlambert 6y ago
    Hi, GitLab product manager here. I'm sorry that we haven't provided a good experience. We try really hard to make sure GitLab deployments "just work", and clearly it does not for either of you. Both Puma and Unicorn should be killed after they exceed a certain size to avoid this situation from happening. It's possible either this is not working in some situations/configurations, or there is a leak elsewhere although this is the first time I have heard reports of this. What configuration is being changed from the defaults? Alternatively if you could open an issue with any additional detail we will try to figure out what is happening and fix it: https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues
hoistbypetard 6y ago
I mean this with no judgement: that is a weird hobby. Is there a reason you do that instead of fixing gitlab or just using something else? Either of those would seem more satisfying to me.
- codetrotter 6y ago
  I doubt that they meant it literally. May have been a reference to the xkcd “my hobby” strips. See https://www.explainxkcd.com/wiki/index.php/Category:My_Hobby https://www.explainxkcd.com/wiki/index.php/Category:My_Hobby
mariusmg 6y ago
And Gitea is usable when running on a Raspberry Pi
clktmr 6y ago
Disabling automatic repository housekeeping might help. It can use lots of RAM for large repositories like the Linux kernel.
- mcny 6y ago
  The Linux kernel isn’t that bad. If you really want to kill your instance of Gitlab, try Google Chrome with all its public repos. Write a gitlab ci yaml file and configure it to build/test with every push. https://chromium.googlesource.com/chromium/ https://chromium.googlesource.com/chromium/
_fizz_buzz_ 6y ago
That isn't my experience. I installed gitlab on a local debian server and it has been running for 3 months straight now.
viraptor 6y ago
So what's the reason for crashing? What's the configuration that you change? I assume it's not random trial and error.
hardwaresofton 6y ago
Don't want to be that guy, but containerization (gitlab omnibus[0]) might actually help you in this case -- in particular the resource utilization caps that you can get with linux containers. While I can't tell you why Gitlab CE is crashing for you, it is much less likely to take down the entire machine from inside a resource-constrained container, and it will probably force you to make the configuration a bit more reproducable as well (and maybe that will help twiddling). Would love to read about the actual config issues you're having and how you're solving them one by one -- are you writing about it anywhere? [0]: https://docs.gitlab.com/omnibus/docker/README.html https://docs.gitlab.com/omnibus/docker/README.html

mattl 6y ago

How much RAM? Are you installing from Omnibus packages? I ask because I have two servers and have near perfect uptime.

ianbooker 6y ago
64 GB on a 16 core machine that has nothing else to do. If any, I think this is maybe too much, every worker just spawns more and more workers. I restricted Postgres from taking all RAM, Sidekiq and Puma to low numbers. But there are more.. I follow the instructions straight from Gitlab.
- brodock 6y ago
  Consider having databases in a separate instance. 2x32GB is better than one 64GB. It's specially important at this size to have git in a different machine than the database as both are I/O constrained.

Jugurtha 6y ago

I have somewhat similar experience. I have 16GB of RAM, 12 CPU. You can reduce the number of unicorn workers in `config/gitlab.rb`.

lbotos 6y ago
what are you seeing in the logs? OOM-killer? Also, newer versions use Puma which lower RAM needs if you are seeing Unicorn OOM Killer
- Jugurtha 6y ago
  Ah, my bad. I have `12.10.6 (833223f2a7f)`. Time to pull, as it really hogged my RAM. The pattern is that it will periodically consume much more resources. Thanks for the heads-up.
ianbooker 6y ago
That was a theory I had: That Unicorn leaks and that Unicorn Killer is not killing any of the workers, since it does so all 160 requests and there are no request happening at the moment. Then I saw the new installation uses Puma ;)
- joshlambert 6y ago
  Hi, GitLab product manager here. I'm sorry that we haven't provided a good experience. We try really hard to make sure GitLab deployments "just work", and clearly it does not for either of you. Both Puma and Unicorn should be killed after they exceed a certain size to avoid this situation from happening. It's possible either this is not working in some situations/configurations, or there is a leak elsewhere although this is the first time I have heard reports of this. What configuration is being changed from the defaults? Alternatively if you could open an issue with any additional detail we will try to figure out what is happening and fix it: https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues

hoistbypetard 6y ago

I mean this with no judgement: that is a weird hobby. Is there a reason you do that instead of fixing gitlab or just using something else? Either of those would seem more satisfying to me.

codetrotter 6y ago
I doubt that they meant it literally. May have been a reference to the xkcd “my hobby” strips. See https://www.explainxkcd.com/wiki/index.php/Category:My_Hobby https://www.explainxkcd.com/wiki/index.php/Category:My_Hobby

mariusmg 6y ago

And Gitea is usable when running on a Raspberry Pi

clktmr 6y ago

Disabling automatic repository housekeeping might help. It can use lots of RAM for large repositories like the Linux kernel.

mcny 6y ago
The Linux kernel isn’t that bad. If you really want to kill your instance of Gitlab, try Google Chrome with all its public repos. Write a gitlab ci yaml file and configure it to build/test with every push. https://chromium.googlesource.com/chromium/ https://chromium.googlesource.com/chromium/

_fizz_buzz_ 6y ago

That isn't my experience. I installed gitlab on a local debian server and it has been running for 3 months straight now.

viraptor 6y ago

So what's the reason for crashing? What's the configuration that you change? I assume it's not random trial and error.

hardwaresofton 6y ago

Don't want to be that guy, but containerization (gitlab omnibus[0]) might actually help you in this case -- in particular the resource utilization caps that you can get with linux containers. While I can't tell you why Gitlab CE is crashing for you, it is much less likely to take down the entire machine from inside a resource-constrained container, and it will probably force you to make the configuration a bit more reproducable as well (and maybe that will help twiddling). Would love to read about the actual config issues you're having and how you're solving them one by one -- are you writing about it anywhere? [0]: https://docs.gitlab.com/omnibus/docker/README.html https://docs.gitlab.com/omnibus/docker/README.html