5 ms·

> We have a plan to make GitLab use less resources I used to use gitlab at work daily and releases would mention how performance is being tackled consistently.

by chmln 8y ago

> We have a plan to make GitLab use less resources I used to use gitlab at work daily and releases would mention how performance is being tackled consistently. However, I have not seen any large improvements at all, suggesting that there is an impenetrable wall made of the underlying technologies. Gitlab is based primarily on Ruby-on-Rails, which doesn't particularly shine with neither memory usage nor performance. What sort of effort would even reduce the use of resources significantly, short of a rewrite?

yutghgh 8y ago
This "rails/ruby is slow" myth really needs to die. Sure, rails is too slow for certain very large companies at very large scales, and if you're in this position you already know it. However, rails is fast enough for 99.5% of usecases, gitlab included. As to memory usage, I will concede it is often somewhat high with rails, but there are solutions, such as jemalloc that have helped me in the past.
- KaoruAoiShiho 8y ago
  Gitlab most certainly not included! Its perf is awful in comparison to any modern stack (including some php ones unfortunately).
  - yutghgh 8y ago
    I didn't say GitLab was fast (I haven't used it enough to know), but blaming its problems on ruby/rails when GitHub (which is pretty fast imo) is also written on rails is misleading at best.
    - chinhodado 8y ago
      Ruby/rails is not the only factor, but it is certainly one of the contributing factors, and it is a big one. There's no need to deny that.
sytse 8y ago
A big thing we're looking at to reduce memory is making the ruby code multi-threaded. https://gitlab.com/gitlab-org/gitlab-ce/issues/3592 https://gitlab.com/gitlab-org/gitlab-ce/issues/3592
- aidos 8y ago
  Where is the bottleneck? I’m not massively familar with Ruby, but I’m assuming the underlying language isn’t the problem and it’s more to do with the general architecture?
  - sytse 8y ago
    We already have it working in development. We're cautious because last time we enabled it in production we got a lot of memory usage and runtime errors. I think the problems are that the native extensions from some dependencies are not multi-threading safe but I might be wrong.
dashundchen 8y ago
I do think they have made improvments to performance. Project search and loading the issues page used to timeout all the time on Gitlab.com, especially for the gitlab repo itself. Now it works much better, loads and searches at an acceptable speed with no timeouts. I will say I've never had a problem with performance on self hosted Gitlab because of the smaller scale.
stanhu 8y ago
There has been a steady march of performance issues that have been made over the last few months that may not have been mentioned in the release post: * 11.8: https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=all&utf8=%E2%9C%93&state=merged&label_name[]=performance&milestone_title=11.8 https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=... * 11.7: https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=all&utf8=%E2%9C%93&state=merged&label_name[]=performance&milestone_title=11.7 https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=... * 11.6: https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=all&utf8=%E2%9C%93&state=merged&label_name[]=performance&milestone_title=11.6 https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=... With regard to memory usage, we've made some progress there as well. For example: 1. We cut 80-100 MB/per Unicorn process by removing a dependency: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21008 https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21008 2. A change in our merge request processing in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22725 https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22725 lowered gigabytes of runtime memory from our worst and most commonly-used Sidekiq background job. 3. In the second-worst performing Sidekiq job, switching to a faster XML processor in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23136 https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23136 also dropped large spikes of runtime memory. That being said, we still have a long way to go. I recently did an analysis to figure out, "Why does each Unicorn process take 400-500 MB RAM?" The note in https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4118#note_141430928 https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4118#not... explains why. In short, there is truth that we're paying a price for Ruby overhead. What can we do about it? I have several ideas: 1. Upgrade to Ruby 2.6 (significant memory improvements for free; see https://youtu.be/ZfgxvNUfQdU https://youtu.be/ZfgxvNUfQdU) 2. Replace many Unicorn processes with a multi-threaded Puma server 3. Continue profiling and optimizing inefficient code paths 4. Make the Ruby interpreter more efficient (e.g. with a compacting garbage collector, see https://gitlab.com/gitlab-org/gitlab-ce/issues/54555 https://gitlab.com/gitlab-org/gitlab-ce/issues/54555) 5. Make Rails more efficient (e.g. https://github.com/rails/rails/pull/34711 https://github.com/rails/rails/pull/34711). Lastly, there may be significant parts of the code base that will need to be rewritten to improve memory and/or performance. Gitaly is a prime example of this: many Git-related calls have been abstracted into a Go process.
- 8y ago

by chmln 8y ago

yutghgh 8y ago

This "rails/ruby is slow" myth really needs to die. Sure, rails is too slow for certain very large companies at very large scales, and if you're in this position you already know it. However, rails is fast enough for 99.5% of usecases, gitlab included. As to memory usage, I will concede it is often somewhat high with rails, but there are solutions, such as jemalloc that have helped me in the past.

KaoruAoiShiho 8y ago
Gitlab most certainly not included! Its perf is awful in comparison to any modern stack (including some php ones unfortunately).
- yutghgh 8y ago
  I didn't say GitLab was fast (I haven't used it enough to know), but blaming its problems on ruby/rails when GitHub (which is pretty fast imo) is also written on rails is misleading at best.
  - chinhodado 8y ago
    Ruby/rails is not the only factor, but it is certainly one of the contributing factors, and it is a big one. There's no need to deny that.

sytse 8y ago

A big thing we're looking at to reduce memory is making the ruby code multi-threaded. https://gitlab.com/gitlab-org/gitlab-ce/issues/3592 https://gitlab.com/gitlab-org/gitlab-ce/issues/3592

aidos 8y ago
Where is the bottleneck? I’m not massively familar with Ruby, but I’m assuming the underlying language isn’t the problem and it’s more to do with the general architecture?
- sytse 8y ago
  We already have it working in development. We're cautious because last time we enabled it in production we got a lot of memory usage and runtime errors. I think the problems are that the native extensions from some dependencies are not multi-threading safe but I might be wrong.

dashundchen 8y ago

I do think they have made improvments to performance. Project search and loading the issues page used to timeout all the time on Gitlab.com, especially for the gitlab repo itself. Now it works much better, loads and searches at an acceptable speed with no timeouts. I will say I've never had a problem with performance on self hosted Gitlab because of the smaller scale.

stanhu 8y ago

There has been a steady march of performance issues that have been made over the last few months that may not have been mentioned in the release post: * 11.8: https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=all&utf8=%E2%9C%93&state=merged&label_name[]=performance&milestone_title=11.8 https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=... * 11.7: https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=all&utf8=%E2%9C%93&state=merged&label_name[]=performance&milestone_title=11.7 https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=... * 11.6: https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=all&utf8=%E2%9C%93&state=merged&label_name[]=performance&milestone_title=11.6 https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=... With regard to memory usage, we've made some progress there as well. For example: 1. We cut 80-100 MB/per Unicorn process by removing a dependency: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21008 https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21008 2. A change in our merge request processing in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22725 https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22725 lowered gigabytes of runtime memory from our worst and most commonly-used Sidekiq background job. 3. In the second-worst performing Sidekiq job, switching to a faster XML processor in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23136 https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23136 also dropped large spikes of runtime memory. That being said, we still have a long way to go. I recently did an analysis to figure out, "Why does each Unicorn process take 400-500 MB RAM?" The note in https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4118#note_141430928 https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4118#not... explains why. In short, there is truth that we're paying a price for Ruby overhead. What can we do about it? I have several ideas: 1. Upgrade to Ruby 2.6 (significant memory improvements for free; see https://youtu.be/ZfgxvNUfQdU https://youtu.be/ZfgxvNUfQdU) 2. Replace many Unicorn processes with a multi-threaded Puma server 3. Continue profiling and optimizing inefficient code paths 4. Make the Ruby interpreter more efficient (e.g. with a compacting garbage collector, see https://gitlab.com/gitlab-org/gitlab-ce/issues/54555 https://gitlab.com/gitlab-org/gitlab-ce/issues/54555) 5. Make Rails more efficient (e.g. https://github.com/rails/rails/pull/34711 https://github.com/rails/rails/pull/34711). Lastly, there may be significant parts of the code base that will need to be rewritten to improve memory and/or performance. Gitaly is a prime example of this: many Git-related calls have been abstracted into a Go process.

8y ago