16 ms·

Degraded performance on GitLab.com

90 points by sexy_devil 7y ago

grafelic 7y ago
Indeed. Incredibly slow. Status page: https://status.gitlab.com/ https://status.gitlab.com/ Issue: https://gitlab.com/gitlab-com/gl-infra/production/issues/928 https://gitlab.com/gitlab-com/gl-infra/production/issues/928
bachmeier 7y ago
I was trying out Gitlab pages for the first time, but gave up since it was so slow, moving my stuff to Github. Guess it's not always that slow...
- Kovah 7y ago
  I can say for sure, that Gitlab is not running that slow all the time. However, the services run noticeably slower when the US awakes (I'm located in Germany), which is in the afternoon and evening around here. I have no stats, but it feels like builds and the site run smoother in the morning.
  - emilycook 7y ago
    In case you're actually curious, we do publish our grafana stats here: https://dashboards.gitlab.com https://dashboards.gitlab.com
- emilycook 7y ago
  Hi! We definitely don't run slowly this often, our dashboard is published here: https://dashboards.gitlab.com https://dashboards.gitlab.com
umvi 7y ago
GitLab is best (imo) as a private instance; we haven't had any issues with it after using it for 3 years now
- mr__y 7y ago
  exactly my experience. Assuming that a powerful enough server is available to do that. Our first attempt to run gitlab on some small cheap VPS (2 cores, 4 gigs of ram) was a total disaster.
  - rococode 7y ago
    What server specs are working well for you?
    - dijit 7y ago
      Not OP but I have a number of gitlab instances: For 5 users we use 18G/4vCPU. For 20 users we use 32G/8vCPU I keep asking to lower the footprint but the gitlab is the very definition of feature creep. And that comes at the cost of optimisation.
      - aprdm 7y ago
        We support 30 developers on 10gb ram and 4 cores. That said, we are probably a year and a half behind version wise and only use it for its SCM capabilities.
    - mr__y 7y ago
      quad core with 16 gigs of ram for a team of 4 people. Most likely 8gb of ram would be sufficient for us but the risk of decreasing our productivity is not worth the insignificant difference in price of 8GB vs 16GB VPS monthly fee.
    - maverwa 7y ago
      We are running it on a 2vCore, 8GB RAM vm. Works good for our ~10 concurrent users on peak. The ~120CI builds per day are running on other hosts though.
- mtgx 7y ago
  Have you also tried Gogs? https://gogs.io https://gogs.io
Kovah 7y ago
Even though my builds won't start as runners seem to be blocked, it's kinda fascinating to see those in-depth details about the service and how the Gitlab dudes try to find the root cause.
- penagwin 7y ago
  Right? This is EXACTLY what I want to see when there's a service disruption. A live, in-depth view of who is doing what, any new leads on the issue, multiple teams chiming in with various diagnostic stats, honestly it's really awesome. I know this can't be expected from most businesses, especially non-open sourced ones, but it's so refreshing to see this instead of the typical "We're working on a potential service disruption" that we normally get.
  - brbrodude 7y ago
    I think it's cool but I'd be totally self-aware and fearful or making the idiot post in the thread for HN to judge :p
    - penagwin 7y ago
      It's certainly something I'd be scared of too. It's honestly really brave of them to approach it this way, but I do enjoy watching them work :D
wolf550e 7y ago
Is the conclusion that redis doesn't handle caching items that reach 99MB size each? I guess I would cache them in blob storage, not in RAM. And make sure to zstd them.
nimbius 7y ago
I know this is probably a controversial post but seeing how big and bloated Gitlab became after years of devops dragon-chasing, i switched to gittea. https://gitea.io/en-us/ https://gitea.io/en-us/ I keep separate CI and a separate wiki, so i can upgrade them independently and not have the entire software development process grind to a halt whenever I need to patch.
- tapoxi 7y ago
  I don't think that's controversial, I think it's different use cases. If I want a simple git repo I'll go gogs/gitea, but GitLab is an all-in-one tool for a software shop.
- emilycook 7y ago
  Hi GitLab employee, just gonna echo what the other person said and say that it isn't controversial to say that! We know that we don't work for every situation, if all you need is a repo, then by design we can't compete with how lightweight something like Gitea is.
samcday 7y ago
I recently started a new position at a company that is using Gitlab. In the last month I've seen a lot of degraded performance and service outages (especially in Gitlab CI). And then I see stuff like this that makes me scratch my head: https://about.gitlab.com/product/service-desk/ https://about.gitlab.com/product/service-desk/ If anyone at Gitlab is reading this ... please, please slow down on chasing new markets + features and just make the stuff you already have work properly, and fill in the missing pieces. Example: https://gitlab.com/gitlab-org/gitlab-ce/issues/63880 https://gitlab.com/gitlab-org/gitlab-ce/issues/63880
- tylfin 7y ago
  > please, please slow down on chasing new markets + features and just make the stuff you already have work properly I really agree with this. I was working at a company where we were exploring switching from Github enterprise to Gitlab for the K8s integrations, the docker registry, and the CI features. Following the helm chart installation instructions was a nightmare because we were on-prem w/ a custom cluster. There were a lot of assumptions we had to find workarounds for (e.g. we used an f5 integration to manage our ingresses and performed SSL termination elsewhere so the nginx thing was awful). The other terrifying bit was the number of services / pods that got brought up with the installation. If I have to monitor my team's services, I really don't want to monitor the health of: NGINX, Postgres, Redis, Minio, Registry, GitLab/sidekiq, GitLab/gitlab-shell, GitLab/gitaly, GitLab/unicorn, and GitLab/migrations. When it comes to on-prem or self-hosted software I actually prefer running a monolithic application that worst-case I can just bounce or reboot the server. Slow down, simplify things, and improve the user experience. Gitlab already has enough features to be competitive for a while with the Github + marketplace model.
  - citruspi 7y ago
    > When it comes to on-prem or self-hosted software I actually prefer running a monolithic application that worst-case I can just bounce or reboot the server. So why not just skip the helm chart and install the omnibus package on a dedicated server? If you want you can even disable stuff like the omnibus Postgres/Redis/etc and manage those yourself elsewhere. That's what I do - run the omnibus package sans fancy helm chart - and it works out pretty well. Granted it does require a fair bit of RAM to run smoothly. > Slow down, simplify things, and improve the user experience. Gitlab already has enough features to be competitive for a while with the Github + marketplace model. Completely agree with this and what the parent said - I think this is part of why running Gitlab seems to require an ever increasing amount of compute resources :(
baq 7y ago
you have a convoy.
kabwj 7y ago
Funny. Gitlab has always had terrible performance for me, both the website and the git server.
jhgg 7y ago
Running Redis on GCP? I’ve seen CPU and latency spike on our production Redis nodes before - solely due to noisy neighbors causing memory bandwidth contention.
yardie 7y ago
Not sure if this is related but I've received a bunch of emails from Gitlab stating my account is locked for too many failed password attempts. Possible DDoS?
- emilycook 7y ago
  We haven't had any other reports of this, please submit a ticket here if you think there's an issue so we can track it: https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=360000515493 https://support.gitlab.com/hc/en-us/requests/new?ticket_form...
awasum_yannick 7y ago
Thank you Gitlab for the amazing product and transparency. Just watching the conversation as everyone over there is trying to get to the root of the problem is inspiring. What a strategy.

90 points by sexy_devil 7y ago

Indeed. Incredibly slow. Status page: https://status.gitlab.com/ https://status.gitlab.com/ Issue: https://gitlab.com/gitlab-com/gl-infra/production/issues/928 https://gitlab.com/gitlab-com/gl-infra/production/issues/928

bachmeier 7y ago

I was trying out Gitlab pages for the first time, but gave up since it was so slow, moving my stuff to Github. Guess it's not always that slow...

Kovah 7y ago
I can say for sure, that Gitlab is not running that slow all the time. However, the services run noticeably slower when the US awakes (I'm located in Germany), which is in the afternoon and evening around here. I have no stats, but it feels like builds and the site run smoother in the morning.
- emilycook 7y ago
  In case you're actually curious, we do publish our grafana stats here: https://dashboards.gitlab.com https://dashboards.gitlab.com
emilycook 7y ago
Hi! We definitely don't run slowly this often, our dashboard is published here: https://dashboards.gitlab.com https://dashboards.gitlab.com

umvi 7y ago

GitLab is best (imo) as a private instance; we haven't had any issues with it after using it for 3 years now

mr__y 7y ago
exactly my experience. Assuming that a powerful enough server is available to do that. Our first attempt to run gitlab on some small cheap VPS (2 cores, 4 gigs of ram) was a total disaster.
- rococode 7y ago
  What server specs are working well for you?
  - dijit 7y ago
    Not OP but I have a number of gitlab instances: For 5 users we use 18G/4vCPU. For 20 users we use 32G/8vCPU I keep asking to lower the footprint but the gitlab is the very definition of feature creep. And that comes at the cost of optimisation.
    - aprdm 7y ago
      We support 30 developers on 10gb ram and 4 cores. That said, we are probably a year and a half behind version wise and only use it for its SCM capabilities.
  - mr__y 7y ago
    quad core with 16 gigs of ram for a team of 4 people. Most likely 8gb of ram would be sufficient for us but the risk of decreasing our productivity is not worth the insignificant difference in price of 8GB vs 16GB VPS monthly fee.
  - maverwa 7y ago
    We are running it on a 2vCore, 8GB RAM vm. Works good for our ~10 concurrent users on peak. The ~120CI builds per day are running on other hosts though.
mtgx 7y ago
Have you also tried Gogs? https://gogs.io https://gogs.io

Kovah 7y ago

Even though my builds won't start as runners seem to be blocked, it's kinda fascinating to see those in-depth details about the service and how the Gitlab dudes try to find the root cause.

penagwin 7y ago
Right? This is EXACTLY what I want to see when there's a service disruption. A live, in-depth view of who is doing what, any new leads on the issue, multiple teams chiming in with various diagnostic stats, honestly it's really awesome. I know this can't be expected from most businesses, especially non-open sourced ones, but it's so refreshing to see this instead of the typical "We're working on a potential service disruption" that we normally get.
- brbrodude 7y ago
  I think it's cool but I'd be totally self-aware and fearful or making the idiot post in the thread for HN to judge :p
  - penagwin 7y ago
    It's certainly something I'd be scared of too. It's honestly really brave of them to approach it this way, but I do enjoy watching them work :D

wolf550e 7y ago

Is the conclusion that redis doesn't handle caching items that reach 99MB size each? I guess I would cache them in blob storage, not in RAM. And make sure to zstd them.

nimbius 7y ago

I know this is probably a controversial post but seeing how big and bloated Gitlab became after years of devops dragon-chasing, i switched to gittea. https://gitea.io/en-us/ https://gitea.io/en-us/ I keep separate CI and a separate wiki, so i can upgrade them independently and not have the entire software development process grind to a halt whenever I need to patch.

tapoxi 7y ago
I don't think that's controversial, I think it's different use cases. If I want a simple git repo I'll go gogs/gitea, but GitLab is an all-in-one tool for a software shop.
emilycook 7y ago
Hi GitLab employee, just gonna echo what the other person said and say that it isn't controversial to say that! We know that we don't work for every situation, if all you need is a repo, then by design we can't compete with how lightweight something like Gitea is.

samcday 7y ago

I recently started a new position at a company that is using Gitlab. In the last month I've seen a lot of degraded performance and service outages (especially in Gitlab CI). And then I see stuff like this that makes me scratch my head: https://about.gitlab.com/product/service-desk/ https://about.gitlab.com/product/service-desk/ If anyone at Gitlab is reading this ... please, please slow down on chasing new markets + features and just make the stuff you already have work properly, and fill in the missing pieces. Example: https://gitlab.com/gitlab-org/gitlab-ce/issues/63880 https://gitlab.com/gitlab-org/gitlab-ce/issues/63880

tylfin 7y ago
> please, please slow down on chasing new markets + features and just make the stuff you already have work properly I really agree with this. I was working at a company where we were exploring switching from Github enterprise to Gitlab for the K8s integrations, the docker registry, and the CI features. Following the helm chart installation instructions was a nightmare because we were on-prem w/ a custom cluster. There were a lot of assumptions we had to find workarounds for (e.g. we used an f5 integration to manage our ingresses and performed SSL termination elsewhere so the nginx thing was awful). The other terrifying bit was the number of services / pods that got brought up with the installation. If I have to monitor my team's services, I really don't want to monitor the health of: NGINX, Postgres, Redis, Minio, Registry, GitLab/sidekiq, GitLab/gitlab-shell, GitLab/gitaly, GitLab/unicorn, and GitLab/migrations. When it comes to on-prem or self-hosted software I actually prefer running a monolithic application that worst-case I can just bounce or reboot the server. Slow down, simplify things, and improve the user experience. Gitlab already has enough features to be competitive for a while with the Github + marketplace model.
- citruspi 7y ago
  > When it comes to on-prem or self-hosted software I actually prefer running a monolithic application that worst-case I can just bounce or reboot the server. So why not just skip the helm chart and install the omnibus package on a dedicated server? If you want you can even disable stuff like the omnibus Postgres/Redis/etc and manage those yourself elsewhere. That's what I do - run the omnibus package sans fancy helm chart - and it works out pretty well. Granted it does require a fair bit of RAM to run smoothly. > Slow down, simplify things, and improve the user experience. Gitlab already has enough features to be competitive for a while with the Github + marketplace model. Completely agree with this and what the parent said - I think this is part of why running Gitlab seems to require an ever increasing amount of compute resources :(

baq 7y ago

you have a convoy.

kabwj 7y ago

Funny. Gitlab has always had terrible performance for me, both the website and the git server.

jhgg 7y ago

Running Redis on GCP? I’ve seen CPU and latency spike on our production Redis nodes before - solely due to noisy neighbors causing memory bandwidth contention.

yardie 7y ago

Not sure if this is related but I've received a bunch of emails from Gitlab stating my account is locked for too many failed password attempts. Possible DDoS?

emilycook 7y ago
We haven't had any other reports of this, please submit a ticket here if you think there's an issue so we can track it: https://support.gitlab.com/hc/en-us/requests/new?ticket_form_id=360000515493 https://support.gitlab.com/hc/en-us/requests/new?ticket_form...

awasum_yannick 7y ago

Thank you Gitlab for the amazing product and transparency. Just watching the conversation as everyone over there is trying to get to the root of the problem is inspiring. What a strategy.