25 ms·

Gitlab.com was experiencing elevated error rates for Git, Web, and API

119 points by kaeruct 6y ago

mondss 6y ago
I can't be the only one that feels that GitLab has extended downtime at least weekly, sometimes bi-weekly.
- dt5702 6y ago
  Do you have info on how different providers stack up?
- RicoElectrico 6y ago
  Obligatory "is bi-weekly 2 or 0.5 times a week" confusion comment ;) Though from the context it's 2.
  - Jenk 6y ago
    This is why we have "fortnightly" in the English language.
    - ntechau 6y ago
      ok
      - aufhebung 6y ago
        The word fortnight is in common parlance. Four score is not. But for the record the term bi-weekly is still confusing.
      - kweinber 6y ago
        Like it or not, fortnightly and “once every two weeks” are your only unambiguous choices. We actually had a programming interview question using biweekly at a previous job. It was about thorough requirements definition and understanding/agreeing to measurements.
    - herodoturtle 6y ago
      TIL: "A fortnight is a unit of time equal to 14 days (2 weeks). The word derives from the Old English term fēowertyne niht, meaning "fourteen nights". https://en.wikipedia.org/wiki/Fortnight https://en.wikipedia.org/wiki/Fortnight
    - KptMarchewa 6y ago
      Or just say "twice a week" and "once every two weeks" and not leave everyone in a meeting thinking you're talking about popular video game.
- 0x008 6y ago
  I use gitlab daily and have not experienced any outage in last 6 months probably.
  - nops 6y ago
    6 months is a pretty low bar for a SCM tool.
    - roblabla 6y ago
      Github has had frequent downtime too in the last 6 months too. For a low bar, it seems to be fairly hard to clear.
  - time0ut 6y ago
    It is usually their shared runners. I experience outages with them a couple times a month. It isn't a big deal since we maintain our own private pool, so I just move jobs to them. (They make you pay for CI minutes so we use them for certain jobs.)
    - wdb 6y ago
      For me even our private runners stopped worked over the last few weeks
      - time0ut 6y ago
        Ouch. We haven't seen that, but we don't run them all the time either.
  - skohan 6y ago
    Same - I've had zero issues.
  - danaris 6y ago
    I use GitLab for my side project (source control only; don't have any kind of CI set up just at the moment) and it's been years since there was any really noticeable slowness for me. I can't say I use it daily, but fairly frequently.
- bromonkey 6y ago
  *any non self-hosted git provider
- stabbles 6y ago
  Github seems to be doing worse
  - deleted 6y ago
    [deleted]
  - rvz 6y ago
    Yes. They (GitHub) have been doing a lot worse. Last time I checked, they where completely down 2 days ago. [0] Before that, GitHub Actions was down 14 days ago: [1]. At least with GitLab, you can set up a self-hosted solution for free as a backup, unlike GitHub (Unless you want to pay a lot for GH Enterprise) where some have 'gone all in on GitHub Actions' and then some couldn't push that critical change [2] before the start of the weekend. Oh dear. Maybe its time to setup a self-hosted backup VCS and not depend entirely on GitHub/GitLab web. [0] https://news.ycombinator.com/item?id=26439075 https://news.ycombinator.com/item?id=26439075 [1] https://news.ycombinator.com/item?id=26301659 https://news.ycombinator.com/item?id=26301659 [2] https://news.ycombinator.com/item?id=26439213 https://news.ycombinator.com/item?id=26439213
    - tomxor 6y ago
      > Maybe its time to setup a self-hosted backup VCS and not depend entirely on GitHub/GitLab web. I've been wanting to do this for some time purely for git hosting (don't care about CI at all) but wasn't sure about how to make it failover... Personally i'd be happy with a headless git server, but that's not fair on everyone else who wants the GUI to browse and organise stuff, so I want gitlab etc to deal with that. What would be nice is to have a headless backup server that allowed everyone to continue pulling/pushing from the CLI with their existing repos when gitlab is down. I can't see a smooth way of doing that since it would require messing with the git remotes, unless the solution is inverted and uses a single remote pointing at the "backup" server which then replicates to gitlab, but I don't think gitlab can be configured to change the default remote when people clone it from the UI. I suppose this is why people just end up self hosting gitlab instead.
      - the_duke 6y ago
- Jamie9912 6y ago
  Our self-hosted GitLab instance is running fine!
  - mikepurvis 6y ago
    Yup; I just updated ours to 13.9.4 and it was seamless as always (EE version).
  - drcursor 6y ago
    Pitty its licensing cost will go up ten fold next year.
    - loloquwowndueo 6y ago
      How so? Pics or it didn’t happen.
      - smcleod 6y ago
        https://about.gitlab.com/blog/2021/01/26/new-gitlab-product-subscription-model/ https://about.gitlab.com/blog/2021/01/26/new-gitlab-product-...
      - drcursor 6y ago
        https://about.gitlab.com/blog/2021/01/26/new-gitlab-product-subscription-model/ https://about.gitlab.com/blog/2021/01/26/new-gitlab-product-...
    - sofixa 6y ago
      Only for the paid tiers, free tier stays free.
      - deleted 6y ago
        [deleted]
      - drcursor 6y ago
        0 ten times fold = 0
- wongarsu 6y ago
  Not downtime, but some kind of issue, yes [1]. However github is not much better [2] and gives much less detail and fewer updates. Bitbucket is more solid recently [3] (but wasn't really better than the others in 2020) 1: https://status.gitlab.com/pages/history/5b36dc6502d06804c08349f7 https://status.gitlab.com/pages/history/5b36dc6502d06804c083... 2: https://www.githubstatus.com/history https://www.githubstatus.com/history (don't get fooled by them collapsing the incident list after three per month) 2: https://bitbucket.status.atlassian.com/history https://bitbucket.status.atlassian.com/history (also collapses after three per month)
mcintyre1994 6y ago
I can't even get to the issue where they're tracking this any more. At least the status page is on different infra though, hopefully they'll keep it updated.
- boleary-gl 6y ago
  GitLab employee here. That is correct - we also will be posting updates here: https://twitter.com/gitlabstatus/ https://twitter.com/gitlabstatus/.
  - deleted 6y ago
    [deleted]
whalesalad 6y ago
This is what happens when you rename master to main.
- makeworld 6y ago
  Oh come on
  - whalesalad 6y ago
    I’m not being facetious, or implying support for master/slave nomenclature. But if you ship a big change like this that impacts very core assumptions in 99% of the Git ecosystem, you’re perhaps in for a treat.
  - rataata_jr 6y ago
    He has a point, priorities under MS ownership.
    - JosephRedfern 6y ago
      Except he doesn't, because GitLab isn't owned by Microsoft.
    - detaro 6y ago
      at least get right which company you are talking about ...
    - tester34 6y ago
      haha I wanted write comment like "since MS bought git the amount of crashes per week has increased" but I guess I'm too late to the party!
loloquwowndueo 6y ago
Oh gitLAB - for a moment there I misread “GitHub” and was worried. (J/k you rock guys!)
INTPenis 6y ago
I love Gitlab because I love open source and competition but Gitlab either on-prem or gitlab.com has been having growing pains ever since I started using it around 2015. And it seems to be related to scaling because it's always iffy problems like higher error rates or jobs not executing. Either way, I love our on-prem Gitlab. It's free and it hosts over 200 projects, Gitops and the whole shebang. I'm just now beginning to use Kubernetes from it. They will sort this out eventually.
- hhh 6y ago
  I have nothing but praise for our enterprise Gitlab implementation. It hosts 2000+ projects and for many teams is mission critical. Our team that maintains it is wonderful and I've only seen it down without planning once for about an hour. Using Gitlab at work actually swayed me to move to it for my personal projects as well.
- irateswami 6y ago
  My company does on-prem gitlab as well, and it's pretty fantastic. We're in the beginning stages of migrating off of Azure DevOps but so far we're wowed by the features.
- kevinob11 6y ago
  +1 for on-prem. I tried gitlab.com first but it was very slow (this was years ago, YMMV) but on-prem has been rock-solid and fast for us hosting ~100 projects. One thing I've been surprised by is how easy updates have been, I don't think we've ever had to even reconfigure anything, including runners. I know it seems like that is how it should work, but I've had a lot of opposite experiences with the open source tier of tools.
qeternity 6y ago
We switched from self-hosted to gitlab.com a couple months back and the migration has been extremely disappointing. Pushing commits often hangs completely, and we've had a number of smaller downtimes/degradations in recent weeks that have been super disruptive. Having been Gitlab users since our inception, we're now seriously considering Github (I know - grass is always greener).
- hughw 6y ago
  We don't use Gitlab CI, but only their hosted git. Just thought I'd throw in our sterling experience over the past three years. We've really not noticed whatever downtime they've had.
- robotmay 6y ago
  Any chance you could drop me some details about pushes hanging? I'm an engineer in the source code team doing a lot of performance work and I might be able to correlate it with known issues or investigate it further. I know it's a bit awkward to get data after-the-fact but just some links to the problem repos would be useful. I might not be very useful if it's related to network/hardware issues but I do work in that aspect of the software at least. You can tag me on an issue on gitlab.com with @robotmay_gitlab if the details can be public, or email them to me at rmay@gitlab.com if not :)
- aaronharnly 6y ago
  What prompted you to move away from self-hosted? What was the maintenance overhead like?
damsta 6y ago
This status page is weird, it says that there is an active incident and that there is a databate latency issues, but everything is green: https://i.imgur.com/oXSiyTG.png https://i.imgur.com/oXSiyTG.png
dnsmichi 6y ago
Hi, GitLab team member here. Heads-up: GitLab.com appears to be stable again. Our engineers are monitoring and investigating the root cause of the incident. https://twitter.com/gitlabstatus/status/1371443832865222658 https://twitter.com/gitlabstatus/status/1371443832865222658
- perihelions 6y ago
  Out of curiosity, why does your "about" page vouch for a suspended Twitter account? Is @GitLab -> @GitLabs a typo? https://about.gitlab.com/handbook/marketing/community-relations/community-operations/workflows/twitter/ https://about.gitlab.com/handbook/marketing/community-relati... <td><a href="https://twitter.com/GitLabs">@GitLab</a></td>
  - sytse 6y ago
    Thanks! It is a typo and I submitted a fix https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/77360/diffs https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_request... Please note that about.gitlab.com contains all our static content, including our 10,000 page handbook. This typo isn’t on our main about page but deep in the marketing handbook.
    - perihelions 6y ago
      Cheers for that. I merely wanted to verify the @ was authentic, and that page was the top "site:gitlab.com" result.
      - sytse 6y ago
        Thanks for raising it.
    - dnsmichi 6y ago
      Thanks Sid, merged :) We've also fixed the formatting at https://about.gitlab.com/handbook/marketing/community-relations/community-operations/workflows/twitter/ https://about.gitlab.com/handbook/marketing/community-relati... in https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/77362 https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_request...
gilfoyle 6y ago
Gitlab is a great service and a great alternative to github and bitbucket. But it has been quite flaky over many years and it's quite surprising they haven't prioritised system stability over everything else. https://twitter.com/gitlabstatus https://twitter.com/gitlabstatus Gitlab has been a fully remote proponent going back a few years and these persistent reliability issues add some doubt to that model - even if there might be altogether other issues in play.
- sithlord 6y ago
  My guess is it has to do with their prioritization of only hiring people with "significant ruby/rails" experience. Kills a lot of the talent pool, which doesn't make sense as easy is it is is to pick up a new high level language and framework.
- SkyPuncher 6y ago
  Fully remotes has nothing to do with it. My company is full remote and we don't have these issue.
  - hamburglar 6y ago
    Agreed. Blaming a remote work culture is a complete non sequitur.
ing33k 6y ago
/r took me a lot of effort to convince my CTO to move from Bitbucket to Gitlab. Since we have started using it ( 3 + months ), we have experienced more than 4 instances of downtime. If some one from Gitlab is reading this message, please consider a code freeze and spend some time on stabilizing the system. time to add a new backup remote to all our repos.
- pbalau 6y ago
  We are using bitbucket atm and I can't fathom a reason to move, care to elaborate?
  - Xelbair 6y ago
    Atlassian is Australian company. Australia has very weird and wide reaching laws regarding software backdoors.
    - danaris 6y ago
      Can you elaborate on a) what kinds of laws on software backdoors? (Mandating them? Forbidding them? What sort?), and b) how that affects the GP's request? Edit: Nvm on (b); somehow I got GitLab and BitBucket mixed up as to which one belonged to Atlassian. Still curious about what sort of backdoor laws they have.
      - herodoturtle 6y ago
        I think it's in reference to this from a few years ago: https://www.wired.com/story/australia-encryption-law-global-impact/ https://www.wired.com/story/australia-encryption-law-global-... EDIT: I've done some more reading up on this, and The Assistance and Access Act of 2018 explicitly states that government cannot use backdoors: https://www.homeaffairs.gov.au/about-us/our-portfolios/national-security/lawful-access-telecommunications/data-encryption https://www.homeaffairs.gov.au/about-us/our-portfolios/natio...
  - ing33k 6y ago
    mainly for DevOps related features. https://about.gitlab.com/devops-tools/bitbucket-vs-gitlab/ https://about.gitlab.com/devops-tools/bitbucket-vs-gitlab/ ( marketing page, but mostly valid points )
ezzato 6y ago
I was just pushing some new code to production when it stopped working. So frustrating.
ahachete 6y ago
For those curious: the root cause for the incident is, ultimately, degraded statistics on a table. There are a set of high frequency (very frequently run queries) that are quite sensible to plan flipping (they normally run with a given execution plan; if the plan changes to a worse one, effects can be dramatic, given their frequency). That leads to a lot of query timeouts (GitLab's database limit query execution time to prevent further damage), which are visualized as database errors. This in turn leads to partial or complete downtime (effectively, if the database is running few effective --non timed out-- traffic). Some of these queries depend on a particular Postgres planner way of working that is less than ideal (in PG11, it is improved in PG12), and may lead to plan flipping. Which in turn, happens when table statistics degrade. In GitLab's case, they are normally close to the threshold, and slight statistics degradation caused a plan change, which in turn lead to many queries time out and excessive load on the fleet. At the end of the day, the fix is quite simple, however: update the table statistics (running ANALYZE). For more information: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3962#note_529314106 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3...
- hobs 6y ago
  s/sensible/sensitive/ Would this be a place where a plan hint would help? It seems like having to re-analyze would mean that this is just a matter of time before it becomes problematic again.
  - ahachete 6y ago
    > s/sensible/sensitive/ Thanks :) > Would this be a place where a plan hint would help? It is one option to consider (I wrote about it last week: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3875#note_528191706 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3...), but it is not an ideal solution, for several reasons. Re-analyze is not the solution either, but as a short-term measure, cron-ed ANALYZEs will be run. Longer term, apart from refactoring some queries which are quite prone to trigger this plan behavior, statistics gathering process is going to be fully reviewed.
    - 6y ago

119 points by kaeruct 6y ago

mondss 6y ago

I can't be the only one that feels that GitLab has extended downtime at least weekly, sometimes bi-weekly.

dt5702 6y ago
Do you have info on how different providers stack up?
RicoElectrico 6y ago
Obligatory "is bi-weekly 2 or 0.5 times a week" confusion comment ;) Though from the context it's 2.
- Jenk 6y ago
  This is why we have "fortnightly" in the English language.
  - ntechau 6y ago
    ok
    - aufhebung 6y ago
      The word fortnight is in common parlance. Four score is not. But for the record the term bi-weekly is still confusing.
    - kweinber 6y ago
      Like it or not, fortnightly and “once every two weeks” are your only unambiguous choices. We actually had a programming interview question using biweekly at a previous job. It was about thorough requirements definition and understanding/agreeing to measurements.
  - herodoturtle 6y ago
    TIL: "A fortnight is a unit of time equal to 14 days (2 weeks). The word derives from the Old English term fēowertyne niht, meaning "fourteen nights". https://en.wikipedia.org/wiki/Fortnight https://en.wikipedia.org/wiki/Fortnight
  - KptMarchewa 6y ago
    Or just say "twice a week" and "once every two weeks" and not leave everyone in a meeting thinking you're talking about popular video game.
0x008 6y ago
I use gitlab daily and have not experienced any outage in last 6 months probably.
- nops 6y ago
  6 months is a pretty low bar for a SCM tool.
  - roblabla 6y ago
    Github has had frequent downtime too in the last 6 months too. For a low bar, it seems to be fairly hard to clear.
- time0ut 6y ago
  It is usually their shared runners. I experience outages with them a couple times a month. It isn't a big deal since we maintain our own private pool, so I just move jobs to them. (They make you pay for CI minutes so we use them for certain jobs.)
  - wdb 6y ago
    For me even our private runners stopped worked over the last few weeks
    - time0ut 6y ago
      Ouch. We haven't seen that, but we don't run them all the time either.
- skohan 6y ago
  Same - I've had zero issues.
- danaris 6y ago
  I use GitLab for my side project (source control only; don't have any kind of CI set up just at the moment) and it's been years since there was any really noticeable slowness for me. I can't say I use it daily, but fairly frequently.
bromonkey 6y ago
*any non self-hosted git provider
stabbles 6y ago
Github seems to be doing worse
- deleted 6y ago
  [deleted]
- rvz 6y ago
  Yes. They (GitHub) have been doing a lot worse. Last time I checked, they where completely down 2 days ago. [0] Before that, GitHub Actions was down 14 days ago: [1]. At least with GitLab, you can set up a self-hosted solution for free as a backup, unlike GitHub (Unless you want to pay a lot for GH Enterprise) where some have 'gone all in on GitHub Actions' and then some couldn't push that critical change [2] before the start of the weekend. Oh dear. Maybe its time to setup a self-hosted backup VCS and not depend entirely on GitHub/GitLab web. [0] https://news.ycombinator.com/item?id=26439075 https://news.ycombinator.com/item?id=26439075 [1] https://news.ycombinator.com/item?id=26301659 https://news.ycombinator.com/item?id=26301659 [2] https://news.ycombinator.com/item?id=26439213 https://news.ycombinator.com/item?id=26439213
  - tomxor 6y ago
    > Maybe its time to setup a self-hosted backup VCS and not depend entirely on GitHub/GitLab web. I've been wanting to do this for some time purely for git hosting (don't care about CI at all) but wasn't sure about how to make it failover... Personally i'd be happy with a headless git server, but that's not fair on everyone else who wants the GUI to browse and organise stuff, so I want gitlab etc to deal with that. What would be nice is to have a headless backup server that allowed everyone to continue pulling/pushing from the CLI with their existing repos when gitlab is down. I can't see a smooth way of doing that since it would require messing with the git remotes, unless the solution is inverted and uses a single remote pointing at the "backup" server which then replicates to gitlab, but I don't think gitlab can be configured to change the default remote when people clone it from the UI. I suppose this is why people just end up self hosting gitlab instead.
    - the_duke 6y ago
Jamie9912 6y ago
Our self-hosted GitLab instance is running fine!
- mikepurvis 6y ago
  Yup; I just updated ours to 13.9.4 and it was seamless as always (EE version).
- drcursor 6y ago
  Pitty its licensing cost will go up ten fold next year.
  - loloquwowndueo 6y ago
    How so? Pics or it didn’t happen.
    - smcleod 6y ago
      https://about.gitlab.com/blog/2021/01/26/new-gitlab-product-subscription-model/ https://about.gitlab.com/blog/2021/01/26/new-gitlab-product-...
    - drcursor 6y ago
      https://about.gitlab.com/blog/2021/01/26/new-gitlab-product-subscription-model/ https://about.gitlab.com/blog/2021/01/26/new-gitlab-product-...
  - sofixa 6y ago
    Only for the paid tiers, free tier stays free.
    - deleted 6y ago
      [deleted]
    - drcursor 6y ago
      0 ten times fold = 0
wongarsu 6y ago
Not downtime, but some kind of issue, yes [1]. However github is not much better [2] and gives much less detail and fewer updates. Bitbucket is more solid recently [3] (but wasn't really better than the others in 2020) 1: https://status.gitlab.com/pages/history/5b36dc6502d06804c08349f7 https://status.gitlab.com/pages/history/5b36dc6502d06804c083... 2: https://www.githubstatus.com/history https://www.githubstatus.com/history (don't get fooled by them collapsing the incident list after three per month) 2: https://bitbucket.status.atlassian.com/history https://bitbucket.status.atlassian.com/history (also collapses after three per month)

mcintyre1994 6y ago

I can't even get to the issue where they're tracking this any more. At least the status page is on different infra though, hopefully they'll keep it updated.

boleary-gl 6y ago
GitLab employee here. That is correct - we also will be posting updates here: https://twitter.com/gitlabstatus/ https://twitter.com/gitlabstatus/.
- deleted 6y ago
  [deleted]

whalesalad 6y ago

This is what happens when you rename master to main.

makeworld 6y ago
Oh come on
- whalesalad 6y ago
  I’m not being facetious, or implying support for master/slave nomenclature. But if you ship a big change like this that impacts very core assumptions in 99% of the Git ecosystem, you’re perhaps in for a treat.
- rataata_jr 6y ago
  He has a point, priorities under MS ownership.
  - JosephRedfern 6y ago
    Except he doesn't, because GitLab isn't owned by Microsoft.
  - detaro 6y ago
    at least get right which company you are talking about ...
  - tester34 6y ago
    haha I wanted write comment like "since MS bought git the amount of crashes per week has increased" but I guess I'm too late to the party!

loloquwowndueo 6y ago

Oh gitLAB - for a moment there I misread “GitHub” and was worried. (J/k you rock guys!)

INTPenis 6y ago

I love Gitlab because I love open source and competition but Gitlab either on-prem or gitlab.com has been having growing pains ever since I started using it around 2015. And it seems to be related to scaling because it's always iffy problems like higher error rates or jobs not executing. Either way, I love our on-prem Gitlab. It's free and it hosts over 200 projects, Gitops and the whole shebang. I'm just now beginning to use Kubernetes from it. They will sort this out eventually.

hhh 6y ago
I have nothing but praise for our enterprise Gitlab implementation. It hosts 2000+ projects and for many teams is mission critical. Our team that maintains it is wonderful and I've only seen it down without planning once for about an hour. Using Gitlab at work actually swayed me to move to it for my personal projects as well.
irateswami 6y ago
My company does on-prem gitlab as well, and it's pretty fantastic. We're in the beginning stages of migrating off of Azure DevOps but so far we're wowed by the features.
kevinob11 6y ago
+1 for on-prem. I tried gitlab.com first but it was very slow (this was years ago, YMMV) but on-prem has been rock-solid and fast for us hosting ~100 projects. One thing I've been surprised by is how easy updates have been, I don't think we've ever had to even reconfigure anything, including runners. I know it seems like that is how it should work, but I've had a lot of opposite experiences with the open source tier of tools.

qeternity 6y ago

We switched from self-hosted to gitlab.com a couple months back and the migration has been extremely disappointing. Pushing commits often hangs completely, and we've had a number of smaller downtimes/degradations in recent weeks that have been super disruptive. Having been Gitlab users since our inception, we're now seriously considering Github (I know - grass is always greener).

hughw 6y ago
We don't use Gitlab CI, but only their hosted git. Just thought I'd throw in our sterling experience over the past three years. We've really not noticed whatever downtime they've had.
robotmay 6y ago
Any chance you could drop me some details about pushes hanging? I'm an engineer in the source code team doing a lot of performance work and I might be able to correlate it with known issues or investigate it further. I know it's a bit awkward to get data after-the-fact but just some links to the problem repos would be useful. I might not be very useful if it's related to network/hardware issues but I do work in that aspect of the software at least. You can tag me on an issue on gitlab.com with @robotmay_gitlab if the details can be public, or email them to me at rmay@gitlab.com if not :)
aaronharnly 6y ago
What prompted you to move away from self-hosted? What was the maintenance overhead like?

damsta 6y ago

This status page is weird, it says that there is an active incident and that there is a databate latency issues, but everything is green: https://i.imgur.com/oXSiyTG.png https://i.imgur.com/oXSiyTG.png

dnsmichi 6y ago

Hi, GitLab team member here. Heads-up: GitLab.com appears to be stable again. Our engineers are monitoring and investigating the root cause of the incident. https://twitter.com/gitlabstatus/status/1371443832865222658 https://twitter.com/gitlabstatus/status/1371443832865222658

perihelions 6y ago
Out of curiosity, why does your "about" page vouch for a suspended Twitter account? Is @GitLab -> @GitLabs a typo? https://about.gitlab.com/handbook/marketing/community-relations/community-operations/workflows/twitter/ https://about.gitlab.com/handbook/marketing/community-relati... <td><a href="https://twitter.com/GitLabs">@GitLab</a></td>
- sytse 6y ago
  Thanks! It is a typo and I submitted a fix https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/77360/diffs https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_request... Please note that about.gitlab.com contains all our static content, including our 10,000 page handbook. This typo isn’t on our main about page but deep in the marketing handbook.
  - perihelions 6y ago
    Cheers for that. I merely wanted to verify the @ was authentic, and that page was the top "site:gitlab.com" result.
    - sytse 6y ago
      Thanks for raising it.
  - dnsmichi 6y ago
    Thanks Sid, merged :) We've also fixed the formatting at https://about.gitlab.com/handbook/marketing/community-relations/community-operations/workflows/twitter/ https://about.gitlab.com/handbook/marketing/community-relati... in https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/77362 https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_request...

gilfoyle 6y ago

Gitlab is a great service and a great alternative to github and bitbucket. But it has been quite flaky over many years and it's quite surprising they haven't prioritised system stability over everything else. https://twitter.com/gitlabstatus https://twitter.com/gitlabstatus Gitlab has been a fully remote proponent going back a few years and these persistent reliability issues add some doubt to that model - even if there might be altogether other issues in play.

sithlord 6y ago
My guess is it has to do with their prioritization of only hiring people with "significant ruby/rails" experience. Kills a lot of the talent pool, which doesn't make sense as easy is it is is to pick up a new high level language and framework.
SkyPuncher 6y ago
Fully remotes has nothing to do with it. My company is full remote and we don't have these issue.
- hamburglar 6y ago
  Agreed. Blaming a remote work culture is a complete non sequitur.

ing33k 6y ago

/r took me a lot of effort to convince my CTO to move from Bitbucket to Gitlab. Since we have started using it ( 3 + months ), we have experienced more than 4 instances of downtime. If some one from Gitlab is reading this message, please consider a code freeze and spend some time on stabilizing the system. time to add a new backup remote to all our repos.

pbalau 6y ago
We are using bitbucket atm and I can't fathom a reason to move, care to elaborate?
- Xelbair 6y ago
  Atlassian is Australian company. Australia has very weird and wide reaching laws regarding software backdoors.
  - danaris 6y ago
    Can you elaborate on a) what kinds of laws on software backdoors? (Mandating them? Forbidding them? What sort?), and b) how that affects the GP's request? Edit: Nvm on (b); somehow I got GitLab and BitBucket mixed up as to which one belonged to Atlassian. Still curious about what sort of backdoor laws they have.
    - herodoturtle 6y ago
      I think it's in reference to this from a few years ago: https://www.wired.com/story/australia-encryption-law-global-impact/ https://www.wired.com/story/australia-encryption-law-global-... EDIT: I've done some more reading up on this, and The Assistance and Access Act of 2018 explicitly states that government cannot use backdoors: https://www.homeaffairs.gov.au/about-us/our-portfolios/national-security/lawful-access-telecommunications/data-encryption https://www.homeaffairs.gov.au/about-us/our-portfolios/natio...
- ing33k 6y ago
  mainly for DevOps related features. https://about.gitlab.com/devops-tools/bitbucket-vs-gitlab/ https://about.gitlab.com/devops-tools/bitbucket-vs-gitlab/ ( marketing page, but mostly valid points )

ezzato 6y ago

I was just pushing some new code to production when it stopped working. So frustrating.

ahachete 6y ago

For those curious: the root cause for the incident is, ultimately, degraded statistics on a table. There are a set of high frequency (very frequently run queries) that are quite sensible to plan flipping (they normally run with a given execution plan; if the plan changes to a worse one, effects can be dramatic, given their frequency). That leads to a lot of query timeouts (GitLab's database limit query execution time to prevent further damage), which are visualized as database errors. This in turn leads to partial or complete downtime (effectively, if the database is running few effective --non timed out-- traffic). Some of these queries depend on a particular Postgres planner way of working that is less than ideal (in PG11, it is improved in PG12), and may lead to plan flipping. Which in turn, happens when table statistics degrade. In GitLab's case, they are normally close to the threshold, and slight statistics degradation caused a plan change, which in turn lead to many queries time out and excessive load on the fleet. At the end of the day, the fix is quite simple, however: update the table statistics (running ANALYZE). For more information: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3962#note_529314106 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3...

hobs 6y ago
s/sensible/sensitive/ Would this be a place where a plan hint would help? It seems like having to re-analyze would mean that this is just a matter of time before it becomes problematic again.
- ahachete 6y ago
  > s/sensible/sensitive/ Thanks :) > Would this be a place where a plan hint would help? It is one option to consider (I wrote about it last week: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3875#note_528191706 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3...), but it is not an ideal solution, for several reasons. Re-analyze is not the solution either, but as a short-term measure, cron-ed ANALYZEs will be run. Longer term, apart from refactoring some queries which are quite prone to trigger this plan behavior, statistics gathering process is going to be fully reviewed.
  - 6y ago