9 ms·

Gitlab was down

143 points by vickyonit 7y ago

unilynx 7y ago
It was already a bit flaky since last friday (although the status was still green then) - perhaps having to roll out two security updates yesterday pushed it over the edge...
the-dude 7y ago
One starts to wonder if they will ever get stability under control.
kburman 7y ago
> Website, API, Git (ssh and https), Pages, Registry, CI/CD, Background Processing, Support Services, packages.gitlab.com, customers.gitlab.com, version.gitlab.com, forum.gitlab.com How come all of them are down all at once.
- beaconstudios 7y ago
  presumably a single point of failure - my guess would be something at the network level.
  - manojlds 7y ago
    It's always DNS!
    - yyx 7y ago
      This time it might be Consul
  - dordoka 7y ago
    Spot on. The status page confirms it's a bad firewall configuration
    - beaconstudios 7y ago
      It's always the network.
- RossM 7y ago
  At the bottom of the page they list availability of third party services used - Fastly has a warning symbol, and I imagine they put that CDN in front of everything.
  - bArray 7y ago
    > Fastly has a warning symbol, and I imagine they put that > CDN in front of everything. Check their status page, it's just a simple reroute since the 13th [1]. [1] https://status.fastly.com/ https://status.fastly.com/
- k_ 7y ago
  Latest tweet by @gitlabstatus said "We've identified an issue with database connectivity", which could explain why so many services are impacted.
  - deleted 7y ago
    [deleted]
- tyingq 7y ago
  Bad firewall change...they just updated the page.
- ownagefool 7y ago
  Likely just means they have a Single Point of Failure. Some guesses would be: Automation/orchestration - They've migrated to k8s (I don't believe they've actually done this yet), but it could be their orchestration / automation tool automated a broken thing everywhere. Database/Auth - Pretty much everything in gitlab will touch the database as far as I'm aware. Otherwise, how do you check whether users are auth'd to take action something. You wouldn't expect this to break the static website, i.e. the sales landing pages, but these could be based off an internal CMS, or could be checking for "guest" role session. DNS/Service Discovery - As a sibling posted, "it's always DNS". It's good practice to use names for services instead of IP addresses, but this means your DNS needs to generally work, or everything will go down. Service Discovery could rely on DNS, but it could also be an API call that finds out DNS addresses or IP addresses directly. CDN - You wouldn't typically put this in front of auth'd usage, and typically a CDN might not be helpful in front of something like SSH, but a quick look at fastly suggests they might support this. The main downside is sharing all the user data / auth tokens. Security Product / CA - All you need is a requirement to encrypt internal traffic and rotate secrets, and you end up with a secret store that sits in the middle of everything. Storage Layer - I believe they were big on Ceph for a while. If everything is backed by Ceph, everything will go down if you fail with Ceph. Obviously, whatever it is, you'd expect them to split up their fail over plan a bit more in the future if it is something like that, but usually there's a single point of failure somewhere.
  - ownagefool 7y ago
    Replying to myself, because it's now on their status page that a firewall change took down the database. This points to there being: - a lack of process and testing on key networking changes. Aren't they doing CI/CD, automated testing and peer review for this? - A SPOF in the database; why couldn't things connect to a secondary for a read-only mode? Quite a lot of the time, things break for stupid reasons. The main difference is when a normal company does something stupid, they can hide it, lie about it, or make it sound more complex. The fact Gitlab publishes their fuck ups, is supposed to force them to do a better job and actually look at root causes and apply proper fixes that we can all judge. I wouldn't hold any particular fuck-up against them.
- m712 7y ago
  The status page has updated to indicate that they misconfigured their firewall. Apparently their entire set of services go through a single firewall (or at least, multiple firewalls with the same config). It's worrying that they don't have a staging setup for these kinds of things. (NOTE: I am speculating here, if they do have a staging system and this wasn't reproduced there then the last sentence doesn't apply.)
- ktsmith 7y ago
  You can look through what happened here: https://gitlab.com/gitlab-com/gl-infra/production/issues/1421 https://gitlab.com/gitlab-com/gl-infra/production/issues/142...
sojmq 7y ago
This strikes me as odd, because if there's something gitlab engineers are known for, is their unmatched competence,
- jensvdh 7y ago
  Like when they dropped their database and took a few days to recover?
  - YorickPeterse 7y ago
    It took about 24 hours to recover, not "a few days".
- qaq 7y ago
  Hmm how did you come to this conclusion?
- iofiiiiiiiii 7y ago
  Good one
pearjuice 7y ago
I don't want to hijack this thread but what was the end result of the decision of Gitlab not hiring _in_ certain countries anymore? A lot of media coverage when the incident happened but no idea what happened afterwards and can't easily find it.
- toupeira 7y ago
  The discussion is still ongoing, the latest official communication I'm aware of was the blog post at https://about.gitlab.com/blog/2019/11/12/update-on-hiring/ https://about.gitlab.com/blog/2019/11/12/update-on-hiring/ Note that this was always only about excluding certain job roles who have administrative access to production servers, not all jobs in general.
- the-dude 7y ago
  IIRC, they are not hiring 'in' certain counties. This is different than 'from' certain countries. AFAIK, nothing material happened after the announcement. There are lots of companies which do not hire in certain countries.
- bArray 7y ago
  [Speculation] Perhaps a DDoS or a disgruntled employee... In all seriousness, hopefully these things are not related.
- UserIsUnused 7y ago
  Usually it's Legal/HR issues with the way to pay your employees. As another one said, their policy in not hiring people "in" rather than "from", as tax rules change from country to country.
Uninen 7y ago
We use GitLab at work and I use it for personal projects as well, it has been very slow for several days now and they had downtime yesterday as well. I don't mind the general sluggishness of the system that much (as I love the platform in general) but when you can't get your work (nor hobbies) done because of tools breaking, it gets really annoying really fast.
- bArray 7y ago
  To be fair, just be thankful you don't use something like Jazz RTC - we had down time at least once every two weeks (from an hour to a whole day at a time). It got so bad we ended up setting up a local network Raspberry Pi as a Git server and emailing patches to remote teams.
- progfix 7y ago
  It's not a tool, it's an online service.
- tadzik_ 7y ago
  > when you can't get your work (...) done because of tools breaking, it gets really annoying really fast I always assumed that the publicly hosted version of gitlab is basically a giant demo version of the enterprise edition you buy to host it yourself. Hell, you can even host the community edition for free. If your work relies on it, why rely on a free online product? EDIT: TIL gitlab.com also has a paid options. I stand corrected. We have a hosted Gitlab at 2 of my clients. Both are up.
  - idiocratic 7y ago
    The online version has non-free plans as well, so I don't see your point of not relying on it.
  - Ndymium 7y ago
    gitlab.com is not only for free clients, plenty of paid clients use it instead of hosting themselves.
  - andrelaszlo 7y ago
beredon 7y ago
Man, I just setup a new project on Gitlab an hour ago!
PhilKunz 7y ago
It seems like GitLab should indeed focus on stability now. The feature set is great. But it is all not worth it if it keeps disrupting work on a constant basis by being unstable.
- toopok4k3 7y ago
  I recommend hosting it yourself, works great and is fast on local network. You can run updates when it's convenient to you, not to others. GitLab is a rare software that enables you to be in control.
  - PhilKunz 7y ago
    I've always been a little cautious of running it myself. When I look at the components involved there are a lot of moving parts. And I don't want to be ever in a position where I can't get things started again. If I'm running on premise I'll switch to gitea with drone ci instead.
    - niceworkbuddy 7y ago
      GitLab has virtual machine image, so it's really unobtrusive for your system.
      - jeltz 7y ago
        They also have a Debian package with sets up the whole environment. It has worked well for me so far. The only thing I had to add was a backup script.
        apple4ever 7y ago
        The omnibus package makes it real easy to install, especially with Ansible. FYI if you are uploading to AWS or other cloud providers, you don’t even need a backup script. You can configure it in the gitlab.rb file: https://docs.gitlab.com/ee/raketasks/backup_restore.html#uploading-backups-to-a-remote-cloud-storage https://docs.gitlab.com/ee/raketasks/backup_restore.html#upl...
castis 7y ago
oof, literally all of it. dont see that too often. on thanksgiving day no less. i root for gitlab when i can, but theres a reason i mirror my repos to github.
bvm 7y ago
I love gitlab, I really do, but their uptime is atrocious for a multi-billion dollar company.
hieudang9 7y ago
I hope this incident is not #HugOps again, waiting for new interesting things from postmortem
the-dude 7y ago
It is worth mentioning their status page is not down.
- spzb 7y ago
  It looks like it's externally hosted at status.io. It makes sense to have your status page on separate hosting otherwise it can go down at the same time as your main infrastructure and that defeats the point of having a status page.
- mtsx 7y ago
  LOL!
dijit 7y ago
Notably they don’t hire any real people with ops experience. Erring instead to go for developers hoping that they can do everything needed. I like gitlab as a product but they don’t have a service mindset, and I think not hiring operations-centric people is a symptom of that which causes these kinds of issues.
- softwarelimits 7y ago
  How do you know that they do not hire "any people with ops experence"? Do you have insight? Is there any public evidence for this or anything you could publish yourself to add some substance to your words? It would be great if you understand that just saying something is not enough on the internet.
  - dijit 7y ago
    What you say is completely fair. I’ve been looking at job postings and watching the way they work too for a little time since it’s all open. DBAs are “ruby devs who have used Postgres” https://about.gitlab.com/jobs/apply/backend-engineer-database-4473989002/ https://about.gitlab.com/jobs/apply/backend-engineer-databas... SREs are “ruby devs who have used docker/kubernetes” (No job listing currently) The only open job labelled “ops” is telling. https://about.gitlab.com/jobs/apply/frontend-engineer---configure-team-4529842002/ https://about.gitlab.com/jobs/apply/frontend-engineer---conf...
    - toupeira 7y ago
      GitLab engineer here. The "Ops" section actually consists of product development teams for features in GitLab itself (the Configure/Monitor "stages"), while the SREs are in the "Infrastructure" department. See https://about.gitlab.com/handbook/engineering/#engineering-departments-sections--teams https://about.gitlab.com/handbook/engineering/#engineering-d.... But yes, we don't seem to be hiring SREs at the moment, but I assume we'll add more openings in the beginning of next year. Regarding DBAs, that job you posted is more of a normal Backend Engineer role with a database specialty. We also have dedicated Database Engineer roles: - https://about.gitlab.com/job-families/engineering/database-engineer/ https://about.gitlab.com/job-families/engineering/database-e... - https://about.gitlab.com/job-families/engineering/database-reliability-engineer/ https://about.gitlab.com/job-families/engineering/database-r... The job description for SREs is here: - https://about.gitlab.com/job-families/engineering/site-reliability-engineer/ https://about.gitlab.com/job-families/engineering/site-relia...
softwarelimits 7y ago
Hmmm, just today I wanted to install a new virtual machine with the latest Gitlab release to check if it makes sense to run that inhouse... does anybod know if there is a VM, a vagrant machine, an ISO or a repository online that still can be used? Thanks! Or does exist a mirror on github.com? ;)
- maxnoe 7y ago
  Get the docker image from dockerhub
- bArray 7y ago
  > Hmmm, just today I wanted to install a new virtual machine > with the latest Gitlab release Just wait till later/tomorrow. Setting up your own instance is relatively easy. Still, you can get most of the way without their repos... https://www.techrepublic.com/article/how-to-set-up-a-gitlab-server-and-host-your-own-git-repositories/ https://www.techrepublic.com/article/how-to-set-up-a-gitlab-... > Or does exist a mirror on github.com? ;) It seems so: https://github.com/gitlabhq/gitlabhq https://github.com/gitlabhq/gitlabhq
- andrelaszlo 7y ago
  If you just want to try it out, the easiest is often Gitlab Development Kit. gem install gitlab-development-kit gdk init cd gitlab-development-kit gdk run I doubt it will work today though; it's probably pulling everything from Gitlab. For Docker, check out https://hub.docker.com/r/gitlab/gitlab-ce https://hub.docker.com/r/gitlab/gitlab-ce I haven't tried it out for myself but it seems popular :)
  - leipert 7y ago
    The GDK installation has changed a bit (just set it up today, you need to run gdk install and gdk start (instead of run) for example). Please also note that it involves installing dependencies on your local machine, like e.g. Ruby / Postgres, etc. Here is the link to the GDK: https://gitlab.com/gitlab-org/gitlab-development-kit https://gitlab.com/gitlab-org/gitlab-development-kit Otherwise I would try the docker container OR just install it with omnibus in a VM: https://about.gitlab.com/install/ https://about.gitlab.com/install/
yRetsyM 7y ago
There has been a theme of instability with Gitlab.com over the last week or two. I'm not sure if it's growth related (they've seen a steady increase of users/traffic) and they've reached a scaling peak. OR if it's technically related - they've been doing a number of different infrastructure changes over the last few weeks which make a material difference to the main layers of the service. For me the real test here is how they respond to this. As a paying customer I want to understand the issue, the efforts to prevent this in future and how they communicate this.
- appkate 7y ago
  Suffering the same instability issue and switched back to Github as that's what a paying customer would do.
tyingq 7y ago
Updated just now...bad firewall change. "[Identified] We have identified firewall misconfiguration was applied that is preventing applications from connecting to the database."
bArray 7y ago
> [Identified] We have identified firewall misconfiguration > was applied that is preventing applications from connecting > to the database. We've rolling back that change and expect > to be operational again shortly. Heh, happens to the best of us! Seems to be coming back online now.
greens231 7y ago
Originally joined gitlab for their free private repos but with the recent downtime/sluggishness, i have jumped over to github (now that they offer free private repos too)
- Dayshine 7y ago
  Gitlab is still the only viable choice for non-commercial groups who want private repos though, the 3 private members and no ability to have mixtures of public/private repos in organisations on Github is very limiting.
  - stevekemp 7y ago
    Bitbucket? (Even though their availability is terrible too.)
  - scaryclam 7y ago
    Sorry, but this isn't true at all. Bitbucket works fine, and GitHub offers plans for non-profit groups for free: https://github.com/nonprofit https://github.com/nonprofit Self-hosting has multiple different options as well.
    - Dayshine 7y ago
      Registered non-profit <> non-commercial. - a private website for my local sailing club. - a mod for a game - an open source project that requires a private repository for a few things - any project relating to a private community None of these are registered non-profits. Bitbucket is capped at 5 users as far as I can see, and self-hosting is just a recipe for lost data. I don't know of many amateur groups that can safely host a server.
m712 7y ago
Ironically, status.gitlab.com takes upwards of a minute to load on my end. I thought it was going to time out.
1337shadow 7y ago
Back in the days GitLab made it clear that it aspires to be a software company rather than an infrastructure company. The reason I favor GitLab over GitHub is that I can install it on a dedicated server and have blazing fast performance, which helps development performance on an every day basis. I've been maintaining a bunch of deployments of all sorts (k8s, docker-compose, baremetal) for the last years for several customers of all sorts of size, the upgrade process has always been pretty smooth, sometimes I ran in edgecases yes but always found a solution. My deployments are always up and kicking, unless I'm messing with the configuration and doing some mistakes. I highly recommend hosting your own GitLab instance, even on a single server.
- oefrha 7y ago
  > have blazing fast performance Sorry, but GitLab has never been blazing fast. The gitlab.com instance is notoriously slow — supposedly improved a lot over the past few years, but still feels pretty sluggish. My self-hosted instance isn’t much better. In fact, a couple of open source maintainer friends looked into migrating to GitLab when GitHub was acquired by MS; they did not precisely because GitLab was too slow.
  - frant-hartm 7y ago
    From my experience a custom GitLab instance is much faster that GitHub.com. But it is apples and oranges and I suppose your friends weren't comparing the two.
    - oefrha 7y ago
      My custom GitLab instance is still slower than github.com. (Yes, I’m giving it enough memory.)
      - Aeolun 7y ago
        It’s slower than github.com, but I wouldn’t call it slow. What’s slow is out bitbucket datacenter edition.
sleepnow 7y ago
Microsoft must have migrated it over to run on Windows Server.
- looperhacks 7y ago
  Maybe you mixed something up. This is about Gitlab, not Github.
jlengrand 7y ago
I used to be a total Gitlab fanboy. I was going to Sid's meetups back in Utrecht 6 years ago, and he's a model of mine to this time. I would have done anything to stay on the platform, for at home and the office and setup gitlab instances in two different offices. Since somewhere last year, I moved back to Github primarily, and I'm sad to hear that my company is likely to make the same choice soon.The only reason is stability, and that's a little sad to me. I really want to love the product, but I need something that just works; not bells and whistles
- kungtotte 7y ago
  Whenever GitHub is down you see comments saying much the same thing only with the names reversed. The thing that strikes me each time is how fragile everyone's setup is if GitHub/lab is a single point of failure for them... At least gitlab let's you self host, which would let you run backups on offsite hosting meaning zero downtime.
  - protomikron 7y ago
    GitHub's uptime is way better than GitLab. I am not really sure gitlab.com even reaches 99% availability (not to mention 99.9%). GitLab may well be focused on providing full-stack dev-services (VCS, build-server, CI and stuff), but in the end they are a hosting company - and for hosting, uptime is one of the most important metrics. EDIT: 99% was exaggerating, but I got so many 50Xs the last days that it was from time to time unusable.
    - tristanperry 7y ago
      I agree that GitHub's uptime is better, but I doubt that Gitlab aren't at 99% availability - that'd mean over 14.4 minutes of downtime each day.
      - toupeira 7y ago
        We have some public Pingdom stats at http://stats.gitlab.com/4932705/history http://stats.gitlab.com/4932705/history, looks like this was the first time we dipped below 99.9% this year (partial outages excluded). But yeah, our response times have been steadily increasing and could definitely be a lot better ;-)
darekkay 7y ago
It is up again. > [Monitoring] GitLab.com is now recovering. We found 2 last DB nodes which had not reverted their change. Apologies for the disruption. [1] [1] https://status.gitlab.com/ https://status.gitlab.com/
joeblau 7y ago
We’re doing this now? For my entire tenure on HN, this site has been a status page for GitHub. Looks like it’s Gitlabs turn.
ifthenelseend 7y ago
You guys should consider switching to GitHub. Much faster, no 500-errors, less downtime.
deleted 7y ago
[deleted]
intellix 7y ago
Not sure if I have anything else to add on top of what people are already saying but the performance of GL is extremely frustrating for the past couple of months. Viewing the diff of a PR can sometimes take around 20 seconds just to load the tab. I think it's about time they dedicated some resources or addressed the public about the efforts they're making to address these issues. It's becoming excruciating. I've used Github every day for years and years and I feel like every page load has been pretty much instant forever.
BlackLotus89 7y ago
It was an iptables problem https://gitlab.com/gitlab-com/gl-infra/production/issues/1421 https://gitlab.com/gitlab-com/gl-infra/production/issues/142... The rollout of new iptables rules blocked database server connections
aabbcc1241 7y ago
Any service on clearnet can go down. p2p alternative like git center (git over zeronet) is more resilient to service provider mul-configuration and censorship git center: http://127.0.0.1:43110/1GitLiXB6t5r8vuU2zC6a8GYj9ME6HMQ4t/ http://127.0.0.1:43110/1GitLiXB6t5r8vuU2zC6a8GYj9ME6HMQ4t/ client: https://zeronet.io/ https://zeronet.io/ proxy: https://zero.acelewis.com/ https://zero.acelewis.com/

143 points by vickyonit 7y ago

It was already a bit flaky since last friday (although the status was still green then) - perhaps having to roll out two security updates yesterday pushed it over the edge...

the-dude 7y ago

One starts to wonder if they will ever get stability under control.

kburman 7y ago

> Website, API, Git (ssh and https), Pages, Registry, CI/CD, Background Processing, Support Services, packages.gitlab.com, customers.gitlab.com, version.gitlab.com, forum.gitlab.com How come all of them are down all at once.

beaconstudios 7y ago
presumably a single point of failure - my guess would be something at the network level.
- manojlds 7y ago
  It's always DNS!
  - yyx 7y ago
    This time it might be Consul
- dordoka 7y ago
  Spot on. The status page confirms it's a bad firewall configuration
  - beaconstudios 7y ago
    It's always the network.
RossM 7y ago
At the bottom of the page they list availability of third party services used - Fastly has a warning symbol, and I imagine they put that CDN in front of everything.
- bArray 7y ago
  > Fastly has a warning symbol, and I imagine they put that > CDN in front of everything. Check their status page, it's just a simple reroute since the 13th [1]. [1] https://status.fastly.com/ https://status.fastly.com/
k_ 7y ago
Latest tweet by @gitlabstatus said "We've identified an issue with database connectivity", which could explain why so many services are impacted.
- deleted 7y ago
  [deleted]
tyingq 7y ago
Bad firewall change...they just updated the page.
ownagefool 7y ago
Likely just means they have a Single Point of Failure. Some guesses would be: Automation/orchestration - They've migrated to k8s (I don't believe they've actually done this yet), but it could be their orchestration / automation tool automated a broken thing everywhere. Database/Auth - Pretty much everything in gitlab will touch the database as far as I'm aware. Otherwise, how do you check whether users are auth'd to take action something. You wouldn't expect this to break the static website, i.e. the sales landing pages, but these could be based off an internal CMS, or could be checking for "guest" role session. DNS/Service Discovery - As a sibling posted, "it's always DNS". It's good practice to use names for services instead of IP addresses, but this means your DNS needs to generally work, or everything will go down. Service Discovery could rely on DNS, but it could also be an API call that finds out DNS addresses or IP addresses directly. CDN - You wouldn't typically put this in front of auth'd usage, and typically a CDN might not be helpful in front of something like SSH, but a quick look at fastly suggests they might support this. The main downside is sharing all the user data / auth tokens. Security Product / CA - All you need is a requirement to encrypt internal traffic and rotate secrets, and you end up with a secret store that sits in the middle of everything. Storage Layer - I believe they were big on Ceph for a while. If everything is backed by Ceph, everything will go down if you fail with Ceph. Obviously, whatever it is, you'd expect them to split up their fail over plan a bit more in the future if it is something like that, but usually there's a single point of failure somewhere.
- ownagefool 7y ago
  Replying to myself, because it's now on their status page that a firewall change took down the database. This points to there being: - a lack of process and testing on key networking changes. Aren't they doing CI/CD, automated testing and peer review for this? - A SPOF in the database; why couldn't things connect to a secondary for a read-only mode? Quite a lot of the time, things break for stupid reasons. The main difference is when a normal company does something stupid, they can hide it, lie about it, or make it sound more complex. The fact Gitlab publishes their fuck ups, is supposed to force them to do a better job and actually look at root causes and apply proper fixes that we can all judge. I wouldn't hold any particular fuck-up against them.
m712 7y ago
The status page has updated to indicate that they misconfigured their firewall. Apparently their entire set of services go through a single firewall (or at least, multiple firewalls with the same config). It's worrying that they don't have a staging setup for these kinds of things. (NOTE: I am speculating here, if they do have a staging system and this wasn't reproduced there then the last sentence doesn't apply.)
ktsmith 7y ago
You can look through what happened here: https://gitlab.com/gitlab-com/gl-infra/production/issues/1421 https://gitlab.com/gitlab-com/gl-infra/production/issues/142...

sojmq 7y ago

This strikes me as odd, because if there's something gitlab engineers are known for, is their unmatched competence,

jensvdh 7y ago
Like when they dropped their database and took a few days to recover?
- YorickPeterse 7y ago
  It took about 24 hours to recover, not "a few days".
qaq 7y ago
Hmm how did you come to this conclusion?
iofiiiiiiiii 7y ago
Good one

pearjuice 7y ago

I don't want to hijack this thread but what was the end result of the decision of Gitlab not hiring _in_ certain countries anymore? A lot of media coverage when the incident happened but no idea what happened afterwards and can't easily find it.

toupeira 7y ago
The discussion is still ongoing, the latest official communication I'm aware of was the blog post at https://about.gitlab.com/blog/2019/11/12/update-on-hiring/ https://about.gitlab.com/blog/2019/11/12/update-on-hiring/ Note that this was always only about excluding certain job roles who have administrative access to production servers, not all jobs in general.
the-dude 7y ago
IIRC, they are not hiring 'in' certain counties. This is different than 'from' certain countries. AFAIK, nothing material happened after the announcement. There are lots of companies which do not hire in certain countries.
bArray 7y ago
[Speculation] Perhaps a DDoS or a disgruntled employee... In all seriousness, hopefully these things are not related.
UserIsUnused 7y ago
Usually it's Legal/HR issues with the way to pay your employees. As another one said, their policy in not hiring people "in" rather than "from", as tax rules change from country to country.

Uninen 7y ago

We use GitLab at work and I use it for personal projects as well, it has been very slow for several days now and they had downtime yesterday as well. I don't mind the general sluggishness of the system that much (as I love the platform in general) but when you can't get your work (nor hobbies) done because of tools breaking, it gets really annoying really fast.

bArray 7y ago
To be fair, just be thankful you don't use something like Jazz RTC - we had down time at least once every two weeks (from an hour to a whole day at a time). It got so bad we ended up setting up a local network Raspberry Pi as a Git server and emailing patches to remote teams.
progfix 7y ago
It's not a tool, it's an online service.
tadzik_ 7y ago
> when you can't get your work (...) done because of tools breaking, it gets really annoying really fast I always assumed that the publicly hosted version of gitlab is basically a giant demo version of the enterprise edition you buy to host it yourself. Hell, you can even host the community edition for free. If your work relies on it, why rely on a free online product? EDIT: TIL gitlab.com also has a paid options. I stand corrected. We have a hosted Gitlab at 2 of my clients. Both are up.
- idiocratic 7y ago
  The online version has non-free plans as well, so I don't see your point of not relying on it.
- Ndymium 7y ago
  gitlab.com is not only for free clients, plenty of paid clients use it instead of hosting themselves.
- andrelaszlo 7y ago

beredon 7y ago

Man, I just setup a new project on Gitlab an hour ago!

PhilKunz 7y ago

It seems like GitLab should indeed focus on stability now. The feature set is great. But it is all not worth it if it keeps disrupting work on a constant basis by being unstable.

toopok4k3 7y ago
I recommend hosting it yourself, works great and is fast on local network. You can run updates when it's convenient to you, not to others. GitLab is a rare software that enables you to be in control.
- PhilKunz 7y ago
  I've always been a little cautious of running it myself. When I look at the components involved there are a lot of moving parts. And I don't want to be ever in a position where I can't get things started again. If I'm running on premise I'll switch to gitea with drone ci instead.
  - niceworkbuddy 7y ago
    GitLab has virtual machine image, so it's really unobtrusive for your system.
    - jeltz 7y ago
      They also have a Debian package with sets up the whole environment. It has worked well for me so far. The only thing I had to add was a backup script.
      - apple4ever 7y ago
        The omnibus package makes it real easy to install, especially with Ansible. FYI if you are uploading to AWS or other cloud providers, you don’t even need a backup script. You can configure it in the gitlab.rb file: https://docs.gitlab.com/ee/raketasks/backup_restore.html#uploading-backups-to-a-remote-cloud-storage https://docs.gitlab.com/ee/raketasks/backup_restore.html#upl...

castis 7y ago

oof, literally all of it. dont see that too often. on thanksgiving day no less. i root for gitlab when i can, but theres a reason i mirror my repos to github.

bvm 7y ago

I love gitlab, I really do, but their uptime is atrocious for a multi-billion dollar company.

hieudang9 7y ago

I hope this incident is not #HugOps again, waiting for new interesting things from postmortem

the-dude 7y ago

It is worth mentioning their status page is not down.

spzb 7y ago
It looks like it's externally hosted at status.io. It makes sense to have your status page on separate hosting otherwise it can go down at the same time as your main infrastructure and that defeats the point of having a status page.
mtsx 7y ago
LOL!

dijit 7y ago

Notably they don’t hire any real people with ops experience. Erring instead to go for developers hoping that they can do everything needed. I like gitlab as a product but they don’t have a service mindset, and I think not hiring operations-centric people is a symptom of that which causes these kinds of issues.

softwarelimits 7y ago
How do you know that they do not hire "any people with ops experence"? Do you have insight? Is there any public evidence for this or anything you could publish yourself to add some substance to your words? It would be great if you understand that just saying something is not enough on the internet.
- dijit 7y ago
  What you say is completely fair. I’ve been looking at job postings and watching the way they work too for a little time since it’s all open. DBAs are “ruby devs who have used Postgres” https://about.gitlab.com/jobs/apply/backend-engineer-database-4473989002/ https://about.gitlab.com/jobs/apply/backend-engineer-databas... SREs are “ruby devs who have used docker/kubernetes” (No job listing currently) The only open job labelled “ops” is telling. https://about.gitlab.com/jobs/apply/frontend-engineer---configure-team-4529842002/ https://about.gitlab.com/jobs/apply/frontend-engineer---conf...
  - toupeira 7y ago
    GitLab engineer here. The "Ops" section actually consists of product development teams for features in GitLab itself (the Configure/Monitor "stages"), while the SREs are in the "Infrastructure" department. See https://about.gitlab.com/handbook/engineering/#engineering-departments-sections--teams https://about.gitlab.com/handbook/engineering/#engineering-d.... But yes, we don't seem to be hiring SREs at the moment, but I assume we'll add more openings in the beginning of next year. Regarding DBAs, that job you posted is more of a normal Backend Engineer role with a database specialty. We also have dedicated Database Engineer roles: - https://about.gitlab.com/job-families/engineering/database-engineer/ https://about.gitlab.com/job-families/engineering/database-e... - https://about.gitlab.com/job-families/engineering/database-reliability-engineer/ https://about.gitlab.com/job-families/engineering/database-r... The job description for SREs is here: - https://about.gitlab.com/job-families/engineering/site-reliability-engineer/ https://about.gitlab.com/job-families/engineering/site-relia...

softwarelimits 7y ago

Hmmm, just today I wanted to install a new virtual machine with the latest Gitlab release to check if it makes sense to run that inhouse... does anybod know if there is a VM, a vagrant machine, an ISO or a repository online that still can be used? Thanks! Or does exist a mirror on github.com? ;)

maxnoe 7y ago
Get the docker image from dockerhub
bArray 7y ago
> Hmmm, just today I wanted to install a new virtual machine > with the latest Gitlab release Just wait till later/tomorrow. Setting up your own instance is relatively easy. Still, you can get most of the way without their repos... https://www.techrepublic.com/article/how-to-set-up-a-gitlab-server-and-host-your-own-git-repositories/ https://www.techrepublic.com/article/how-to-set-up-a-gitlab-... > Or does exist a mirror on github.com? ;) It seems so: https://github.com/gitlabhq/gitlabhq https://github.com/gitlabhq/gitlabhq
andrelaszlo 7y ago
If you just want to try it out, the easiest is often Gitlab Development Kit. gem install gitlab-development-kit gdk init cd gitlab-development-kit gdk run I doubt it will work today though; it's probably pulling everything from Gitlab. For Docker, check out https://hub.docker.com/r/gitlab/gitlab-ce https://hub.docker.com/r/gitlab/gitlab-ce I haven't tried it out for myself but it seems popular :)
- leipert 7y ago
  The GDK installation has changed a bit (just set it up today, you need to run gdk install and gdk start (instead of run) for example). Please also note that it involves installing dependencies on your local machine, like e.g. Ruby / Postgres, etc. Here is the link to the GDK: https://gitlab.com/gitlab-org/gitlab-development-kit https://gitlab.com/gitlab-org/gitlab-development-kit Otherwise I would try the docker container OR just install it with omnibus in a VM: https://about.gitlab.com/install/ https://about.gitlab.com/install/

yRetsyM 7y ago

There has been a theme of instability with Gitlab.com over the last week or two. I'm not sure if it's growth related (they've seen a steady increase of users/traffic) and they've reached a scaling peak. OR if it's technically related - they've been doing a number of different infrastructure changes over the last few weeks which make a material difference to the main layers of the service. For me the real test here is how they respond to this. As a paying customer I want to understand the issue, the efforts to prevent this in future and how they communicate this.

appkate 7y ago
Suffering the same instability issue and switched back to Github as that's what a paying customer would do.

tyingq 7y ago

Updated just now...bad firewall change. "[Identified] We have identified firewall misconfiguration was applied that is preventing applications from connecting to the database."

bArray 7y ago

> [Identified] We have identified firewall misconfiguration > was applied that is preventing applications from connecting > to the database. We've rolling back that change and expect > to be operational again shortly. Heh, happens to the best of us! Seems to be coming back online now.

greens231 7y ago

Originally joined gitlab for their free private repos but with the recent downtime/sluggishness, i have jumped over to github (now that they offer free private repos too)

Dayshine 7y ago
Gitlab is still the only viable choice for non-commercial groups who want private repos though, the 3 private members and no ability to have mixtures of public/private repos in organisations on Github is very limiting.
- stevekemp 7y ago
  Bitbucket? (Even though their availability is terrible too.)
- scaryclam 7y ago
  Sorry, but this isn't true at all. Bitbucket works fine, and GitHub offers plans for non-profit groups for free: https://github.com/nonprofit https://github.com/nonprofit Self-hosting has multiple different options as well.
  - Dayshine 7y ago
    Registered non-profit <> non-commercial. - a private website for my local sailing club. - a mod for a game - an open source project that requires a private repository for a few things - any project relating to a private community None of these are registered non-profits. Bitbucket is capped at 5 users as far as I can see, and self-hosting is just a recipe for lost data. I don't know of many amateur groups that can safely host a server.

m712 7y ago

Ironically, status.gitlab.com takes upwards of a minute to load on my end. I thought it was going to time out.

1337shadow 7y ago

Back in the days GitLab made it clear that it aspires to be a software company rather than an infrastructure company. The reason I favor GitLab over GitHub is that I can install it on a dedicated server and have blazing fast performance, which helps development performance on an every day basis. I've been maintaining a bunch of deployments of all sorts (k8s, docker-compose, baremetal) for the last years for several customers of all sorts of size, the upgrade process has always been pretty smooth, sometimes I ran in edgecases yes but always found a solution. My deployments are always up and kicking, unless I'm messing with the configuration and doing some mistakes. I highly recommend hosting your own GitLab instance, even on a single server.

oefrha 7y ago
> have blazing fast performance Sorry, but GitLab has never been blazing fast. The gitlab.com instance is notoriously slow — supposedly improved a lot over the past few years, but still feels pretty sluggish. My self-hosted instance isn’t much better. In fact, a couple of open source maintainer friends looked into migrating to GitLab when GitHub was acquired by MS; they did not precisely because GitLab was too slow.
- frant-hartm 7y ago
  From my experience a custom GitLab instance is much faster that GitHub.com. But it is apples and oranges and I suppose your friends weren't comparing the two.
  - oefrha 7y ago
    My custom GitLab instance is still slower than github.com. (Yes, I’m giving it enough memory.)
    - Aeolun 7y ago
      It’s slower than github.com, but I wouldn’t call it slow. What’s slow is out bitbucket datacenter edition.

sleepnow 7y ago

Microsoft must have migrated it over to run on Windows Server.

looperhacks 7y ago
Maybe you mixed something up. This is about Gitlab, not Github.

jlengrand 7y ago

I used to be a total Gitlab fanboy. I was going to Sid's meetups back in Utrecht 6 years ago, and he's a model of mine to this time. I would have done anything to stay on the platform, for at home and the office and setup gitlab instances in two different offices. Since somewhere last year, I moved back to Github primarily, and I'm sad to hear that my company is likely to make the same choice soon.The only reason is stability, and that's a little sad to me. I really want to love the product, but I need something that just works; not bells and whistles

kungtotte 7y ago
Whenever GitHub is down you see comments saying much the same thing only with the names reversed. The thing that strikes me each time is how fragile everyone's setup is if GitHub/lab is a single point of failure for them... At least gitlab let's you self host, which would let you run backups on offsite hosting meaning zero downtime.
- protomikron 7y ago
  GitHub's uptime is way better than GitLab. I am not really sure gitlab.com even reaches 99% availability (not to mention 99.9%). GitLab may well be focused on providing full-stack dev-services (VCS, build-server, CI and stuff), but in the end they are a hosting company - and for hosting, uptime is one of the most important metrics. EDIT: 99% was exaggerating, but I got so many 50Xs the last days that it was from time to time unusable.
  - tristanperry 7y ago
    I agree that GitHub's uptime is better, but I doubt that Gitlab aren't at 99% availability - that'd mean over 14.4 minutes of downtime each day.
    - toupeira 7y ago
      We have some public Pingdom stats at http://stats.gitlab.com/4932705/history http://stats.gitlab.com/4932705/history, looks like this was the first time we dipped below 99.9% this year (partial outages excluded). But yeah, our response times have been steadily increasing and could definitely be a lot better ;-)

darekkay 7y ago

It is up again. > [Monitoring] GitLab.com is now recovering. We found 2 last DB nodes which had not reverted their change. Apologies for the disruption. [1] [1] https://status.gitlab.com/ https://status.gitlab.com/

joeblau 7y ago

We’re doing this now? For my entire tenure on HN, this site has been a status page for GitHub. Looks like it’s Gitlabs turn.

ifthenelseend 7y ago

You guys should consider switching to GitHub. Much faster, no 500-errors, less downtime.

deleted 7y ago

[deleted]

intellix 7y ago

Not sure if I have anything else to add on top of what people are already saying but the performance of GL is extremely frustrating for the past couple of months. Viewing the diff of a PR can sometimes take around 20 seconds just to load the tab. I think it's about time they dedicated some resources or addressed the public about the efforts they're making to address these issues. It's becoming excruciating. I've used Github every day for years and years and I feel like every page load has been pretty much instant forever.

BlackLotus89 7y ago

It was an iptables problem https://gitlab.com/gitlab-com/gl-infra/production/issues/1421 https://gitlab.com/gitlab-com/gl-infra/production/issues/142... The rollout of new iptables rules blocked database server connections

aabbcc1241 7y ago

Any service on clearnet can go down. p2p alternative like git center (git over zeronet) is more resilient to service provider mul-configuration and censorship git center: http://127.0.0.1:43110/1GitLiXB6t5r8vuU2zC6a8GYj9ME6HMQ4t/ http://127.0.0.1:43110/1GitLiXB6t5r8vuU2zC6a8GYj9ME6HMQ4t/ client: https://zeronet.io/ https://zeronet.io/ proxy: https://zero.acelewis.com/ https://zero.acelewis.com/