9 ms·

Gitlab 500 error

120 points by sbhn 8y ago

txmjs 8y ago
Had quite some trouble pushing this afternoon. Hope it gets resolved soon.
deleted 8y ago
[deleted]
jedberg 8y ago
I gotta say I love but am surprised by the public google doc of the outages. If I worked there I would personally advocate for not doing that, only because I feel like it would slow down resolution if I have to constantly be worrying about not putting "secret" things into the doc. Unless they have someone who's job is to take their real chat and transcribe it, in which case, that's great because that person is useful for a postmortem as well. Speaking of postmortems, reading through the Google Doc of their outage timeline, my first postmortem question would be why is the website dependent on the database being up? Shouldn't there at least be a fallback that says "We're offline right now sorry" when the database is unavailable? Or in the name of extreme transparency, have it frame or show a link to the google doc of the outage? My second question would be how often do we test failures of a read slave, of the master, and of pgbouncer? Do we at least have a practiced procedure for manual failover if not an automatic failover? My third question would be what sort of alerts do we have, if any, on replication lag? It's been a while since I've had to run a major outage, this was an interesting exercise for me!
- otterlicious 8y ago
  why is the website dependent on the database being up? Shouldn't there at least be a fallback that says "We're offline right now sorry" when the database is unavailable? According to the outage Google Doc it's supposed to do this but it stopped working a little while ago: 16h33 | Yorick: Why have the 502s on GitLab.com switched to 500s? 16h46 | Slight concern that 500s have not yet gone away, but let’s focus on standing up postgres-04 first...
  - jedberg 8y ago
    Right exactly, my first question would be why did this happen, because it could lead to some interesting insight about an unknown dependency.
- YorickPeterse 8y ago
  > Speaking of postmortems, reading through the Google Doc of their outage timeline, my first postmortem question would be why is the website dependent on the database being up? You can't do anything with GitLab without the database being available. > Shouldn't there at least be a fallback that says "We're offline right now sorry" when the database is unavailable? Or in the name of extreme transparency, have it frame or show a link to the google doc of the outage? We have a deploy page, but it requires us to explicitly enable it, which can be easy to forget. I do think adding a link to our Twitter status to the error pages could be useful, or maybe a widget of some kind. > My second question would be how often do we test failures of a read slave, of the master, and of pgbouncer? In the past we tested certain scenarios whenever necessary. For example, when introducing new parts of the database load balancer we'd test what happens if we terminate X or Y. However, we do not yet do periodic chaos monkey style testing, which has been on our todo list for quite a while. > Do we at least have a practiced procedure for manual failover if not an automatic failover? Failovers are actively worked on (https://gitlab.com/gitlab-com/database/issues/91 https://gitlab.com/gitlab-com/database/issues/91, https://gitlab.com/gitlab-com/database/issues/86 https://gitlab.com/gitlab-com/database/issues/86, https://gitlab.com/gitlab-com/database/issues/40 https://gitlab.com/gitlab-com/database/issues/40). Right now our failover procedure is still quite painful, and it's easy to make mistakes. We held off using this approach for our maintenance because of these reasons, and also because a failover usually results in 2-3 hours of slow loading times. Ironically we ended up with just that. > My third question would be what sort of alerts do we have, if any, on replication lag? Plenty: https://gitlab.com/gitlab-com/runbooks/blob/810824765ec455344bdae35300a352be8c147dbb/alerts/postgresqls.yml https://gitlab.com/gitlab-com/runbooks/blob/810824765ec45534... In this particular instance they go to a Slack channel called "#database", and this channel has been on fire with alerts since the problem started.
  - jedberg 8y ago
    Wow sweet, I didn't actually expect a response! Nice work. > We have a deploy page, but it requires us to explicitly enable it, which can be easy to forget. I do think adding a link to our Twitter status to the error pages could be useful, or maybe a widget of some kind. Or make it automatic when the database isn't available? > In this particular instance they go to a Slack channel called "#database", and this channel has been on fire with alerts since the problem started. Good to know. My next question would be "were there any signs of impending doom before the event happened that could have warned us, and do we have alerts on that?"
- Rjevski 8y ago
  > why is the website dependent on the database being up? I believe they use "GitLab Pages" to host their main, public-facing website. If Gitlab.com (the app I mean) is down, then so is their website.
- graystevens 8y ago
  I'm happy to be corrected here, but in response to: I have to constantly be worrying about not putting "secret" things into the doc. Unless they have someone who's job is to take their real chat and transcribe it, in which case, that's great because that person is useful for a postmortem as well. I think they did only have one person updating the document, from what I could as I was following the live updates - only very occasionally did I see someone other than 'Andrew' update the document.
philip1209 8y ago
Time for another outage tshirt?
deleted 8y ago
[deleted]
GrayShade 8y ago
I always found it disheartening to read their changelogs and see how every release (I've looked at) fixes some "N+1 SELECTs" problem [1]. This will be an unpopular opinion here, but I feel we've (collectively) failed. Between ORMs and the dynamic language du jour, we have become so far removed from how computers work that most software development is a waste of electricity. We're using tens of servers and complex architectures for simple sites, just because we think that's the only way to scale them [2]. And our computers are slower than they ever were [3] [4]. I only gave a couple of examples, but look around, you can find them everywhere. Everything is awful. [1] The issue tracker is now down, but you can try it when it comes back: https://about.gitlab.com/releases/ https://about.gitlab.com/releases/. [2] https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos... [3] On older computers GNOME can't even keep up to update the mouse cursor. [4] I'm honestly not sure that Windows 10 boots faster from an SSD than XP from an HDD.
- overcast 8y ago
  SQL optimization is a dying art. Amazes me when I dig through software we've purchased to find abominations that I have to refactor. I've sent emails to developers with the fixes for shit we've paid for.
  - pc86 8y ago
    Ah yes I love receiving Smug Reports[0] in my inbox. A 30,000' fix provided by someone with a 10' view of the code base and no business understanding outside their own organization. [0] https://blog.codinghorror.com/new-programming-jargon/ https://blog.codinghorror.com/new-programming-jargon/ Number 4
    - slacka 8y ago
      Could it be that you have the 30,000' view, while they had a ground view and climbed up a mountain to fix the view for all those in their town? Out of frustration I've broke out IDA Pro enough times to scratch my own itch that it's turned me to into a near Stallman-esque fan of open source software. When you work for small companies, bug reports on proprietary software issue trackers often get neglected.
sergiotapia 8y ago
Please send some love to the engineer responsible. Been on the causing end of a 500 error, even though I fixed it in 10 minutes, it's never fun!
MrFurious 8y ago
GitLab with problems?, again?.
cyphar 8y ago
It's a little bit concerning to note that the last GitLab outage was also caused by Postgres replication issues, which eventually culminated in some pretty bad data loss and a realisation that the most recent backups they had weren't usable. Maybe they should reach out to someone from Postgres to help figure out why their replication appears to have some sort of systemic issue...
phonon 8y ago
Should have used AWS Aurora Postgres
deleted 8y ago
[deleted]
LoneWolf 8y ago
Curious StackOverflow also has a 500, could they be related?
nAwYz 8y ago
Google doc with some information: https://docs.google.com/document/d/1WmOMKq63Rap2wfQ-yy7sHyocFvmbJgIt1UIj2xEXEco/preview https://docs.google.com/document/d/1WmOMKq63Rap2wfQ-yy7sHyoc...
dpcx 8y ago
Gitlab working document https://docs.google.com/document/d/1WmOMKq63Rap2wfQ-yy7sHyocFvmbJgIt1UIj2xEXEco/preview https://docs.google.com/document/d/1WmOMKq63Rap2wfQ-yy7sHyoc...
- haskellandchill 8y ago
  I’ve been through similar scenarios and they went down like this. Scrambling, making notes about things that should have been done while putting full understanding beside to fight the fire. I used to think there was a better way, now I’m convinced there is but it would take too much time across the stack and still go down anyways because you can’t control everything all the time forever.
- Moter8 8y ago
  Even that took 10s to load for me, haha.
- sbhn 8y ago
  alas poor yorick
- rhacker 8y ago
  Super transparent at least
- lunixbochs 8y ago
  > Bash scripts did not have set -x so continued to run even after the stop command had failed. I hope that's a typo in transcription and they actually used `set -e` to fix (and maybe `set -o pipefail`), `set -x` does something entirely different.
  - YorickPeterse 8y ago
    Yes that's a typo, it should have been `set -e`
- batat 8y ago
  18h45 Yorick is doing the vacuuming. Vacuuming is not doing it’s thing. Why…. 18h48 Still not vacuuming. 18h50 Yorick: still not sure why it’s not vacuuming. We need a roomba. I love them :)
dustfinger 8y ago
Yes, I confirm the same. They are both returning server 500 errors. That is quite a coincidence.
dustfinger 8y ago
> https://outage.report/stackoverflow https://outage.report/stackoverflow
dustfinger 8y ago
stackoverflow is up now. gitlab still down.
dustfinger 8y ago
stackoverflow was actually returning a 503, not a 500.
yathomasi1 8y ago
WOW back at the right moment.
winslow 8y ago
I'm also seeing a 5xx Server error on instagram. Edit: ~15 mins later Instagram seems to be back up. Edit #2: Seems to be back to 5xx server error ~1hr now.
- LusoTycoon 8y ago
  still seeing it
dubcanada 8y ago
They do database maintenance at 11am EST on a Tuesday? Seems like a terrible time to do database maintenance?
sbhn 8y ago
Excellent, my fork-me-on-gitlab project is viewable again. https://gitlab.com/seanwasere/fork-me-on-gitlab https://gitlab.com/seanwasere/fork-me-on-gitlab
sbuttgereit 8y ago
I have growing concerns over GitLab. I appreciate the openness and forthrightness of their handling of these sorts of incidents and in regard to other issues, but there comes a point where none of that matters if the product itself is inherently unreliable. This post isn't about any one incident, including this one, but an ongoing trend that I don't get the sense is improving. To be completely fair, I'm on the free plans and have no rightful expectation of any sort of performance level. Having said that, I have considered moving to paid tiers, and I do advise others in regard to these sorts of services (whether cloud based or on-premise). Every time I see a planned maintenance, of "under 1m", I simply realize that I shouldn't plan anything important for GitLab that day. I can't see that this would be any different with GitLab.com paid plans and I have to imagine there is something inherently difficult in managing the software if the developer of that software has issues with common maintenance; this colors my impressions of what this might be like in-house. It seems a beast, I get that there are scaling issues that are different between something like gitlab.com and GitLab on-premise, but history of these things is coloring my impression. At some point moving fast and breaking things needs to give way to the pursuit of enough stability that users aren't overly concerned about whether or not the tools they depend on, or more importantly the data they're storing, are regularly at risk. I don't want to buy services/products from a company that is just trustworthy and open when things go wrong. I want to buy services/products from companies that only need to prove those qualities with some rarity.
- jazoom 8y ago
  It always seems to be either their storage abstraction (Ceph?) or Postgres. I've never used Postgres but from what I see it's not really designed for enormous scale (it's a very old project). Perhaps they would be well served to see if CockroachDB gives them more stability. I've started using it at a small scale and the clustering aspect seems legit.
  - sbuttgereit 8y ago
    I work with PostgreSQL quite a lot and I don't think this is the case. Yes, PostgreSQL is a mature project. But that has little or no bearing by itself on what degree it can scale; Linux is just about as old, yet it still drives most of the infrastructure we're talking about. In context, PostgreSQL has been under continuous development since inception and no more so than in the past decade or so with substantial community and corporate backing. To conflate project age with current robustness is to indulge in fallacious thinking (in this case "cum hoc ergo propter hoc", if I'm not mistaken). The project has advanced with the times. There are good examples of PostgreSQL running at scale, including at companies operating at scale such as Instagram, Skype (up to at least the point of the Microsoft acquisition), Pandora, as well as others. Most of these, I'd bet, are/were using PostgreSQL at scales substantially larger than that faced by GitLab. Relational databases require good management and good design to function properly. There were people who specialized in this called Data Base Administrators (at least the management piece anyway). My feeling (perhaps unjustified) is in start-upish environments there is a tendency to minimize DBA expertise in favor of having more conventional developers that can "get the database to work"... which is a different standard than "getting the database to perform". That's mostly not my world, so I may be jumping to conclusions (I'm an enterprise systems guy on most days). I think GitLab probably has a complex software product and lacks the correct expertise in infrastructure (including, yes, DBAs) for their SaaS offering. I'm reading tea leaves, but that would be a perfect storm which would produce exactly what we see no matter how robust any one piece of this puzzle might be.
thinkindie 8y ago
They probably need stronger arguments than "Micro$oft is really bad" if they want to have an edge against Github. Professional tool solidity is way more important than the company process transparence.
geekrax 8y ago
So now that there's no outage, this post has no significance and no way to see what was happening during the outage. Does anyone have a cached version of the page or a screenshot for some context?
KaiserPro 8y ago
The hosted product, whilst far more fully featured than github, is horridly unstable. Its got to the point where I am seriously considering moving our repos (~250) to a self hosted CE version. Looking at the outage reports, it doesn't fill me with confidence. A lack of redundancy, no systematic testing of backups, a seeming lack of QA. Gitlab has an outage of average once every two weeks, compared to github's once every 6 _months_ (look at the status pages)

Gitlab 500 error

120 points by sbhn 8y ago

txmjs 8y ago

Had quite some trouble pushing this afternoon. Hope it gets resolved soon.

deleted 8y ago

[deleted]

jedberg 8y ago

I gotta say I love but am surprised by the public google doc of the outages. If I worked there I would personally advocate for not doing that, only because I feel like it would slow down resolution if I have to constantly be worrying about not putting "secret" things into the doc. Unless they have someone who's job is to take their real chat and transcribe it, in which case, that's great because that person is useful for a postmortem as well. Speaking of postmortems, reading through the Google Doc of their outage timeline, my first postmortem question would be why is the website dependent on the database being up? Shouldn't there at least be a fallback that says "We're offline right now sorry" when the database is unavailable? Or in the name of extreme transparency, have it frame or show a link to the google doc of the outage? My second question would be how often do we test failures of a read slave, of the master, and of pgbouncer? Do we at least have a practiced procedure for manual failover if not an automatic failover? My third question would be what sort of alerts do we have, if any, on replication lag? It's been a while since I've had to run a major outage, this was an interesting exercise for me!

otterlicious 8y ago
why is the website dependent on the database being up? Shouldn't there at least be a fallback that says "We're offline right now sorry" when the database is unavailable? According to the outage Google Doc it's supposed to do this but it stopped working a little while ago: 16h33 | Yorick: Why have the 502s on GitLab.com switched to 500s? 16h46 | Slight concern that 500s have not yet gone away, but let’s focus on standing up postgres-04 first...
- jedberg 8y ago
  Right exactly, my first question would be why did this happen, because it could lead to some interesting insight about an unknown dependency.
YorickPeterse 8y ago
> Speaking of postmortems, reading through the Google Doc of their outage timeline, my first postmortem question would be why is the website dependent on the database being up? You can't do anything with GitLab without the database being available. > Shouldn't there at least be a fallback that says "We're offline right now sorry" when the database is unavailable? Or in the name of extreme transparency, have it frame or show a link to the google doc of the outage? We have a deploy page, but it requires us to explicitly enable it, which can be easy to forget. I do think adding a link to our Twitter status to the error pages could be useful, or maybe a widget of some kind. > My second question would be how often do we test failures of a read slave, of the master, and of pgbouncer? In the past we tested certain scenarios whenever necessary. For example, when introducing new parts of the database load balancer we'd test what happens if we terminate X or Y. However, we do not yet do periodic chaos monkey style testing, which has been on our todo list for quite a while. > Do we at least have a practiced procedure for manual failover if not an automatic failover? Failovers are actively worked on (https://gitlab.com/gitlab-com/database/issues/91 https://gitlab.com/gitlab-com/database/issues/91, https://gitlab.com/gitlab-com/database/issues/86 https://gitlab.com/gitlab-com/database/issues/86, https://gitlab.com/gitlab-com/database/issues/40 https://gitlab.com/gitlab-com/database/issues/40). Right now our failover procedure is still quite painful, and it's easy to make mistakes. We held off using this approach for our maintenance because of these reasons, and also because a failover usually results in 2-3 hours of slow loading times. Ironically we ended up with just that. > My third question would be what sort of alerts do we have, if any, on replication lag? Plenty: https://gitlab.com/gitlab-com/runbooks/blob/810824765ec455344bdae35300a352be8c147dbb/alerts/postgresqls.yml https://gitlab.com/gitlab-com/runbooks/blob/810824765ec45534... In this particular instance they go to a Slack channel called "#database", and this channel has been on fire with alerts since the problem started.
- jedberg 8y ago
  Wow sweet, I didn't actually expect a response! Nice work. > We have a deploy page, but it requires us to explicitly enable it, which can be easy to forget. I do think adding a link to our Twitter status to the error pages could be useful, or maybe a widget of some kind. Or make it automatic when the database isn't available? > In this particular instance they go to a Slack channel called "#database", and this channel has been on fire with alerts since the problem started. Good to know. My next question would be "were there any signs of impending doom before the event happened that could have warned us, and do we have alerts on that?"
Rjevski 8y ago
> why is the website dependent on the database being up? I believe they use "GitLab Pages" to host their main, public-facing website. If Gitlab.com (the app I mean) is down, then so is their website.
graystevens 8y ago
I'm happy to be corrected here, but in response to: I have to constantly be worrying about not putting "secret" things into the doc. Unless they have someone who's job is to take their real chat and transcribe it, in which case, that's great because that person is useful for a postmortem as well. I think they did only have one person updating the document, from what I could as I was following the live updates - only very occasionally did I see someone other than 'Andrew' update the document.

philip1209 8y ago

Time for another outage tshirt?

deleted 8y ago

[deleted]

GrayShade 8y ago

I always found it disheartening to read their changelogs and see how every release (I've looked at) fixes some "N+1 SELECTs" problem [1]. This will be an unpopular opinion here, but I feel we've (collectively) failed. Between ORMs and the dynamic language du jour, we have become so far removed from how computers work that most software development is a waste of electricity. We're using tens of servers and complex architectures for simple sites, just because we think that's the only way to scale them [2]. And our computers are slower than they ever were [3] [4]. I only gave a couple of examples, but look around, you can find them everywhere. Everything is awful. [1] The issue tracker is now down, but you can try it when it comes back: https://about.gitlab.com/releases/ https://about.gitlab.com/releases/. [2] https://www.usenix.org/system/files/conference/hotos15/hotos15-paper-mcsherry.pdf https://www.usenix.org/system/files/conference/hotos15/hotos... [3] On older computers GNOME can't even keep up to update the mouse cursor. [4] I'm honestly not sure that Windows 10 boots faster from an SSD than XP from an HDD.

overcast 8y ago
SQL optimization is a dying art. Amazes me when I dig through software we've purchased to find abominations that I have to refactor. I've sent emails to developers with the fixes for shit we've paid for.
- pc86 8y ago
  Ah yes I love receiving Smug Reports[0] in my inbox. A 30,000' fix provided by someone with a 10' view of the code base and no business understanding outside their own organization. [0] https://blog.codinghorror.com/new-programming-jargon/ https://blog.codinghorror.com/new-programming-jargon/ Number 4
  - slacka 8y ago
    Could it be that you have the 30,000' view, while they had a ground view and climbed up a mountain to fix the view for all those in their town? Out of frustration I've broke out IDA Pro enough times to scratch my own itch that it's turned me to into a near Stallman-esque fan of open source software. When you work for small companies, bug reports on proprietary software issue trackers often get neglected.

sergiotapia 8y ago

Please send some love to the engineer responsible. Been on the causing end of a 500 error, even though I fixed it in 10 minutes, it's never fun!

MrFurious 8y ago

GitLab with problems?, again?.

cyphar 8y ago

It's a little bit concerning to note that the last GitLab outage was also caused by Postgres replication issues, which eventually culminated in some pretty bad data loss and a realisation that the most recent backups they had weren't usable. Maybe they should reach out to someone from Postgres to help figure out why their replication appears to have some sort of systemic issue...

phonon 8y ago

Should have used AWS Aurora Postgres

deleted 8y ago

[deleted]

LoneWolf 8y ago

Curious StackOverflow also has a 500, could they be related?

nAwYz 8y ago

Google doc with some information: https://docs.google.com/document/d/1WmOMKq63Rap2wfQ-yy7sHyocFvmbJgIt1UIj2xEXEco/preview https://docs.google.com/document/d/1WmOMKq63Rap2wfQ-yy7sHyoc...

dpcx 8y ago

Gitlab working document https://docs.google.com/document/d/1WmOMKq63Rap2wfQ-yy7sHyocFvmbJgIt1UIj2xEXEco/preview https://docs.google.com/document/d/1WmOMKq63Rap2wfQ-yy7sHyoc...

haskellandchill 8y ago
I’ve been through similar scenarios and they went down like this. Scrambling, making notes about things that should have been done while putting full understanding beside to fight the fire. I used to think there was a better way, now I’m convinced there is but it would take too much time across the stack and still go down anyways because you can’t control everything all the time forever.
Moter8 8y ago
Even that took 10s to load for me, haha.
sbhn 8y ago
alas poor yorick
rhacker 8y ago
Super transparent at least
lunixbochs 8y ago
> Bash scripts did not have set -x so continued to run even after the stop command had failed. I hope that's a typo in transcription and they actually used `set -e` to fix (and maybe `set -o pipefail`), `set -x` does something entirely different.
- YorickPeterse 8y ago
  Yes that's a typo, it should have been `set -e`
batat 8y ago
18h45 Yorick is doing the vacuuming. Vacuuming is not doing it’s thing. Why…. 18h48 Still not vacuuming. 18h50 Yorick: still not sure why it’s not vacuuming. We need a roomba. I love them :)

dustfinger 8y ago

Yes, I confirm the same. They are both returning server 500 errors. That is quite a coincidence.

dustfinger 8y ago

> https://outage.report/stackoverflow https://outage.report/stackoverflow

dustfinger 8y ago

stackoverflow is up now. gitlab still down.

dustfinger 8y ago

stackoverflow was actually returning a 503, not a 500.

yathomasi1 8y ago

WOW back at the right moment.

winslow 8y ago

I'm also seeing a 5xx Server error on instagram. Edit: ~15 mins later Instagram seems to be back up. Edit #2: Seems to be back to 5xx server error ~1hr now.

LusoTycoon 8y ago
still seeing it

dubcanada 8y ago

They do database maintenance at 11am EST on a Tuesday? Seems like a terrible time to do database maintenance?

sbhn 8y ago

Excellent, my fork-me-on-gitlab project is viewable again. https://gitlab.com/seanwasere/fork-me-on-gitlab https://gitlab.com/seanwasere/fork-me-on-gitlab

sbuttgereit 8y ago

I have growing concerns over GitLab. I appreciate the openness and forthrightness of their handling of these sorts of incidents and in regard to other issues, but there comes a point where none of that matters if the product itself is inherently unreliable. This post isn't about any one incident, including this one, but an ongoing trend that I don't get the sense is improving. To be completely fair, I'm on the free plans and have no rightful expectation of any sort of performance level. Having said that, I have considered moving to paid tiers, and I do advise others in regard to these sorts of services (whether cloud based or on-premise). Every time I see a planned maintenance, of "under 1m", I simply realize that I shouldn't plan anything important for GitLab that day. I can't see that this would be any different with GitLab.com paid plans and I have to imagine there is something inherently difficult in managing the software if the developer of that software has issues with common maintenance; this colors my impressions of what this might be like in-house. It seems a beast, I get that there are scaling issues that are different between something like gitlab.com and GitLab on-premise, but history of these things is coloring my impression. At some point moving fast and breaking things needs to give way to the pursuit of enough stability that users aren't overly concerned about whether or not the tools they depend on, or more importantly the data they're storing, are regularly at risk. I don't want to buy services/products from a company that is just trustworthy and open when things go wrong. I want to buy services/products from companies that only need to prove those qualities with some rarity.

jazoom 8y ago
It always seems to be either their storage abstraction (Ceph?) or Postgres. I've never used Postgres but from what I see it's not really designed for enormous scale (it's a very old project). Perhaps they would be well served to see if CockroachDB gives them more stability. I've started using it at a small scale and the clustering aspect seems legit.
- sbuttgereit 8y ago
  I work with PostgreSQL quite a lot and I don't think this is the case. Yes, PostgreSQL is a mature project. But that has little or no bearing by itself on what degree it can scale; Linux is just about as old, yet it still drives most of the infrastructure we're talking about. In context, PostgreSQL has been under continuous development since inception and no more so than in the past decade or so with substantial community and corporate backing. To conflate project age with current robustness is to indulge in fallacious thinking (in this case "cum hoc ergo propter hoc", if I'm not mistaken). The project has advanced with the times. There are good examples of PostgreSQL running at scale, including at companies operating at scale such as Instagram, Skype (up to at least the point of the Microsoft acquisition), Pandora, as well as others. Most of these, I'd bet, are/were using PostgreSQL at scales substantially larger than that faced by GitLab. Relational databases require good management and good design to function properly. There were people who specialized in this called Data Base Administrators (at least the management piece anyway). My feeling (perhaps unjustified) is in start-upish environments there is a tendency to minimize DBA expertise in favor of having more conventional developers that can "get the database to work"... which is a different standard than "getting the database to perform". That's mostly not my world, so I may be jumping to conclusions (I'm an enterprise systems guy on most days). I think GitLab probably has a complex software product and lacks the correct expertise in infrastructure (including, yes, DBAs) for their SaaS offering. I'm reading tea leaves, but that would be a perfect storm which would produce exactly what we see no matter how robust any one piece of this puzzle might be.

thinkindie 8y ago

They probably need stronger arguments than "Micro$oft is really bad" if they want to have an edge against Github. Professional tool solidity is way more important than the company process transparence.

geekrax 8y ago

So now that there's no outage, this post has no significance and no way to see what was happening during the outage. Does anyone have a cached version of the page or a screenshot for some context?

KaiserPro 8y ago

The hosted product, whilst far more fully featured than github, is horridly unstable. Its got to the point where I am seriously considering moving our repos (~250) to a self hosted CE version. Looking at the outage reports, it doesn't fill me with confidence. A lack of redundancy, no systematic testing of backups, a seeming lack of QA. Gitlab has an outage of average once every two weeks, compared to github's once every 6 _months_ (look at the status pages)