7 ms·

GitLab.com Downtime Postmortem

123 points by jobvandervoort 12y ago

sytse 12y ago
GitLab B.V. CEO here, please let us know if you have any questions (you can also leave a comment or suggestion in the doc if you want). This whole real-time postmortem is something we thought of to contribute back after having downtime. Feel free to let us know what you think of it.
- dvirsky 12y ago
  I haven't read it but I like the idea in general. One question: aren't you concerned about accidentally exposing security critical info over the doc?
  - sytse 12y ago
    Glad you like the idea. We're worried about exposing sensitive information. In general we're pretty careful about credentials so that will be OK and our setup is not a secret (a lot is open source and we share the High Availability details with standard subscribers). But it could happen that we share user information such as projects names in the log-files by accident. We think that there are also benefits of working out in the open and thing that the trade-off is worth it. We hope that people let us know if they see anything sensitive in the doc so we can quickly remove it.
    - dvirsky 12y ago
      Thanks, I've now read it. Very nice. I guess having your architecture out in the open is liberating in that sense. So now two specific questions: 1. So why did you have a big read spike? And 2. are you going to share the TODOs as well? :) BTW We're using gitlab internally (at EverythingMe) and are very happy about it, especially the flow of fixes and features implemented.
      - sytse 12y ago
        1. We don't know (and the read spike might be an effect rather than the cause) 2. Yes, most of the things after the hashrockets (=>) currently in the doc are TODO's BTW Awesome to hear that you are happy users of GitLab.
        dvirsky 12y ago
        I bet it's an effect rather than a cause. We had some server incident recently that started with a huge write spike on some machines. Turned out it was the nginx error log and the problem itself was something else.
        sytse 12y ago
        Right now it looks like 1 was caused by an extremely large repo (18 GB) being pushed. Only 0.1% of repos are bigger than 1GB. We're still investigating.
- LeonM 12y ago
  It's only after I saw the "B.V." I realized Gitlab is a Dutch company, cool! :) Just one question, I saw a lot of logs being copy-pasted, aren't you concerned about any security sensitive data leaking out?
  - sytse 12y ago
    Thanks! We are based in the Netherlands but we're mostly a remote company https://about.gitlab.com/2014/07/03/how-gitlab-works-remotely/ https://about.gitlab.com/2014/07/03/how-gitlab-works-remotel... We're worried about sensitive data, see my other answer https://news.ycombinator.com/item?id=8003800 https://news.ycombinator.com/item?id=8003800
- ismail 12y ago
  have you tried HackPad instead of docs?
  - Numberwang 12y ago
    I have yet to understand why HN:ers like HackPad. It is at best almost as good as GDoc:s on some of the features.
    - wpietri 12y ago
      The reason I've seen it used is that it's open source and therefore can be hosted by the entity involved.
      - scott_karana 12y ago
        It is?! Where's the source code? Their GitHub page is rather sparse, and the only hint I've found is this tweet: https://twitter.com/hackpad/status/407651583598813184 https://twitter.com/hackpad/status/407651583598813184
        wpietri 12y ago
        Sorry, I'm thinking of Etherpad: http://etherpad.org/ http://etherpad.org/ The source code of which is here: https://github.com/ether https://github.com/ether Hackpad is apparently a fork of that: https://hackpad.com/Whats-the-difference-between-Hackpad-and-Etherpad-AA3sVQt9rJA https://hackpad.com/Whats-the-difference-between-Hackpad-and...
        scott_karana 12y ago
        Cool, thanks!
pilif 12y ago
I wonder what their drbd is using as its backing store and why it's needed at all. Personally, I honestly wouldn't trust to run a database on top of drbd and I'm unsure whether using a drbd volume as a database directory will even cause the other replica to have correct and usable data in case of an error. Personally, I would use a real disk or maybe LVM as the store for the database in order to reduce points of failures and remove less tested components from the picture. Then of course you need a database slave, but it looks like you had that anyways which, again, leads me to question why drbd was even involved.
- sytse 12y ago
  We use EXT4 on DRBD on LVM on RAID10. We already have DRBD for the git repo's and prefer to reuse our experience for the database. There are other reasons too some of which are articulated in http://wiki.postgresql.org/images/0/07/Ha_postgres.pdf http://wiki.postgresql.org/images/0/07/Ha_postgres.pdf
  - sytse 12y ago
    I gave a presentation yesterday about our HA setup https://docs.google.com/presentation/d/1gIzmp-d5X86jJMQz7Ixs_dzyhC1OTVK0w3uV_Rtc_Us/edit#slide=id.p https://docs.google.com/presentation/d/1gIzmp-d5X86jJMQz7Ixs...
- Rapzid 12y ago
  Why would you not trust a database server on DRBD? It's a very solid product and using it for database replication/failover is a common pattern. Word on the street is that Amazon RDS uses DRBD for multi-az failover.
- caw 12y ago
  I used DRBD before it was cool (before it was added into the kernel), and it works great for highly available directories like home dirs, which was my use case. Even then it was pretty solid, and it's very similar to using a high end NAS product like NetApp or EMC with automatic failover, except you don't have to shell out $100K+++ for it. If you used it for a database then everything committed to disk is immediately available on the partner, which you can spin up instantly with some scripts. And look, your transaction log is exactly where you left it! With a database and built-in replication you could go either way, but there are some applications with persistency that you need to make highly available, and that's another area DRBD would shine.
sytse 12y ago
We're talking a break until 16:00 CEST, many questions are answered already and TODO's are given after the => in the document.
- sytse 12y ago
  We're back again.
  - sytse 12y ago
    And we're done for today, thanks for stopping by everyone.
fsiefken 12y ago
Could this be caused by DRBD not reading the changes from the Postgresql database quickly enough? Or is it some issue with the interaction between the software RAID10, DRBD under high postgresql I/O load? What are the respective versions of the kernel, database and DRBD? Are there disk I/O, network I/O and cpu logs available of the time leading up to the crash?
- sytse 12y ago
  We're not sure. There was a lot of disk locking going on at the time.
felixgallo 12y ago
One important learning from the postmortem: always set your servers up to be in UTC rather than any other time zone. Helps debugging and log correlation and eliminates confusion during incidents.
- sytse 12y ago
  Very true, we'll update the graph server that was misconfigured.
  - Alupis 12y ago
    This is a very open and candid write-up. Very much appreciated by the community. Too many companies try to hide/cover-up their outages, but Gitlab is letting it all hang out, their mess-ups as well as things outside their control. I think that speaks a lot to the character of the company, and shows they really care.
    - sytse 12y ago
      Thanks Alipus, we're trying to be a part of the GitLab community in everything we do. Even if our operations go south we want that to inform others.
- wpietri 12y ago
  Is that the lesson? Or just that everything should be in the same time zone? If the company's staff is all in one time zone, I'm inclined to use that for servers, as otherwise people have to mentally juggle two time zones: local and UTC.
  - res0nat0r 12y ago
    This starts out to be the way most companies think, then before you know it you are big enough to start opening datacenters in other countries, then you have to deal with timestamp conversion.
    - 12y ago
wpietri 12y ago
The bit about the large repositories reminds me tangentially of something I read about Amazon: when tuning for performance, they don't look at, say, the median response time. They look at the 99.9% level. If I recall rightly, their example was a customer searching their old orders. The customers with the slowest response times were their very best customers, and they wanted those people to have at least as good as an experience as the median customer. That definitely change my attitude to performance tests: now whenever I'm picking metrics, I think hard the real-world implications of the levels I'm setting.
- sytse 12y ago
  Very true, and you can be sure we'll pay a bit more attention to the 0.1% biggest repositories from now on.
- res0nat0r 12y ago
  Yup, AWS uses this metric everywhere (TP99). If 1% of your requests take 1000% more time than everything else, then you need to optimize and ensure this doesn't screw everyone. Here is a bit more info: http://codesith.blogspot.com/2012/06/tp99.html http://codesith.blogspot.com/2012/06/tp99.html
jbk 12y ago
Well, this is not back for us: our repositories are corrupted...
- sytse 12y ago
  I'm sorry to hear you are experiencing problems. This is the first report we see of possible corruption. Please contact support@gitlab.com and note the urls and commands that do not work for you. Edit: there another report I just found: https://gitlab.com/gitlab-com/support-forum/issues/2 https://gitlab.com/gitlab-com/support-forum/issues/2
sciurus 12y ago
"6. Production filesystem IS NOT mounted at this point 9. gitlab-ctl start starts the GitLab with the staging data in /var/opt/gitlab on the root filesystem that doesn’t have any production data. At this point logs report that the production db doesn’t exist which is correct because we are not on the production file system. No production data has been touched at this point. The Gitlab web UI is not responding (502 error from nginx)" This is what strikes me as needing addressing. There shouldn't be staging data that is normally hidden by mounting the production filesystem. If the production database isn't there, Postgres should fail to start. The Postgres team is pretty adamant that otherwise bad things can happen; see http://www.postgresql.org/message-id/12168.1312921709@sss.pgh.pa.us http://www.postgresql.org/message-id/12168.1312921709@sss.pg...
- sytse 12y ago
  We fixed part of the root issue by restarting GitLab automatically when we configure a server as master, preventing it from continuing with the staging data. The staging data was created automatically by gitlab-ctl. I agree that this is confusing and created the item: "Why did GitLab start when it shouldn't?" in the postmortem. Thanks for raising this point.

123 points by jobvandervoort 12y ago

sytse 12y ago

GitLab B.V. CEO here, please let us know if you have any questions (you can also leave a comment or suggestion in the doc if you want). This whole real-time postmortem is something we thought of to contribute back after having downtime. Feel free to let us know what you think of it.

dvirsky 12y ago
I haven't read it but I like the idea in general. One question: aren't you concerned about accidentally exposing security critical info over the doc?
- sytse 12y ago
  Glad you like the idea. We're worried about exposing sensitive information. In general we're pretty careful about credentials so that will be OK and our setup is not a secret (a lot is open source and we share the High Availability details with standard subscribers). But it could happen that we share user information such as projects names in the log-files by accident. We think that there are also benefits of working out in the open and thing that the trade-off is worth it. We hope that people let us know if they see anything sensitive in the doc so we can quickly remove it.
  - dvirsky 12y ago
    Thanks, I've now read it. Very nice. I guess having your architecture out in the open is liberating in that sense. So now two specific questions: 1. So why did you have a big read spike? And 2. are you going to share the TODOs as well? :) BTW We're using gitlab internally (at EverythingMe) and are very happy about it, especially the flow of fixes and features implemented.
    - sytse 12y ago
      1. We don't know (and the read spike might be an effect rather than the cause) 2. Yes, most of the things after the hashrockets (=>) currently in the doc are TODO's BTW Awesome to hear that you are happy users of GitLab.
      - dvirsky 12y ago
        I bet it's an effect rather than a cause. We had some server incident recently that started with a huge write spike on some machines. Turned out it was the nginx error log and the problem itself was something else.
      - sytse 12y ago
        Right now it looks like 1 was caused by an extremely large repo (18 GB) being pushed. Only 0.1% of repos are bigger than 1GB. We're still investigating.
LeonM 12y ago
It's only after I saw the "B.V." I realized Gitlab is a Dutch company, cool! :) Just one question, I saw a lot of logs being copy-pasted, aren't you concerned about any security sensitive data leaking out?
- sytse 12y ago
  Thanks! We are based in the Netherlands but we're mostly a remote company https://about.gitlab.com/2014/07/03/how-gitlab-works-remotely/ https://about.gitlab.com/2014/07/03/how-gitlab-works-remotel... We're worried about sensitive data, see my other answer https://news.ycombinator.com/item?id=8003800 https://news.ycombinator.com/item?id=8003800
ismail 12y ago
have you tried HackPad instead of docs?
- Numberwang 12y ago
  I have yet to understand why HN:ers like HackPad. It is at best almost as good as GDoc:s on some of the features.
  - wpietri 12y ago
    The reason I've seen it used is that it's open source and therefore can be hosted by the entity involved.
    - scott_karana 12y ago
      It is?! Where's the source code? Their GitHub page is rather sparse, and the only hint I've found is this tweet: https://twitter.com/hackpad/status/407651583598813184 https://twitter.com/hackpad/status/407651583598813184
      - wpietri 12y ago
        Sorry, I'm thinking of Etherpad: http://etherpad.org/ http://etherpad.org/ The source code of which is here: https://github.com/ether https://github.com/ether Hackpad is apparently a fork of that: https://hackpad.com/Whats-the-difference-between-Hackpad-and-Etherpad-AA3sVQt9rJA https://hackpad.com/Whats-the-difference-between-Hackpad-and...
        scott_karana 12y ago
        Cool, thanks!

pilif 12y ago

I wonder what their drbd is using as its backing store and why it's needed at all. Personally, I honestly wouldn't trust to run a database on top of drbd and I'm unsure whether using a drbd volume as a database directory will even cause the other replica to have correct and usable data in case of an error. Personally, I would use a real disk or maybe LVM as the store for the database in order to reduce points of failures and remove less tested components from the picture. Then of course you need a database slave, but it looks like you had that anyways which, again, leads me to question why drbd was even involved.

sytse 12y ago
We use EXT4 on DRBD on LVM on RAID10. We already have DRBD for the git repo's and prefer to reuse our experience for the database. There are other reasons too some of which are articulated in http://wiki.postgresql.org/images/0/07/Ha_postgres.pdf http://wiki.postgresql.org/images/0/07/Ha_postgres.pdf
- sytse 12y ago
  I gave a presentation yesterday about our HA setup https://docs.google.com/presentation/d/1gIzmp-d5X86jJMQz7Ixs_dzyhC1OTVK0w3uV_Rtc_Us/edit#slide=id.p https://docs.google.com/presentation/d/1gIzmp-d5X86jJMQz7Ixs...
Rapzid 12y ago
Why would you not trust a database server on DRBD? It's a very solid product and using it for database replication/failover is a common pattern. Word on the street is that Amazon RDS uses DRBD for multi-az failover.
caw 12y ago
I used DRBD before it was cool (before it was added into the kernel), and it works great for highly available directories like home dirs, which was my use case. Even then it was pretty solid, and it's very similar to using a high end NAS product like NetApp or EMC with automatic failover, except you don't have to shell out $100K+++ for it. If you used it for a database then everything committed to disk is immediately available on the partner, which you can spin up instantly with some scripts. And look, your transaction log is exactly where you left it! With a database and built-in replication you could go either way, but there are some applications with persistency that you need to make highly available, and that's another area DRBD would shine.

sytse 12y ago

We're talking a break until 16:00 CEST, many questions are answered already and TODO's are given after the => in the document.

sytse 12y ago
We're back again.
- sytse 12y ago
  And we're done for today, thanks for stopping by everyone.

fsiefken 12y ago

Could this be caused by DRBD not reading the changes from the Postgresql database quickly enough? Or is it some issue with the interaction between the software RAID10, DRBD under high postgresql I/O load? What are the respective versions of the kernel, database and DRBD? Are there disk I/O, network I/O and cpu logs available of the time leading up to the crash?

sytse 12y ago
We're not sure. There was a lot of disk locking going on at the time.

felixgallo 12y ago

One important learning from the postmortem: always set your servers up to be in UTC rather than any other time zone. Helps debugging and log correlation and eliminates confusion during incidents.

sytse 12y ago
Very true, we'll update the graph server that was misconfigured.
- Alupis 12y ago
  This is a very open and candid write-up. Very much appreciated by the community. Too many companies try to hide/cover-up their outages, but Gitlab is letting it all hang out, their mess-ups as well as things outside their control. I think that speaks a lot to the character of the company, and shows they really care.
  - sytse 12y ago
    Thanks Alipus, we're trying to be a part of the GitLab community in everything we do. Even if our operations go south we want that to inform others.
wpietri 12y ago
Is that the lesson? Or just that everything should be in the same time zone? If the company's staff is all in one time zone, I'm inclined to use that for servers, as otherwise people have to mentally juggle two time zones: local and UTC.
- res0nat0r 12y ago
  This starts out to be the way most companies think, then before you know it you are big enough to start opening datacenters in other countries, then you have to deal with timestamp conversion.
  - 12y ago

wpietri 12y ago

The bit about the large repositories reminds me tangentially of something I read about Amazon: when tuning for performance, they don't look at, say, the median response time. They look at the 99.9% level. If I recall rightly, their example was a customer searching their old orders. The customers with the slowest response times were their very best customers, and they wanted those people to have at least as good as an experience as the median customer. That definitely change my attitude to performance tests: now whenever I'm picking metrics, I think hard the real-world implications of the levels I'm setting.

sytse 12y ago
Very true, and you can be sure we'll pay a bit more attention to the 0.1% biggest repositories from now on.
res0nat0r 12y ago
Yup, AWS uses this metric everywhere (TP99). If 1% of your requests take 1000% more time than everything else, then you need to optimize and ensure this doesn't screw everyone. Here is a bit more info: http://codesith.blogspot.com/2012/06/tp99.html http://codesith.blogspot.com/2012/06/tp99.html

jbk 12y ago

Well, this is not back for us: our repositories are corrupted...

sytse 12y ago
I'm sorry to hear you are experiencing problems. This is the first report we see of possible corruption. Please contact support@gitlab.com and note the urls and commands that do not work for you. Edit: there another report I just found: https://gitlab.com/gitlab-com/support-forum/issues/2 https://gitlab.com/gitlab-com/support-forum/issues/2

sciurus 12y ago

"6. Production filesystem IS NOT mounted at this point 9. gitlab-ctl start starts the GitLab with the staging data in /var/opt/gitlab on the root filesystem that doesn’t have any production data. At this point logs report that the production db doesn’t exist which is correct because we are not on the production file system. No production data has been touched at this point. The Gitlab web UI is not responding (502 error from nginx)" This is what strikes me as needing addressing. There shouldn't be staging data that is normally hidden by mounting the production filesystem. If the production database isn't there, Postgres should fail to start. The Postgres team is pretty adamant that otherwise bad things can happen; see http://www.postgresql.org/message-id/12168.1312921709@sss.pgh.pa.us http://www.postgresql.org/message-id/12168.1312921709@sss.pg...

sytse 12y ago
We fixed part of the root issue by restarting GitLab automatically when we configure a server as master, preventing it from continuing with the staging data. The staging data was created automatically by gitlab-ctl. I agree that this is confusing and created the item: "Why did GitLab start when it shouldn't?" in the postmortem. Thanks for raising this point.