19 ms·

Postmortem of database outage of January 31

377 points by mbrain 10y ago

greenrd 10y ago
GitHub also lost a bunch of PRs and issues sitewide early in their history. They claimed to have restored all the PRs from backup, but I was pretty sure I had opened a PR and it never came back. I emailed support and they basically told me tough luck.
oli5679 10y ago
I found this entertaining, even if they did later admit that it was a hoax: http://serverfault.com/questions/587102/monday-morning-mistake-sudo-rm-rf-no-preserve-root http://serverfault.com/questions/587102/monday-morning-mista...
gr2020 10y ago
Reading this, the thing that stuck out to me was how remarkably lucky they were to have the two snapshots. The one from 6 hours earlier was there seemingly by chance, as an engineer had created it for unrelated reasons. And for both the 6- and 24-hour snapshots, it seems just lucky that neither had any breaking changes made to them by pre-production code (they _were_ dev/staging snapshots, after all). I'm glad it all worked out in the end!
- sytse 10y ago
  We too are glad we had those snapshots. And while it was the worst thing that ever happend at GitLab it is humbling to know that it could have been worse.
  - animex 10y ago
    What do you think would have happened if you had total data loss/failure?
    - sb8244 10y ago
      I don't work for gitlab. Isn't that a put you out of business moment? I guess there is enterprise installations but the main website would be toast. I guess there is the chance there were still older backups taken intentionally or by chance that would have been worse but recoverable from
      - gtaylor 10y ago
        Most of their income probably comes from customers that run their own GitLab Enterprises installs. This would have really sucked for all of their non paying users, though.
samat 10y ago
Am I missing something or didn't they mention 'test recovery, not backups'?
- sytse 10y ago
  Two of the issues linked from the article deal with testing the backups: - Automated testing of recovering PostgreSQL database backups https://gitlab.com/gitlab-com/infrastructure/issues/1102 https://gitlab.com/gitlab-com/infrastructure/issues/1102 - Build Streaming Database Restore https://gitlab.com/gitlab-com/infrastructure/issues/1152 https://gitlab.com/gitlab-com/infrastructure/issues/1152
nierman 10y ago
yes, wal archiving would have helped (archive_command = rsync standby ...), but it's also very easy in postgres 9.4+ to add a replication slot on the master so that wal is kept until it is no longer needed by the standby. simply reference the slot in the standby's recovery.conf file. definitely monitor your replication lag--or at least disk usage on the master--with this approach (in case wal starts piling up there).
isoos 10y ago
sytse and GitLab folks: thank you for the transparency.
- sytse 10y ago
  You're welcome. Thanks for all the kind responses we received https://twitter.com/i/moments/826818668948549632 https://twitter.com/i/moments/826818668948549632
meowface 10y ago
>Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed. I could feel the sweat drops just from reading this. I'd bet every one of us has experienced the panicked Ctrl+C of Death at some point or another.
- a_t48 10y ago
  Misread that as "The engineer was terminated" at first. Poor guy.
  - haldean 10y ago
    Potential outage prevention plan: put an alias on all production machines that emails HR to schedule a disciplinary meeting every time you run `rm -rf`.
- rectang 10y ago
  For me, it was when I meant to execute this... rm -rf ~/foo ... but executed this instead: rm -rf ~ /foo
  - BinaryIdiot 10y ago
    I did the Windows equivalent once a long time ago (I think it's deltree?) and I did it on a university computer system. It cleared out a TON of files and the computer itself pretty much stopped working. I had to hard turn it off. Fortunately the University was using some tool that can re-image a computer each time it boots before hitting Windows so starting it back up and all the deleted system and application files were back.
    - overcast 10y ago
      Probably Ghost server, that's why schools do this.
grhmc 10y ago
Thank you for redacting who the engineer was. Great write-up. Thank you!
- sytse 10y ago
  You're very welcome. By the way the engineer was cool with being named, but we think this is good practice for future postmortems. I do hope it doesn't come soon.
jsperson 10y ago
>An ideal environment is one in which you can make mistakes but easily and quickly recover from them with minimal to no impact. This is a great attitude. Too often opportunity cost isn't considered when making rules to protect folks from doing something stupid.
illumin8 10y ago
I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, and automates failover, including updating the DNS CNAME that the clients connect to during a failover (so it is seamless to the clients, other than requiring a reconnect), and ensuring that you don't lose a single transaction during a failover (the magic of synchronous replication over a low latency link between datacenters). For a company like Gitlab, that is public about wanting to exit the cloud, I feel like they could have really benefited from a fully managed relational database service. This entire tragic situation could have never happened if they were willing to acknowledge the obvious: managing relational databases is hard, and allowed someone with better operational automation, like AWS, to do it for them.
- overcast 10y ago
  This likely would have never happened, if one of their one hundred and sixty employees, just took the time to make sure backups were setup at all. You also need to be a sufficiently large enough organization to warrant the prices that the "cloud" services demand. As stated below, cloud computing is just someone else's server somewhere, and they are making lots of money doing it. Unless you need that level of scalability, and processing, then it's not worth it. I think Gitlab stated their entire PostgreSQL database was only a few hundred gigabytes. That's not exactly huge.
  - RhodesianHunter 10y ago
    "As stated below, cloud computing is just someone else's server somewhere. Unless you need that level of scalability, and processing, then it's not worth it." I keep seeing people throw this around as if it's God's truth and it frustrates the hell out of me. It may be the case for your organization but everywhere I have worked (from startups to Fortune 500) the cloud allowed our engineers to focus on our product rather than infrastructure maintenance and contributed massively to our success.
yarper 10y ago
It's amazing how quickly it descends into "one of the engineers" did x or y. Who was steering this ship exactly? It's really simple to point the finger and try to find a single cause of failure - but it's a fools errand - comparable to finding the single source behind a great success.
- jdavis703 10y ago
  Do you expect management to be staring over your shoulder every time you do some kind of `rm` on a production server? With great power comes great responsibility.
  - chadcmulligan 10y ago
    doesn't everyone alias rm to rm -i on prod? likewise all tty's have red backgrounds on prod.
    - nathancahill 10y ago
      I'm not sure if you're making an equally sarcastic point as your parent or not..
      - chadcmulligan 10y ago
        nope serious. I was at a place that the dba did exactly what happened at gitlab but in sql select * from table > script @script. (drop all the tables) It was in prod, he thought it was a dev db, the backups had never worked. After this the edict was all terminals for prod will be red. A simple solution
  - yarper 10y ago
    People get tired, sick, frustrated, panic - part of being a responsible engineer is accepting you're as fallible as the next person and building in protection against your own errors. However, if "the engineer" that caused this happens to read this, the above is not a sign that you should quit the profession and become a hermit. A chain of events caused this, you just happened to be the one without a chair to sit in when the music stopped.
nowarninglabel 10y ago
Thanks so much for the post and transparency Gitlab! We had just finished recovering from our own outage (stemming for a power loss and subsequent cascading failures) and were scheduled to do our post-mortem on 2/1 so the original document was a refreshing and reassuring read.
- sytse 10y ago
  Glad to hear it was of use to you.
ky738 10y ago
RIP the engineer
- YorickPeterse 10y ago
  Still here, and doing just fine.
  - boulos 10y ago
    I enjoy your updated profile.
  - sslalready 10y ago
    I love your tag line here on HN! Some 15 yrs ago I was hired as a UNIX administrator at some larger company. Despite being fresh from school I already had plenty of experience from spending the 90's hacking and programming on whatever UNIX system fyodor or rootshell.com had an exploit for. When the DBA was leaving for vaccation they didn't hesitate in letting me take over his daily routines. On this particular summer day, I had a simple job: dump the production database and load the data into the test environment. I had to make sure the dump was finished before 4pm when the daily production run started (which, FWIW, continued well into the evening). This was an Oracle shop so I believe the commands were mostly "exp" and "imp" -- with the caveat that the imp command would need an additional parameter to select the test database instead of the production one that was the default. Yeah, you see where this is going. The prod dump finished in time and shortly before leaving work I started importing the data. Then I sat around for a while before I realized I had forgotten about that additional "use the test environment" parameter -- and now I was importing a several hours old dump into the production database while the daily production run was running. I had to call company execs and explain the catastrophe to them, who in turn had to call in the vendor that sold us the system. Those were some pretty scary hours for a 20 yr old kid. Luckily it was just a matter of aborting the production run, reload the prod dump and then reschedule the production run for the day. The next day I had to start my day at the vendor's place to get some shaming, but also a good piece of advice - "always say destructive things out loud before doing them". Then they continued to tell me stories of people they had worked with who really messed things up, and we all had some good, evil laughs. Mistakes build experience, and hard learned lessons even more so. You now have a pretty good conversation starter to put on your CV. Personally I'd rather hire someone who was a "removal" specialist over someone who hadn't learned the skill yet. :) I believe both GitLab and the community in general will come out stronger from this incident. Thank you all for being so transparent about it.
pradeepchhetri 10y ago
Just want to add here that using tools like safe-rm[1] across your infrastructure would help in preventing data losses by running rm on unintended directories. [1]: https://launchpad.net/safe-rm https://launchpad.net/safe-rm
encoderer 10y ago
If you want to up your cron job monitoring game there's a link in my profile.
nurgleza 10y ago
I'm all for the spirit of openness & postmortems, but live streaming the actual troubleshooting is laughable, and I would resign at any company that forces such bullshit.
tschellenbach 10y ago
Shouldn't the conclusion of this post mortem be a move to a managed database service like RDS? The database doesn't sound huge, RDS is affordable enough, sounds to me that you spend less money and have better uptime and sleep by moving away from this in-house solution.
- carterehsmith 10y ago
  According to this article: > https://www.theregister.co.uk/2016/11/14/gitlab_to_dump_cloud_for_its_own_bare_metal_ceph_boxen/ https://www.theregister.co.uk/2016/11/14/gitlab_to_dump_clou... ...they were moving away from the cloud, to their own servers.
  - sytse 10y ago
    We changed our minds. We're working on a post in https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/4426/diffs https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/... but we wanted to post this first.
    - carterehsmith 10y ago
      It would be great for all of us if you provided your reasoning: to cloud, or not to cloud :) Thanks
      - sytse 10y ago
        The thing I linked contains a draft blog post with our reasons. But feel free to ask questions after reading that.
tschellenbach 10y ago
Shouldn't the conclusion of this post mortem be a move to a managed database service like RDS? The database doesn't sound huge, RDS is affordable enough, sounds to me that you spend less money and have better uptime and sleep by moving away from this in-house solution.
EnFinlay 10y ago
Most destructive troll ever.
- jolopes 10y ago
  Glad you read it till the end.
NPegasus 10y ago
> Root Cause Analysis > [...] > [List of technical problems] No, the root cause is you have no senior engineers who have been through this before. A collection of distributed remote employees, none of whom has enough experience to know any of the list of "Basic Knowledge Needed to Run a Website at Scale" that you list as the root causes. $30 million in funding and still running the company like a hobby project among college roommates. Mark my words, the board members from the VC firms will be removed by the VC partners due to letting the kids run the show. Then VC firms will put an experienced CEO and CTO in place to clean up the mess and get the company on track. Unfortunately they will probably have wasted a couple years and be down to the last million $ before they take action.
- dorianm 10y ago
  Exactly, I feel like it was nobody's job to make sure everything was resilient to failures
- gumby 10y ago
  > Mark my words, the board members from the VC firms will be removed by the VC partners I have never, ever, seen this. Every firm has its own internal politics [1] but rarely if ever would they do this. They are more likely to just ignore it. There is a belief that it takes a decade to tell if a VC is any good or not, and that includes "learning experiences" (all on the LP's dime of course). > Then VC firms will put an experienced CEO and CTO in place Now this I have seen. It even works sometimes (e.g. Eric Schmidt/Google). [1] I have had a firm invest in which all decisions were made by a single partner. I also had a firm invest (a sizable sum!) in which other senior partners never met me and only learned what my company even did when I gave a presentation at one of their LP meetings. Also some very large funds allow senior partners to make small seed investments ("science projects") without formal approval from the partnership.
- deleted 10y ago
  [deleted]
- elliotec 10y ago
  They also openly pay well below market rate, so they should expect to get what they pay for.
_Marak_ 10y ago
I've noticed a lot of other positive activity and press for Gitlab for in the past month. It's unfortunate they had this technical issue, but it's good to see others ( besides Github ) operating in this space. I should give Gitlab a try sometime.
- sperglord 10y ago
  We just switched from Github to Gitlab for our private repos. The choice (based upon cost alone) was between them and Bitbucket, and the professional way that this was handled and the transparent communication was really nice to see.
voidlogic 10y ago
>When we went to look for the pg_dump backups we found out they were not there. The S3 bucket was empty, and there was no recent backup to be found anywhere. Upon closer inspection we found out that the backup procedure was using pg_dump 9.2, while our database is running PostgreSQL 9.6 (for Postgres, 9.x releases are considered major). A difference in major versions results in pg_dump producing an error, terminating the backup procedure. Yikes. One common practice that would have avoided this is by using the just taken backup to populate stage. If the restore fails pages go out. If integration tests that run after a successful restore/populate fail- pages go out. Live and learn I guess.
nstj 10y ago
@sytse were you in contact with MS/Azure during the restore? If so did they offer any assistance, e.g in speeding up restoration disk speed etc
nodesocket 10y ago
My main question is still: >> Why did replication stop? - A spike in database load caused the database replication process to stop. This was due to the primary removing WAL segments before the secondary could replicate them. Is this a bug/defect in PostgreSQL then? Incorrect PostgreSQL configuration? Insufficient hardware? What was the root cause of Postgres primary removing the WAL segments?
- ploxiln 10y ago
  There's always some limit. At $PREVIOUS_JOB I think it was at least 48 hours, probably over 72 hours, of replication log retention (usually measured in GiB though). So it's surprising that in GitLab's case it must have been less than 6 hours (IIRC from the original google doc the slave had more than 4 hours replication lag due to load, initially ...)
- cookiecaper 10y ago
  The cause is bad configuration. PgSQL, Mongo, and MySQL all use a transaction stream like this for replication and they all have to put some kind of cap on it or risk running out of disk space, but the cap should be made sufficiently large to allow automatic resumption of disconnected slaves without manual redumping, except in extraordinary circumstances. Log retention should be long enough to last at least a long weekend so that someone can come in and poke the DB back into action on Tuesday morning, but preferably more like 1 week. Alarms should be configured to fire well before replication lag gets anywhere near the log expiration timeout. In particular, PostgreSQL has a feature that allows automatic WAL archiving (i.e., it confirms that the WAL has been successfully shipped to a separate system before it removes it from the master) and a feature called "replication slots" that ensures that all WALs are kept if a regular subscriber is offline. If either of these features had been correctly configured, there would've been no need to do a full resync; the secondary database would've come back and immediately picked up where it left off. Additionally, if one must resync the full database (and I've had to do this many times), tools like pg_basebackup and innobackupex are basically required to consistently perform the process of pulling the master dumps, and the old (unsynced) data directory should be allowed to linger until the full master snapshot has been fully confirmed and is ready to resync. It's very reckless to go around removing binary data directories until you're certain that the new stuff is running, even if you're "just on the replicant". With pg_basebackup, you run it on the replicant server and it streams down the files, no need to log into the master server at all. With innobackupex, you need to have read access to the master's binary data directory, but should achieve this safely through something like a read-only NFS mount. mydumper is a possible alternative to innobackupex that tries to capture the binlog coords and doesn't require any direct access to the host beneath the database server.
KayEss 10y ago
The engineers still seem to have a physical server mindset rather than a cloud mindset. Deleting data is always extremely dangerous and there was no need for it in this situation. They should have spun up a new server to act as secondary the moment replication failed. This new server is the one you run all of these commands on, and if you make a mistake you spin up a new one. Only when the replication is back in good order do you go through and kill the servers you no longer need. The procedure for setting up these new servers should be based on the same scripts that spin up new UAT servers for each release. You spin up a server that is a near copy of production and then do the upgrade to new software on that. Only when you've got a successful deployment do you kill the old UAT server. This way all of these processes are tested time and time again and you know exactly how long they'll take and iron out problems in the automation.
- matt_wulfeck 10y ago
  This type of thing always sounds good and all, but the reality is people get desperate and emotional when their website is down and everyone wants it up ASAP.
  - KayEss 10y ago
    I certainly don't disagree with that, but if you have this automated it is also the fastest way to get it back up and running. Besides, the site wouldn't have been down if they had this.
  - jjirsa 10y ago
    Desperate and emotional is no way to run a business
    - IA21 10y ago
      I agree but engineers are people, not machines. They're bound to make mistakes under pressure.
      - sneak 10y ago
matt_wulfeck 10y ago
> Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed. I can only image this engineer's poor old heart after the realization of removing that directory on the master. A sinking, awful feeling of dread. I've had a few close calls in my career. Each time it's made me pause and thank my luck it wasn't prod.
khazhou 10y ago
Every internal Ops manual needs to begin with the simple phrase: DON'T PANIC
atmosx 10y ago
Great to have a full-featured, professional post-mortem. Incidentally I work at a company that suffered data loss because of this outage and we're looking for ways to move out of GL. My 2 cents... I might be the only one, but I don't like the way GL handled this case. I understand transparency as a core value and all, but they've gotten a bit too far. IMHO this level of exposure has far-reaching, privacy implications for the ppl who work there. Implications that cannot be assessed now. The engineer in question might have not suffered a PTSD, but some other engineer might haven been. Who knows how a bad public experience might play out? It's a fairly small circle, I'm not sure I would like to be part of a company that would expose me in a similar fashion, if I happen to screw up. On the corporate side of things there is a saying in Greek: "Τα εν οίκω μη εν δήμω" meaning don't wash your dirty linen in public. Although they're getting praised by bloggers and other small-size startups, in the end of the day exposing your 6-layer broken backup policy and other internal flaws in between, while being funded at the tune of 25.62M in 4 rounds, does not look good.
- sytse 10y ago
  Hi Panagiotis. I'm glad to hear you like the postmortem. I'm very sorry your company suffered data loss. If you want to move from GitLab.com please know that you can easily export projects and import them on a self-hosted instance https://gitlab.com/help/user/project/settings/import_export.md https://gitlab.com/help/user/project/settings/import_export.... (and if in the future we regain your trust you can also go the other way). It is not our intent to have one of our team members implicated by the transparency. That is why we redacted their name to team-member-1 and in any future incidents we'll do the same. It should be their choice to be identified or not. We are very aware of the stress that such a mistake might cause and the rest of the team has been very supportive. I agree that we don't look good because of the broken backup policy. The way to fix that is to improve our processes. We recognize the risk to the company of being transparent, but your values are defined by what you do when it is hard.
  - AsyncAwait 10y ago
    > your values are defined by what you do when it is hard. Precisely. Most companies would stay as quiet about this as possible, you guys remained transparent and this is why I'll remain a customer.
cookiecaper 10y ago
I really hate to pile on, but after reading through this whole thread and the whole post-mortem, there are a few basic things that are troubling besides the widely-acknowledged backup methodology. I don't see issues directly related to addressing these things. 1. notifications go through regular email. Email should be only one channel used to dispatch notifications of infrastructure events. Tools like VictorOps or PagerDuty should be employed as notification brokers/coordinators and notifications should go to email, team chat, and phone/SMS if severity warrants, and have an attached escalation policy so that it doesn't all hinge on one guy's phone not being dead. 2. there was a single database, whose performance problems had impacted production multiple times before (the post lists 4 incidents). One such performance problem was contributing to breakage at this very moment. I understand that was the thing that was trying to be fixed here, but what process allowed this to cause 4 outages over the preceding year without moving to the top of the list of things to address? Wouldn't it be wise to tweak the PgSQL configuration and/or upgrade the server before trying to integrate the hot standby to serve some read-only queries? And since a hot standby can only service reads (and afaik this is not a well-supported option in PgSQL), wouldn't most of the performance issues, which appear write-related, remain? The process seriously needs to be reviewed here. And am I reading this right, the one and only production DB server was restarted to change a configuration value in order to try to make pg_basebackup work? What impact did that have on the people trying to use the site a) while the database was restarting, and b) while the kernel settings were tweaked to accommodate the too-high max_connections value? Is it normal for GitLab to cause intermittent, few-minute downtimes like that? Or did that occur while the site was already down? 3. Spam reports can cause mass hard deletion of user data? Has this happened to other users? The target in this instance was a GitLab employee. Who has been trolled this way such that performance wasn't impacted? What's the remedy for wrongly-targeted persons? It's clear that backups of this data are not available. And is the GitLab employee's data gone now too? How could something so insufficient have been released to the public, and how can you disclose this apparently-unresolved vulnerability? By so doing, you're challenging the public to come and try to empty your database. Good thing you're surely taking good backups now! (We're going to glance over the fact that GitLab just told everyone its logical DB backups are 3 days behind and that we shouldn't worry because LVM snapshots now occur hourly, and that it only takes 16 hours to transfer LVM snapshots between environments :) ) 4. the PgSQL master deleted its WALs within 4 hours of the replica "beginning to lag" (<interrobang here>). That really needs to be fixed. Again, you probably need a serious upgrade to your PgSQL server because it apparently doesn't have enough space to hold more than a couple of hours of WALs (unless this was just a naive misconfiguration of the [min|max]_wal_size parameter, like the max_connections parameter?). I understand that transaction logs can get very large, but the disk needs to accommodate (usually a second disk array is used for WALs to ease write impact) and replication lag needs to be monitored and alarmed on. There were a few other things (including someone else downthread who pointed out that your CEO re-revealed your DB's hostnames in this write-up, and that they're resolvable via public DNS and have running sshds on port 22), but these are the big standouts for me. P.S. bonus point, just speculative: Not sure how fast your disks were, but 300GB gone in "a few seconds" sounds like a stretch. Some data may've been recoverable with some disk forensics. Especially if your Postgres server was running at the time of the deletion, some data and file descriptors also likely could've been extracted from system memory. Linux doesn't actually delete files if another process is holding their handle open; you can go into the /proc virtual filesystem and grab the file descriptor again to redump the files to live disk locations. Since your database was 400GB and too big to keep 100% in RAM, this probably wouldn't have been a full recovery, but it may have been able to provide a partial. The theoretically best thing to do in such a situation would probably be to unplug the machine ASAP after ^C (without going through formal shutdown processes that may try to "clean up" unfinished disk work), remove the disk, attach it to a machine with a write blocker, and take a full-disk image for forensics purposes. This would maximize the ability to extract any data that the system was unable to eat/destroy. In theory, I believe pulling the plug while a process kept the file descriptor open should keep you in reasonably good shape, as far as that goes after you've accidentally deleted 3/4 of your production database. The process never closes and the disk stops and the contents remain on disk, just pending unlink when the OS stops the process (this is one reason why it'd be important to block writes to the disk/be extremely careful while mounting; if the journal plays back, it may destroy these files on the next boot anyway). But someone more familiar with the FS internals would have to say definitively if it works that way or not. I recognize that such speculative/experimental recovery measures may have been intentionally forgone since they're labor intensive, may have delayed the overall recovery, and very possibly wouldn't have returned useful data anyway. Mentioning it mainly as an option to remain aware of.
dustinmoris 10y ago
Watching GitLab is somewhat painful. I feel like they make every possible mistake you could do as an IT startup and because they are transparent about it people seem to love the fact that they screw up all the time. I don't know if I share the same mentality, because at the end of the day I don't trust GitLab even with the simplest task, let alone any valuable work of mine. It's good to be humble and know that mistakes can happen to anyone and learn from it, etc., but when you do in 2017 still the same stupid mistakes that people did a million times since 1990 and it's all well documented and there's systems built to avoid these same basic mistakes and you still do them today then I just think it cannot be described any different than absolute stupidity and incompetence. I know they have many fans who just look past every mistake no matter how bad it was only because they are open about it, but common, this is now just taking the piss no?
- egwor 10y ago
  I had exactly the same feeling. One of my friends was a doctor at a hospital and there was a serious mistake which my friend reported to the consultant. The consultant made a good point that it was never just one mistake that caused the serious mistake; it was a series of smaller mistakes that hadn't been checked or addressed. (the argument was that there were processes to avoid these kinds of mistakes) If you read the post they also noted that they were also in the process of accidentally removing one of their own employees which make me think that there are other problems going on here. Maybe part of the problem is that I think that the industry has a problem that it isn't willing to learn from the experiences of others? (I feel like we have 'just enough learning' and 'experienced folk who raise concerns are considered stuck in their outdated ways' and 'people who make a silly mistake like that must be an idiot'). I think that since we clump together those that have had formal training with those that haven't we aren't encouraging the value of this education. I'm also fully aware that some self-learned developers are much more competent rather than a college educated developer.
  - dustinmoris 10y ago
    I don't actually think it has anything to do with education. I think it really comes down to common sense. Any half intelligent engineer would always first research good practices, pitfalls and existing information which has been gathered from decades of other experienced engineers before doing anything stupid on their own. It seems that GitLab is lacking this attitude.
Achshar 10y ago
Does anyone have a link to the YouTube stream they's talking about? Can't seem to find it on their channel. And the link in the doc is redirecting to the live link [1] which doesn't list the stream. [1] - https://www.youtube.com/c/Gitlab/live https://www.youtube.com/c/Gitlab/live
- cmatija 10y ago
  Yup, Thanks for taking the interest to check it out. It's an unlisted YT video, so that's why it might be hard to find. Here it is: https://www.youtube.com/watch?v=nc0hPGerSd4 https://www.youtube.com/watch?v=nc0hPGerSd4
  - Achshar 10y ago
    Thanks! I'll check it out.
- deleted 10y ago
  [deleted]
ancarda 10y ago
>Unfortunately DMARC was not enabled for the cronjob emails, resulting in them being rejected by the receiver. This means we were never aware of the backups failing, until it was too late. At my dayjob, we gradually stopped using email for almost all alerts, instead we have several Slack channels like #database-log where errors to MySQL go. Any cron jobs that fail post in #general-log. Uptime monitoring tools post in #status. So on... Email has so much anti-spam stuff like DMARC that make it less reliable your mail will be delivered. For something failing like a backup or database query, it's too important to have potentially not reach someone who can make sure it gets fixed. My 2 cents.
- gl- 10y ago
  This is a step in the right direction but still misses a big part of it IMO: push versus pull notifications. If the agent stops functioning correctly or someone makes a config change, the alerts just stop and no one notices. At the very least you want some kinda dead-man's switch that gets pissed if it's seen no events in the last x amount of time. Ideally you want to be polling the box in a stateful way; although with ephemeral nodes & flexible infra being all the rage that's fallen to the side a bit lately.
  - ancarda 10y ago
    Absolutely, that's a great idea! You could also check for evidence a run has been successful, although that does depend on what you're doing exactly. For our backup system, we're going to build an audit cron job on our main server that checks all our Azure containers to see if each server has pushed a file lately. It'll alert us if a file hasn't been uploaded in a few days or if it's smaller than a few MB (which is suspiciously small; we'd expect a few hundred MB for mysqldump+files).
- mkopinsky 10y ago
  How do you set cron to post to Slack?
  - ancarda 10y ago
aabajian 10y ago
This is an outstanding writeup, but I wonder if it glosses over the real problem: >>The standby (secondary) is only used for failover purposes. >>One of the engineers went to the secondary and wiped the data directory, then ran pg_basebackup. IMO, secondaries should be treated exactly as their primaries. No operation should be done on a secondary unless you'd be OK doing that same operation on the primary. You can always create another instance for these operations.
jsingleton 10y ago
TIL GitLab runs on Azure. If your CI servers or deployment targets are also on Azure then the latency should be pretty low (assuming you get the correct region). Good to know. I moved from AWS to Azure years ago. Mainly because I run mostly .NET workloads and the support is better. I've recently done some .NET stuff on AWS again and am remembering why I switched.
XorNot 10y ago
The backup situation stands out to me as a problem no one has really adequately solved. Verifying a task has happened in a way where the notifications are noticed is actually a really hard problem that it feels like we collectively ignore in this business. How do you reliably check if something didn't happen? Is the backup server alive? Did the script work? Did the backup work? Is the email server working? Is the dashboard working? Is the user checking their emails (think: wildcard mail sorting rule dumping a slight change in failure messages to the wrong folder). And the converse answer isn't much better: send a success notification...but if it mostly succeeds, how do you keep people paying attention to it when it doesn't (i.e. no failure message, but no success message)? The best answer I've got, personally, is to use positive notifications combined with visibility - dashboard your really important tasks with big, distinctive colors - use time based detection and put a clock on your dashboard (because dashboards which mostly don't change might hang and no one notice).
AlexCoventry 10y ago
Thank you for this informative postmortem and mitigation outline. Are any organizational changes planned in response to the development friction which led to the outage? It seems to have arisen from long-standing operational issues, and an analysis of how prior attempts to address those issues got bogged down would be very interesting.
dancryer 10y ago
Can't help but notice that the new backup monitoring tool suggests that the latest PGSQL backup is almost six days old... Is that correct? http://monitor.gitlab.net/dashboard/db/backups?from=1485941950082&to=1486978750114 http://monitor.gitlab.net/dashboard/db/backups?from=14859419...

377 points by mbrain 10y ago

GitHub also lost a bunch of PRs and issues sitewide early in their history. They claimed to have restored all the PRs from backup, but I was pretty sure I had opened a PR and it never came back. I emailed support and they basically told me tough luck.

oli5679 10y ago

I found this entertaining, even if they did later admit that it was a hoax: http://serverfault.com/questions/587102/monday-morning-mistake-sudo-rm-rf-no-preserve-root http://serverfault.com/questions/587102/monday-morning-mista...

gr2020 10y ago

Reading this, the thing that stuck out to me was how remarkably lucky they were to have the two snapshots. The one from 6 hours earlier was there seemingly by chance, as an engineer had created it for unrelated reasons. And for both the 6- and 24-hour snapshots, it seems just lucky that neither had any breaking changes made to them by pre-production code (they _were_ dev/staging snapshots, after all). I'm glad it all worked out in the end!

sytse 10y ago
We too are glad we had those snapshots. And while it was the worst thing that ever happend at GitLab it is humbling to know that it could have been worse.
- animex 10y ago
  What do you think would have happened if you had total data loss/failure?
  - sb8244 10y ago
    I don't work for gitlab. Isn't that a put you out of business moment? I guess there is enterprise installations but the main website would be toast. I guess there is the chance there were still older backups taken intentionally or by chance that would have been worse but recoverable from
    - gtaylor 10y ago
      Most of their income probably comes from customers that run their own GitLab Enterprises installs. This would have really sucked for all of their non paying users, though.

samat 10y ago

Am I missing something or didn't they mention 'test recovery, not backups'?

sytse 10y ago
Two of the issues linked from the article deal with testing the backups: - Automated testing of recovering PostgreSQL database backups https://gitlab.com/gitlab-com/infrastructure/issues/1102 https://gitlab.com/gitlab-com/infrastructure/issues/1102 - Build Streaming Database Restore https://gitlab.com/gitlab-com/infrastructure/issues/1152 https://gitlab.com/gitlab-com/infrastructure/issues/1152

nierman 10y ago

yes, wal archiving would have helped (archive_command = rsync standby ...), but it's also very easy in postgres 9.4+ to add a replication slot on the master so that wal is kept until it is no longer needed by the standby. simply reference the slot in the standby's recovery.conf file. definitely monitor your replication lag--or at least disk usage on the master--with this approach (in case wal starts piling up there).

isoos 10y ago

sytse and GitLab folks: thank you for the transparency.

sytse 10y ago
You're welcome. Thanks for all the kind responses we received https://twitter.com/i/moments/826818668948549632 https://twitter.com/i/moments/826818668948549632

meowface 10y ago

>Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed. I could feel the sweat drops just from reading this. I'd bet every one of us has experienced the panicked Ctrl+C of Death at some point or another.

a_t48 10y ago
Misread that as "The engineer was terminated" at first. Poor guy.
- haldean 10y ago
  Potential outage prevention plan: put an alias on all production machines that emails HR to schedule a disciplinary meeting every time you run `rm -rf`.
rectang 10y ago
For me, it was when I meant to execute this... rm -rf ~/foo ... but executed this instead: rm -rf ~ /foo
- BinaryIdiot 10y ago
  I did the Windows equivalent once a long time ago (I think it's deltree?) and I did it on a university computer system. It cleared out a TON of files and the computer itself pretty much stopped working. I had to hard turn it off. Fortunately the University was using some tool that can re-image a computer each time it boots before hitting Windows so starting it back up and all the deleted system and application files were back.
  - overcast 10y ago
    Probably Ghost server, that's why schools do this.

grhmc 10y ago

Thank you for redacting who the engineer was. Great write-up. Thank you!

sytse 10y ago
You're very welcome. By the way the engineer was cool with being named, but we think this is good practice for future postmortems. I do hope it doesn't come soon.

jsperson 10y ago

>An ideal environment is one in which you can make mistakes but easily and quickly recover from them with minimal to no impact. This is a great attitude. Too often opportunity cost isn't considered when making rules to protect folks from doing something stupid.

illumin8 10y ago

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, and automates failover, including updating the DNS CNAME that the clients connect to during a failover (so it is seamless to the clients, other than requiring a reconnect), and ensuring that you don't lose a single transaction during a failover (the magic of synchronous replication over a low latency link between datacenters). For a company like Gitlab, that is public about wanting to exit the cloud, I feel like they could have really benefited from a fully managed relational database service. This entire tragic situation could have never happened if they were willing to acknowledge the obvious: managing relational databases is hard, and allowed someone with better operational automation, like AWS, to do it for them.

overcast 10y ago
This likely would have never happened, if one of their one hundred and sixty employees, just took the time to make sure backups were setup at all. You also need to be a sufficiently large enough organization to warrant the prices that the "cloud" services demand. As stated below, cloud computing is just someone else's server somewhere, and they are making lots of money doing it. Unless you need that level of scalability, and processing, then it's not worth it. I think Gitlab stated their entire PostgreSQL database was only a few hundred gigabytes. That's not exactly huge.
- RhodesianHunter 10y ago
  "As stated below, cloud computing is just someone else's server somewhere. Unless you need that level of scalability, and processing, then it's not worth it." I keep seeing people throw this around as if it's God's truth and it frustrates the hell out of me. It may be the case for your organization but everywhere I have worked (from startups to Fortune 500) the cloud allowed our engineers to focus on our product rather than infrastructure maintenance and contributed massively to our success.

yarper 10y ago

It's amazing how quickly it descends into "one of the engineers" did x or y. Who was steering this ship exactly? It's really simple to point the finger and try to find a single cause of failure - but it's a fools errand - comparable to finding the single source behind a great success.

jdavis703 10y ago
Do you expect management to be staring over your shoulder every time you do some kind of `rm` on a production server? With great power comes great responsibility.
- chadcmulligan 10y ago
  doesn't everyone alias rm to rm -i on prod? likewise all tty's have red backgrounds on prod.
  - nathancahill 10y ago
    I'm not sure if you're making an equally sarcastic point as your parent or not..
    - chadcmulligan 10y ago
      nope serious. I was at a place that the dba did exactly what happened at gitlab but in sql select * from table > script @script. (drop all the tables) It was in prod, he thought it was a dev db, the backups had never worked. After this the edict was all terminals for prod will be red. A simple solution
- yarper 10y ago
  People get tired, sick, frustrated, panic - part of being a responsible engineer is accepting you're as fallible as the next person and building in protection against your own errors. However, if "the engineer" that caused this happens to read this, the above is not a sign that you should quit the profession and become a hermit. A chain of events caused this, you just happened to be the one without a chair to sit in when the music stopped.

nowarninglabel 10y ago

Thanks so much for the post and transparency Gitlab! We had just finished recovering from our own outage (stemming for a power loss and subsequent cascading failures) and were scheduled to do our post-mortem on 2/1 so the original document was a refreshing and reassuring read.

sytse 10y ago
Glad to hear it was of use to you.

ky738 10y ago

RIP the engineer

YorickPeterse 10y ago
Still here, and doing just fine.
- boulos 10y ago
  I enjoy your updated profile.
- sslalready 10y ago
  I love your tag line here on HN! Some 15 yrs ago I was hired as a UNIX administrator at some larger company. Despite being fresh from school I already had plenty of experience from spending the 90's hacking and programming on whatever UNIX system fyodor or rootshell.com had an exploit for. When the DBA was leaving for vaccation they didn't hesitate in letting me take over his daily routines. On this particular summer day, I had a simple job: dump the production database and load the data into the test environment. I had to make sure the dump was finished before 4pm when the daily production run started (which, FWIW, continued well into the evening). This was an Oracle shop so I believe the commands were mostly "exp" and "imp" -- with the caveat that the imp command would need an additional parameter to select the test database instead of the production one that was the default. Yeah, you see where this is going. The prod dump finished in time and shortly before leaving work I started importing the data. Then I sat around for a while before I realized I had forgotten about that additional "use the test environment" parameter -- and now I was importing a several hours old dump into the production database while the daily production run was running. I had to call company execs and explain the catastrophe to them, who in turn had to call in the vendor that sold us the system. Those were some pretty scary hours for a 20 yr old kid. Luckily it was just a matter of aborting the production run, reload the prod dump and then reschedule the production run for the day. The next day I had to start my day at the vendor's place to get some shaming, but also a good piece of advice - "always say destructive things out loud before doing them". Then they continued to tell me stories of people they had worked with who really messed things up, and we all had some good, evil laughs. Mistakes build experience, and hard learned lessons even more so. You now have a pretty good conversation starter to put on your CV. Personally I'd rather hire someone who was a "removal" specialist over someone who hadn't learned the skill yet. :) I believe both GitLab and the community in general will come out stronger from this incident. Thank you all for being so transparent about it.

pradeepchhetri 10y ago

Just want to add here that using tools like safe-rm[1] across your infrastructure would help in preventing data losses by running rm on unintended directories. [1]: https://launchpad.net/safe-rm https://launchpad.net/safe-rm

encoderer 10y ago

If you want to up your cron job monitoring game there's a link in my profile.

nurgleza 10y ago

I'm all for the spirit of openness & postmortems, but live streaming the actual troubleshooting is laughable, and I would resign at any company that forces such bullshit.

tschellenbach 10y ago

Shouldn't the conclusion of this post mortem be a move to a managed database service like RDS? The database doesn't sound huge, RDS is affordable enough, sounds to me that you spend less money and have better uptime and sleep by moving away from this in-house solution.

carterehsmith 10y ago
According to this article: > https://www.theregister.co.uk/2016/11/14/gitlab_to_dump_cloud_for_its_own_bare_metal_ceph_boxen/ https://www.theregister.co.uk/2016/11/14/gitlab_to_dump_clou... ...they were moving away from the cloud, to their own servers.
- sytse 10y ago
  We changed our minds. We're working on a post in https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/4426/diffs https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/... but we wanted to post this first.
  - carterehsmith 10y ago
    It would be great for all of us if you provided your reasoning: to cloud, or not to cloud :) Thanks
    - sytse 10y ago
      The thing I linked contains a draft blog post with our reasons. But feel free to ask questions after reading that.

tschellenbach 10y ago

EnFinlay 10y ago

Most destructive troll ever.

jolopes 10y ago
Glad you read it till the end.

NPegasus 10y ago

> Root Cause Analysis > [...] > [List of technical problems] No, the root cause is you have no senior engineers who have been through this before. A collection of distributed remote employees, none of whom has enough experience to know any of the list of "Basic Knowledge Needed to Run a Website at Scale" that you list as the root causes. $30 million in funding and still running the company like a hobby project among college roommates. Mark my words, the board members from the VC firms will be removed by the VC partners due to letting the kids run the show. Then VC firms will put an experienced CEO and CTO in place to clean up the mess and get the company on track. Unfortunately they will probably have wasted a couple years and be down to the last million $ before they take action.

dorianm 10y ago
Exactly, I feel like it was nobody's job to make sure everything was resilient to failures
gumby 10y ago
> Mark my words, the board members from the VC firms will be removed by the VC partners I have never, ever, seen this. Every firm has its own internal politics [1] but rarely if ever would they do this. They are more likely to just ignore it. There is a belief that it takes a decade to tell if a VC is any good or not, and that includes "learning experiences" (all on the LP's dime of course). > Then VC firms will put an experienced CEO and CTO in place Now this I have seen. It even works sometimes (e.g. Eric Schmidt/Google). [1] I have had a firm invest in which all decisions were made by a single partner. I also had a firm invest (a sizable sum!) in which other senior partners never met me and only learned what my company even did when I gave a presentation at one of their LP meetings. Also some very large funds allow senior partners to make small seed investments ("science projects") without formal approval from the partnership.
deleted 10y ago
[deleted]
elliotec 10y ago
They also openly pay well below market rate, so they should expect to get what they pay for.

_Marak_ 10y ago

I've noticed a lot of other positive activity and press for Gitlab for in the past month. It's unfortunate they had this technical issue, but it's good to see others ( besides Github ) operating in this space. I should give Gitlab a try sometime.

sperglord 10y ago
We just switched from Github to Gitlab for our private repos. The choice (based upon cost alone) was between them and Bitbucket, and the professional way that this was handled and the transparent communication was really nice to see.

voidlogic 10y ago

>When we went to look for the pg_dump backups we found out they were not there. The S3 bucket was empty, and there was no recent backup to be found anywhere. Upon closer inspection we found out that the backup procedure was using pg_dump 9.2, while our database is running PostgreSQL 9.6 (for Postgres, 9.x releases are considered major). A difference in major versions results in pg_dump producing an error, terminating the backup procedure. Yikes. One common practice that would have avoided this is by using the just taken backup to populate stage. If the restore fails pages go out. If integration tests that run after a successful restore/populate fail- pages go out. Live and learn I guess.

nstj 10y ago

@sytse were you in contact with MS/Azure during the restore? If so did they offer any assistance, e.g in speeding up restoration disk speed etc

nodesocket 10y ago

My main question is still: >> Why did replication stop? - A spike in database load caused the database replication process to stop. This was due to the primary removing WAL segments before the secondary could replicate them. Is this a bug/defect in PostgreSQL then? Incorrect PostgreSQL configuration? Insufficient hardware? What was the root cause of Postgres primary removing the WAL segments?

ploxiln 10y ago
There's always some limit. At $PREVIOUS_JOB I think it was at least 48 hours, probably over 72 hours, of replication log retention (usually measured in GiB though). So it's surprising that in GitLab's case it must have been less than 6 hours (IIRC from the original google doc the slave had more than 4 hours replication lag due to load, initially ...)
cookiecaper 10y ago
The cause is bad configuration. PgSQL, Mongo, and MySQL all use a transaction stream like this for replication and they all have to put some kind of cap on it or risk running out of disk space, but the cap should be made sufficiently large to allow automatic resumption of disconnected slaves without manual redumping, except in extraordinary circumstances. Log retention should be long enough to last at least a long weekend so that someone can come in and poke the DB back into action on Tuesday morning, but preferably more like 1 week. Alarms should be configured to fire well before replication lag gets anywhere near the log expiration timeout. In particular, PostgreSQL has a feature that allows automatic WAL archiving (i.e., it confirms that the WAL has been successfully shipped to a separate system before it removes it from the master) and a feature called "replication slots" that ensures that all WALs are kept if a regular subscriber is offline. If either of these features had been correctly configured, there would've been no need to do a full resync; the secondary database would've come back and immediately picked up where it left off. Additionally, if one must resync the full database (and I've had to do this many times), tools like pg_basebackup and innobackupex are basically required to consistently perform the process of pulling the master dumps, and the old (unsynced) data directory should be allowed to linger until the full master snapshot has been fully confirmed and is ready to resync. It's very reckless to go around removing binary data directories until you're certain that the new stuff is running, even if you're "just on the replicant". With pg_basebackup, you run it on the replicant server and it streams down the files, no need to log into the master server at all. With innobackupex, you need to have read access to the master's binary data directory, but should achieve this safely through something like a read-only NFS mount. mydumper is a possible alternative to innobackupex that tries to capture the binlog coords and doesn't require any direct access to the host beneath the database server.

KayEss 10y ago

The engineers still seem to have a physical server mindset rather than a cloud mindset. Deleting data is always extremely dangerous and there was no need for it in this situation. They should have spun up a new server to act as secondary the moment replication failed. This new server is the one you run all of these commands on, and if you make a mistake you spin up a new one. Only when the replication is back in good order do you go through and kill the servers you no longer need. The procedure for setting up these new servers should be based on the same scripts that spin up new UAT servers for each release. You spin up a server that is a near copy of production and then do the upgrade to new software on that. Only when you've got a successful deployment do you kill the old UAT server. This way all of these processes are tested time and time again and you know exactly how long they'll take and iron out problems in the automation.

matt_wulfeck 10y ago
This type of thing always sounds good and all, but the reality is people get desperate and emotional when their website is down and everyone wants it up ASAP.
- KayEss 10y ago
  I certainly don't disagree with that, but if you have this automated it is also the fastest way to get it back up and running. Besides, the site wouldn't have been down if they had this.
- jjirsa 10y ago
  Desperate and emotional is no way to run a business
  - IA21 10y ago
    I agree but engineers are people, not machines. They're bound to make mistakes under pressure.
    - sneak 10y ago

matt_wulfeck 10y ago

> Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed. I can only image this engineer's poor old heart after the realization of removing that directory on the master. A sinking, awful feeling of dread. I've had a few close calls in my career. Each time it's made me pause and thank my luck it wasn't prod.

khazhou 10y ago

Every internal Ops manual needs to begin with the simple phrase: DON'T PANIC

atmosx 10y ago

Great to have a full-featured, professional post-mortem. Incidentally I work at a company that suffered data loss because of this outage and we're looking for ways to move out of GL. My 2 cents... I might be the only one, but I don't like the way GL handled this case. I understand transparency as a core value and all, but they've gotten a bit too far. IMHO this level of exposure has far-reaching, privacy implications for the ppl who work there. Implications that cannot be assessed now. The engineer in question might have not suffered a PTSD, but some other engineer might haven been. Who knows how a bad public experience might play out? It's a fairly small circle, I'm not sure I would like to be part of a company that would expose me in a similar fashion, if I happen to screw up. On the corporate side of things there is a saying in Greek: "Τα εν οίκω μη εν δήμω" meaning don't wash your dirty linen in public. Although they're getting praised by bloggers and other small-size startups, in the end of the day exposing your 6-layer broken backup policy and other internal flaws in between, while being funded at the tune of 25.62M in 4 rounds, does not look good.

sytse 10y ago
Hi Panagiotis. I'm glad to hear you like the postmortem. I'm very sorry your company suffered data loss. If you want to move from GitLab.com please know that you can easily export projects and import them on a self-hosted instance https://gitlab.com/help/user/project/settings/import_export.md https://gitlab.com/help/user/project/settings/import_export.... (and if in the future we regain your trust you can also go the other way). It is not our intent to have one of our team members implicated by the transparency. That is why we redacted their name to team-member-1 and in any future incidents we'll do the same. It should be their choice to be identified or not. We are very aware of the stress that such a mistake might cause and the rest of the team has been very supportive. I agree that we don't look good because of the broken backup policy. The way to fix that is to improve our processes. We recognize the risk to the company of being transparent, but your values are defined by what you do when it is hard.
- AsyncAwait 10y ago
  > your values are defined by what you do when it is hard. Precisely. Most companies would stay as quiet about this as possible, you guys remained transparent and this is why I'll remain a customer.

cookiecaper 10y ago

I really hate to pile on, but after reading through this whole thread and the whole post-mortem, there are a few basic things that are troubling besides the widely-acknowledged backup methodology. I don't see issues directly related to addressing these things. 1. notifications go through regular email. Email should be only one channel used to dispatch notifications of infrastructure events. Tools like VictorOps or PagerDuty should be employed as notification brokers/coordinators and notifications should go to email, team chat, and phone/SMS if severity warrants, and have an attached escalation policy so that it doesn't all hinge on one guy's phone not being dead. 2. there was a single database, whose performance problems had impacted production multiple times before (the post lists 4 incidents). One such performance problem was contributing to breakage at this very moment. I understand that was the thing that was trying to be fixed here, but what process allowed this to cause 4 outages over the preceding year without moving to the top of the list of things to address? Wouldn't it be wise to tweak the PgSQL configuration and/or upgrade the server before trying to integrate the hot standby to serve some read-only queries? And since a hot standby can only service reads (and afaik this is not a well-supported option in PgSQL), wouldn't most of the performance issues, which appear write-related, remain? The process seriously needs to be reviewed here. And am I reading this right, the one and only production DB server was restarted to change a configuration value in order to try to make pg_basebackup work? What impact did that have on the people trying to use the site a) while the database was restarting, and b) while the kernel settings were tweaked to accommodate the too-high max_connections value? Is it normal for GitLab to cause intermittent, few-minute downtimes like that? Or did that occur while the site was already down? 3. Spam reports can cause mass hard deletion of user data? Has this happened to other users? The target in this instance was a GitLab employee. Who has been trolled this way such that performance wasn't impacted? What's the remedy for wrongly-targeted persons? It's clear that backups of this data are not available. And is the GitLab employee's data gone now too? How could something so insufficient have been released to the public, and how can you disclose this apparently-unresolved vulnerability? By so doing, you're challenging the public to come and try to empty your database. Good thing you're surely taking good backups now! (We're going to glance over the fact that GitLab just told everyone its logical DB backups are 3 days behind and that we shouldn't worry because LVM snapshots now occur hourly, and that it only takes 16 hours to transfer LVM snapshots between environments :) ) 4. the PgSQL master deleted its WALs within 4 hours of the replica "beginning to lag" (<interrobang here>). That really needs to be fixed. Again, you probably need a serious upgrade to your PgSQL server because it apparently doesn't have enough space to hold more than a couple of hours of WALs (unless this was just a naive misconfiguration of the [min|max]_wal_size parameter, like the max_connections parameter?). I understand that transaction logs can get very large, but the disk needs to accommodate (usually a second disk array is used for WALs to ease write impact) and replication lag needs to be monitored and alarmed on. There were a few other things (including someone else downthread who pointed out that your CEO re-revealed your DB's hostnames in this write-up, and that they're resolvable via public DNS and have running sshds on port 22), but these are the big standouts for me. P.S. bonus point, just speculative: Not sure how fast your disks were, but 300GB gone in "a few seconds" sounds like a stretch. Some data may've been recoverable with some disk forensics. Especially if your Postgres server was running at the time of the deletion, some data and file descriptors also likely could've been extracted from system memory. Linux doesn't actually delete files if another process is holding their handle open; you can go into the /proc virtual filesystem and grab the file descriptor again to redump the files to live disk locations. Since your database was 400GB and too big to keep 100% in RAM, this probably wouldn't have been a full recovery, but it may have been able to provide a partial. The theoretically best thing to do in such a situation would probably be to unplug the machine ASAP after ^C (without going through formal shutdown processes that may try to "clean up" unfinished disk work), remove the disk, attach it to a machine with a write blocker, and take a full-disk image for forensics purposes. This would maximize the ability to extract any data that the system was unable to eat/destroy. In theory, I believe pulling the plug while a process kept the file descriptor open should keep you in reasonably good shape, as far as that goes after you've accidentally deleted 3/4 of your production database. The process never closes and the disk stops and the contents remain on disk, just pending unlink when the OS stops the process (this is one reason why it'd be important to block writes to the disk/be extremely careful while mounting; if the journal plays back, it may destroy these files on the next boot anyway). But someone more familiar with the FS internals would have to say definitively if it works that way or not. I recognize that such speculative/experimental recovery measures may have been intentionally forgone since they're labor intensive, may have delayed the overall recovery, and very possibly wouldn't have returned useful data anyway. Mentioning it mainly as an option to remain aware of.

dustinmoris 10y ago

Watching GitLab is somewhat painful. I feel like they make every possible mistake you could do as an IT startup and because they are transparent about it people seem to love the fact that they screw up all the time. I don't know if I share the same mentality, because at the end of the day I don't trust GitLab even with the simplest task, let alone any valuable work of mine. It's good to be humble and know that mistakes can happen to anyone and learn from it, etc., but when you do in 2017 still the same stupid mistakes that people did a million times since 1990 and it's all well documented and there's systems built to avoid these same basic mistakes and you still do them today then I just think it cannot be described any different than absolute stupidity and incompetence. I know they have many fans who just look past every mistake no matter how bad it was only because they are open about it, but common, this is now just taking the piss no?

egwor 10y ago
I had exactly the same feeling. One of my friends was a doctor at a hospital and there was a serious mistake which my friend reported to the consultant. The consultant made a good point that it was never just one mistake that caused the serious mistake; it was a series of smaller mistakes that hadn't been checked or addressed. (the argument was that there were processes to avoid these kinds of mistakes) If you read the post they also noted that they were also in the process of accidentally removing one of their own employees which make me think that there are other problems going on here. Maybe part of the problem is that I think that the industry has a problem that it isn't willing to learn from the experiences of others? (I feel like we have 'just enough learning' and 'experienced folk who raise concerns are considered stuck in their outdated ways' and 'people who make a silly mistake like that must be an idiot'). I think that since we clump together those that have had formal training with those that haven't we aren't encouraging the value of this education. I'm also fully aware that some self-learned developers are much more competent rather than a college educated developer.
- dustinmoris 10y ago
  I don't actually think it has anything to do with education. I think it really comes down to common sense. Any half intelligent engineer would always first research good practices, pitfalls and existing information which has been gathered from decades of other experienced engineers before doing anything stupid on their own. It seems that GitLab is lacking this attitude.

Achshar 10y ago

Does anyone have a link to the YouTube stream they's talking about? Can't seem to find it on their channel. And the link in the doc is redirecting to the live link [1] which doesn't list the stream. [1] - https://www.youtube.com/c/Gitlab/live https://www.youtube.com/c/Gitlab/live

cmatija 10y ago
Yup, Thanks for taking the interest to check it out. It's an unlisted YT video, so that's why it might be hard to find. Here it is: https://www.youtube.com/watch?v=nc0hPGerSd4 https://www.youtube.com/watch?v=nc0hPGerSd4
- Achshar 10y ago
  Thanks! I'll check it out.
deleted 10y ago
[deleted]

ancarda 10y ago

>Unfortunately DMARC was not enabled for the cronjob emails, resulting in them being rejected by the receiver. This means we were never aware of the backups failing, until it was too late. At my dayjob, we gradually stopped using email for almost all alerts, instead we have several Slack channels like #database-log where errors to MySQL go. Any cron jobs that fail post in #general-log. Uptime monitoring tools post in #status. So on... Email has so much anti-spam stuff like DMARC that make it less reliable your mail will be delivered. For something failing like a backup or database query, it's too important to have potentially not reach someone who can make sure it gets fixed. My 2 cents.

gl- 10y ago
This is a step in the right direction but still misses a big part of it IMO: push versus pull notifications. If the agent stops functioning correctly or someone makes a config change, the alerts just stop and no one notices. At the very least you want some kinda dead-man's switch that gets pissed if it's seen no events in the last x amount of time. Ideally you want to be polling the box in a stateful way; although with ephemeral nodes & flexible infra being all the rage that's fallen to the side a bit lately.
- ancarda 10y ago
  Absolutely, that's a great idea! You could also check for evidence a run has been successful, although that does depend on what you're doing exactly. For our backup system, we're going to build an audit cron job on our main server that checks all our Azure containers to see if each server has pushed a file lately. It'll alert us if a file hasn't been uploaded in a few days or if it's smaller than a few MB (which is suspiciously small; we'd expect a few hundred MB for mysqldump+files).
mkopinsky 10y ago
How do you set cron to post to Slack?
- ancarda 10y ago

aabajian 10y ago

This is an outstanding writeup, but I wonder if it glosses over the real problem: >>The standby (secondary) is only used for failover purposes. >>One of the engineers went to the secondary and wiped the data directory, then ran pg_basebackup. IMO, secondaries should be treated exactly as their primaries. No operation should be done on a secondary unless you'd be OK doing that same operation on the primary. You can always create another instance for these operations.

jsingleton 10y ago

TIL GitLab runs on Azure. If your CI servers or deployment targets are also on Azure then the latency should be pretty low (assuming you get the correct region). Good to know. I moved from AWS to Azure years ago. Mainly because I run mostly .NET workloads and the support is better. I've recently done some .NET stuff on AWS again and am remembering why I switched.

XorNot 10y ago

The backup situation stands out to me as a problem no one has really adequately solved. Verifying a task has happened in a way where the notifications are noticed is actually a really hard problem that it feels like we collectively ignore in this business. How do you reliably check if something didn't happen? Is the backup server alive? Did the script work? Did the backup work? Is the email server working? Is the dashboard working? Is the user checking their emails (think: wildcard mail sorting rule dumping a slight change in failure messages to the wrong folder). And the converse answer isn't much better: send a success notification...but if it mostly succeeds, how do you keep people paying attention to it when it doesn't (i.e. no failure message, but no success message)? The best answer I've got, personally, is to use positive notifications combined with visibility - dashboard your really important tasks with big, distinctive colors - use time based detection and put a clock on your dashboard (because dashboards which mostly don't change might hang and no one notice).

AlexCoventry 10y ago

Thank you for this informative postmortem and mitigation outline. Are any organizational changes planned in response to the development friction which led to the outage? It seems to have arisen from long-standing operational issues, and an analysis of how prior attempts to address those issues got bogged down would be very interesting.

dancryer 10y ago

Can't help but notice that the new backup monitoring tool suggests that the latest PGSQL backup is almost six days old... Is that correct? http://monitor.gitlab.net/dashboard/db/backups?from=1485941950082&to=1486978750114 http://monitor.gitlab.net/dashboard/db/backups?from=14859419...