31 ms·
GitLab Database Incident – Live Report
- nickjj 10y agoI wonder if this could have been avoided by using different subdomains. For example, instead of db1.cluster, what if it were named db-production.cluster. Would it have still happened? Probably not. I could totally see myself accidentally typing db1 out of muscle memory, but there's no way to accidentally type db-production.
- nacs 10y ago> there's no way to accidentally type db-production That's what auto-complete is for. rm -rf db<tab><tab><enter>
- why-el 10y agoThat's what we do. `service-1-dev` and `service-1-prod`. Can't ssh to `prod` unless you are a really really qualified devops.
- wolfgang42 10y agoThey already do this. Both db1.cluster and db2.cluster are production machines; their staging equivalents are db1.staging and db2.staging. The confusion was between two production instances--one of which had the latest data, and the other was replicating. The intent was to delete the partially replicated data but the command was run on the current master database server instead.
- sciurus 10y agoThis is why you have lots of copies of your data. Matt Raney has a great talk about designing for failure that includes details on Uber's "worst outage ever." It too involved postgresql replication and mistaking on host for another, but they didn't lose any data because they had more than a dozen live copies of their database. https://www.youtube.com/watch?v=bNeZYVIfskc&t=26m54s https://www.youtube.com/watch?v=bNeZYVIfskc&t=26m54s This isn't an alternative to working backups, of course, but it is an additional safety net. Plus it can give you a lot more options when handling an incident.
- yellowbeard 10y agoMore copies / replicas also means a larger attack surface.
- CameronBanga 10y agoNot sure if the doc here is refreshing or scary. But Godspeed GitLab team. I've loved the product for about two years now, so curious to see how this plays out.
- sbuttgereit 10y agoIt's both. I very much appreciate their forthrightness and the way they conduct their company generally. Having said that, I have the code I work on, related content, and a number of clients on the service. [edit for additional point] They need the infrastructure guy they've been looking for sooner than later. I hope there's good progress on that front.
- sytse 10y agoWe've hired some great new people recently but as you can see there is still a lot work to do. https://about.gitlab.com/jobs/production-engineer/ https://about.gitlab.com/jobs/production-engineer/
- Perihelion 10y agoThere's also this one if anyone is interested in helping out with the DB side of things: https://about.gitlab.com/jobs/specialist/database/ https://about.gitlab.com/jobs/specialist/database/
- mrmondo 10y agoI've just DM'd you in twitter with some PostgreSQL advice ;)
- mrmondo 10y agoNote for the person that downvoted my comment: GitLab are fantastic, I'm a big advocate for their product & support. My comment to Sid was in support of helping based on some of the notes I saw in their very transparent report, Sid & I have talked several times in the past & I have quite a bit of PostgreSQL experience - so my comment was a positive one offering support when / if needed, not a negative / piss take if it came off as such.
- TimWolla 10y agoAre 'YP' the initials of an employee or is this an acronym I don't know?
- Perihelion 10y agoYes, those are the initials of an employee here. Sorry for the confusion!
- detaro 10y agoAs much as I appreciate GitLabs extreme openness, that's maybe something that by policy shouldn't be part of published reports. Internal process is one thing, if something goes really bad customers might not be so good at "blameless postmortems" if they have a name to blame.
- sytse 10y agoThat is why we went with initials. And I hope customers understand the blame is with all of us, starting with me. Not with the person in the arena. https://twitter.com/sytses/status/826598260831842308 https://twitter.com/sytses/status/826598260831842308
- grhmc 10y agoIt seems to me that, as a customer, it is blame-shifting away from the company to a particular person. Blameless post-mortems are great, but when speaking to people outside the company I think it is important to own it collectively, "after a second or two we notice we ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com." I believe this isn't your intention, but that is how I interpreted it.
- greenleafjacob 10y agoIn our postmortems we explicitly avoid referring to names and only refer to "engineers" or specific teams. There is no reason to refer to specific names if your intention is a systems/process fix.
- polygot 10y ago"So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place."
- deleted 10y ago[deleted]
- echelon 10y agoDoes this mean whatever was in that database is gone, with no available backups? Is this an SOA where important data might lie in another service or data store, or is this a monolithic app and DB that is responsible for many (or all) things? What was stored in that database? Does this affect user data? Code?
- YorickPeterse 10y agoWe have snapshots, but they're not very recent (see the document for more info). The most recent snapshot is roughly 6 hours old (relative to the data loss). The data loss only affects database data, Git repositories and Wikis still exist (though they are fairly useless without a corresponding project).
- echelon 10y agoBest of luck with the recovery! I know this must be stressful. :(
- TimWolla 10y agoThe doc says that there is a LVM snapshot being 6 hours old. <strike>And there should be a regular logical backup with at most 24 hours age as well (they just can't find it for whatever reason).</strike> (Scratch that, my doc did not update, despite Google saying it should automatically update). Regarding what's gone: The production PostgreSQL database. This suggests that the code itself is fine, but the mappings to the users are gone. But git is a distributed VCS after all, so all the code should be on the developer's machines as well.
- ams6110 10y ago23:00-ish YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com Good lesson on the risks of working on a live production system late at night when you're tired and/or frustrated.
- eric_h 10y agoGood lesson on making command prompts on machines always tell you exactly what machine you're working on.
- m_fam_wa_k 10y agoI like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.
- joshaidan 10y agoHow do you go about colour coding your terminal?
- TimWolla 10y agoI assume he color coded the prompt. You can use ANSI color escape codes in there to e.g. color your hostname. Here's a generator for Bash: http://bashrcgenerator.com/ http://bashrcgenerator.com/, the prompt's format string is stored in the $PS1 variable.
- TimWolla 10y agoIn this case it looks like it has been a confusion between two different replicated Production databases. So this would not have helped.
- deleted 10y ago[deleted]
- 10y ago
- mnarayan01 10y agoStart at: > At this point frustration begins to kick in. Earlier this night YP explicitly mentioned he was going to sign off as it was getting late (23:00 or so local time), but didn’t due to the replication problems popping up all of a sudden. This is why I'm not a fan of emergency pager duty.
- leesalminen 10y agoThis is the stuff my nightmares consist of after 900 consecutive days of being on call (and counting).
- slezakattack 10y agoAre you a one man team or...? My wife would probably leave me if I was on-call for that long.
- leesalminen 10y agoI am :/ ... currently maintaining service for 500 high-volume businesses 24x7x365 in 9 timezones. Luckily the product and infrastructure is pretty stable and problems occur maybe once a quarter. But the constant nagging in the back of your head that shit can go wrong at any second is draining and has been the biggest stressor in my life for a long time now. My S.O. still gets mildly upset when I pack up the laptop on our way out to a fancy dinner, or disappear with my laptop when visiting her parents, but the fact that our life goals are aligned is the saving grace of all these situations. We both know what we want out of the next 5 years of our lives and are willing to sacrifice to achieve this goal (long term financial security).
- overcast 10y agoI hope you are being SERIOUSLY compensated.
- leesalminen 10y agoCash salary today is well under market for my skill set. But, I do own 1/3 of the company so it's not all bad :).
- overcast 10y agoWith all of that clout, why aren't you hiring?
- gizmo 10y agoThis is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize the steps. If there is no coworker, talk to your rubber ducky or stapler on your desk. This will help you catch mistakes. Then when the entire plan looks sensible, go through the steps one by one. Don't deviate from the plan. Don't get distracted and start switching between terminal windows. While making the checklist ask yourself if what you're doing is A) absolutely necessary and B) risks making things worse. Even when the angry emails are piling up you can't allow that pressure to cloud your judgment. Every startup has moments when last-minute panic-patching of a critical part of the server infrastructure is needed, but if you use a checklist you're not likely to mess up badly, even when tired.
- connorshea 10y agoYup, it's never the fault of a person, always of the system. Once we get this resolved we'll definitely look at ways to prevent anything like it in the future.
- deleted 10y ago[deleted]
- duncan_bayne 10y agoWishing you the best of luck partly out of sympathy, and partly as an active user wanting my account operative again :)
- wereHamster 10y agoAt some point you have to blame the person. If the person did it wilfully, deliberately, ignoring the checks and safeguards. Or because there's already so much system that having any more of it would place severe restrictions on everyday tasks. I'm not saying that's the case here, because it does seem that GitLab has systemic deficiencies. But "never" and "always" are such strong statements. Prosecutor: Mr. Accused, here is evidence that you murdered that other person. Accused: It's not my fault, but the systems'. Judge: Oh, ok. You are a free man.
- ploxiln 10y agoAmazingly transparent and honest. Unfortunately, this kind of situation, "only the ideal case ever worked at all", is not uncommon. I've seen it before ... when doing things the right way, dotting 'I's and crossing 'T's, requires an experienced employee a good week or two, it's very tempting for a lean startup to bang out something that seems to work in a couple days and move on.
- Walkman 10y agoSeems like very basic mistakes were made, not at the event but way long before. If you don't test to restore your backups, you don't have a backup. How does it go unnoticed that S3 backups don't work for so long?
- tetrep 10y agoYeah, the "You don't have backups unless you can restore them" stikes again. Virtually the only way to lose data is to not have backups. We live in such fancy times that there's no reason to ever lose data that you care about.
- koolba 10y agoNot "can restore them", it's "have restored them". Best way to ensure that is to have backup restoration be a regularly scheduled event. For most apps I work on, that's either daily or (worst case) weekly, with prod being entirely rebuilt in a lower environment. Works great for creating a test lane too!
- mschuster91 10y ago> How does it go unnoticed that S3 backups don't work for so long? My uneducated guess (this one hit a friend of mine): expired/revoked AWS credentials combined with a backup script that doesn't exit(1) on failure and just writes the exception trace to stderr.
- koolba 10y agoI bet it writes to a log file. It just doesn't alert anyone on failure so the log just grows and grows daily with the same error.
- marcosdumay 10y agoOr it alerts people, but on the same channel every other piece of infrastructure alerts them, and they have a severe case of false positives. I've seen that many more times than I've seen the "no alert" option.
- ArtDev 10y agoI noticed the issue when I was pushing code earlier today. Hopefully this gets resolved soon. You guys are doing a great job. Keep up the good work!
- sytse 10y agoThanks, not feeling great about the job we're doing today, but we'll learn from this.
- jdhendrickson 10y agoMore than loving your service we like you guys as people and trust you to get things straightened out. I wouldn't host my code with anyone else despite this.
- knocte 10y agoLosing 6 hours of data is not a big deal! Kudos!
- sytse 10y agoIt is a big deal to us. It should never be more than an hour and we should be able to recover much quicker. But thanks for the kudos.
- antocv 10y agoThe way you are open about this and do not blame the engineer who did "rm -Rvf" (he knows he fucked up, suffers enough already), and seeing that improvements can be made and you're willing to do them. Applying for work with you now, and moving all my stuff to GitLab.
- sytse 10y agoThanks for your interest in working at us.
- mahnunchik 10y ago
- graphememes 10y agoIf you haven't tested your backups, you don't have backups.
- piinbinary 10y agoIf you haven't done so recently, TEST YOUR BACKUPS.
- sashk 10y agoBackups sucked for the starting in 8.15 on our instances of GLE, because someone decided to add "readable" date stamp in addition to unix timestamp in backup file name without proper testing, which caused many issues. It was somewhat fixed, but I do still issues in 8.16. I'm not complaining, but backup/restore is important part, with 100% test coverage and daily backup/restore runs.
- btgeekboy 10y ago> Our backups to S3 apparently don’t work either: the bucket is empty followed by > So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. is no way to be running a public service with customer data. Did the person who set up that S3 job simply write a script or something and just go "yep, it's done" and walk away? Seriously?
- ChuckMcM 10y ago> Did the person who set up that S3 job simply write a > script or something and just go "yep, it's done" and > walk away? I don't know of course, but one failure mode that has to be explicitly tested for is continual monitoring that the existing backup process is still working. We had a backup process at Blekko which stopped working once when an S3 credential that appeared unrelated was removed, as I recall it was a Nagios test that detected that the next set of backups were too small and got that fixed.
- problems 10y agoYeah, the only solution is to test your backups - either automatically or manually through failure. My small business uses a global network of cheap servers to provide low latency to customers, so we have encountered significant lengthy failures of disk or network once every few months, but hey, we can be damn sure our backups and deploy scripts work because we're forced to restore them all too often.
- Elizzy 10y agoTo be fair the script is probably the same as what is in their repos, it probably wasn't set to report to a logging service that'd raise red flags.
- overcast 10y agoApparently the following insane interviewing process, wasn't enough to find someone competent enough to cover the basics. https://about.gitlab.com/jobs/production-engineer/ https://about.gitlab.com/jobs/production-engineer/ ------------------- Applicants for this position can expect the hiring process to follow the order below. Please keep in mind that applicants can be declined from the position at any stage of the process. To learn more about someone who may be conducting the interview, find her/his job title on our team page. Qualified applicants receive a short questionnaire and coding exercise from our Global Recruiters The review process for this role can take a little longer than usual but if in doubt, check in with the Global recruiter at any point. Selected candidates will be invited to schedule a 45min screening call with our Global Recruiters Next, candidates will be invited to schedule a first 45 minute behavioral interview with the Infrastructure Lead Candidates will then be invited to schedule a 45 minute technical interview with a Production Engineer Candidates will be invited to schedule a third interview with our VP of Engineering Finally, candidates will have a 50 minute interview with our CEO Successful candidates will subsequently be made an offer via email
- ChuckMcM 10y agoSo in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. Ouch. That is so harsh. Sorry to hear about the incident. Testing ones backups can be a pain to do but it is so very important.
- jongar_xyz 10y agoWow. This is a joke. Gitlab is a joke. Just moved my stuff to BitBucket.
- huula 10y agoI once rm-ed my home directory when I was writing and testing a script, but turned out the stuff like .m2, .ivy2 are huge and they are the first ones by default to be deleted by 'rm -rf'. So they kind of gave me some buffering time to figure out that something was wrong.
- niftich 10y agoI applaud their forthrightness and hope that it's recoverable so that most of the disaster is averted. To me the most illuminating lesson is that debugging 'weird' issues is enough of a minefield; doing it in production is fraught with even more peril. Perhaps we as users (or developers with our 'user' hat on) expect so much availability as to cause companies to prioritize it so high, but (casually, without really being on the hook for any business impact) I'd say availability is nice to have, while durability is mandatory. To me, an emergency outage would've been preferable to give the system time to catch up or recover, with the added bonus of also kicking off the offending user causing spurious load. My other observation is that troubleshooting -- the entire workflow -- is inevitably pure garbage. We engineer systems to work well -- these days often with elaborate instrumentation to spin up containers of managed services and whatnot, but once they no longer work well we have to dip down to the lowest adminable levels, tune obscure flags, restart processes to see if it's any better, muck about with temp files, and use shell commands that were designed 40 years ago for when it was a different time. This is a terrible idea. I don't have an easy solution for the 'unknown unknowns', but the collective state of 'what to do if this application is fucking up' feels like it's in the stone ages compared to what we've accomplished on the side of when things are actually working.
- zenlikethat 10y agoEngineering things to work well and the troubleshooting process at a low level are one and the same. It's just that in some cases other people found these bugs and issues before you and fixed them. But this is the cost of OSS, you get awesome stuff for free (as in beer) and are expected to be on the hook for helping with this process. If you don't like it, pay somebody. Really everyone could benefit from learning more about the systems they rely on such as Linux and observability tools like vmstat, etc. The less lucky guesses or cargo culted solutions you use the better.
- chatmasta 10y agoBe careful not to overlook the benefits of instrumentation even in the "unknown unknowns" scenario. If you implement it properly, the instruments will alert you to where the problem is, saving you time from debugging in the wrong place. The initial goal of instrumentation should be to provide sufficient cover to a broad area of failure scenarios (database, network, CPU, etc), so that in the event of a failure, you immediately know where to look. Then, once those broad areas are covered, move onto more fine-grained instrumentation, preferably prioritized by failure rates and previous experience. A bug should never be undetectable a second time. As a contrived example, it was "instrumentation," albeit crudely targeted, that alerted GitLab the problem was with the database. This instrumentation only pointed them to the general area of the problem, but of course that's a necessary first step. Now that they had this problem, they can improve their database-specific instrumentation and catch the error faster next time.
- ckdarby 10y ago"Our backups to S3 apparently don’t work either: the bucket is empty" 6/6 failed backup procedures. Looks like they are going to be hiring a new sysadmin/devops person...
- Washuu 10y agoThe best system administrator is the one that has learned from their catastrophic fuck up. To that effect, I still have the same job as I did before I ran "yum update" without knowing it attempts to do in place kernel upgrades. Which resulted in a corrupted RedHat installation on a server we could not turn off.
- overcast 10y agoThere is learning from a catastrophic fuck up, and then there is incompetence. Backups is like Day 1, SysAdmin 101. I can't quite grasp how so many different backup systems were left unchecked. Every morning I receive messages saying everything is fine, yet I still go into the backup systems, to make sure they actually did run. In case there was issue with the system alerting me.
- wtbob 10y ago> There is learning from a catastrophic fuck up, and then there is incompetence. We all start at incompetence, but eventually we — wait for it — learn from our experiences. Would you believe that Caesar, Michael Jordan and Steve Wozniak once were so incompetent that they couldn't even control their bowels or tie their shoes? They learned. Is it possible that the guys in the team running GitLab's operations were misplaced? Certainly — that's a management issue. And I can guar-an-tee you that GitLab now has a team of ops guys who viscerally understand the need for good backups: they'd be insane to disperse that team to the winds.
- overcast 10y agoThere's no excuse for backups not being setup, period. For such a high profile site, and the rigorous hiring circus they put candidates through. This doesn't fall under "a learning experience". I wish them luck, but this is just gross negligence.
- zenlikethat 10y agoHang in there GitLab, it sucks now but mistakes like this can happen to anyone. The important thing is how you deal with it.
- tony-allan 10y agoIt's good when companies are open and honest about problems. I imagine they will have a great multi-level tested backup process in the next day or two!
- connorshea 10y agoIt'll definitely be a priority now!
- elementalest 10y ago>1. LVM snapshots are by default only taken once every 24 hours. YP happened to run one manually about 6 hours prior to the outage >2. Regular backups seem to also only be taken once per 24 hours, though YP has not yet been able to figure out where they are stored. According to JN these don’t appear to be working, producing files only a few bytes in size. >3. SH: It looks like pg_dump may be failing because PostgreSQL 9.2 binaries are being run instead of 9.6 binaries. This happens because omnibus only uses Pg 9.6 if data/PG_VERSION is set to 9.6, but on workers this file does not exist. As a result it defaults to 9.2, failing silently. No SQL dumps were made as a result. Fog gem may have cleaned out older backups. >4. Disk snapshots in Azure are enabled for the NFS server, but not for the DB servers. The synchronisation process removes webhooks once it has synchronised data to staging. Unless we can pull these from a regular backup from the past 24 hours they will be lost The replication procedure is super fragile, prone to error, relies on a handful of random shell scripts, and is badly documented >5. Our backups to S3 apparently don’t work either: the bucket is empty >So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. Sounds like it was only a matter of time before something like this happened. How could so many systems be not working and no one notice?
- gizmo 10y agoWhat if I told you all of society is held together by duct tape? If you're surprised that startups cut corners you're in for a rude awakening. I'm frequently amazed anything works at all.
- cnst 10y agoStartups only, you say? Everything, everywhere, is held together by ducttape!
- saosebastiao 10y agoTrue. I watched a 60 year old manufacturing plant shut down for 7 days once because someone saved an excel spreadsheet in the wrong format and it deleted a macro written more than a decade prior that held together all of their operations planning.
- alyandon 10y agoThank you for the transparency. This is a good read and I'm going to be sharing it with coworkers tomorrow. :)
- connorshea 10y agoGlad to hear others will learn from our mistakes, we certainly are :)
- jeffmcjunkin 10y agoFor item 3h under recovery, consider: chattr +i /var/opt/gitlab/postgresql/data Yes, it doesn't completely stop foot-guns, but it means you have to shoot twice [0]. [0]: chattr -i /whatever rm /whatever
- artursapek 10y agoDoes that prevent postgres from modifying it?
- ocdtrekkie 10y agoFirst of all, I want to say: I think the GitLab people are incredible. They're open and transparent about everything they do, and I think that in a world dominated by GitHub, the fact that GitLab exists is hugely important to decentralization and the continued march of open source. This document continues to demonstrate how the GitLab people are great, transparent people. That being said, this is why you shouldn't entrust a cloud service to keep your data safe: "So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place." My backups work. I know they work, because I run them and I test them. People entrusting cloud services to have good backups cannot say that.
- dorianm 10y ago> 2017/01/31 23:00-ish > YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. > After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com > 2017/01/31 23:27 > YP terminates the removal, but it’s too late. Of around 310 GB only about 4.5 GB is left The naming couldn't be more confusing
- richardwhiuk 10y agoIn what way? They have two production database servers, db1 and db2.
- kkirsche 10y agoWhile this is sad to see it is a lesson to us all and I've shared it with coworkers who haven't taken disaster recovery as seriously as we should on our projects. Hopefully this will help raise this as a priority.
- technion 10y agoa user for using a repository as some form of CDN, resulting in 47 000 IPs signing in using the same account I'd be interested in how this occurs. Simply linking a raw file in a repository would surely not require a sign in. Did someone come up with some way of automatically using credentials on a download link? 47 000 simultaneous users suggests that wouldn't be a small project that did so.
- justinclift 10y agoAs a complete guess, something like using sessions persisted back to the PostgreSQL database, without something like memcached in front of it. With that kind of approach it could be trying to update a session table (with new IP address?) for literally every page load by the 47,000 people. Which would probably suck. ;)
- totally 10y agoI had a great manager a little while back who said they had an expression in Spain: > "The person who washes the dishes is the one who breaks them." Not, like, all the time. But sometimes. If you don't have one of these under your belt, you might ask yourself if you're moving too slow. If that didn't help, he would also point out: > "This is not a hospital." Whatever the crisis, and there were some good ones, we weren't going to save anyone's life by running around. Sure, data loss sucks, but nobody died today because of this. I really appreciate the raw timeline. I feel your pain. Get some sleep. Tomorrow is a new day.
- drewmate 10y ago> Get some sleep. Definitely get sleep, but it would be nice if the site were back online before that. I actually just created a new GitLab account and project a couple days ago for a project I needed to work on with a collaborator tonight. This is not a good first impression.
- sidlls 10y agoPaid or unpaid account and project?
- deleted 10y ago[deleted]
- cs2818 10y agoWow, very intriguing to read. I appreciate honest event descriptions like this. This has reminded me how important it is to perform regular rehearsals of data recovery scenarios. I'd much rather find these failures in a practice run. Thanks to GitLab for continuing to openly share their experience.
- crankshift 10y agoFirst they censored that list of vulnerable ecommerce sites, now they're getting pwnd... Are there any git hosts that are not PC obsessed and complete garbage?
- jtchang 10y agoYou know this actually makes me want to try out gitlab. They are super transparent and the fact that they ran into this issue now means they will just be better off in the long run. Does it suck? Sure. But this is why you backup your own data anyway.
- dboreham 10y agoWell...never delete. Rename the directory someplace else so you can get it back if the deletion was a bad plan. Also helpful to make the window background color different, or some other highly conspicuous visual difference when working on several very similar production machines.
- DanielDent 10y agoI'm a huge Gitlab fan. But I long ago lost faith in their ability to run a production service at scale. Nothing important of mine is allowed to live exclusively on Gitlab.com. It seems like they are just growing too fast for their level of investment in their production environment. One of the only reasons I was comfortable using Gitlab.com in the first place was because I knew I could migrate off it without too much disruption if I needed to (yay open source!). Which I ended up forced to do on short notice when their CI system became unusable for people who use their own runners (overloaded system + an architecture which uses a database as a queue. ouch.). Which put an end to what seemed like constant performance issues. It was overdue, and made me sleep well about things like backups :). A while back one of their database clusters went into split brain mode, which I could tell as an outsider pretty quickly... but for those on the inside, it took them a while before they figured it out. My tweet on the subject ended up helping document when the problem had started. If they are going to continue offering Gitlab.com I think they need to seriously invest in their talent. Even with highly skilled folks doing things efficiently, at some point you just need more people to keep up with all the things that need to be done. I know it's a hard skillset to recruit for - us devopish types are both quite costly and quite rare - but I think operating the service as they do today seriously tarnishes the Gitlab brand. I don't like writing things like this because I know it can be hard to hear/demoralizing. But it's genuine feedback that, taken in the kind spirit is intended, will hopefully be helpful to the Gitlab team.
- CaliforniaKarl 10y agoIf I understand correctly, the public Gitlab is similar to what you can get with a private Gitlab instance. That makes me wonder, instead of trying to scale the one platform up, would it be OK to spin up a second public silo? I mean yeah, it would be a different silo, but for something free I'd say "meh". I think it's totally fine admitting when you've stopped being able to scale up, and need to start scaling out.
- DanielDent 10y agoThey could, and as a stopgap measure that might work, but.. (1) Some of the collaboration features (e.g. work on Gitlab itself) depend on having everyone on the same instance. (2) Gitlab.com gives them a nice dogfood-esque environment for what it's like to actually operate Gitlab at scale. If they are having problems scaling it, then potentially so are their customers. Fixing the root cause is usually a good thing and is often an imperative to avoid being drowned in technical debt. (3) It moves the problem in some respects. Modern devops techniques mostly allow the number of like servers to be largely irrelevant, but still.. the more unique instances of Gitlab, the more overhead there will be managing those instances (and figuring out which projects/people go on which instances). It's a simple approach which I'm sure would work, but it also means a bunch of new problems are introduced which don't currently exist.
- nummy 10y ago6 hour rewind is really not that bad, even if it was 24 hours, that's still really good for the scope.
- CaliforniaKarl 10y agoFirst off, my heartfelt commiserations for the GitLab team here. My suggestion: Start watching an hour-long video; the rsync will finish right when it gets to the good part! I wonder if a future project might be to have the DB-stored stuff use Git as a replication back-end. Like, for example, having each issue be a directory, and individual comments be JSON files. It would never (normally) be the data store "of record" (the DB would), but maybe that would work as a backup?
- jamesmiller5 10y agoGerrit is working on just this. It uses `refs/meta/config` branch for project configuration and is moving its database dependencies into a git database. Reviews are stored in refs/changes/* . Backing-up a project & verifying it's integrity is simple as `git clone --mirror`
- xfactor973 10y agoI'm surprised they didn't take any hourly backups.
- umbrai_nation 10y ago"Of around 310 GB only about 4.5 GB is left" What is gitlab storing in their database? From what I understand, the repos were untouched by the DB problems, so what is taking up a third of a terabyte of DB space?
- YorickPeterse 10y agoissues, merge requests (titles, descriptions, etc), comments, events ("X pushed to Y"), labels, projects, milestones, users, permissions, project statistics, CI builds information, abuse reports, the list goes on.
- X-Istence 10y agoTickets, tags, labels, users information/data and more. Sounds like they also host snippets out of the PostgreSQL database. 310 GB for a database is not all that large.
- neandrake 10y agoThe database might have tables for commit data so history etc. can be searched, and possibly used for managing on-server operations like merges etc.
- soheil 10y agobitbucket offers free private repos.
- knocte 10y agogitlab too, plus they are opensource
- coding123 10y agoWe're using the Gitlab CE for our internal use, but one of our customer's is on Gitlab.com. We're likely still going to recommend Gitlab.com after this, but also enable mirroring to our internal instance. As a side note, I just checked our S3 Gitlab backup bucket and it does have backups for every day for the last year (1.8 GB each yikes) so instead of failing to create the backups, its actually failing to delete the older ones! :)
- bm5k 10y agoGitlab's automagic backup cleaner specifically does not work for S3 backups. To automatically clean up your s3 bucket use S3's built in lifecycle settings.
- nanch 10y agoI remember when we accidentally deleted our customers' data. That was the worst feeling I ever had running our business. It was about 4% of our entire storage set and had to let our customers know and start the resyncs. Those first 12 hours of panic were physically and emotionally debilitating - more than they have the right to be. I learned an important lesson that day: Business is business and personal is personal. I remember it like it was yesterday, the momement I conciously decided I would no longer allow business operations determine my physical health (stress level, heart rate, sleep schedule). For what it's worth, it was a lesson worth learning despite what seemed like catastrophic world-ending circumstances. We survived, and GitLab will too. GitLab has been an extraordinary service since the beginning. Even if their repos were to get wiped (which seems not to be the case), I'd still continue supporting them (after I re-up'd from my local repos). I appreciate their transparency and hope that they can turn this situation into a positive lesson in the long run. Best of luck to GitLab sysops and don't forget to get some sleep and relax.
- derricgilling 10y agoI like to always try to automate stuff as much as possible to remove human error. It's easy to forget an arg or other parameter on a script or even know what the arg was for in the first place. Sounds like their last backup was 24 hrs ago. Having backups are like having good security, you don't realize how important they are until it's too late. Reminds me of this old meme: http://www2.rafaelguimaraes.net/wp-content/uploads/2015/12/giphy2.gif http://www2.rafaelguimaraes.net/wp-content/uploads/2015/12/g...
- duckbill 10y agohttps://memegenerator.net/instance/75228340 https://memegenerator.net/instance/75228340
- yjlim5 10y agoMy team and I switched from bitbucket to gitlab a few months ago and we love the transition. Gitlab provides a lot of value to me and my team who are learning how to code while working on side projects. Although we cannot send merge requests today because of this issue, we are all cheering them on. I’m very happy that they are so transparent about their issues because my team and I are learning so much from their report and insights here on HN comments. Good luck!
- eblanshey 10y agoWould you mind explaining what you like about gitlab better than bitbucket? They seem to be on par with each other, including integrated CI.
- 0x0 10y agoFor me, privately self-hosted CI runners was a killer feature. Bitbucket only seemed to offer limited-minutes shared runners when I looked.
- yjlim5 10y agoOne major reason was that our team was growing and Gitlab allowed us to add them, whereas BitBucket was limited to 5 devs.
- songzme 10y agoI like gitlab's issue tracking. Its faster (from a productivity standpoint) and easier to manage compared to Jira + Bucket, which feels a bit too bloated.
- anonfunction 10y agoYou don't have a production ready backup system until you've used it.
- fsiefken 10y agoYes, when I was responsible for databases and servers 'red alert' file where all the worst case scenario's were described with the recovery procedures which were tested every half year or so. This came about after these scenario's happened one after the other and I had to fix them manually. One of them, a hard disk crash, were a tape backup failed for some reason, and the tape from two days ago was to outdated. I did a filesystem recovery, then a database recovery and consistency check, mounted the files and looked at which tables were corrupted and restored those from backup. I didn't want to through this (with a huge time pressure) ever again. After that we decided to check if all backups succeeded and were consistent so everyone on the team could restore within an hour.
- jincheker 10y agoIncidents after incidents, feels like a toy by new grads.
- jlgaddis 10y agoApparently the BOFH works for Gitlab these days: > It's backup day today so I'm pissed off. Being the BOFH, however, does have it's advantages. I reassign null to be the tape device - it's so much more economical on my time as I don't have to keep getting up to change tapes every 5 minutes. And it speeds up backups too, so it can't be all bad can it? Of course not. --bofh, episode #1
- riebschlager 10y agoI'm another happy GitLab user, but things like this always kinda freak me out. Do any of you use any repo-mirroring strategy? Something a little more automated than pushing to and maintaining separate remotes? For example, would it be worth it to spin up a self-hosted GitLab instance and then script nightly pulls from GitLab.com? Edit: Answered my own question! If anyone else was curious: http://stackoverflow.com/questions/14288288/gitlab-repository-mirroring http://stackoverflow.com/questions/14288288/gitlab-repositor...
- louiz 10y agoWhy would you use gitlab.com if you have a self-hosted Gitlab instance?
- riebschlager 10y agoBecause I trust(ed) GitLab's backup procedures more than my own.
- jamesmiller5 10y agoAs I read the report I notice a lot of PostgreSQL "backup" systems depend on snapshotting from the FS & Rsync. This may work for database write logs, but it certainly will corrupt live git repositories that use local file system locking guarantees. NFS also requires special attention (a symlink lock) as writes can be acknowledged concurrently for byte offsets unless NFSv4 locking & compatible storage software is used. Git repo corruption from snapshotting tech (tarball, zfs, rsync, etc): http://web.archive.org/web/20130326122719/http://jefferai.org/2013/03/24/screw-the-mirrors/ http://web.archive.org/web/20130326122719/http://jefferai.or... Prev. Hacker News submission: https://news.ycombinator.com/item?id=5431409 https://news.ycombinator.com/item?id=5431409 Gitlab, I know you are all under pressure atm but when the storm passes feel free to reach out to my HN handle at jmiller5.com and I'd be happy to let you know if any of your repository backup solutions are dangerous/prone to corruption.
- kentonv 10y ago> Git repo corruption from snapshotting tech (tarball, zfs, rsync, etc): The link discusses why rsync and tarballs are not good backup solutions. But, those wouldn't rightly be called "snapshots". "Snapshot" implies atomic, right? Surely an atomic snapshot would not corrupt git -- I would expect git, like any database, is designed to be recoverable after power failure, to which recovering from an atomic snapshot should be equivalent. Or is that not the case?
- jamesmiller5 10y agoThat is actually not the case. Say a git server is in the middle of a write to refs/heads/master . You atomically snapshot the FS and then a power outage kills the server. The repository state will have a small chance of lock files that are never removed. Depending on the lock, future writes to a ref or to the repository can fail. Not the worst situation as data lose won't occur but definitely not a stable state. If you treat git repositories as a service that needs a recovery step it would be fine, unfortunately most don't. (edit) source: http://joeyh.name/blog/entry/difficulties_in_backing_up_live_git_repositories/ http://joeyh.name/blog/entry/difficulties_in_backing_up_live...
- justinclift 10y ago"Removed a user for using a repository as some form of CDN, resulting in 47 000 IPs signing in using the same account (causing high DB load)." No memcached in front of PostgreSQL?
- foo101 10y agoCan someone elaborate who or what is YP?
- grzm 10y agoHe's identified elsewhere in the thread as Yorick Peterse: https://news.ycombinator.com/item?id=13537132 https://news.ycombinator.com/item?id=13537132
- vhost- 10y agoI haven't seen or done anything of this scale before, but I did have a very sobering moment while working on a large online retailers stack as a systems engineer. We were rolling out a new stack in another data center across the country and before replication went live, I decided to connect and check things out. Our chef work hadn't completed for the database hosts, so I decided to install some OS updates by hand using pssh on all the MySQL hosts and saw a kernel update. So I thought, the DC isn't live yet, no replication is running, I'll just restart these servers. So I did using pssh again and then I caught a glimps at the domain in some output and my face went completely pale. I restarted the production databases... all of them. And they all had 256GB of ecc memory. It takes a very long time for each of those machines to POST. I contacted the client and said the maintenance page was my fault and was fully expecting to be fired on the spot, but they just grilled me about being careful in the future, and then laughed it off. I've been the most careful ever since then. It scared me straight. Always make sure you are in the right environment before you do anything that requires a write operation.
- olig15 10y agoExactly the right response. You're not going to make that same mistake again, but if you were fired, your replacement very well might.
- 1945 10y agoI applaud their honesty, most companies would choose to keep customers in the dark while the investigate "an incident."
- LeonidBugaev 10y agoNot so long ago GitLab decided to move from using AWS Cloud to managing own hardware. I wonder if such situation could happen if they used managed Postgres with automatic backups. Most of us use Cloud because OP's is hard, and human related risks are too high.
- lbotos 10y agoThat choice to go to bare metal was reversed: https://gitlab.com/gitlab-com/infrastructure/issues/727#note_20044060 https://gitlab.com/gitlab-com/infrastructure/issues/727#note... https://webcache.googleusercontent.com/search?q=cache:M2CRY7xD6v4J:https://gitlab.com/gitlab-com/infrastructure/issues/727+&cd=1&hl=en&ct=clnk&gl=us https://webcache.googleusercontent.com/search?q=cache:M2CRY7... (cache link until GL.com is back up.)
- motles 10y agoDoes Azure not have managed postgres?
- rrdelaney 10y agoAzure does not provide a managed postgres service. However it does have Azure SQL[0], which is based on SQL Server. Disclaimer: I work for Microsoft [0] https://azure.microsoft.com/en-us/services/sql-database/ https://azure.microsoft.com/en-us/services/sql-database/
- frik 10y agoThe major Azure problems like two years ago were documented by GitLab in similar manner. I find the openess a good thing, even in not so good times. Thumbs up.
- dimitrie 10y agoThanks for the thumbsup!
- scurvy 10y agoAt my $currentJob, we restore 30TB worth of database restores every day in automated processes. We do this because the DBA insisted that the DB backup process was fine. We tried to restore 3 backups as a test, and they all failed. We no longer have DBA's. We have automated procedures and very thorough testing. Zero failed restorations since then.
- ishitatsuyuki 10y agoI was using GitLab for the huge number of features, but moved away to GitHub due to the awful uptime and server speed. BTW, overhosting is always a big risk, since it takes along period to catch up for the incident.
- susi22 10y agoI was a Sysadmin for a university for a while. During those years I also messed up once. I learnt one thing: If you run a command that cannot be undone, you pause, take a few seconds and think. Is it the right directory? Is it the right server (yes this has happened to me too). Are those the right parameters? Can I simulate the command beforehand? I introduced this double check for myself, and it has actually caught a few commands I was about to run.
- jorblumesea 10y agoSo, maybe a stupid question from a frontend dev who doesn't deal with these systems at all, but aren't these systems usually part of a cluster with read replicas? Blowing away the contents of one box shouldn't destroy the cluster right? I thought the primary/secondary pattern was really common among relational databases and failover boxes and other measures were standard practice. Was the command executed on all machines? Is the cluster treated as one file system? Please excuse the ignorance.
- yeukhon 10y agoYou'd be surprised how many companies out there do not run read replica and don't really have anything for disaster recovery (same data center, same country good luck when there's an outage).
- terom 10y agoReplication systems can break. They work working on fixing the broken db2 read replica when they accidentially nuked the primary db1 server. > db2.cluster refuses to replicate, /var/opt/gitlab/postgresql/data is wiped to ensure a clean replication > [...] decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com That ops fail on db1 wouldn't have been such an issue if they weren't in such a vulnerable position with an invalidated read replica.
- onetom 10y agoThank you GitLab people for writing a log! It's just as important as the service you are providing. Keep up the good job and never be afraid to talk about your mistakes. It would be also very educational if you could try to do a "5 whys" session and share it too. The person who made the mistake deserves a bit of rest, that's for sure. I wish she or he is supported by the team emotionally and not just being blamed.
- sytse 10y agoTotally agree with doing the 5 why's, see https://news.ycombinator.com/item?id=13539595 https://news.ycombinator.com/item?id=13539595
- 234dd57d2c8db 10y agoOnly having two production database servers? That is a mistake, because as soon as you need to do maint. on one, you have just voided your DR recovery plan. Two is one, one is none.
- huula 10y agoAs a user of GitLab who commits several times everyday, I have all the respect for the great work you guys have done. Keep hacking and don't worry!
- perlgeek 10y ago>2. Regular backups seem to also only be taken once per 24 hours, though YP has not yet been able to figure out where they are stored. According to JN these don’t appear to be working, producing files only a few bytes in size. >3. SH: It looks like pg_dump may be failing because PostgreSQL 9.2 binaries are being run instead of 9.6 binaries. This happens because omnibus only uses Pg 9.6 if data/PG_VERSION is set to 9.6, but on workers this file does not exist. As a result it defaults to 9.2, failing silently. No SQL dumps were made as a result. Fog gem may have cleaned out older backups. >5. Our backups to S3 apparently don’t work either: the bucket is empty I think we've all seen that with some kind of report or backup or that's regularly reported: an empty file or none at all is generated due to some silent error. I highly recommend creating a monitoring check for file size for each automatically generated file. At $work, we also generate quite a few config files (for Radius, DHCP servers, web servers, mail servers....). For those we have a mechanism that diffs the old and the new version, and rejects the new version if the diff exceeds a pre-defined percentage of the file size, escalating the decision to a human.
- pmontra 10y agoIs anybody using a procedure to automatically perform restores and check that they match production data? I guess it's not easy for complex systems and non trivial amounts of data. Do you want to share the details and give us an idea of the costs? Thanks.
- stephenr 10y agoPoint #3 is a good example of why "omnibus" packages are a fucking terrible idea.
- SeriousM 10y ago"somehow disallow rm - rf"... Well, you didn't understood the linux
- samblr 10y agoHave been a happy gitlab user. Reading this worries me, as honestly I do not know what I have personal backup of. But I appreciate transparency with which Gitlab are dealing the issue. And I hope Gitlab will bounce back for good.
- beezischillin 10y agoAs a small team made up of 3 companies that work closely together, we all use GitLab's services daily for our work. Thank you for the great service and we wish you a speedy and painless recovery!
- arkh 10y ago> Create issue to change terminal PS1 format/colours to make it clear whether you’re using production or staging (red production, yellow staging) QubesOS anyone? It could be a good idea to have a Qube per environment targeted.
- jlengrand 10y agoGotta love the tweet though : "We accidentally deleted production data and might have to restore from backup." https://status.gitlab.com/ https://status.gitlab.com/ As usual, I really love the transparency they are showing in how they are taking care of the issue. Lots to learn from
- musicmatze 10y agoThey say that git data (repos and wikis) are not affected... well ... if they would have had their PRs and Issues in git repositories, too... Disclaimer: Worked on a POC for exactly this last semester and going to publish my results in the next few weeks.
- talex5 10y agoYes, doing this would make it easier to use services like GitHub. For example, DataKit's GitHub bridge service uses webhooks to track changes in GitHub metadata (e.g. the status of branches, tags and PRs) and stores the state in another Git branch. The DataKitCI monitors this branch to know when to build things (and writes the status back there): https://github.com/docker/datakit/tree/master/ci https://github.com/docker/datakit/tree/master/ci Would be simpler if GitHub offered this data in Git format directly.
- jschulenklopper 10y ago> if they would have had their PRs and Issues in git repositories, too Well, perhaps a git repository isn't the right persistency structure for non-code items like issues and pull requests. That a git repository is a good structure for document-like entities (like code files) and tracking changes/versions, doesn't mean that it's a good choice for highly-relational or highly-dynamic objects.
- musicmatze 10y agoWell, from what we've experienced during the implementation and while using our tool, it works _really_ well!
- procrastitron 10y agoGitLab has an open issue (https://gitlab.com/gitlab-org/gitlab-ce/issues/4084 https://gitlab.com/gitlab-org/gitlab-ce/issues/4084) to use git-appraise (https://github.com/google/git-appraise https://github.com/google/git-appraise) for storing pull requests in the repository.
- 10y ago
- oelmekki 10y agoI had to restore a rails app from a day old backup once. I actually manage to bring last day data as well by parsing POST/PUT/PATCH lines in rails log. This is painful, and you have to keep track of new ids for relations, but it "works" (obviously, there is info you can't retrieve that way, but in those situations, anything more than nothing is good).
- aidos 10y agoI really feel for everyone involved. Knowing Gitlab, they'll learn and become better for it. I've been using the PS1 trick they mention for the last couple of years and I've found it to be a really good visual check (red=prod, yellow=staging, green=dev). We then also apply the colorscheme to the header in our admin pages too. Those of us that are jumping between environments are a big risk to data :-)
- zamalek 10y ago> Work is interrupted due to this, and due to spam/high load on GitLab.com This is why spam should be illegal. The advertisers, the ISPs harboring them or their country should be taken on for damages. This not only prevented Gitlab from doing business, but also people who depend on them from doing business. It's criminal.
- V-2 10y agoI know it's not a laughing matter (and suddenly my working day just seems much easier), but when I read "YP says it’s best for him not to run anything with sudo any more today" I couldn't but smile. Yeah, well, that's a good point probably : )
- jfindley 10y ago"The replication procedure is super fragile, prone to error, relies on a handful of random shell scripts, and is badly documented" This is true of many databases, but in my experience is particularly true of postgres. It's a marvelous single-instance product, but I've never really found any replication/ha tech for it that I've been that happy with. I've always been a bit nervous about postgres-backed products for this very reason. I'd be interested what other people's take on this is, though.
- dijit 10y agoI don't share your observation at all. If we're talking normal replication then I can tell you for certain that the built in replication system is completely rock solid (especially compared to MySQL). If you're talking multi-master replication, then Citus is pretty solid, but not nearly as solid as replication. If you're talking statement based replication, well, it's the same as all databases. Here be dragons.
- jfindley 10y agoThe built-in replication may be good - but it's pretty new. Replication slots, which are IMO vital for making the built-in replication non-fragile, only arrived in 9.4, which is pretty recent of a release. I wonder how widely tested it is, given that? How many people are actually using it for large workloads? Citrus I hadn't heard of though - that's interesting, thanks.
- dmichulke 10y ago> only arrived in 9.4, which is pretty recent of a release. Release Date: 2014-12-18 (from https://www.postgresql.org/docs/9.4/static/release-9-4.html https://www.postgresql.org/docs/9.4/static/release-9-4.html) Two years do surely not confer the status of "battle-tested" but I wouldn't call it "recent" either. Then again, DB service level and a standard application service level might differ by a few nines/sigma here. Is it that what you're referring to?
- 10y ago
- scandox 10y agoA lot of sympathy for YP. I could have been that person (on a smaller scale) many times. An expensive lesson in this case.
- Traubenfuchs 10y agoSo tl;dr: Gitlab is experiencing heavy DOS attacks that created so much data that replication stopped working. In the process of getting replication to work again, "YP" wanted to delete the empty data directory of the slave DB server, but accidentally deleted it on the master DB server. Out of 5 backup/replication techniques they use not one is working reliably. YP manually created the backup they could use 6 hours ago by chance. Tell me if I misunderstood something. I hope the customer I met last week does not remember I ever recommended GitLab to him.
- fbreduc 10y ago>Tell me if I misunderstood something. I hope the customer I met last week does not remember I ever recommended GitLab to him. this is how i feel right now, i suggested gitlab to half a dozen people who eventually moved to it for their business work... now i'm such an asshole :(
- pits 10y agoJe Suis YP
- z3t4 10y agoThere's really no point doing intensive congnitive work for more then 8 hours straight. After that you go by instinct and muscle memory. Surprisingly a lot of tasks can be done, but you shouldn't do anything critical.
- aubreykilian 10y agoThis happened to me one night late a few years back, with Oracle on a CentOS server. rm -rf /data/oradata/ on the wrong machine. I managed to get the data back though, as Oracle was still running and had the files open. "lsof | grep '(deleted)'" and /proc/<ORACLEPIDHERE>/fd/* saved my life. I managed to stop all connections to the database, copy all the (deleted) files into a temp directory, stop Oracle, copy the files to their rightful place, and start up Oracle, with no data lost.
- Heliosmaster 10y agothey are livestreaming their recovery process: https://www.youtube.com/watch?v=nc0hPGerSd4 https://www.youtube.com/watch?v=nc0hPGerSd4
- asmosoinio 10y agoWhich is pretty amazing in my opinion!
- arc_of_descent 10y agoDon't prepare a backup plan. Prepare a recovery plan.
- pmlnr 10y agoI agree with this. Prepare for destroyed (burnt-down level) machine, for datacentre failure, for stolen home server, for scratched blu-ray archives - in short, for the worst. And of course, hope for the best.
- djsumdog 10y agoIt's easier said than done in some companies where stake holders are always pushing for new stuff. Unless those who fund what you're doing understand why disaster recovery is vital, you're going to see this. Ideally you want devops in such a state that you create new lower environments that mirror production, complete with state/backup restoration, that's run automatically every week.
- travisby 10y agoAmazing document. Thank you for sharing. Taking it back to my company to make sure we can learn from it and to know what to check (like our logical backups... I know we've seen issues with our 9.5 servers and RHEL7 defaulting to 9.1 or 9.2 on our host where we take the backups from! Verifying exit code here we come...) @sytse, I noticed you _do_ use streaming WAL replication, but I didn't notice any mention of attempting Point In Time Recovery. Have you taken a look into archiving the WAL files in S3? Those, along with frequent pg_basebackup's (frequent because replaying a WAL file has been painfully slow for us) could allow you to point in time recover to either a timestamp, or a transaction (and before or after). https://www.postgresql.org/docs/9.6/static/continuous-archiving.html https://www.postgresql.org/docs/9.6/static/continuous-archiv... We use https://github.com/wal-e/wal-e https://github.com/wal-e/wal-e to manage our uploading to swift (no S3 at our company heh) and then inhouse tooling to build a recovery.conf. Note we actually have our asynchronous followers work off of this system too so they're not taking bandwidth from the primary. (note this is can lead to ~ 1 WAL file of data loss, but is acceptable for us.) I doubt I could be of any help, since reading the report definitely shows y'all having an up on me with pg knowledge, but if there's anything I can do / talk about feel free to reach out.
- YorickPeterse 10y agoI don't think we use PITR, certainly not to S3. I believe wal-e was discussed internally some time in the past, but we never really did anything with it.
- kogepathic 10y agoI know it's not wal-e, but Barman recently added support for streaming WAL from postgres, so in theory you shouldn't lose any data if the master crashes. Note that this does require a replication slot on the master to implement. <rant> It's also stupid that you still have to set up WAL shipping (e.g. via rsync or scp) before taking a base backup even if you have streaming replication enabled. </rant> That being said though, I have not been happy with the restore performance of barman, though admittedly this may be I/O related.
- dx034 10y agoThe recovery process seems very slow with ~50mbit/sec. Could that be an issue related to cloud providers? I heard that issue quite often when dealing with AWS/Azure. Even HDDs should have much higher throughput for that kind of transfer. If they had dedicated hardware in 2 datacentres on the same continent, copying between those servers should easily be possible at 250mbit/s or more (from my experience). Especially as they seem to copy at the US east coast, where it's now night. For me, that would be a serious issue dealing with cloud providers. If I have a server with a 250mbit connection, I expect to be able to copy data between datacentres at that speed. And I never had problems with OVH, Hetzner and the like.
- CrLf 10y agoThe public report is nice and we can see a sequence of mishaps from it, that shouldn't have been allowed to happen but which (unfortunately) are not that uncommon. I've done my share of mistakes, I know what's like to be in emergency mode and too tired to think straight, so I'm going to refrain from criticizing individual actions. What I'm going to criticize is the excess of transparency: You absolutely DO NOT publish postmortems referencing actions by NAMED individuals, EVER. From reading the whole report it's clear that the group is at fault, not a single individual. But most people won't read the whole thing, even less people will try to understand the whole picture. That's why failures are always attributed publicly to the whole team, and actions by individuals are handled internally only. And they're making it even worse by livestreaming the thing! It's like having your boss looking over your shoulder but a million times worse...
- jjawssd 10y agoMaybe these individuals don't mind, it could just be a cultural thing.
- deleted 10y ago[deleted]
- mverwijs 10y agoAt one company I worked for we had a saying: "You're not one of the team until you've brought down the network. " We all mess up. Much respect to gitlab for being open about.
- drvdevd 10y agoThat's the kind of team I want! All hands on deck. No lame responsibility shifters.
- nanch 10y agoAnybody whose opinion matters understands that this type of event is a process problem, not a person problem. GitLab has always blazed their own trail with their transparency, whether through their open run books, open source code, or in this case their open problem resolution. Kudos to them in whatever manner they want to do it in (with or without names). To be honest, through all of the comments, yours seems the most high-strung, and you're the one complaining about high-pressure situations like having your boss looking over your shoulder. Relax buddy. :)
- neolawliet 10y agoThis is kind of tangential. Does anyone have any sort of DevOps 101 resource (books, links, etc.) for new DevOps engineers?
- INTPenis 10y agohttp://checkyourbackups.work/ http://checkyourbackups.work/ It hits home. I've seen three backup methods fail when it came time for an emergency restore, due to lack of competence, confusion and lack of regular restore tests.
- jjawssd 10y agoTheir network file transfer is super slow
- hardwaresofton 10y agoWorking on a Gitlab project right now, just noticed the site was down, thanks to the team for working so hard to fix/rectify this mistake and being totally open about it. Appreciate the openness and utility of Gitlab (as I've said in other threads), I'm sure it's frustrating to have this happen, but hang in there! services generally have 99.9% uptime anyway :)
- msimpson 10y agoMistakes will happen, sure. But given the following: > The replication procedure is super fragile, prone to error, relies on a handful of random shell scripts, and is badly documented > Our backups to S3 apparently don’t work either: the bucket is empty It seems like a lot of their backup and restore procedures were never even tested.
- gagabity 10y agoNuts, doesn't Azure have a managed Database option that can be used like AWS?
- ausjke 10y agoThat's why I do not use github/gitlab/whatever to host the part of my code that is too critical to me. I push it to my ssh/git server and use local UI to interact with it instead. Sometimes source code is very valuable and you just can not make any mistakes with it.
- jschulenklopper 10y ago> I do not use github/gitlab/whatever to host the part of my code that is too critical to me In this specific case, GitLab mentioned that code repositories are fine. It's the database part with issues and pull requests that they are restoring. https://twitter.com/gitlabstatus/status/826662763577618432 https://twitter.com/gitlabstatus/status/826662763577618432
- mdekkers 10y agoYP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com My heart honestly goes out to YP. This is a terrible sysops _oh shit_ moment.
- thosemagicstars 10y agoWe all fuck up, time to recovery and lessons learned is all that matters.
- bkbridge 10y agoThis stuff is complicated. No one gets that, no one.
- wruza 10y agorm is a part of coreutils, right? Why not just substitute it with a less destructive script moving files to /mnt/oops/<origpath-date>/<filename> in the next release? Badasses can echo badass > /etc/rm.conf to get the original behavior. Admins have to reinvent that bicycle for decades, stop it now, please!
- adamkittelson 10y agoI think the fact that no one seems to even remember that GitHub had a similar incident in 2010 https://github.com/blog/744-today-s-outage https://github.com/blog/744-today-s-outage is a good thing to keep in mind for the GitLab team. This too shall pass etc.
- elliottcarlson 10y agoRegarding making mistakes: Tom Watson Jr., CEO of IBM between 1956 and 1971, was a key figure in the information revolution. Watson repeatedly demonstrated his abilities as a leader, never more so than in our first short story. A young executive had made some bad decisions that cost the company several million dollars. He was summoned to Watson’s office, fully expecting to be dismissed. As he entered the office, the young executive said, “I suppose after that set of mistakes you will want to fire me.” Watson was said to have replied, “Not at all, young man, we have just spent a couple of million dollars educating you.”
- luckystartup 10y agoI have at least 10 private repos on GitLab, and many public ones. Even so, this is no big deal to me. That's the beauty of git. Even if all of their backups fail, I can just do a push and everything is back up there. I just hope my laptop doesn't die before they get it back online. EDIT: Was fun to put this little command together. Run this from your code directory, and it will push all of your gitlab repos. I'm going to run it when GitLab is back online. find . -maxdepth 3 -type d -name '.git' -exec bash -c 'cd ${@%/.git} && git remote -v | grep -q gitlab.com && echo "Pushing $PWD..." && git push' -- {} \;
- wolfgang42 10y agoFrom the incident report: > Git repositories are NOT lost, we can recreate all of the projects whose user/group existed before the data loss, but we cannot restore any of these projects issues, etc. Your fancy snippet will report that it has pushed no changes. The data that was lost was new issues, PRs, issue comments, and so on; I've never heard of anyone keeping backups of these on their local laptops.
- luckystartup 10y ago> I've never heard of anyone keeping backups of these on their local laptops. Hmm... That's an interesting idea! You could do that on a separate (empty) branch. Maybe call it `__project`, and you could just have folders of markdown files. You could have two root folders for `issues/` and 'pull_requests/', and two subfolders in each for `./open/` and `./closed/`. And a simple command-line tool + web UI. You could just edit the file to add a comment. It would be really nice to have a history and backup of all of your issues. I also like the fact that you could create or edit issues offline. Then you could also set up a 2-way sync between your repo and GitLab / GitHub / Trello.
- zokier 10y agoThat sort of "inline" issue tracking is a thing. I think Bugs Everywhere[1] is one of the more mature systems based on the idea. There are several others too[2], most of them unmaintained. There are also wiki-style systems based on the same idea. [1] http://www.bugseverywhere.org/ http://www.bugseverywhere.org/ [2] http://www.cs.unb.ca/~bremner/blog/posts/git-issue-trackers/ http://www.cs.unb.ca/~bremner/blog/posts/git-issue-trackers/
- perryprog 10y agoAnd this is why I still use GitHub. It's a shame as GitLab was looking like a nice alliterative (for self hosting, might still try it some time), but if this sort of thing could even possibly happen..... I didn't read to much into this, but they really didn't haven any backup on the databases?
- Slaul 10y agoA very similar thing happened at GitHub a while back: https://github.com/blog/744-today-s-outage https://github.com/blog/744-today-s-outage
- thepumpkin1979 10y agoI wonder if they have some kind of protocol to modify production environments that was somehow overwritten, one person creates the bash scripts and a second one reviews and executes, never a single person.
- grovegames 10y agoTransparency should be like a sheer dress, enough to see, but still covering the bits that need to stay hidden. This is just nudity.
- rocky1138 10y agoThere should be a 15 second warm-up after any `rm -rf` command in order to give admins time to cancel a stupid move.
- TeeWEE 10y agoGitLab is run completely decentralized, and they are underpaying their developers. I think thats one of the reasons for this failure to happen. Decentralized work, can work. But face2face communication is important.
- therealmarv 10y agoI only hope you prepare for DDoS attacks of this kind. Github was fighting with this stuff the last several years... you maybe only have luck. Please prepare!
- js2 10y agoIf you're a sys admin long enough, it will eventually happen to you that you'll execute a destructive command on the wrong machine. I'm fortunate that it happened to me very early in my career, and I made two changes in how I work at the suggestion of a wiser SA. 1) Before executing a destructive command, pause. Take your hands off the keyboard and perform a mental check that you're executing the right command on the right machine. I was explicitly told to literally sit on my hands while doing this check, and for a long time I did so. Now I just remove my hands from the keyboard and lower them to my side while re-considering my action. 2) Make your production shells visually distinct. I setup staging machine shells with a yellow prompt and production shells with a red prompt, with full hostname in the prompt. You can also color your terminal window background. Or use a routine such as: production terminal windows are always on the right of the screen. Close/hide all windows that aren't relevant to the production task at hand. It should always be obvious what machine you're executing a commmand on and especially whether it is production. (edit: I see this is in outage the remeditation steps.) One last thing: I try never to run 'rm -rf /some/dir' straight out. I'll almost always rename the directory and create a new directory. I don't remove the old directory till I confirm everything is working as expected. Really, 'rm -rf' should trigger red-alerts in your brain, especially if a glob is involved, no matter if you're running it in production or anywhere else. DANGER WILL ROBINSON plays in my brain every time. Lastly, I'm sorry for your loss. I've been there, it sucks.
- js2 10y agoHere's the .bashrc/.bash_login I use: https://gist.github.com/jaysoffian/8c75e661f7a61b0d094703e265d8d5b4 https://gist.github.com/jaysoffian/8c75e661f7a61b0d094703e26...
- danaliv 10y ago> 2017/01/31 23:00-ish YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com A long, frustrating day. Running destructive commands at 11pm. This is why pilots have duty time limits. YP should've been relieved by another engineer who was physically and mentally fresher. Human beings have limits, and when we reach them, we make more—and worse—mistakes. Any process that fails to account for this is broken.
- wildchild 10y agoTransparency maybe ok, but live stream is a theatre indeed.
- tcpipcowboy 10y agoI'm curious to know what strategy has been developed out of this regarding delivery of spam through creation of snippets. In the original GitLab First Incident report it noted "spammers were hammering the database by creating snippets, making it unstable". So many easily accessible platforms are out there that this method of spamming could be used on that it seems like a necessity to evaluate current workflows and identify where checks/balances can be inserted that would prevent this from happening again. Short of removal of snippets, there must be some method of snippet grepping that would put a pause on suspicious snippets, preventing the bulk of submissions along the lines of what GitLab initially received.
- lallysingh 10y agoI'm commenting a bit late, but I hope it's still read by the gitlab team. First, you kept heads and didn't turn on each other. That's a major success, and gives me more confidence in gitlab. The rest you can improve on only if you have this right. Second, I'm sure you've gotten the message to test your backups and recovery plan. It's a good time to read the Google SRE book, and consider how to put together full integration tests that build up a db, back it up, nuke the original, and recover from the db. With containers this isn't actually awful to do. But I didn't see much mentioned about load tests. A few simple scripts that hit your server (or test instance!) hard can help you find points where things fall apart under load. Even if you don't have a good way to gracefully do anything else other than alert a human, you can figure out what to monitor and how to make sure your backup/recovery plans can deal with a shit-ton of spammer data suddenly in your DB.
- cabargas 10y agoThank you for your feedback. I will add your suggestions to our document! We will be implementing new policies on backups and you can totally expect us doing load tests in the future. The whole team wants to make sure this will not happen again.
- ygersie 10y agoFunny that I haven't seen the following anecdote yet: One is none, two is one (== none). AFAICT there were only 2 Postgres instances? What gives? How would you ever feel comfortable when one goes down? How we deal with recovery: - run DB servers on ZFS - built a tool to orchestrate snapshotting (every 15 minutes) using an external Mutex to distribute snapshot creation for best recover accuracy. You could also have increased retention over time like: - keep 6 snapshots of 5 minutes - 4 hourly - 1 daily - 1 weekly Recover: choose point in time closest to fckup, the tool automatically elects the DB with closest (earlier than given time) snapshot. All other slaves are restored before that point in time and roll forward to the active state of new "master". Instead of executing worst case recovery plans by copying data to at least 6 (minimal) db read slaves we can recover in minutes with minimal data loss (especially when you consider downtime == data loss). There are cases where a setup like this would be a no go (think of companies where having lost transactions are absolutely devastating) but I don't think Gitlab is one of those. Side effect of ZFS is being able to ship blocks of data as offsite backups (instead of dumping), able to `zpool import` anywhere, checksumming, compression etc etc..