26 ms·

Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-

• ParadisoShlee 10y ago
  I have nothing, but respect for the GitLab team.... offering live notes for the recovery is stunning.
  • edelans 10y ago
    Agreed ! Even when they screw things up, they make people benefit from it! Kudos for such a good spirit and dedication to your customers. Everybody screw up from time to time in our industry. And when this happens, you have 2 types of guy : those who try to hide it, and those like Giltlab team who communicate as fast as they can because they respect their customers. Paradoxically, to me, it creates more trust than it damages it !
    • sytse 10y ago
      Thanks for the kind words. I'm sorry for letting our users down. We'll ask the the 5 why's https://en.wikipedia.org/wiki/5_Whys https://en.wikipedia.org/wiki/5_Whys We need to go from the initial mistake (wrong machine, solve by better hostname display and colors), to the second (not having a recent backup), to the third (not testing backups), to the fourth (not having a script for backup restores), to the fifth (nobody in charge of data durability and no written plan). The solutions above are just guesses at this point, we'll dive into this in the coming days and will communicate what we will do in a blog post.
      • throwaway8124 10y ago
        Good morning (posted from throwaway for reasons Ill describe). I feel for you greatly here, and I commend your openness about how data restoration caused 6 hours of data loss. I too work in a critical area where even minutes DB lost is bad. We just had our own test event recently. We make sure that we can fail everything over, and run on all secondaries. I found out how that worked; we failed. The problem with this, is I found out after the fact. Due to the secrecy, not even the teams knew why things failed the way they did. I had to piece it together from disjoint hearsay, and now I believe I have a competent picture. So yes, when I read your post mortem and RCA, it reminded me greatly of what happened here as well. But we all can learn from your example. As for me, I'm posting it as a throwaway due to likely threats on my job.
        • sytse 10y ago
          I agree that 6 hours is way too much.
      • mozair 10y ago
        make sure everyone is doing ok, don't let them beat themselves over it all the love and support in the world towards the team
        • sytse 10y ago
          Thanks mozair. Everyone is OK but sad. I sent this tweet earlier https://twitter.com/sytses/status/826598260831842308 https://twitter.com/sytses/status/826598260831842308 Thanks for the support, we've received a lot of kind reactions and are very grateful for them.
      • djm_ 10y ago
        Just know that your transparency here in this situation has put you leagues and bounds ahead of other vendors in my mind. Thank you for sharing.
      • jameskegel 10y ago
        I admire the response and quick action, but after reading this, I get the strong feeling Gitlab et al didn't know they were running a real business with real projects and real people who trusted you to be a good custodian of their data.
  • nadaviv 10y ago
    Completely agree. This is 1000x better than the typical cooperate PR response that says nothing which we'll usually get. Kudos for their honesty and openness, even in a somewhat embarrassing situation such as that.
  • toyg 10y ago
    it is also very, very risky from a liability perspective, both for insurance purposes and third-party indemnification. That's why most companies will keep as silent as possible.
    • michaelt 10y ago
      I've never heard of anyone successfully suing AWS/Google/Azure over lost data or downtime. And presumably even if your notes are kept private, it'll still come out in discovery if you end up in court? What are the risks here? I'd be interested to learn more.
      • toyg 10y ago
        > I've never heard of anyone successfully suing AWS/Google/Azure over lost data Those companies have huge legal departments writing iron-clad contracts and stocking a lot of very sharp knives for any such eventuality. Smaller companies make for much easier targets. > even if your notes are kept private, it'll still come out in discovery Those notes might have been lost by then - that is, if their existence at any given time is even known. It is perfectly reasonable for people working quickly after an outage, to not actually write down every step or observation they make. > What are the risks here? Not prison, but you might be forced to pay a bunch if someone manages to get a ruling against you for negligence or the likes.
      • korzun 10y ago
        > I've never heard of anyone successfully suing AWS/Google/Azure over lost data or downtime. Doesn't mean it never happens.
        • chiefalchemist 10y ago
          Imagine how deep their pockets are. Now imagine how deep your pockets would need to be to go toe to toe with them. Sure plenty of uber larges Incs are "in the cloud" but with mission critical (read: high value worth sueing over) applications and data? That's a pretty small universe.
• yeukhon 10y ago
  I think this is a big slap to company when they need to talk to their enterprise customers. I am not sure how we can really prevent this from happening by implementing an industry standard.
  • cyberferret 10y ago
    You know, I remember reading years ago that during the launch preparation for Apollo 16(?), they did a routine pressure test of the Command Module atop the rocket a few days before launch. The technician in charge made the simple error of forgetting to open a pressure release valve during the test which lead to an overpressurisation of the CM causing significant damage, including separation of the heat shield. The ENTIRE Saturn 5 rocket had to be wheeled back to the hangar and the CM dismantled and rebuilt, resulting in a month long delay in the launch. When the Apollo launchpad manager was asked if he had fired the technician in question, then answer was allegedly "Nope. He is the one guy on the next launch team that I know will NEVER make the same mistake again." I am willing to bet that Gitlab is now a company that will never slacken off their backup checking in the future.
    • hueving 10y ago
      >When the Apollo launchpad manager was asked if he had fired the technician in question, then answer was allegedly "Nope. He is the one guy on the next launch team that I know will NEVER make the same mistake again." This same general story appears in different forms with million dollar trading errors, etc. But I have to wonder if it's really that good of a lesson. If you truly take it to heart, you could potentially keep people on the team that are legitimately incompetent which could result in another catastrophic failure.
      • safeaim 10y ago
        And if one starts applying your principle at a workplace, you end up with the whole company being afraid of doing anything at all, to avoid being fired.
        • 10y ago
• tawayway 10y ago
  > Imagine if you're the CEO of Gitlab and seeing this right now I kind of hope the CEO of Gitlab isn't reading HN right now
  • hrrsn 10y ago
    He was a few hours ago ;) https://news.ycombinator.com/threads?id=sytse https://news.ycombinator.com/threads?id=sytse
  • ocdtrekkie 10y ago
    If he's getting some much needed sleep right now, he'll probably read it tomorrow. The CEO of GitLab is really active on HN, and a pretty classy guy.
    • sytse 10y ago
      Thanks. Just landed in Europe for Git Merge. It is heartwarming to read some of the comments here. We realize that we only get this leniency once and we'll learn from this and communicate our lessons in a few days from now.
      • harshreality 10y ago
        The lesson here has been learned over and over again, painfully. For every backup system you have, test restores, from scratch, periodically. The more critical the data is to your business, the more frequently and more automated you want those backup checks. Of course you also want procedures to try to prevent tired admins from deleting production databases. To the greatest degree possible, implement systems to prevent manual tampering with production data; often there's an alternative. But databases can get broken or corrupt without admin interference, so preventing manual database removal or corruption is not an ultimate solution.
      • erikbye 10y ago
        Why is it you (GitLab) and every other company has to learn from your own mistakes again and again instead of learning from others? Just a cursory glance at a few log files would have told you that your backups are not happening. Nevermind verifying actual backups... Do you have DBAs? Are they completely inept at their job? Don't answer.
• mrmondo 10y ago
  I could not agree more @soheil, let me also say that Sid aka @sytse is not only a top notch CEO but he's also a hard working, caring individual who truly not only cares about the product but the whole community so, so much.
  • gorodetsky 10y ago
    Wholeheartedly agree. Incidents are inevitable and it's important to have a proper RCA/Service-Disruption process in place to handle those. As mentioned on the other thread, maybe Gitlab doesn't have enough operational/SRE expertise in-house yet, but that was the case in every fast-growing company I've worked for last decade.
• dspillett 10y ago
  > even then it could still just stop working one day This is where automated testing helps. My mail server has a sister VM out in the wild that once a day picks up the latest backup from the offsite-online backups and restores it, sending me a copy of the last message received according to the data in that backup. If I don't get the message, restoring the backup failed. If the message looks too old then making or transferring the backup failed. My source control services and static web servers do something similar. None of the shadow copies are available to the world, though I can see them via VPN to perform manual check occasionally and if something nasty happens they are only a few firewall and routing rules away from being used as DR sites (they are slower as in their normal just-testing-the-backups operation they don't need nearly the same resources as the live copies, but slow is better than no!). This won't catch everything of course, but it catches many things that not doing it would not. The time spent maintaining the automation (which itself could have faults of course) is time well spent if done intelligently. For a system as large in scale as GitLabs then a full restore daily is probably not practical so a more selective heuristic will need to be chosen if you are operating at such a scale. My arrangement still needs some manual checking and sometimes I'm too busy or just forget, so again it isn't perfect, but the risk of making it more clever and inviting failure that way is at this point worse than the risk of my being lazy at exactly the wrong time. One thing my arrangement doesn't test is point-in-time restores (because sometimes the problem happened last week, so last night's backup is no use) but there is a limit to how much you can practically do. > The problem of restoring non-existing backups should be treated as a more serious problem in our industry It is by people that care about it, but not enough people care, and too many people see the resources needed to get it right as an expense rather than an investment for future mental health. It isn't just non-existent backups. Any backup arrangement could be subject to corruption either through hardware fault, process fault, or deliberate action (the old "they hacked us and took out our backups too" - I really must get around to publishing my notes on soft-offline backups...). > Let's identify the diseases of this sort in our industry Apathy mainly. The people who care most are either naturally paranoid (like me), have lost important data at some point in the past so know the feeling first hand (thankfully not me, though in part to having a backup strategy that worked) or have had to have the difficult conversation with another party (sorry, I can't magic your data back for you, it really is gone) and watch the pitiful expressions as they beg for the impossible. The only way to enforce the correct due diligence is to make someone responsible for it, it is more a management problem than a technical one because the technical approaches needed pretty much all exist and for the most part are well studies and documented. Of course to an extent you have to accept reasonable risks. It is usually not practical to do everything that could be done, and understandable human error always needs to be accounted for as do "acts of Murphy". But someone needs to be responsible for deciding what sort of risk to take (by not doing something, or doing something less ideal) rather than them just being taken by general inaction.
• hvidgaard 10y ago
  Is it too much to ask for a few days dedicated to this? I mean, pretent that your critical infastructure goes up in smoke - can you recover, and how quickly? For most companies it simply means that you try to restore the machines from your backup. You can do this as everything else continues as normal.
• nicky0 10y ago
  This page did a lot to raise my awareness of my own inadequate backup situation. Any hurt feeling of the CEO caused by this page will be outweighed by the greater good of the message being spread.
• marcosdumay 10y ago
  > We should probably treat this issue as something more like a disease like high blood pressure, namely, you don't know you have it, but it is probably doing irreparable damage to your internal organs. That's why we need awareness days.
  • evilDagmar 10y ago
    Precisely this. Things that aren't tested can pretty much be counted on to fail a non-trivial percentage of the time. If it's going to be business critical, then it needs to be tested. (Lesson hopefully learned here) By the same token, things which aren't monitored can also be counted upon to fail a non-trivial percentage of the time and even worse, go unnoticed. (Another teachable point here) ...but people being what they are tend to learn these things the hard way unless we've got concrete, real-world incidents like this for them to learn from.
• elastic_church 10y ago
  oh I thought GitLab was testing because of this day.
• AsyncAwait 10y ago
  Indeed, I just started planning for a side project to solve this by automating as much of the backup sanity checking procedure as possible.