5 ms·

I have nothing, but respect for the GitLab team.... offering live notes for the recovery is stunning.

edelans 10y ago
Agreed ! Even when they screw things up, they make people benefit from it! Kudos for such a good spirit and dedication to your customers. Everybody screw up from time to time in our industry. And when this happens, you have 2 types of guy : those who try to hide it, and those like Giltlab team who communicate as fast as they can because they respect their customers. Paradoxically, to me, it creates more trust than it damages it !
- sytse 10y ago
  Thanks for the kind words. I'm sorry for letting our users down. We'll ask the the 5 why's https://en.wikipedia.org/wiki/5_Whys https://en.wikipedia.org/wiki/5_Whys We need to go from the initial mistake (wrong machine, solve by better hostname display and colors), to the second (not having a recent backup), to the third (not testing backups), to the fourth (not having a script for backup restores), to the fifth (nobody in charge of data durability and no written plan). The solutions above are just guesses at this point, we'll dive into this in the coming days and will communicate what we will do in a blog post.
  - throwaway8124 10y ago
    Good morning (posted from throwaway for reasons Ill describe). I feel for you greatly here, and I commend your openness about how data restoration caused 6 hours of data loss. I too work in a critical area where even minutes DB lost is bad. We just had our own test event recently. We make sure that we can fail everything over, and run on all secondaries. I found out how that worked; we failed. The problem with this, is I found out after the fact. Due to the secrecy, not even the teams knew why things failed the way they did. I had to piece it together from disjoint hearsay, and now I believe I have a competent picture. So yes, when I read your post mortem and RCA, it reminded me greatly of what happened here as well. But we all can learn from your example. As for me, I'm posting it as a throwaway due to likely threats on my job.
    - sytse 10y ago
      I agree that 6 hours is way too much.
  - mozair 10y ago
    make sure everyone is doing ok, don't let them beat themselves over it all the love and support in the world towards the team
    - sytse 10y ago
      Thanks mozair. Everyone is OK but sad. I sent this tweet earlier https://twitter.com/sytses/status/826598260831842308 https://twitter.com/sytses/status/826598260831842308 Thanks for the support, we've received a lot of kind reactions and are very grateful for them.
  - djm_ 10y ago
    Just know that your transparency here in this situation has put you leagues and bounds ahead of other vendors in my mind. Thank you for sharing.
  - jameskegel 10y ago
    I admire the response and quick action, but after reading this, I get the strong feeling Gitlab et al didn't know they were running a real business with real projects and real people who trusted you to be a good custodian of their data.
  - throwaway1236t4 10y ago
    I also would ask Why are we running our own Postgre setup and not using RDS? And then Why do we not have production Postgre DBAs on staff that would do this rather than an engineer?
    - AsyncAwait 10y ago
      They want to have more control over the database and moving providers etc., which is why they're not using RDS, as was explained on the live stream.
  - sumbry 10y ago
    If you do the 5 why's and are honest you'll discover the real root cause is not having a backup and restore procedure that is programmatic. All outages are blameless. It's always a process failure or lack of a proper system or tool.
nadaviv 10y ago
Completely agree. This is 1000x better than the typical cooperate PR response that says nothing which we'll usually get. Kudos for their honesty and openness, even in a somewhat embarrassing situation such as that.
toyg 10y ago
it is also very, very risky from a liability perspective, both for insurance purposes and third-party indemnification. That's why most companies will keep as silent as possible.
- michaelt 10y ago
  I've never heard of anyone successfully suing AWS/Google/Azure over lost data or downtime. And presumably even if your notes are kept private, it'll still come out in discovery if you end up in court? What are the risks here? I'd be interested to learn more.
  - toyg 10y ago
    > I've never heard of anyone successfully suing AWS/Google/Azure over lost data Those companies have huge legal departments writing iron-clad contracts and stocking a lot of very sharp knives for any such eventuality. Smaller companies make for much easier targets. > even if your notes are kept private, it'll still come out in discovery Those notes might have been lost by then - that is, if their existence at any given time is even known. It is perfectly reasonable for people working quickly after an outage, to not actually write down every step or observation they make. > What are the risks here? Not prison, but you might be forced to pay a bunch if someone manages to get a ruling against you for negligence or the likes.
  - korzun 10y ago
    > I've never heard of anyone successfully suing AWS/Google/Azure over lost data or downtime. Doesn't mean it never happens.
    - chiefalchemist 10y ago
      Imagine how deep their pockets are. Now imagine how deep your pockets would need to be to go toe to toe with them. Sure plenty of uber larges Incs are "in the cloud" but with mission critical (read: high value worth sueing over) applications and data? That's a pretty small universe.

Agreed ! Even when they screw things up, they make people benefit from it! Kudos for such a good spirit and dedication to your customers. Everybody screw up from time to time in our industry. And when this happens, you have 2 types of guy : those who try to hide it, and those like Giltlab team who communicate as fast as they can because they respect their customers. Paradoxically, to me, it creates more trust than it damages it !

sytse 10y ago
Thanks for the kind words. I'm sorry for letting our users down. We'll ask the the 5 why's https://en.wikipedia.org/wiki/5_Whys https://en.wikipedia.org/wiki/5_Whys We need to go from the initial mistake (wrong machine, solve by better hostname display and colors), to the second (not having a recent backup), to the third (not testing backups), to the fourth (not having a script for backup restores), to the fifth (nobody in charge of data durability and no written plan). The solutions above are just guesses at this point, we'll dive into this in the coming days and will communicate what we will do in a blog post.
- throwaway8124 10y ago
  Good morning (posted from throwaway for reasons Ill describe). I feel for you greatly here, and I commend your openness about how data restoration caused 6 hours of data loss. I too work in a critical area where even minutes DB lost is bad. We just had our own test event recently. We make sure that we can fail everything over, and run on all secondaries. I found out how that worked; we failed. The problem with this, is I found out after the fact. Due to the secrecy, not even the teams knew why things failed the way they did. I had to piece it together from disjoint hearsay, and now I believe I have a competent picture. So yes, when I read your post mortem and RCA, it reminded me greatly of what happened here as well. But we all can learn from your example. As for me, I'm posting it as a throwaway due to likely threats on my job.
  - sytse 10y ago
    I agree that 6 hours is way too much.
- mozair 10y ago
  make sure everyone is doing ok, don't let them beat themselves over it all the love and support in the world towards the team
  - sytse 10y ago
    Thanks mozair. Everyone is OK but sad. I sent this tweet earlier https://twitter.com/sytses/status/826598260831842308 https://twitter.com/sytses/status/826598260831842308 Thanks for the support, we've received a lot of kind reactions and are very grateful for them.
- djm_ 10y ago
  Just know that your transparency here in this situation has put you leagues and bounds ahead of other vendors in my mind. Thank you for sharing.
- jameskegel 10y ago
  I admire the response and quick action, but after reading this, I get the strong feeling Gitlab et al didn't know they were running a real business with real projects and real people who trusted you to be a good custodian of their data.
- throwaway1236t4 10y ago
  I also would ask Why are we running our own Postgre setup and not using RDS? And then Why do we not have production Postgre DBAs on staff that would do this rather than an engineer?
  - AsyncAwait 10y ago
    They want to have more control over the database and moving providers etc., which is why they're not using RDS, as was explained on the live stream.
- sumbry 10y ago
  If you do the 5 why's and are honest you'll discover the real root cause is not having a backup and restore procedure that is programmatic. All outages are blameless. It's always a process failure or lack of a proper system or tool.

nadaviv 10y ago

Completely agree. This is 1000x better than the typical cooperate PR response that says nothing which we'll usually get. Kudos for their honesty and openness, even in a somewhat embarrassing situation such as that.

toyg 10y ago

it is also very, very risky from a liability perspective, both for insurance purposes and third-party indemnification. That's why most companies will keep as silent as possible.

michaelt 10y ago
I've never heard of anyone successfully suing AWS/Google/Azure over lost data or downtime. And presumably even if your notes are kept private, it'll still come out in discovery if you end up in court? What are the risks here? I'd be interested to learn more.
- toyg 10y ago
  > I've never heard of anyone successfully suing AWS/Google/Azure over lost data Those companies have huge legal departments writing iron-clad contracts and stocking a lot of very sharp knives for any such eventuality. Smaller companies make for much easier targets. > even if your notes are kept private, it'll still come out in discovery Those notes might have been lost by then - that is, if their existence at any given time is even known. It is perfectly reasonable for people working quickly after an outage, to not actually write down every step or observation they make. > What are the risks here? Not prison, but you might be forced to pay a bunch if someone manages to get a ruling against you for negligence or the likes.
- korzun 10y ago
  > I've never heard of anyone successfully suing AWS/Google/Azure over lost data or downtime. Doesn't mean it never happens.
  - chiefalchemist 10y ago
    Imagine how deep their pockets are. Now imagine how deep your pockets would need to be to go toe to toe with them. Sure plenty of uber larges Incs are "in the cloud" but with mission critical (read: high value worth sueing over) applications and data? That's a pretty small universe.