11 ms·

The public report is nice and we can see a sequence of mishaps from it, that shouldn't have been allowed to happen but which (unfortunately) are not that uncomm

by CrLf 10y ago

The public report is nice and we can see a sequence of mishaps from it, that shouldn't have been allowed to happen but which (unfortunately) are not that uncommon. I've done my share of mistakes, I know what's like to be in emergency mode and too tired to think straight, so I'm going to refrain from criticizing individual actions. What I'm going to criticize is the excess of transparency: You absolutely DO NOT publish postmortems referencing actions by NAMED individuals, EVER. From reading the whole report it's clear that the group is at fault, not a single individual. But most people won't read the whole thing, even less people will try to understand the whole picture. That's why failures are always attributed publicly to the whole team, and actions by individuals are handled internally only. And they're making it even worse by livestreaming the thing! It's like having your boss looking over your shoulder but a million times worse...

jjawssd 10y ago
Maybe these individuals don't mind, it could just be a cultural thing.
- deleted 10y ago
  [deleted]
mverwijs 10y ago
At one company I worked for we had a saying: "You're not one of the team until you've brought down the network. " We all mess up. Much respect to gitlab for being open about.
- drvdevd 10y ago
  That's the kind of team I want! All hands on deck. No lame responsibility shifters.
nanch 10y ago
Anybody whose opinion matters understands that this type of event is a process problem, not a person problem. GitLab has always blazed their own trail with their transparency, whether through their open run books, open source code, or in this case their open problem resolution. Kudos to them in whatever manner they want to do it in (with or without names). To be honest, through all of the comments, yours seems the most high-strung, and you're the one complaining about high-pressure situations like having your boss looking over your shoulder. Relax buddy. :)
- CrLf 10y ago
  In a few years the guy doing the `rm -rf` is going to be on a job interview and someone will recall bits of this report. Enough bits to remember the guy, not enough bits to remember that it wasn't his (individual) fault. Transparency doesn't mean publicly throwing people under the bus. I'm not a GitLab customer, I'm relaxed. :)
  - avar 10y ago
    That guy is going to be interviewing at some company with someone who's obsessive enough about outage reports to remember a then-obscure one years later, but enough of an idiot to not understand that people aren't personally to blame for this sort of stuff? Sounds like even in that very contrived scenario the guy involved would dodge a bullet in not being hired by a bunch of idiots.
    - nsxwolf 10y ago
      "I worked at GitLab." Googles name + GitLab, finds postmortem Highly likely, and now you don't get to tell your own story and emphasize what you want to.
      - woof 10y ago
        Awsome postmortem -is there any thing you would do differently today? What's your most valuable lesson from that incident? You're hired!
    - drvdevd 10y ago
      also, maybe some people on here are perfect, but if you've used Unix for more than half your life (as I have) you've 'rm -rf'-ed some stuff. I think people who've been through disasters have a much better understanding of the importance and methods of not ending up there than those with a perfectly clean record. IOW, I'd hire the "rm -rf" guy first if he owns it.
  - OskarS 10y ago
    Honestly, if I were interviewing the guy, that would almost be a bonus! Like, everyone makes mistakes, we're all human, but I can guarantee you that THAT person will never make that particular mistake ever again. And he's going to be 10 times more diligent than the average engineer in making sure there are good backup/restore procedures.
mateus1 10y ago
Yep, I'd feel awful if I were the employee in this headline "GitLab Goes Down After Employee Deletes the Wrong Folder" [0]. It's the process and the team who are at fault. [0] https://www.bleepingcomputer.com/news/hardware/gitlab-goes-down-after-employee-deletes-the-wrong-folder/ https://www.bleepingcomputer.com/news/hardware/gitlab-goes-d...
sbuttgereit 10y ago
I completely agree. Trying to get low level details out to the public while in the heat of the issue is a misstep; you can still be transparent while not risking over communication that could haunt you later. While I think most of the HN audience understands that some days you have a bad day and that sometimes very small actions, like typing a single command at a prompt, can have dramatic consequences in technology, there are nonetheless less enlightened souls in hiring positions that simply might find fragments of this in a search on the name when that time comes. Being too transparent could also encourage legal problems, too, if someone decides that they had a material loss over this, at least for the company. Terms of service or likelihood of a challenge prevailing doesn't necessarily matter: you can be sued for any reason and since there's no loser pay provision in any U.S. jurisdiction that I know of, even a win in court could be very costly. Being overly transparent in a case like this can bolster a claim of gross negligence (justified or not) and the law/courts/judges/juries cannot be relied upon to be consistently rational or properly informed. Part of the problem is that this isn't actually a postmortem: they're basic live blogging/streaming in real time. What would be helpful for us (users) and them (GitLab) in terms of real-time transparency: * Acknowledge there were problem during a maintenance and data may/may not have been lost. * If some data is known to be safe: what data that is. * What stage are we at. Still figuring it out? Waiting for backups to restore? Verification? * Broad -estimated- time to recovery: make clear it's a guess. Even coarsely: days away, 10's of hours away, etc. * When to expect the next public update on the issue. None of this needs to be very detailed and likely shouldn't include actual technical detail. It just needs to be honest, forthright, and timely. That meets the transparency test while also protecting employees and the company. Later, when there is time for thoughtful consideration, a technical postmortem at a good level of detail is completely appropriate. [edit for clarity]
Ajedi32 10y ago
Am I missing something? Where in this report are any individuals actually named? My understanding was that they're using initials in place of names specifically because they want to _avoid_ naming anyone.
- sbuttgereit 10y ago
  The original versions of the document had names. Those were later replaced with initials. I think the issue was in part that this document didn't appear to be a public "here's what's going on doc" as much as it was a doc they seemed to be using as a focal point for their own coordination efforts.
YorickPeterse 10y ago
I myself initially added my name to the document in various parts, this was later changed to just initials. I specifically told my colleagues it was OK to keep it in the document. I have no problems taking responsibility for mistakes, and making sure they don't happen ever again.
- jawilson2 10y ago
  That's awesome, but why publicize it? This isn't an act of contrition for you, no one outside your team really needs to see your dirty laundry, and actually comes off as unprofessional to me. The gitlab team is a team, and you take responsibility as a team. Placing names and initials in the liveblog makes it look like SOMEONE is trying to assign and pass off blame, even if that is not what is happening. Presumably in the coming days there will be a number of team meetings where you discover what went wrong, and what the action items are for everyone moving forward. The public looking info just needs to say what went wrong, how it is being fixed, and what will be done in the future to prevent it from happening again. I don't need names to get that.
  - Shank 10y ago
    There are quite a few comments on this very thread about how this creates trust, not ruins it. Thinking that there's a "right" and a "wrong" way (or an "unprofessional"/"dirty laundry" situation) is quite contrarian to the Gitlab transparency model and their culture. If you don't like their culture, don't work there.
  - cjbprime 10y ago
    You're assuming the only reason for wanting to do it would be as contrition, but it sounds like that's not the reason here. Possibly the GitLab team cares about transparency to the extent that they simply prefer to be transparent.
    - throwaway91111 10y ago
      It's not transparent to name people; it's unnecessary and betrays a misunderstanding of how to handle an incident.
ejcx 10y ago
Many have echo'ed you, but I agree. The person who made the error is just the straw that broke the camels back. I'm sure these folks knew that they needed to prioritize their backups but other things kept getting in the way. You don't throw people under the bus.
beamatronic 10y ago
Modern companies tend to have people in roles and teams. What I've done in postmortems is to use role names and team names, not person's names. Even if the team is just one person. This helps keep it about the team and the process. We're all professionals doing our best and striving for continuous improvement.

by CrLf 10y ago

jjawssd 10y ago

Maybe these individuals don't mind, it could just be a cultural thing.

deleted 10y ago
[deleted]

mverwijs 10y ago

At one company I worked for we had a saying: "You're not one of the team until you've brought down the network. " We all mess up. Much respect to gitlab for being open about.

drvdevd 10y ago
That's the kind of team I want! All hands on deck. No lame responsibility shifters.

nanch 10y ago

Anybody whose opinion matters understands that this type of event is a process problem, not a person problem. GitLab has always blazed their own trail with their transparency, whether through their open run books, open source code, or in this case their open problem resolution. Kudos to them in whatever manner they want to do it in (with or without names). To be honest, through all of the comments, yours seems the most high-strung, and you're the one complaining about high-pressure situations like having your boss looking over your shoulder. Relax buddy. :)

CrLf 10y ago
In a few years the guy doing the `rm -rf` is going to be on a job interview and someone will recall bits of this report. Enough bits to remember the guy, not enough bits to remember that it wasn't his (individual) fault. Transparency doesn't mean publicly throwing people under the bus. I'm not a GitLab customer, I'm relaxed. :)
- avar 10y ago
  That guy is going to be interviewing at some company with someone who's obsessive enough about outage reports to remember a then-obscure one years later, but enough of an idiot to not understand that people aren't personally to blame for this sort of stuff? Sounds like even in that very contrived scenario the guy involved would dodge a bullet in not being hired by a bunch of idiots.
  - nsxwolf 10y ago
    "I worked at GitLab." Googles name + GitLab, finds postmortem Highly likely, and now you don't get to tell your own story and emphasize what you want to.
    - woof 10y ago
      Awsome postmortem -is there any thing you would do differently today? What's your most valuable lesson from that incident? You're hired!
  - drvdevd 10y ago
    also, maybe some people on here are perfect, but if you've used Unix for more than half your life (as I have) you've 'rm -rf'-ed some stuff. I think people who've been through disasters have a much better understanding of the importance and methods of not ending up there than those with a perfectly clean record. IOW, I'd hire the "rm -rf" guy first if he owns it.
- OskarS 10y ago
  Honestly, if I were interviewing the guy, that would almost be a bonus! Like, everyone makes mistakes, we're all human, but I can guarantee you that THAT person will never make that particular mistake ever again. And he's going to be 10 times more diligent than the average engineer in making sure there are good backup/restore procedures.

mateus1 10y ago

Yep, I'd feel awful if I were the employee in this headline "GitLab Goes Down After Employee Deletes the Wrong Folder" [0]. It's the process and the team who are at fault. [0] https://www.bleepingcomputer.com/news/hardware/gitlab-goes-down-after-employee-deletes-the-wrong-folder/ https://www.bleepingcomputer.com/news/hardware/gitlab-goes-d...

sbuttgereit 10y ago

I completely agree. Trying to get low level details out to the public while in the heat of the issue is a misstep; you can still be transparent while not risking over communication that could haunt you later. While I think most of the HN audience understands that some days you have a bad day and that sometimes very small actions, like typing a single command at a prompt, can have dramatic consequences in technology, there are nonetheless less enlightened souls in hiring positions that simply might find fragments of this in a search on the name when that time comes. Being too transparent could also encourage legal problems, too, if someone decides that they had a material loss over this, at least for the company. Terms of service or likelihood of a challenge prevailing doesn't necessarily matter: you can be sued for any reason and since there's no loser pay provision in any U.S. jurisdiction that I know of, even a win in court could be very costly. Being overly transparent in a case like this can bolster a claim of gross negligence (justified or not) and the law/courts/judges/juries cannot be relied upon to be consistently rational or properly informed. Part of the problem is that this isn't actually a postmortem: they're basic live blogging/streaming in real time. What would be helpful for us (users) and them (GitLab) in terms of real-time transparency: * Acknowledge there were problem during a maintenance and data may/may not have been lost. * If some data is known to be safe: what data that is. * What stage are we at. Still figuring it out? Waiting for backups to restore? Verification? * Broad -estimated- time to recovery: make clear it's a guess. Even coarsely: days away, 10's of hours away, etc. * When to expect the next public update on the issue. None of this needs to be very detailed and likely shouldn't include actual technical detail. It just needs to be honest, forthright, and timely. That meets the transparency test while also protecting employees and the company. Later, when there is time for thoughtful consideration, a technical postmortem at a good level of detail is completely appropriate. [edit for clarity]

Ajedi32 10y ago

Am I missing something? Where in this report are any individuals actually named? My understanding was that they're using initials in place of names specifically because they want to _avoid_ naming anyone.

sbuttgereit 10y ago
The original versions of the document had names. Those were later replaced with initials. I think the issue was in part that this document didn't appear to be a public "here's what's going on doc" as much as it was a doc they seemed to be using as a focal point for their own coordination efforts.

YorickPeterse 10y ago

I myself initially added my name to the document in various parts, this was later changed to just initials. I specifically told my colleagues it was OK to keep it in the document. I have no problems taking responsibility for mistakes, and making sure they don't happen ever again.

jawilson2 10y ago
That's awesome, but why publicize it? This isn't an act of contrition for you, no one outside your team really needs to see your dirty laundry, and actually comes off as unprofessional to me. The gitlab team is a team, and you take responsibility as a team. Placing names and initials in the liveblog makes it look like SOMEONE is trying to assign and pass off blame, even if that is not what is happening. Presumably in the coming days there will be a number of team meetings where you discover what went wrong, and what the action items are for everyone moving forward. The public looking info just needs to say what went wrong, how it is being fixed, and what will be done in the future to prevent it from happening again. I don't need names to get that.
- Shank 10y ago
  There are quite a few comments on this very thread about how this creates trust, not ruins it. Thinking that there's a "right" and a "wrong" way (or an "unprofessional"/"dirty laundry" situation) is quite contrarian to the Gitlab transparency model and their culture. If you don't like their culture, don't work there.
- cjbprime 10y ago
  You're assuming the only reason for wanting to do it would be as contrition, but it sounds like that's not the reason here. Possibly the GitLab team cares about transparency to the extent that they simply prefer to be transparent.
  - throwaway91111 10y ago
    It's not transparent to name people; it's unnecessary and betrays a misunderstanding of how to handle an incident.

ejcx 10y ago

Many have echo'ed you, but I agree. The person who made the error is just the straw that broke the camels back. I'm sure these folks knew that they needed to prioritize their backups but other things kept getting in the way. You don't throw people under the bus.

beamatronic 10y ago

Modern companies tend to have people in roles and teams. What I've done in postmortems is to use role names and team names, not person's names. Even if the team is just one person. This helps keep it about the team and the process. We're all professionals doing our best and striving for continuous improvement.