5 ms·

I myself initially added my name to the document in various parts, this was later changed to just initials. I specifically told my colleagues it was OK to keep

I myself initially added my name to the document in various parts, this was later changed to just initials. I specifically told my colleagues it was OK to keep it in the document. I have no problems taking responsibility for mistakes, and making sure they don't happen ever again.

jawilson2 10y ago
That's awesome, but why publicize it? This isn't an act of contrition for you, no one outside your team really needs to see your dirty laundry, and actually comes off as unprofessional to me. The gitlab team is a team, and you take responsibility as a team. Placing names and initials in the liveblog makes it look like SOMEONE is trying to assign and pass off blame, even if that is not what is happening. Presumably in the coming days there will be a number of team meetings where you discover what went wrong, and what the action items are for everyone moving forward. The public looking info just needs to say what went wrong, how it is being fixed, and what will be done in the future to prevent it from happening again. I don't need names to get that.
- Shank 10y ago
  There are quite a few comments on this very thread about how this creates trust, not ruins it. Thinking that there's a "right" and a "wrong" way (or an "unprofessional"/"dirty laundry" situation) is quite contrarian to the Gitlab transparency model and their culture. If you don't like their culture, don't work there.
- cjbprime 10y ago
  You're assuming the only reason for wanting to do it would be as contrition, but it sounds like that's not the reason here. Possibly the GitLab team cares about transparency to the extent that they simply prefer to be transparent.
  - throwaway91111 10y ago
    It's not transparent to name people; it's unnecessary and betrays a misunderstanding of how to handle an incident.
- omouse 10y ago
  On the contrary, it comes off as very professional. All other companies would hide this, they would show off a very cleaned up post-mortem and say "problem solved" and that's it. Ok so what does that mean, does it mean the process will change for the future or that they just fixed it for today? This is also an awesome advert to see how they work remotely all together and I'm sure they're hiring for DevOps people now ;)
  - throwaway91111 10y ago
    Naming individuals is not professional. Even allowing it with permission does not set a good standard for operation.
  - riralk 10y ago
    I agree the transparency was probably appreciated by most users. I also think the human factor should be considered not just the technical aspect. Here's my take on the issue... https://forgetfulprogrammer.wordpress.com/2017/02/04/key-learning-from-the-gitlab-incident/ https://forgetfulprogrammer.wordpress.com/2017/02/04/key-lea...
- mastre_ 10y ago
  IMO the idea of secrecy equating to professionalism is _the_ problem with many things. "Information wants to be free." It's also more personable, especially to those who use their product - to me, it shows they're on top of it, they care and are taking responsibility. Gives you a sense like you're part of the team (or they part of yours).
  - scaryclam 10y ago
    Keeping names out of incident reports isn't about secrecy. There's nothing stopping the folks at Gitlab posting up a retrospective blog post. Incident reports are formal documents published to let users and customers know what's going on. The names can come later, if all parties are OK with it.
    - yuhong 10y ago
      And they initially put in only initials and the developer posted in this thread only after it was published.
RA_Fisher 10y ago
That's courageous but this wasn't your mistake, it was the CEO's mistake. They owe you a vacation and apology for putting you in that terrible position!
- YorickPeterse 10y ago
  I'm not sure if it's the CEO's mistake, or any specific individual's mistake for that matter. In this particular case many different problems coalesced, producing one giant problem. If it wasn't me, somebody else would have eventually made the same mistake; perhaps with even greater impact.
  - pc86 10y ago
    Any blame that would be generalized to the company as a whole is also specifically the CEO's fault. The buck has to stop somewhere. That is part of the deal for the big chair/title/paycheck/expense account (at any company, not just Gitlab or in SV).
    - RA_Fisher 10y ago
      Exactly. The CEO clearly didn't properly weigh the impact of a failure like this on the reputation of the company. GitLab will lose many customers over this. Unfortunately choices like calling in (or allowing) the original engineer to work on recovery, not protecting the engineer from mistakes of the organization, etc. don't signal recognition of the required corrective action. That corrective action requires the CEO to take full responsibility, protect individual employees from process-driven failure, study the cultural aspects that allowed the failure to occur, etc.
      - sytse 10y ago
        I agree with the CEO being responsible. As I mentioned 15 hours ago in this HN post in https://news.ycombinator.com/item?id=13537245 https://news.ycombinator.com/item?id=13537245 "the blame is with all of us, starting with me. Not with the person in the arena."
testUser69 10y ago
I don't know why people on hacker news are against transparency. I'm glad you guys are live streaming this, others would feel too inadequate to do so. Being this transparent only makes me want to use (and contribute to) gitlab even more.
- Tephlon 10y ago
  I'm guessing they feel strongly about getting singled out if something like this would happen to them. Possibly because they have been used as a scapegoat by a employer or team mate once. /ArmchairPsychology
- rhizome 10y ago
  I don't know why people on hacker news are against transparency. It appears to be only a couplefew people on HN who have a real problem with it, so maybe temper your generalizations in a way that they aren't.
- toomuchtodo 10y ago
  > I don't know why people on hacker news are against transparency. > Being this transparent only makes me want to use (and contribute to) gitlab even more. I hope you'll be there when someone doesn't hire the person responsible for their mistake so you can vouch for them. You don't have radical transparency because the world is not the understanding meritocracy you think it is. There is no value to the employee for having radical transparency in a post mortem.
scaryclam 10y ago
I think perhaps, you want to not do this in the future. Incident reports are about focusing on the "what" and "when" not the "who". This is not about taking responsibility (you don't need to be published on the internet to do that) and you can always have a follow up post after the incident report has been published as a "what I learned during incident X". While it's great you're OK with publishing your name, you've now set a precedent that says it's OK to do this to other developers. A blanket policy on keeping names out of the incident report protects others who may not be as willing to get their name of HN (as well as not having to make amendments or retractions if the initial assumptions are incorrect). It also keeps a sense of professionalism as it's clear that no blame is being assigned. I know that you guys are not assigning blame, but if I was to show this to someone outside of this discussion, they'd assume that it was a fingerpointing exercise, which does not reflect well on Gitlab.
- rhizome 10y ago
  I think you're blowing it out of proportion. If you showed it to someone and they told you they assumed it was about fingerpo...Look, it's not that big a deal. They decided to do it, not everything is a blame game.
  - user15672 10y ago
    Woah there! I think you may have misread the parent as it looks like some friendly advice to me (with actual reasons and stuff), rather than the "You shouldn't have done that! You've destroyed your company!!!!" you seem to have read it as. Heck, they didn't even say to retract anything from the report, just maybe to leave adding names to things until a later date in future incidents.
    - rhizome 10y ago
      I'm not sure impugning their professionalism qualifies as friendly advice.
      - user15672 10y ago

jawilson2 10y ago

That's awesome, but why publicize it? This isn't an act of contrition for you, no one outside your team really needs to see your dirty laundry, and actually comes off as unprofessional to me. The gitlab team is a team, and you take responsibility as a team. Placing names and initials in the liveblog makes it look like SOMEONE is trying to assign and pass off blame, even if that is not what is happening. Presumably in the coming days there will be a number of team meetings where you discover what went wrong, and what the action items are for everyone moving forward. The public looking info just needs to say what went wrong, how it is being fixed, and what will be done in the future to prevent it from happening again. I don't need names to get that.

Shank 10y ago
There are quite a few comments on this very thread about how this creates trust, not ruins it. Thinking that there's a "right" and a "wrong" way (or an "unprofessional"/"dirty laundry" situation) is quite contrarian to the Gitlab transparency model and their culture. If you don't like their culture, don't work there.
cjbprime 10y ago
You're assuming the only reason for wanting to do it would be as contrition, but it sounds like that's not the reason here. Possibly the GitLab team cares about transparency to the extent that they simply prefer to be transparent.
- throwaway91111 10y ago
  It's not transparent to name people; it's unnecessary and betrays a misunderstanding of how to handle an incident.
omouse 10y ago
On the contrary, it comes off as very professional. All other companies would hide this, they would show off a very cleaned up post-mortem and say "problem solved" and that's it. Ok so what does that mean, does it mean the process will change for the future or that they just fixed it for today? This is also an awesome advert to see how they work remotely all together and I'm sure they're hiring for DevOps people now ;)
- throwaway91111 10y ago
  Naming individuals is not professional. Even allowing it with permission does not set a good standard for operation.
- riralk 10y ago
  I agree the transparency was probably appreciated by most users. I also think the human factor should be considered not just the technical aspect. Here's my take on the issue... https://forgetfulprogrammer.wordpress.com/2017/02/04/key-learning-from-the-gitlab-incident/ https://forgetfulprogrammer.wordpress.com/2017/02/04/key-lea...
mastre_ 10y ago
IMO the idea of secrecy equating to professionalism is _the_ problem with many things. "Information wants to be free." It's also more personable, especially to those who use their product - to me, it shows they're on top of it, they care and are taking responsibility. Gives you a sense like you're part of the team (or they part of yours).
- scaryclam 10y ago
  Keeping names out of incident reports isn't about secrecy. There's nothing stopping the folks at Gitlab posting up a retrospective blog post. Incident reports are formal documents published to let users and customers know what's going on. The names can come later, if all parties are OK with it.
  - yuhong 10y ago
    And they initially put in only initials and the developer posted in this thread only after it was published.

RA_Fisher 10y ago

That's courageous but this wasn't your mistake, it was the CEO's mistake. They owe you a vacation and apology for putting you in that terrible position!

YorickPeterse 10y ago
I'm not sure if it's the CEO's mistake, or any specific individual's mistake for that matter. In this particular case many different problems coalesced, producing one giant problem. If it wasn't me, somebody else would have eventually made the same mistake; perhaps with even greater impact.
- pc86 10y ago
  Any blame that would be generalized to the company as a whole is also specifically the CEO's fault. The buck has to stop somewhere. That is part of the deal for the big chair/title/paycheck/expense account (at any company, not just Gitlab or in SV).
  - RA_Fisher 10y ago
    Exactly. The CEO clearly didn't properly weigh the impact of a failure like this on the reputation of the company. GitLab will lose many customers over this. Unfortunately choices like calling in (or allowing) the original engineer to work on recovery, not protecting the engineer from mistakes of the organization, etc. don't signal recognition of the required corrective action. That corrective action requires the CEO to take full responsibility, protect individual employees from process-driven failure, study the cultural aspects that allowed the failure to occur, etc.
    - sytse 10y ago
      I agree with the CEO being responsible. As I mentioned 15 hours ago in this HN post in https://news.ycombinator.com/item?id=13537245 https://news.ycombinator.com/item?id=13537245 "the blame is with all of us, starting with me. Not with the person in the arena."

testUser69 10y ago

I don't know why people on hacker news are against transparency. I'm glad you guys are live streaming this, others would feel too inadequate to do so. Being this transparent only makes me want to use (and contribute to) gitlab even more.

Tephlon 10y ago
I'm guessing they feel strongly about getting singled out if something like this would happen to them. Possibly because they have been used as a scapegoat by a employer or team mate once. /ArmchairPsychology
rhizome 10y ago
I don't know why people on hacker news are against transparency. It appears to be only a couplefew people on HN who have a real problem with it, so maybe temper your generalizations in a way that they aren't.
toomuchtodo 10y ago
> I don't know why people on hacker news are against transparency. > Being this transparent only makes me want to use (and contribute to) gitlab even more. I hope you'll be there when someone doesn't hire the person responsible for their mistake so you can vouch for them. You don't have radical transparency because the world is not the understanding meritocracy you think it is. There is no value to the employee for having radical transparency in a post mortem.

scaryclam 10y ago

I think perhaps, you want to not do this in the future. Incident reports are about focusing on the "what" and "when" not the "who". This is not about taking responsibility (you don't need to be published on the internet to do that) and you can always have a follow up post after the incident report has been published as a "what I learned during incident X". While it's great you're OK with publishing your name, you've now set a precedent that says it's OK to do this to other developers. A blanket policy on keeping names out of the incident report protects others who may not be as willing to get their name of HN (as well as not having to make amendments or retractions if the initial assumptions are incorrect). It also keeps a sense of professionalism as it's clear that no blame is being assigned. I know that you guys are not assigning blame, but if I was to show this to someone outside of this discussion, they'd assume that it was a fingerpointing exercise, which does not reflect well on Gitlab.

rhizome 10y ago
I think you're blowing it out of proportion. If you showed it to someone and they told you they assumed it was about fingerpo...Look, it's not that big a deal. They decided to do it, not everything is a blame game.
- user15672 10y ago
  Woah there! I think you may have misread the parent as it looks like some friendly advice to me (with actual reasons and stuff), rather than the "You shouldn't have done that! You've destroyed your company!!!!" you seem to have read it as. Heck, they didn't even say to retract anything from the report, just maybe to leave adding names to things until a later date in future incidents.
  - rhizome 10y ago
    I'm not sure impugning their professionalism qualifies as friendly advice.
    - user15672 10y ago