4 ms·

As someone who's worked on similar situations, I don't expect any pat on the back, but see it as my responsibility to make sure it doesn't happen in the first p

by nomilk 5y ago

As someone who's worked on similar situations, I don't expect any pat on the back, but see it as my responsibility to make sure it doesn't happen in the first place, and when it does, my responsibility to fix it without complaint. Some Atlassian customers might have had much more severe (mental health and other) problems than the Atlassian staff, so I think it can be perceived as mildly solipsistic to be praising the staff and (being perceived to) forget about the customers' wellbeing (I know you didn't say anything about forgetting about the customers, but affected customers may perceive it that way).

eyelidlessness 5y ago
This is a fascinating response. I’ll address the last part first, because I don’t want it to get lost. > I know you didn't say anything about forgetting about the customers, but affected customers may perceive it that way No, I didn’t mean to suggest or imply this, and hope it won’t be taken this way by anyone. It was perhaps a mistake taking it as read that obviously their customers were also harmed. But I do think it’s a reasonable point to make more explicit, and I do agree that it likely had a similar impact on customers to the experience I described. I’ll repeat that no one should experience that. That said… > As someone who's worked on similar situations, I don't expect any pat on the back, but see it as my responsibility to make sure it doesn't happen in the first place, and when it does, my responsibility to fix it without complaint. This seems like you’re addressing something entirely outside the actual content of my comment, and perhaps projecting your own priorities onto it. First of all, I was in no way suggesting anyone get special reward. And I was in no way referencing any Atlassian employee’s complaints nor airing my own. I mean it sincerely that I hope they are okay and that their mental health needs are being respected. It was entirely a statement of human compassion, and an observation that it’s one which can go unstated/understated in these discussions. Secondly, I added my own experience as a personal reflection on the toll it can take. It’s not easy to say in a public forum that I’ve suffered years of chronic pain after addressing an incident. I added this for context because I think it is easy for people to dismiss the impact serious incidents have on the people responsible to them. A side note before I get to thirdly: I think this is also true for people in many careers where incident response is a primary job responsibility. Sometimes it prompts explicit acknowledgement, often it does not. I just think the world would be better off if more people’s legitimate pain and challenges were acknowledged. Third point: I took special care above to say “responsible to”, not “responsible for” or simply “responsible”. I’m certain, given Atlassian’s size and the impact of this outage, that many of the people involved in recovery efforts played no role in the mistakes leading to the outage. In my own anecdote, I played no role in causing the incident. I have to walk a fine professional line here because I have no intention or desire to criticize anyone else involved. I think I can reasonably say this. I did what you described: > see it as my responsibility to make sure it doesn't happen in the first place, and when it does, my responsibility to fix it without complaint. Even so, I’m experiencing chronic pain years later. And now I will complain: it sucks being in pain every day for years. I wouldn’t go back and do anything differently as an IC, except perhaps to tell past me when to slow down and that I have more ability to affect short term prioritization than I once realized. Lastly, > praising the staff I sincerely hope this hasn’t had the negative impact on you which you’re describing as hypothetical for Atlassian customers generally. If it has, I hope you’ll hear me when I say any sharp tone in this response is not from lack of solidarity. But if it hasn’t and you’re replying only out of work ethic and blame-placing: kindly go re-read my comment and recognize that the only praise I expressed was for the humanity of human people whose experience I wish to include in the conversation.
- nomilk 5y ago
  Thanks a lot for sharing the experience, especially here in a public forum. Not an easy thing to do. I only have pure respect for you, and am grateful for you sharing the experience. Sorry for my lack of clarity, I intended for my comment to regard Atlassian and in general, and definitely not to downplay anyone's specific situation.
cgio 5y ago
Speaking as someone who is no longer involved at the coal face of things like that, my role in an equivalent incident resolution would be to attend some of the 3-hourly calls (this frequency sounds extreme past the first 3-4days btw), and therefore having nothing to lose from siding with you on this, I’ll disagree. Accountability would be mine, responsibility only to fix of the team. I.e. the team needs many pats on the back, as they break their backs to solve something the organisation has done wrong, such as in this case not implementing proper controls on the processes. It’s never about the running code, it’s about how it ended up in production. People make mistakes, are assigned work outside their comfort zone, are junior. And these failure modes are also operational growth modalities. But the processes need to support that growth and focusing on the debugging of the issue is necessary for resolution but is wrong when the focus moves to long term avoidance. You can praise staff but cannot praise the customer unfortunately as that would be inappropriate. You can work on relationship of course, by not limiting your reconciliation to contractual credits on SLA. These will mean nothing to the customer staff that took significant part of the hit in work stress. Not sure what would be a good way to regain the trust at that level. Edit: fixed accountability vs responsibility in wrong order
Aeolun 5y ago
I find it hard to imagine mental health being affected negatively by atlassian services disappearing.
- eyelidlessness 5y ago
  I find it exceedingly hard to imagine how it wouldn’t. Think of how people use their products, and the cascading effect of that use being disrupted with long term interrupted access to information that was probably stored only there. Think of whole organizations scrambling to recreate knowledge that’s only partially in their heads.
daniel_iversen 5y ago
I think that’s a false trade off, you can empathise with both employees and customers at the same time. And while sure maybe a lot of the Atlassians might have fixed the issue “without complaint” as you said, it can still be a nightly stressful situation (probably even leading to burnout) and I hope they are looked after even if they don’t speak up.

by nomilk 5y ago

eyelidlessness 5y ago

This is a fascinating response. I’ll address the last part first, because I don’t want it to get lost. > I know you didn't say anything about forgetting about the customers, but affected customers may perceive it that way No, I didn’t mean to suggest or imply this, and hope it won’t be taken this way by anyone. It was perhaps a mistake taking it as read that obviously their customers were also harmed. But I do think it’s a reasonable point to make more explicit, and I do agree that it likely had a similar impact on customers to the experience I described. I’ll repeat that no one should experience that. That said… > As someone who's worked on similar situations, I don't expect any pat on the back, but see it as my responsibility to make sure it doesn't happen in the first place, and when it does, my responsibility to fix it without complaint. This seems like you’re addressing something entirely outside the actual content of my comment, and perhaps projecting your own priorities onto it. First of all, I was in no way suggesting anyone get special reward. And I was in no way referencing any Atlassian employee’s complaints nor airing my own. I mean it sincerely that I hope they are okay and that their mental health needs are being respected. It was entirely a statement of human compassion, and an observation that it’s one which can go unstated/understated in these discussions. Secondly, I added my own experience as a personal reflection on the toll it can take. It’s not easy to say in a public forum that I’ve suffered years of chronic pain after addressing an incident. I added this for context because I think it is easy for people to dismiss the impact serious incidents have on the people responsible to them. A side note before I get to thirdly: I think this is also true for people in many careers where incident response is a primary job responsibility. Sometimes it prompts explicit acknowledgement, often it does not. I just think the world would be better off if more people’s legitimate pain and challenges were acknowledged. Third point: I took special care above to say “responsible to”, not “responsible for” or simply “responsible”. I’m certain, given Atlassian’s size and the impact of this outage, that many of the people involved in recovery efforts played no role in the mistakes leading to the outage. In my own anecdote, I played no role in causing the incident. I have to walk a fine professional line here because I have no intention or desire to criticize anyone else involved. I think I can reasonably say this. I did what you described: > see it as my responsibility to make sure it doesn't happen in the first place, and when it does, my responsibility to fix it without complaint. Even so, I’m experiencing chronic pain years later. And now I will complain: it sucks being in pain every day for years. I wouldn’t go back and do anything differently as an IC, except perhaps to tell past me when to slow down and that I have more ability to affect short term prioritization than I once realized. Lastly, > praising the staff I sincerely hope this hasn’t had the negative impact on you which you’re describing as hypothetical for Atlassian customers generally. If it has, I hope you’ll hear me when I say any sharp tone in this response is not from lack of solidarity. But if it hasn’t and you’re replying only out of work ethic and blame-placing: kindly go re-read my comment and recognize that the only praise I expressed was for the humanity of human people whose experience I wish to include in the conversation.

nomilk 5y ago
Thanks a lot for sharing the experience, especially here in a public forum. Not an easy thing to do. I only have pure respect for you, and am grateful for you sharing the experience. Sorry for my lack of clarity, I intended for my comment to regard Atlassian and in general, and definitely not to downplay anyone's specific situation.

cgio 5y ago

Speaking as someone who is no longer involved at the coal face of things like that, my role in an equivalent incident resolution would be to attend some of the 3-hourly calls (this frequency sounds extreme past the first 3-4days btw), and therefore having nothing to lose from siding with you on this, I’ll disagree. Accountability would be mine, responsibility only to fix of the team. I.e. the team needs many pats on the back, as they break their backs to solve something the organisation has done wrong, such as in this case not implementing proper controls on the processes. It’s never about the running code, it’s about how it ended up in production. People make mistakes, are assigned work outside their comfort zone, are junior. And these failure modes are also operational growth modalities. But the processes need to support that growth and focusing on the debugging of the issue is necessary for resolution but is wrong when the focus moves to long term avoidance. You can praise staff but cannot praise the customer unfortunately as that would be inappropriate. You can work on relationship of course, by not limiting your reconciliation to contractual credits on SLA. These will mean nothing to the customer staff that took significant part of the hit in work stress. Not sure what would be a good way to regain the trust at that level. Edit: fixed accountability vs responsibility in wrong order

Aeolun 5y ago

I find it hard to imagine mental health being affected negatively by atlassian services disappearing.

eyelidlessness 5y ago
I find it exceedingly hard to imagine how it wouldn’t. Think of how people use their products, and the cascading effect of that use being disrupted with long term interrupted access to information that was probably stored only there. Think of whole organizations scrambling to recreate knowledge that’s only partially in their heads.

daniel_iversen 5y ago

I think that’s a false trade off, you can empathise with both employees and customers at the same time. And while sure maybe a lot of the Atlassians might have fixed the issue “without complaint” as you said, it can still be a nightly stressful situation (probably even leading to burnout) and I hope they are looked after even if they don’t speak up.