5 ms·

I appreciate the story and the pride he must have felt. But this is the entirely wrong message. It perpetuates that the superstars are those jumping in last sec

I appreciate the story and the pride he must have felt. But this is the entirely wrong message. It perpetuates that the superstars are those jumping in last second in some heroic effort to save the day. This is not how projects should work, and we should stop perpetuating these heroic stories. The actual superstars are all those people doing the less-than-glamorous grunt work. Planning things on time. Getting the infra ready, stable, well-tested, documented. Making schedules that have margins. The people carefully designing things so they are robust and don't break left and right. Thinking through their systematic testing. In short, preparing everything so that heroic acts are not necessary. Imagine the same story but somebody two months earlier had voiced a concern that Jabberwocky and ChaChing wouldn't play well together. Pushing for the APIs to be harmonized so that they could play together and integrate. Pushing all this against pressure that they are just delaying everything and should not worry so much and be more positive and agile. Imagine they had succeeded and everything had gone smoothly. This would have been much better for the company (since the existential risk wouldn't have existed in the first place). But that somebody wouldn't have ended being the superstar with the promotion and the bonus. It would likely have stuck to have been the guy just delaying things with their worries. That's what needs to change in sw eng culture.

jwatzman 4y ago
Author here. For what it's worth, I (mostly) think you're right! IMO heroics were a huge problem at that company, and are a huge problem at the company I work at now too. The gruntwork is really really important. I tried to get at this in the last section, that it's not about going out trying to be a hero, and it's unfortunate it didn't get across. That was my point that you shouldn't go look for company priorities to fix, to be a hero. Just look around you, see what's going on, and do good shit. Sometimes yeah that does involve heroics, sometimes it's just gruntwork. I don't want to go back and edit the essay now but I think it could be much clearer indeed! > Imagine the same story but somebody two months earlier had voiced a concern that Jabberwocky and ChaChing wouldn't play well together. Pushing for the APIs to be harmonized so that they could play together and integrate. [...] Disagree here, though it's maybe not obvious from the bits of the story I told why. ChaChing was one of... maybe three or four dozen experiments like it. All of the others failed. There was no reason to believe ChaChing would be any different, and if it did work, even the best-case expectations were well below what it ended up doing. It really was a lightning strike. So it would have been a huge waste of time to do all of that work, delaying Jabberwocky, for a bunch of things which never ended up shipping.
- gopalv 4y ago
  > you shouldn't go look for company priorities to fix, to be a hero The "to be a hero" is the problem, you can always feel good about solving hard problems and finding your boundaries, but "avoiding a crisis" is a thankless job & we should thank those people more than the "only I can fix it" people. The core problem for me is that the constructive "avoid crisis" people are baked out of these sort of events, rather than born risk averse in the first place. > Just look around you, see what's going on, and do good shit. One of the biggest side-effects of WFH was that this whole "who's around" mechanic has faded away without a proper replacement. Most of my satisfying work came from random interactions like this (not the most impactful or the most purposeful). I was a principal engineer and most of my 10 AM to 2 PM was taken up by meetings which mostly involved prevention of work duplication or to stop someone driving down to where I knew there was a dead-end (like, "I see you're using alpine docker images, why?" or more management boundary stuff like "the difference between these two ideas is ~140k over a year for all customers - can we agree that the expensive one is better for customers, because they don't see a 12$ month bill and it costs more than 140k$ to build the cheap one in NRE?"). Most of that work is tiring and always unproductive - the actual work is done by smart people who answer my questions carefully, the dumber the questions, usually the better the results. But sometime around 3 pm, I'd be line to get a coffee and I would run into a PM or someone from support or dev who would have a question for me which sounds nonsensical at first - but having an hour & half to go down that rabbit hole looking over someone's shoulder usually ended up with mostly "hey, I know what's happening, I've seen this before" or the classic "hmm, that's strange, how could that even happen?". Six to ten weeks later, I would be in a meeting where I would get some sense of "deja vu" when someone describes their approach and reach back to that day & that problem, to ask "how would you detect X" (something like "machine reboots, comes back with same hostname, but different IP" - would you restablish connections to IP or hostname, what about krb5 ... yada yada). That part is the real value as an engineer, but entirely unappreciated from the management's point of view. With WFH, I would go get a coffee, go for a run/walk and as good as it was for my state of well being, I would eventually get pulled into these crisis situations Thursday at 9 pm instead of Tuesday at 3. I like WFH for the work I actually have to do, but this sort of temporal serendipity is not easy to recreate on Zoom.
  - 4y ago
not2b 4y ago
While this is true, it's also true that the people who are doing the less-than-glamorous grunt work are often unrecognized by management (which arguably is a management problem). I remember being at an all-hands company meeting years ago and listening carefully to what the people who were given the top awards got them for, and about three in a row were for diving in, working nights and weekends to come up with emergency fixes to save projects or products that had gone seriously wrong, and these were different projects. It was so blatant then even some senior managers finally noticed "Hey, we have a problem".
morelisp 4y ago
I think there's two stories intertwining here. One which you've zeroed in on is "don't require heroism". We have a lot of these stories already. (Unfortunately also a lot of exciting stories of "heroism".) The more interesting one though, is that sometimes high-impact work is "just there" to be done, if you're looking. IMO this is also really what "hero" stories are about: something goes so wrong that either people are finally forced to either look for once, or forced to bring in someone who will look. As a senior engineer, for the past couple years I've tried to pitch parts of what I see "just there" intentionally, either before it becomes a crisis or just as a potential win. Sometimes people still don't see it; maybe they're stuck in "one year of experience, ten times" or sometimes they're just built out of stop energy. But the best way I've found to detect promising junior engineers who will benefit from mentorship is to find who does bite, and more often respond "what if" instead of "but".
- jwatzman 4y ago
  > The more interesting one though, is that sometimes high-impact work is "just there" to be done, if you're looking. Author here. Yes, this was exactly my point! > But the best way I've found to detect promising junior engineers who will benefit from mentorship is to find who does bite, and more often respond "what if" instead of "but". Indeed. I've mentored a bunch of otherwise-competent engineers trying to get to the next level, and my advice is similar: ask yourself "ok, now what's next". After doing the cool thing, there's inevitably half a dozen other things around it, and things might be obvious to you (as the now-expert in the cool thing) but not others.
  - necovek 4y ago
    > Author here. Yes, this was exactly my point! Your original example shows you doing extra hours and long days to achieve this: if this was your point, the example is terrible. I do agree with high-impact work just being there, but finding the time to do it other than "on your own time" is where the tension is between management and engineers who have a different belief of what the highest priority is.
    - 4y ago
aidenn0 4y ago
Firemen will always get more praise than people who change the batteries in smoke detectors.
- dilyevsky 4y ago
  False. People who quickly and cheaply build houses that kill everyone inside in case of small fires get all the praise, not firemen or whoever changes batteries (usually nobody)
  - aidenn0 4y ago
    Both of our analogies are true, but what they are analogs for is different. In my case, the house is the business goals of the organization. In your case the house is the product made by the organization.
necovek 4y ago
And not to be discounted, all those prideful references to long days and late nights: another symptom of bad planning and management. It also puts a bunch of people who have actual life commitments (like kids, health issues, family issues or whatever else) at a disadvantage simply because somebody can afford to spend the extra time on something. And if those are the people who properly plan and execute, they get a double whammy. The moral of the story should not be that when there is an opportunity to make a significant contribution by exchanging your life for your work you do that, but that you should be able to excel at work and be recognized for it without exchanging your life for it. Sure, the author of the post does not have anything to do with most of the management failures, except the failure to not work extra hours for a very artificial deadline (one does not need a hero to save face if you've messed up yourself — exposing the failure is usually a net win).
tonyarkles 4y ago
You are totally right. 100%. For over a decade, I basically sold myself as a hero-for-hire (or as one of my clients called me, a mercenary). My general schtick was two-fold: help the company/team out of the immediate problem they were facing, and, if they were game, spend some time with the team to help figure out how the problem came to be and how they might acquire the skills to prevent it (ideally) or solve it on their own next time. Looking back, I am still pretty amazed at the turn-around I helped facilitate at a bunch of smaller companies. Presently, I am an employee, and this is one of the most painful uphill battles I have ever had to fight. I have tried warning colleagues about issues that seem obvious to me, but face exceptional pushback all the time, only to have things blow up exactly how I predicted a few months (or sometimes weeks) later. In my own work, people generally let me be and do good work, but as soon as it becomes a group effort the “damn the torpedoes full speed ahead” minority steam rolls ahead and makes a mess of it. The worst part is, I know exactly what the problem is, because I’ve been on the other side of it: organizations pay way more attention to external consultants than internal resources. 90% of the advice I have companies for how to change was suggestions gathered from their own employees, slightly curated and abridged into something easily digestible. And so, while I’m exhausted from trying to change things from inside, when shit hits the fan, I step up and do the old mercenary thing to solve problems at the last minute and great expense. I keep trying because I do have a ton of exceptionally bright colleagues who I love working with, as well as an actually world changing technology. We’ll see… after this next demo… maybe we’ll take some time to breathe and get back on track.

jwatzman 4y ago

Author here. For what it's worth, I (mostly) think you're right! IMO heroics were a huge problem at that company, and are a huge problem at the company I work at now too. The gruntwork is really really important. I tried to get at this in the last section, that it's not about going out trying to be a hero, and it's unfortunate it didn't get across. That was my point that you shouldn't go look for company priorities to fix, to be a hero. Just look around you, see what's going on, and do good shit. Sometimes yeah that does involve heroics, sometimes it's just gruntwork. I don't want to go back and edit the essay now but I think it could be much clearer indeed! > Imagine the same story but somebody two months earlier had voiced a concern that Jabberwocky and ChaChing wouldn't play well together. Pushing for the APIs to be harmonized so that they could play together and integrate. [...] Disagree here, though it's maybe not obvious from the bits of the story I told why. ChaChing was one of... maybe three or four dozen experiments like it. All of the others failed. There was no reason to believe ChaChing would be any different, and if it did work, even the best-case expectations were well below what it ended up doing. It really was a lightning strike. So it would have been a huge waste of time to do all of that work, delaying Jabberwocky, for a bunch of things which never ended up shipping.

gopalv 4y ago
> you shouldn't go look for company priorities to fix, to be a hero The "to be a hero" is the problem, you can always feel good about solving hard problems and finding your boundaries, but "avoiding a crisis" is a thankless job & we should thank those people more than the "only I can fix it" people. The core problem for me is that the constructive "avoid crisis" people are baked out of these sort of events, rather than born risk averse in the first place. > Just look around you, see what's going on, and do good shit. One of the biggest side-effects of WFH was that this whole "who's around" mechanic has faded away without a proper replacement. Most of my satisfying work came from random interactions like this (not the most impactful or the most purposeful). I was a principal engineer and most of my 10 AM to 2 PM was taken up by meetings which mostly involved prevention of work duplication or to stop someone driving down to where I knew there was a dead-end (like, "I see you're using alpine docker images, why?" or more management boundary stuff like "the difference between these two ideas is ~140k over a year for all customers - can we agree that the expensive one is better for customers, because they don't see a 12$ month bill and it costs more than 140k$ to build the cheap one in NRE?"). Most of that work is tiring and always unproductive - the actual work is done by smart people who answer my questions carefully, the dumber the questions, usually the better the results. But sometime around 3 pm, I'd be line to get a coffee and I would run into a PM or someone from support or dev who would have a question for me which sounds nonsensical at first - but having an hour & half to go down that rabbit hole looking over someone's shoulder usually ended up with mostly "hey, I know what's happening, I've seen this before" or the classic "hmm, that's strange, how could that even happen?". Six to ten weeks later, I would be in a meeting where I would get some sense of "deja vu" when someone describes their approach and reach back to that day & that problem, to ask "how would you detect X" (something like "machine reboots, comes back with same hostname, but different IP" - would you restablish connections to IP or hostname, what about krb5 ... yada yada). That part is the real value as an engineer, but entirely unappreciated from the management's point of view. With WFH, I would go get a coffee, go for a run/walk and as good as it was for my state of well being, I would eventually get pulled into these crisis situations Thursday at 9 pm instead of Tuesday at 3. I like WFH for the work I actually have to do, but this sort of temporal serendipity is not easy to recreate on Zoom.
- 4y ago

not2b 4y ago

While this is true, it's also true that the people who are doing the less-than-glamorous grunt work are often unrecognized by management (which arguably is a management problem). I remember being at an all-hands company meeting years ago and listening carefully to what the people who were given the top awards got them for, and about three in a row were for diving in, working nights and weekends to come up with emergency fixes to save projects or products that had gone seriously wrong, and these were different projects. It was so blatant then even some senior managers finally noticed "Hey, we have a problem".

morelisp 4y ago

I think there's two stories intertwining here. One which you've zeroed in on is "don't require heroism". We have a lot of these stories already. (Unfortunately also a lot of exciting stories of "heroism".) The more interesting one though, is that sometimes high-impact work is "just there" to be done, if you're looking. IMO this is also really what "hero" stories are about: something goes so wrong that either people are finally forced to either look for once, or forced to bring in someone who will look. As a senior engineer, for the past couple years I've tried to pitch parts of what I see "just there" intentionally, either before it becomes a crisis or just as a potential win. Sometimes people still don't see it; maybe they're stuck in "one year of experience, ten times" or sometimes they're just built out of stop energy. But the best way I've found to detect promising junior engineers who will benefit from mentorship is to find who does bite, and more often respond "what if" instead of "but".

jwatzman 4y ago
> The more interesting one though, is that sometimes high-impact work is "just there" to be done, if you're looking. Author here. Yes, this was exactly my point! > But the best way I've found to detect promising junior engineers who will benefit from mentorship is to find who does bite, and more often respond "what if" instead of "but". Indeed. I've mentored a bunch of otherwise-competent engineers trying to get to the next level, and my advice is similar: ask yourself "ok, now what's next". After doing the cool thing, there's inevitably half a dozen other things around it, and things might be obvious to you (as the now-expert in the cool thing) but not others.
- necovek 4y ago
  > Author here. Yes, this was exactly my point! Your original example shows you doing extra hours and long days to achieve this: if this was your point, the example is terrible. I do agree with high-impact work just being there, but finding the time to do it other than "on your own time" is where the tension is between management and engineers who have a different belief of what the highest priority is.
  - 4y ago

aidenn0 4y ago

Firemen will always get more praise than people who change the batteries in smoke detectors.

dilyevsky 4y ago
False. People who quickly and cheaply build houses that kill everyone inside in case of small fires get all the praise, not firemen or whoever changes batteries (usually nobody)
- aidenn0 4y ago
  Both of our analogies are true, but what they are analogs for is different. In my case, the house is the business goals of the organization. In your case the house is the product made by the organization.

necovek 4y ago

And not to be discounted, all those prideful references to long days and late nights: another symptom of bad planning and management. It also puts a bunch of people who have actual life commitments (like kids, health issues, family issues or whatever else) at a disadvantage simply because somebody can afford to spend the extra time on something. And if those are the people who properly plan and execute, they get a double whammy. The moral of the story should not be that when there is an opportunity to make a significant contribution by exchanging your life for your work you do that, but that you should be able to excel at work and be recognized for it without exchanging your life for it. Sure, the author of the post does not have anything to do with most of the management failures, except the failure to not work extra hours for a very artificial deadline (one does not need a hero to save face if you've messed up yourself — exposing the failure is usually a net win).

tonyarkles 4y ago

You are totally right. 100%. For over a decade, I basically sold myself as a hero-for-hire (or as one of my clients called me, a mercenary). My general schtick was two-fold: help the company/team out of the immediate problem they were facing, and, if they were game, spend some time with the team to help figure out how the problem came to be and how they might acquire the skills to prevent it (ideally) or solve it on their own next time. Looking back, I am still pretty amazed at the turn-around I helped facilitate at a bunch of smaller companies. Presently, I am an employee, and this is one of the most painful uphill battles I have ever had to fight. I have tried warning colleagues about issues that seem obvious to me, but face exceptional pushback all the time, only to have things blow up exactly how I predicted a few months (or sometimes weeks) later. In my own work, people generally let me be and do good work, but as soon as it becomes a group effort the “damn the torpedoes full speed ahead” minority steam rolls ahead and makes a mess of it. The worst part is, I know exactly what the problem is, because I’ve been on the other side of it: organizations pay way more attention to external consultants than internal resources. 90% of the advice I have companies for how to change was suggestions gathered from their own employees, slightly curated and abridged into something easily digestible. And so, while I’m exhausted from trying to change things from inside, when shit hits the fan, I step up and do the old mercenary thing to solve problems at the last minute and great expense. I keep trying because I do have a ton of exceptionally bright colleagues who I love working with, as well as an actually world changing technology. We’ll see… after this next demo… maybe we’ll take some time to breathe and get back on track.