4 ms·

Maybe we could kill two birds with one stone here and tie production/maintenance outcomes to promotions? Rather than making everyone be on-call for free (or sli

Maybe we could kill two birds with one stone here and tie production/maintenance outcomes to promotions? Rather than making everyone be on-call for free (or slightly more depending on what "extra" is), dispense with the usual circus that is performance reviews and start tracking when bad code causes outages. Blame assignment is super counter-productive in the moment of emergency, but it seems like it could be useful for measuring developer effectiveness, incentivizing shipping features but also shipping features with a small amount of bugs. I have written my fair share of bugs that have snuck into test/staging/production (I'm a prolific at writing bugs), but that's the kind of thing that should come up in a yearly review (and I expect it to) and hurt my chances for raises/promotions, instead of the bullshit musical chairs, politics and level/rank setting (how many more years until you reach Senior Staff Software Engineer IV with distinction again?) that happens right now. Also my ideal on-call situation (which probably doesn't exist): - optional - paid by supply/demand (price per hour on-call increases until someone decides to do it) Companies should go back to hiring competent night staff for truly critical business processes and paying them whatever is appropriate. The on-call system as it sits now is heavily tilted in favor of business at the expense of employees -- the attention of a $100/hr+ professional for free, or some small percentage of the actual cost. Also BTW if you write software and don't care that it's bug-free or don't take responsibility for it, you're a bad software engineer/developer. You don't have to be passionate about code but being a professional generally means producing quality work, and quality work is reasonably robust whenever it can be. One of the differences between a junior and senior software engineer is knowledge of what constitutes enough "quality" in context.

beejiu 8y ago
Blame assignment is counterproductive because it encourages a culture of risk aversion. I suspect highly productive members produce more bugs simply because of the size of their contribution, and that's acceptable in some (perhaps most) businesses.
- hardwaresofton 8y ago
  I think it's fair to say that it's counter-productive for more than one reason. That said, I think if you want to encourage risk taking, do it directly -- incentivize it with money/prestige or hire more people who take risks (and give them free reign). In recent history more and more "labs"-type positions have been opening up at companies, with the aim being to lure in people who want to do interesting work. As an engineer, a labs position is 1000% more interesting to me than any other senior whatever position because of this ability to take risks and possibly reap large rewards (even if they go to the company). As far as your point about productive members producing more bugs, you can incentivize/dis-incentivize this by changing how you perform reviews. Incentivize productivity, but not at the cost of introducing more bugs. Shower cash/prestige/autonomy on developers that produce lots of features with low error-counts and people will optimize for that if that's what drives them.
partycoder 8y ago
An outage could be caused by infrastructure, by library code, by configuration, by increased activity, not necessarily your own software.
- hardwaresofton 8y ago
  Yes, and there's at least two ways you could handle this: - Have the entire org take a hit - Penalize the infra team The thing about blame assignment is that no one wants to get blamed for anything (so like if you try #2 the infra team would likely find someone to blame as quick as possible), which ordinarily makes it pretty toxic but I think you can use it for good here with proper communication and goal-setting (which is of course harder than it sounds). The case of outages completely caused by infrastructure I think is pretty rare, but if it's really something like S3 going down or whatever just don't count it. If it's that someone pushed an invalid load balancer setting to an ELB, then there's gonna be someone at fault, whether it's the infra team for letting it be possible in the first place or the developer that did it for doing it. Good infra teams try to make bad settings impossible and good settings self-servicable in my experience. All this said, it really doesn't need to be super heavy handed, I mean don't make some orwellian report-your-neighbor for points system, but enforce accountability and link it to something people care about.
  - partycoder 8y ago
    What you want is to have fewer outages. Penalization is a non-goal.
pjc50 8y ago
Remember that you can either have an inquiry that finds out what happened or one that assigns blame, but not both; if there is a hint that real blame with consequences will be assigned people will clam up, hide the evidence, or even start destroying evidence and framing their colleagues. About the most consequential thing you can get away with without wrecking trust is mild humourous social shame like making someone wear a silly hat. And for things which have gone badly wrong that seems inappropriate. This kind of thing is what blogger Alex Harrowell brilliantly coined "Coasian hell" megaprojects don't work: http://www.harrowell.org.uk/blog/2018/01/31/in-the-eternal-inferno-fiends-torment-ronald-coase-with-the-fate-of-his-ideas/ http://www.harrowell.org.uk/blog/2018/01/31/in-the-eternal-i...
- hardwaresofton 8y ago
  I'm agree that real blame with real consequences will encourage people to clam up/hide evidence/start framing, but I don't think it can be 100% true/the only outcome because if this was the case then no consequences-based governance system would ever work. Also I admit it's a bit naive to say but maybe we should also be focusing on not hiring people that do egregious things when consequences arrive? A certain amount is normal but if you're working with people that destroy evidence and frame others when shit hits the fan... That's kind of a red flag no? No one hires for integrity anymore? I also disagree about it wrecking trust -- a well built & fairly applied system should build trust -- it's when people put their trust in systems with no power/hidden manipulation that trust gets wrecked the fastest. You could even make it opt in, and tie bonuses to the risk taken by those who decide to have raises/promotions manipulated in the context of the system. IMO this is basically just a sub-problem of the general "how do we govern societies" general problem and "don't have consequences" doesn't seem like a good plan either. [EDIT] I want to add that I really would like to hear other suggestions for how to solve these kinds of issues. I could only imagine a truly no-consequences style working in a xerox parc-ish environment which is only possible when there's more than enough money (both on the corporate and the people side) so desperation isn't producing rash actions, and most people are being motivated by something other than the normal money/prestige.

beejiu 8y ago

Blame assignment is counterproductive because it encourages a culture of risk aversion. I suspect highly productive members produce more bugs simply because of the size of their contribution, and that's acceptable in some (perhaps most) businesses.

hardwaresofton 8y ago
I think it's fair to say that it's counter-productive for more than one reason. That said, I think if you want to encourage risk taking, do it directly -- incentivize it with money/prestige or hire more people who take risks (and give them free reign). In recent history more and more "labs"-type positions have been opening up at companies, with the aim being to lure in people who want to do interesting work. As an engineer, a labs position is 1000% more interesting to me than any other senior whatever position because of this ability to take risks and possibly reap large rewards (even if they go to the company). As far as your point about productive members producing more bugs, you can incentivize/dis-incentivize this by changing how you perform reviews. Incentivize productivity, but not at the cost of introducing more bugs. Shower cash/prestige/autonomy on developers that produce lots of features with low error-counts and people will optimize for that if that's what drives them.

partycoder 8y ago

An outage could be caused by infrastructure, by library code, by configuration, by increased activity, not necessarily your own software.

hardwaresofton 8y ago
Yes, and there's at least two ways you could handle this: - Have the entire org take a hit - Penalize the infra team The thing about blame assignment is that no one wants to get blamed for anything (so like if you try #2 the infra team would likely find someone to blame as quick as possible), which ordinarily makes it pretty toxic but I think you can use it for good here with proper communication and goal-setting (which is of course harder than it sounds). The case of outages completely caused by infrastructure I think is pretty rare, but if it's really something like S3 going down or whatever just don't count it. If it's that someone pushed an invalid load balancer setting to an ELB, then there's gonna be someone at fault, whether it's the infra team for letting it be possible in the first place or the developer that did it for doing it. Good infra teams try to make bad settings impossible and good settings self-servicable in my experience. All this said, it really doesn't need to be super heavy handed, I mean don't make some orwellian report-your-neighbor for points system, but enforce accountability and link it to something people care about.
- partycoder 8y ago
  What you want is to have fewer outages. Penalization is a non-goal.

pjc50 8y ago

Remember that you can either have an inquiry that finds out what happened or one that assigns blame, but not both; if there is a hint that real blame with consequences will be assigned people will clam up, hide the evidence, or even start destroying evidence and framing their colleagues. About the most consequential thing you can get away with without wrecking trust is mild humourous social shame like making someone wear a silly hat. And for things which have gone badly wrong that seems inappropriate. This kind of thing is what blogger Alex Harrowell brilliantly coined "Coasian hell" megaprojects don't work: http://www.harrowell.org.uk/blog/2018/01/31/in-the-eternal-inferno-fiends-torment-ronald-coase-with-the-fate-of-his-ideas/ http://www.harrowell.org.uk/blog/2018/01/31/in-the-eternal-i...

hardwaresofton 8y ago
I'm agree that real blame with real consequences will encourage people to clam up/hide evidence/start framing, but I don't think it can be 100% true/the only outcome because if this was the case then no consequences-based governance system would ever work. Also I admit it's a bit naive to say but maybe we should also be focusing on not hiring people that do egregious things when consequences arrive? A certain amount is normal but if you're working with people that destroy evidence and frame others when shit hits the fan... That's kind of a red flag no? No one hires for integrity anymore? I also disagree about it wrecking trust -- a well built & fairly applied system should build trust -- it's when people put their trust in systems with no power/hidden manipulation that trust gets wrecked the fastest. You could even make it opt in, and tie bonuses to the risk taken by those who decide to have raises/promotions manipulated in the context of the system. IMO this is basically just a sub-problem of the general "how do we govern societies" general problem and "don't have consequences" doesn't seem like a good plan either. [EDIT] I want to add that I really would like to hear other suggestions for how to solve these kinds of issues. I could only imagine a truly no-consequences style working in a xerox parc-ish environment which is only possible when there's more than enough money (both on the corporate and the people side) so desperation isn't producing rash actions, and most people are being motivated by something other than the normal money/prestige.