7 ms·

It's not your fault

82 points by thcipriani 5y ago

hprotagonist 5y ago
perhaps the best example i can think of was the gitlab incident. and if we’re honest it’s because the engineer’s name is Yorick.
wly_cdgr 5y ago
- ivraatiems 5y ago
  Their argument: "The great man theory is wrong because [actual reasons]." Your argument: "The great man theory is right, you're dumb if you disagree." Sure, I'm pretty dumb, but why should I believe you?
  - wly_cdgr 5y ago
    A good reason would be the countless examples available freely on sites like GitHub. Also the fact that it is this way in music, art, sculpture, writing, mathematics, game design, sport, cooking, fashion,....it would be shocking indeed if it was any different in programming
    - ivraatiems 5y ago
      ...it's your argument. Not on me to prove it for you!
      - wly_cdgr 5y ago
        Sure. But I got better things to do than prove the obvious to people who don't want to believe it
        exBarrelSpoiler 5y ago
        Then you cede the argument entirely.
    - Underphil 5y ago
      If you release on GitHub you have no skin in the game. If you break something or abandon the project entirely (which happens with a breathtaking frequency) you can just move on to something else. Not the same at all.
    - drewcoo 5y ago
      Aha. I disagreed with you until you made this point. Now I think I see where you're right and people are talking past each other. Art masterpieces are not built by committee. You list mostly what I'd consider arts. You claim that programming is (like) an art. And I think that's the difference. The opinion you're arguing against comes from the perspective that programming on a larger team to solve real world problems is (like) an engineering discipline. Engineering is both an art and a science, but more importantly engineering is a collaborative process. And that's why the focus of the piece was on social aspects of engineering and avoiding blame culture. We need better terms than were using. "Programming" is about as useful in this discussion as "building things with rock." Does that mean stacking cairns? Does that mean fine art sculpture? Does that mean designing highways and dams?
  - PeterisP 5y ago
    Their entire argument against the great man theory, quoting the original article, is "But it’s also wrong, and toxic to sustainable development and equitable environments." No reasons or arguments are provided, just an assertion. On the other hand, "a lot of the very best software is made by either one person or a very small band of very close collaborators." is an argument - not very strong, but at least something, with prominent examples such as git, tex, Calibre, emacs, OBS, Minecraft, Stardew Valley, Sublime Text and many others.
- _3u10 5y ago
  They might not build all of it but I’m pretty sure Linux never gets built without Linus.
_aleph2c_ 5y ago
This may be true, but when things break people will look for a scapegoat. So when things break, and you are mostly-responsible for initiating the failure, use collective language ("we" and not "I"), frame the failure as a systems failure when you are talking to management or the executives, look cool even if you are feeling stressed out. Manage the narrative! Sure, you flipped the switch or whatever, but try and survive the event. Just because you think its a system's failure doesn't mean other people share this belief, don't volunteer to be thrown off the bus.
- ChrisMarshallNY 5y ago
  In my experience, working at a "classic" Japanese engineering firm, scapegoating was discouraged. During postmortems, we would often decide something like "Chris made an erroneous assumption that the fix introduced no bugs." (That's a classic "oldtimer" mistake, BTW. I make it all the time -I'm a slow learner). Absolutely no blame would be affixed. It was really important for Chris (that's me) to assume Responsibility for the error, and the team would develop a solution. This being a Japanese company, of course, said "solution" usually ended up being another punchlist item, like "Perform complete regression tests for even the smallest bug fix release," etc. I'm not thrilled with people using "hero programmer syndrome," or "bus factor" as an excuse to write naive or deliberately dumbed-down code, though. Sometimes, a program needs to be maintained by skilled, experienced, well-paid, and motivated people. If a company insists on developing code, using advanced techniques, then turning over maintenance to junior staff, or do a bad job, writing a program, because they want it to be maintained by the absolute cheapest programmers possible, that's a problem.
_3u10 5y ago
It being your fault is a good way to go through life. It focuses your mind on what you could do to avoid those situations in the first place. If it’s your fault prod broke you can fix the process or you can look for a new job where the process is already fixed. Or you can find a role that doesn’t involve pushing code to prod, maybe in R&D, etc.
- ivraatiems 5y ago
  It being your fault in the sense of "you know you made a mistake and you're committed to remedying that mistake" is fine. What we don't need is "you know you made a mistake and now you must endure abuse and have your job threatened over a mistake."
  - _3u10 5y ago
    Yeah shitty managers exist. I was a lead when one of the engineers shipped a debug build that made it past App Store review. (Our debug builds were obvious). My manager says Mike (name changed) isn’t cutting releases anymore. I say Mike is cutting releases because he’s now the one person I trust on the team to not fuck it up. If you need it in writing so you can fire me if mike fucks it up, let me know. Manager mike and I all cut the next release at Mikes workstation with him knowing my ass was on the line if we shipped another debug build. Mike never shipped another debug build.
    - rdtwo 5y ago
      Only works if Mike has skin in the game. If every time Mike cuts a debug build we blame the process then mikes never too worried about it
      - _3u10 5y ago
        True. Most people have no skin in the game.
        rdtwo 5y ago
ivraatiems 5y ago
I have some paradoxical feelings about "blameless" retro culture that I'll try to sum up. In general, I'm in favor of the approach. I don't think singling people out and bullying or shaming them for their mistakes ever works. I think most well-intentioned engineers will already beat themselves up plenty for making a serious mistake, and they don't need any encouragement to do so. I know I do. On the other hand, there is a red line. At a place I worked, a DBA was let go after he repeatedly brought production down for 45 minutes to an hour at a time by running intensive queries of his own design for data-gathering, in some cases, after being explicitly told not to do that against the prod database. This was a person whose job description required him to have access to prod. There were process problems, maybe - being allowed to run whatever queries you want on production under your own authority, sure - but his cavalier attitude towards a production environment was still unacceptable. Process can only help when people are well-intentioned and doing their best; if people are malicious or negligent or just not good at their jobs, adding more process to get around that only makes things worse.
- benjiweber 5y ago
  Reminded of https://twitter.com/allspaw/status/931543941966647297 https://twitter.com/allspaw/status/931543941966647297
- tuckerman 5y ago
  I think there should be a difference between a postmortem process and a performance management process and just because the first is blameless doesn’t mean that the second can’t look back to find problems or negligence. That said, even when there is obvious negligence, having the postmortem process look at the issue with blamelessness is important to build up tooling/changes that could prevent it from happening again. For example, maybe you could revoke individuals having direct access to the production database without multi-party authentication.
  - SilasX 5y ago
    >I think there should be a difference between a postmortem process and a performance management process and just because the first is blameless doesn’t mean that the second can’t look back to find problems or negligence. That doesn't make sense. The moment that you look back at a postmortem for use in penalizing someone via performance management, the postmortem is no longer blameless.
blakesterz 5y ago
I feel like, for me at least, being able to say "That was my fault" makes me remember what I did. It makes me document that failure. And makes me tell others to avoid what I did. I've been training a JR lately and I will always say "Here's all the things I did wrong when I did this, so don't do this things" If I did it, he could easily do it, and if we can all avoid my mistakes, so much the better. It WAS my fault quite a few times, and I'm ok with that. And luckily, everyone else around here is too. I'd hate to work somewhere that punishes honest mistakes. (there are limits, of course)
- jjice 5y ago
  I've only been a full time developer for a little while now, but I've absolutely gotten some good lessons out of these mistakes. I'll tell you what, I always make sure to keep the updated_at column unchanged when I'm doing a manual update query. I also always double check and make sure I run DB changes before merging code, and I do a double check on all my code changes in a PR before I merge it in case I left in something behind. All of these are rooted in screw ups on my end that my company was very understanding of and made me grow as an engineer. Would it have been better to never have made these mistakes at all? Yes, but that's probably unrealistic. I've seen the hires after me make mistakes and I made sure to let them know of my first prod bug too, the way my boss did for me.
mberning 5y ago
Sorry, sometimes breaking prod is your fault. As a senior engineer there is no excuse for forgetting a where clause on your SQL delete. And doubly no excuse for not having somebody else look at it.
- theli0nheart 5y ago
  Humans forget things and make mistakes, no matter what your job title is. Good systems and organizations take this into account and are fault tolerant.
  - mattm 5y ago
    That's true but a senior engineer is expected to know when they are doing something potentially risky and should know when to ask someone to double check their work.
    - fsociety 5y ago
      That’s all and good if your senior engineer is a program. But they are a human, and humans are prone to making silly mistakes. Instead systems should be designed such that you can’t easily take them down due to operator errors.
- dilyevsky 5y ago
  This is a bad take. Everyone makes mistakes. I once saw principal engineer take down 15 thousand machine cluster offline bc he fat-fingered a command. The “blameless” part of pm culture is designed to address process and architecture rather than individual failures. Ofc if someone is intentionally subverting the process that’s a different story…
  - darkerside 5y ago
    Are you saying that wasn't his fault? I agree it's a false dichotomy between the process failing and people sharing blame.
dj_mc_merlin 5y ago
"Great Men are the cause of everything!" "No, you're wrong, it's Systems!" "No, it's Great Men!" "No, it's Systems!" The author found out there's another way to think about events other than "someone did it". Now he thinks he can apply that thought to everything. Sometimes it's the system, sometimes the person. You can't generalize easily.
- beaconstudios 5y ago
  If we're talking about the historical theory, the systems angle doesn't deny that smart people exist. It just highlights that Einstein was born at the right time and in the right environment to become a great person - 50 years earlier and the groundwork he relied on would not have existed, and he would've been stuck laying that groundwork himself. I feel like Kuhn's "the structure of scientific revolutions" lays out the case well, in that incremental improvements set up for a massive paradigm shift, but both the iteration and the revolution are conducted by smart people.
  - marginalia_nu 5y ago
    This is assuming you can separate a person from their environment, which is a highly dubious counterfactual that becomes stranger the more you consider it. If through some freak accident in probability and a family tradition of extramarital affairs with Germanic slaves someone genetically identical to Einstein was born gens Claudia in the early imperial period of Rome, he simply wouldn't be the same person. A person's experiences and circumstances are an inexorable part of that person. They are part of what makes them great (or not). You simply can't remove the person from their environment.
    - beaconstudios 5y ago
      That is also very true, there are many layers to a systemic (or critical, if you prefer) perspective.
NewEntryHN 5y ago
The way to attribute accountability is protocols. Create and maintain protocols that are known ways to do something safely. If you broke something by disregarding the protocol, then you've fucked up. If you broke something by complying with the protocol, then you discovered the protocol needs to be updated.
alexashka 5y ago
> breaking prod is not your fault. Put differently: breaking prod is a systems failure, not an individual one. False dichotomy.
slibhb 5y ago
The question isn't whether it's your fault, it's whether you take responsibility for it. If no one takes responsibility for anything then you get nowhere. And if you take responsibility for it, it's your fault if it goes wrong. > "Hero Programmer" is a derogatory name for a programmer who chooses to fix problems in epic, caffeine-fueled 36-hour coding sessions that frequently just kick the can down the road to the next heroic 36-hour coding blitz. Hero programmers would rather react than plan. Projects with hero programmers working on them often make a lot of progress initially, but never arrive at a stable state of completion Maybe there are workplaces where people get together to collaborate on a design and then break the design down into tasks and assign those tasks to programmers to implement. Maybe this process is performed until the project is done. Maybe. But I've never seen it. I see people taking responsibility for small and large tasks, and the large ones sometimes involve a single person re-implementing entire systems spread across thousands of files (though not necessarily in "36-hour coding blitzes").
- darkerside 5y ago
  Agreed. > Projects with hero programmers working on them often make a lot of progress initially, but never arrive at a stable state of completion No project is ever really finished except the ones nobody cares about. Probably because they stopped being maintained by the hero programmer.
  - watwut 5y ago
    This is not true. We and I had finished tons of projects. They are done in the true sense. We are OK with their state and they rarely ever change. They work. We moved on other projects.
- marginalia_nu 5y ago
  Honestly that whole hero programmer bit seems like a bit of a strawman. What's being described sounds like a talented but inexperienced developer (which doesn't necessarily mean they are young or fresh to the field; some people manage to stay beginners for decades). Doesn't mean there aren't highly talented developers that can get a lot of work done in a short amount of time if you let them. The failure in that case is not having a more senior developer mentor the kid.
ozzythecat 5y ago
> When you look at things from this lens, all the successes of a website, an application, or an organization flow from the talents and genius of a few individuals. It’s a compelling outlook because, well, empirically it can definitely appear this way, and it’s naturally aligned with the other dominant societal ideas we have about individuality. In many large organizations, much of the success comes from the foresight, insight, and hard "work" comes from a few benefitting many. The reality is, it is individuals and not some collective group or "teams".
commandlinefan 5y ago
> would rather react than plan Can’t help but notice that the people who throw around accusations like this are… still blaming the individual.
ilaksh 5y ago
The point (which many people missed) is that there should be a process or tooling in place to make it hard or ideally impossible for a mistake like that to completely disable the system. And that it is normal for humans to make mistakes every once and awhile. If the system can't deal with that, then it's not a robust system.
aaron695 5y ago
throwawaylinux 5y ago
It is your fault. If you introduce a bug and let's say in part due to insufficient documentation in the code, code reviews that didn't catch it, your change passed all tests and you even made some new tests to exactly test corner cases of the change you were making, it ran fine in beta testing. Then it is still (in part) your fault! Everybody makes mistakes, and systems should be resilient to mistakes. That doesn't mean nobody is at fault for anything. It means everybody has faults, and if somebody doesn't think they have any faults then why should they think they have anything to learn or improve? Fault finding should be done. People who wrote the documentation should be given notice and opportunity to improve. People who reviewed the change should as well. And certainly the person who introduced the bug. If people can not cope with accepting the blame for a problem they had responsibility for introducing, they should not be in that line of work.
michelpp 5y ago
This is the default approach taken by the airline industry, just culture: "Just culture is a concept related to systems thinking which emphasizes that mistakes are generally a product of faulty organizational cultures, rather than solely brought about by the person or persons directly involved." https://en.wikipedia.org/wiki/Just_culture https://en.wikipedia.org/wiki/Just_culture
mandevil 5y ago
I've been at multiple companies with formalized postmortem processes that involved final documents that were widely available within the company. And at each of them, I spent a few hours one afternoon reading down the last 30 or so. And I was struck by how many of them had the same root cause (different at each company, because each companies culture and tech-stack is unique, which creates unique failure modes). So at $firstCompany, 75% of their errors were "Dev made a change in the big ball of mud that is our product to fix a bug in feature A, didn't realize that code was reused in feature B, and since we don't have sufficient automated testing feature B was never checked before it hit prod." At $secondCompany, a software consultancy, all of the failures were because each deployment was a unique special snowflake with no backups or dev-test environment (they had originally been created with hardware for those, but that hardware was inevitably cannibalized to keep prod going or used to add a new feature to prod, without back-fill). At $thirdCompany, they had a massively complex roll-out process with larger and larger rings of internal dog-fooding, and developers kept screwing it up, advancing software that was known bad or hurrying things along too quickly or missing roll-out dates or whatever. Reading a bunch of these documents in a row made the failure mode for the corporate culture clear and obvious. The difference between effective and ineffective teams is whether they decided to fix the underlying problem or not. At $firstCompany I asked my bosses to take a dev team and fix enough stuff so that we could run a full automated test suite reliably before every deployment, was told it wasn't a good idea, and left the company shortly afterwards. At $secondCompany they tried things to get that to work, but had major cultural problems with getting individuals on each team to have the discipline to backfill the hardware and actually keep prod and dev-test identical. At $thirdCompany they looked at the same problem of individual discipline and created a team of professional non-developers who were full time focused on controlling software roll-outs so that every new roll-out went to the right level at the right time. We devs gratefully handed over this complex process to that team, and the professionals did a great job. Be like $thirdCompany: figure out your larger internal pain points and fix those; you actually need to be hit over the head with the data on this to make it clear how your culture and tech stack are failing.
- hinkley 5y ago
  > The difference between effective and ineffective teams is whether they decided to fix the underlying problem or not. In Good to Great this is an aspect of Confronting the Brutal Facts. https://www.jimcollins.com/concepts/confront-the-brutal-facts.html https://www.jimcollins.com/concepts/confront-the-brutal-fact...

82 points by thcipriani 5y ago

perhaps the best example i can think of was the gitlab incident. and if we’re honest it’s because the engineer’s name is Yorick.

wly_cdgr 5y ago

ivraatiems 5y ago
Their argument: "The great man theory is wrong because [actual reasons]." Your argument: "The great man theory is right, you're dumb if you disagree." Sure, I'm pretty dumb, but why should I believe you?
- wly_cdgr 5y ago
  A good reason would be the countless examples available freely on sites like GitHub. Also the fact that it is this way in music, art, sculpture, writing, mathematics, game design, sport, cooking, fashion,....it would be shocking indeed if it was any different in programming
  - ivraatiems 5y ago
    ...it's your argument. Not on me to prove it for you!
    - wly_cdgr 5y ago
      Sure. But I got better things to do than prove the obvious to people who don't want to believe it
      - exBarrelSpoiler 5y ago
        Then you cede the argument entirely.
  - Underphil 5y ago
    If you release on GitHub you have no skin in the game. If you break something or abandon the project entirely (which happens with a breathtaking frequency) you can just move on to something else. Not the same at all.
  - drewcoo 5y ago
    Aha. I disagreed with you until you made this point. Now I think I see where you're right and people are talking past each other. Art masterpieces are not built by committee. You list mostly what I'd consider arts. You claim that programming is (like) an art. And I think that's the difference. The opinion you're arguing against comes from the perspective that programming on a larger team to solve real world problems is (like) an engineering discipline. Engineering is both an art and a science, but more importantly engineering is a collaborative process. And that's why the focus of the piece was on social aspects of engineering and avoiding blame culture. We need better terms than were using. "Programming" is about as useful in this discussion as "building things with rock." Does that mean stacking cairns? Does that mean fine art sculpture? Does that mean designing highways and dams?
- PeterisP 5y ago
  Their entire argument against the great man theory, quoting the original article, is "But it’s also wrong, and toxic to sustainable development and equitable environments." No reasons or arguments are provided, just an assertion. On the other hand, "a lot of the very best software is made by either one person or a very small band of very close collaborators." is an argument - not very strong, but at least something, with prominent examples such as git, tex, Calibre, emacs, OBS, Minecraft, Stardew Valley, Sublime Text and many others.
_3u10 5y ago
They might not build all of it but I’m pretty sure Linux never gets built without Linus.

_aleph2c_ 5y ago

This may be true, but when things break people will look for a scapegoat. So when things break, and you are mostly-responsible for initiating the failure, use collective language ("we" and not "I"), frame the failure as a systems failure when you are talking to management or the executives, look cool even if you are feeling stressed out. Manage the narrative! Sure, you flipped the switch or whatever, but try and survive the event. Just because you think its a system's failure doesn't mean other people share this belief, don't volunteer to be thrown off the bus.

ChrisMarshallNY 5y ago
In my experience, working at a "classic" Japanese engineering firm, scapegoating was discouraged. During postmortems, we would often decide something like "Chris made an erroneous assumption that the fix introduced no bugs." (That's a classic "oldtimer" mistake, BTW. I make it all the time -I'm a slow learner). Absolutely no blame would be affixed. It was really important for Chris (that's me) to assume Responsibility for the error, and the team would develop a solution. This being a Japanese company, of course, said "solution" usually ended up being another punchlist item, like "Perform complete regression tests for even the smallest bug fix release," etc. I'm not thrilled with people using "hero programmer syndrome," or "bus factor" as an excuse to write naive or deliberately dumbed-down code, though. Sometimes, a program needs to be maintained by skilled, experienced, well-paid, and motivated people. If a company insists on developing code, using advanced techniques, then turning over maintenance to junior staff, or do a bad job, writing a program, because they want it to be maintained by the absolute cheapest programmers possible, that's a problem.

_3u10 5y ago

It being your fault is a good way to go through life. It focuses your mind on what you could do to avoid those situations in the first place. If it’s your fault prod broke you can fix the process or you can look for a new job where the process is already fixed. Or you can find a role that doesn’t involve pushing code to prod, maybe in R&D, etc.

ivraatiems 5y ago
It being your fault in the sense of "you know you made a mistake and you're committed to remedying that mistake" is fine. What we don't need is "you know you made a mistake and now you must endure abuse and have your job threatened over a mistake."
- _3u10 5y ago
  Yeah shitty managers exist. I was a lead when one of the engineers shipped a debug build that made it past App Store review. (Our debug builds were obvious). My manager says Mike (name changed) isn’t cutting releases anymore. I say Mike is cutting releases because he’s now the one person I trust on the team to not fuck it up. If you need it in writing so you can fire me if mike fucks it up, let me know. Manager mike and I all cut the next release at Mikes workstation with him knowing my ass was on the line if we shipped another debug build. Mike never shipped another debug build.
  - rdtwo 5y ago
    Only works if Mike has skin in the game. If every time Mike cuts a debug build we blame the process then mikes never too worried about it
    - _3u10 5y ago
      True. Most people have no skin in the game.
      - rdtwo 5y ago

ivraatiems 5y ago

I have some paradoxical feelings about "blameless" retro culture that I'll try to sum up. In general, I'm in favor of the approach. I don't think singling people out and bullying or shaming them for their mistakes ever works. I think most well-intentioned engineers will already beat themselves up plenty for making a serious mistake, and they don't need any encouragement to do so. I know I do. On the other hand, there is a red line. At a place I worked, a DBA was let go after he repeatedly brought production down for 45 minutes to an hour at a time by running intensive queries of his own design for data-gathering, in some cases, after being explicitly told not to do that against the prod database. This was a person whose job description required him to have access to prod. There were process problems, maybe - being allowed to run whatever queries you want on production under your own authority, sure - but his cavalier attitude towards a production environment was still unacceptable. Process can only help when people are well-intentioned and doing their best; if people are malicious or negligent or just not good at their jobs, adding more process to get around that only makes things worse.

benjiweber 5y ago
Reminded of https://twitter.com/allspaw/status/931543941966647297 https://twitter.com/allspaw/status/931543941966647297
tuckerman 5y ago
I think there should be a difference between a postmortem process and a performance management process and just because the first is blameless doesn’t mean that the second can’t look back to find problems or negligence. That said, even when there is obvious negligence, having the postmortem process look at the issue with blamelessness is important to build up tooling/changes that could prevent it from happening again. For example, maybe you could revoke individuals having direct access to the production database without multi-party authentication.
- SilasX 5y ago
  >I think there should be a difference between a postmortem process and a performance management process and just because the first is blameless doesn’t mean that the second can’t look back to find problems or negligence. That doesn't make sense. The moment that you look back at a postmortem for use in penalizing someone via performance management, the postmortem is no longer blameless.

blakesterz 5y ago

I feel like, for me at least, being able to say "That was my fault" makes me remember what I did. It makes me document that failure. And makes me tell others to avoid what I did. I've been training a JR lately and I will always say "Here's all the things I did wrong when I did this, so don't do this things" If I did it, he could easily do it, and if we can all avoid my mistakes, so much the better. It WAS my fault quite a few times, and I'm ok with that. And luckily, everyone else around here is too. I'd hate to work somewhere that punishes honest mistakes. (there are limits, of course)

jjice 5y ago
I've only been a full time developer for a little while now, but I've absolutely gotten some good lessons out of these mistakes. I'll tell you what, I always make sure to keep the updated_at column unchanged when I'm doing a manual update query. I also always double check and make sure I run DB changes before merging code, and I do a double check on all my code changes in a PR before I merge it in case I left in something behind. All of these are rooted in screw ups on my end that my company was very understanding of and made me grow as an engineer. Would it have been better to never have made these mistakes at all? Yes, but that's probably unrealistic. I've seen the hires after me make mistakes and I made sure to let them know of my first prod bug too, the way my boss did for me.

mberning 5y ago

Sorry, sometimes breaking prod is your fault. As a senior engineer there is no excuse for forgetting a where clause on your SQL delete. And doubly no excuse for not having somebody else look at it.

theli0nheart 5y ago
Humans forget things and make mistakes, no matter what your job title is. Good systems and organizations take this into account and are fault tolerant.
- mattm 5y ago
  That's true but a senior engineer is expected to know when they are doing something potentially risky and should know when to ask someone to double check their work.
  - fsociety 5y ago
    That’s all and good if your senior engineer is a program. But they are a human, and humans are prone to making silly mistakes. Instead systems should be designed such that you can’t easily take them down due to operator errors.
dilyevsky 5y ago
This is a bad take. Everyone makes mistakes. I once saw principal engineer take down 15 thousand machine cluster offline bc he fat-fingered a command. The “blameless” part of pm culture is designed to address process and architecture rather than individual failures. Ofc if someone is intentionally subverting the process that’s a different story…
- darkerside 5y ago
  Are you saying that wasn't his fault? I agree it's a false dichotomy between the process failing and people sharing blame.

dj_mc_merlin 5y ago

"Great Men are the cause of everything!" "No, you're wrong, it's Systems!" "No, it's Great Men!" "No, it's Systems!" The author found out there's another way to think about events other than "someone did it". Now he thinks he can apply that thought to everything. Sometimes it's the system, sometimes the person. You can't generalize easily.

beaconstudios 5y ago
If we're talking about the historical theory, the systems angle doesn't deny that smart people exist. It just highlights that Einstein was born at the right time and in the right environment to become a great person - 50 years earlier and the groundwork he relied on would not have existed, and he would've been stuck laying that groundwork himself. I feel like Kuhn's "the structure of scientific revolutions" lays out the case well, in that incremental improvements set up for a massive paradigm shift, but both the iteration and the revolution are conducted by smart people.
- marginalia_nu 5y ago
  This is assuming you can separate a person from their environment, which is a highly dubious counterfactual that becomes stranger the more you consider it. If through some freak accident in probability and a family tradition of extramarital affairs with Germanic slaves someone genetically identical to Einstein was born gens Claudia in the early imperial period of Rome, he simply wouldn't be the same person. A person's experiences and circumstances are an inexorable part of that person. They are part of what makes them great (or not). You simply can't remove the person from their environment.
  - beaconstudios 5y ago
    That is also very true, there are many layers to a systemic (or critical, if you prefer) perspective.

NewEntryHN 5y ago

The way to attribute accountability is protocols. Create and maintain protocols that are known ways to do something safely. If you broke something by disregarding the protocol, then you've fucked up. If you broke something by complying with the protocol, then you discovered the protocol needs to be updated.

alexashka 5y ago

> breaking prod is not your fault. Put differently: breaking prod is a systems failure, not an individual one. False dichotomy.

slibhb 5y ago

The question isn't whether it's your fault, it's whether you take responsibility for it. If no one takes responsibility for anything then you get nowhere. And if you take responsibility for it, it's your fault if it goes wrong. > "Hero Programmer" is a derogatory name for a programmer who chooses to fix problems in epic, caffeine-fueled 36-hour coding sessions that frequently just kick the can down the road to the next heroic 36-hour coding blitz. Hero programmers would rather react than plan. Projects with hero programmers working on them often make a lot of progress initially, but never arrive at a stable state of completion Maybe there are workplaces where people get together to collaborate on a design and then break the design down into tasks and assign those tasks to programmers to implement. Maybe this process is performed until the project is done. Maybe. But I've never seen it. I see people taking responsibility for small and large tasks, and the large ones sometimes involve a single person re-implementing entire systems spread across thousands of files (though not necessarily in "36-hour coding blitzes").

darkerside 5y ago
Agreed. > Projects with hero programmers working on them often make a lot of progress initially, but never arrive at a stable state of completion No project is ever really finished except the ones nobody cares about. Probably because they stopped being maintained by the hero programmer.
- watwut 5y ago
  This is not true. We and I had finished tons of projects. They are done in the true sense. We are OK with their state and they rarely ever change. They work. We moved on other projects.
marginalia_nu 5y ago
Honestly that whole hero programmer bit seems like a bit of a strawman. What's being described sounds like a talented but inexperienced developer (which doesn't necessarily mean they are young or fresh to the field; some people manage to stay beginners for decades). Doesn't mean there aren't highly talented developers that can get a lot of work done in a short amount of time if you let them. The failure in that case is not having a more senior developer mentor the kid.

ozzythecat 5y ago

> When you look at things from this lens, all the successes of a website, an application, or an organization flow from the talents and genius of a few individuals. It’s a compelling outlook because, well, empirically it can definitely appear this way, and it’s naturally aligned with the other dominant societal ideas we have about individuality. In many large organizations, much of the success comes from the foresight, insight, and hard "work" comes from a few benefitting many. The reality is, it is individuals and not some collective group or "teams".

commandlinefan 5y ago

> would rather react than plan Can’t help but notice that the people who throw around accusations like this are… still blaming the individual.

ilaksh 5y ago

The point (which many people missed) is that there should be a process or tooling in place to make it hard or ideally impossible for a mistake like that to completely disable the system. And that it is normal for humans to make mistakes every once and awhile. If the system can't deal with that, then it's not a robust system.

aaron695 5y ago

throwawaylinux 5y ago

It is your fault. If you introduce a bug and let's say in part due to insufficient documentation in the code, code reviews that didn't catch it, your change passed all tests and you even made some new tests to exactly test corner cases of the change you were making, it ran fine in beta testing. Then it is still (in part) your fault! Everybody makes mistakes, and systems should be resilient to mistakes. That doesn't mean nobody is at fault for anything. It means everybody has faults, and if somebody doesn't think they have any faults then why should they think they have anything to learn or improve? Fault finding should be done. People who wrote the documentation should be given notice and opportunity to improve. People who reviewed the change should as well. And certainly the person who introduced the bug. If people can not cope with accepting the blame for a problem they had responsibility for introducing, they should not be in that line of work.

michelpp 5y ago

This is the default approach taken by the airline industry, just culture: "Just culture is a concept related to systems thinking which emphasizes that mistakes are generally a product of faulty organizational cultures, rather than solely brought about by the person or persons directly involved." https://en.wikipedia.org/wiki/Just_culture https://en.wikipedia.org/wiki/Just_culture

mandevil 5y ago

I've been at multiple companies with formalized postmortem processes that involved final documents that were widely available within the company. And at each of them, I spent a few hours one afternoon reading down the last 30 or so. And I was struck by how many of them had the same root cause (different at each company, because each companies culture and tech-stack is unique, which creates unique failure modes). So at $firstCompany, 75% of their errors were "Dev made a change in the big ball of mud that is our product to fix a bug in feature A, didn't realize that code was reused in feature B, and since we don't have sufficient automated testing feature B was never checked before it hit prod." At $secondCompany, a software consultancy, all of the failures were because each deployment was a unique special snowflake with no backups or dev-test environment (they had originally been created with hardware for those, but that hardware was inevitably cannibalized to keep prod going or used to add a new feature to prod, without back-fill). At $thirdCompany, they had a massively complex roll-out process with larger and larger rings of internal dog-fooding, and developers kept screwing it up, advancing software that was known bad or hurrying things along too quickly or missing roll-out dates or whatever. Reading a bunch of these documents in a row made the failure mode for the corporate culture clear and obvious. The difference between effective and ineffective teams is whether they decided to fix the underlying problem or not. At $firstCompany I asked my bosses to take a dev team and fix enough stuff so that we could run a full automated test suite reliably before every deployment, was told it wasn't a good idea, and left the company shortly afterwards. At $secondCompany they tried things to get that to work, but had major cultural problems with getting individuals on each team to have the discipline to backfill the hardware and actually keep prod and dev-test identical. At $thirdCompany they looked at the same problem of individual discipline and created a team of professional non-developers who were full time focused on controlling software roll-outs so that every new roll-out went to the right level at the right time. We devs gratefully handed over this complex process to that team, and the professionals did a great job. Be like $thirdCompany: figure out your larger internal pain points and fix those; you actually need to be hit over the head with the data on this to make it clear how your culture and tech stack are failing.

hinkley 5y ago
> The difference between effective and ineffective teams is whether they decided to fix the underlying problem or not. In Good to Great this is an aspect of Confronting the Brutal Facts. https://www.jimcollins.com/concepts/confront-the-brutal-facts.html https://www.jimcollins.com/concepts/confront-the-brutal-fact...