5 ms·

There’s a good lecture about this, called “The Normalization of Deviance”: https://m.youtube.com/watch?v=Ljzj9Msli5o&pp=ygUZbm9ybWFsaXphdGlvbiBvZiBkZXZpYW5jZQ%

There’s a good lecture about this, called “The Normalization of Deviance”: https://m.youtube.com/watch?v=Ljzj9Msli5o&pp=ygUZbm9ybWFsaXphdGlvbiBvZiBkZXZpYW5jZQ%3D%3D https://m.youtube.com/watch?v=Ljzj9Msli5o&pp=ygUZbm9ybWFsaXp...

steveBK123 2y ago
Exactly the concept why you don't want to let whatever dashboards/alerts/etc you maintain on your systems have a "normal amount of reds/fails/spurious texts". At some point you become immune. It's a lot harder to notice theres 4 red lights today than the usual 2-3 vs noticing 1 when there are normally exactly 0.
- OutOfHere 2y ago
  Yes. The causative issue is the way in which projects are managed. Employees have no ownership of the project. If employees had ownership over which changes they think are best, a good employee would act on bringing the alerts back to zero before they take on new features or a new project. There are some obstacles: 1. Employees not having a say in which issues to work on. This pretty much leads to the death of a project in the medium term due to near-total disregard of maintenance issues and alerts. 2. Big-team ownership of a project. When everyone is in charge, no one is. This is why I advocate for a team size of exactly two for each corporate project. 3. Employees being unreasonably pressured for time. Perhaps the right framing for employees to think about it is: "If it were their own business or product, how would they do it?" This framing, combined with the backlog, should automatically help avoid spending more time than is necessary on an issue.
  - whodidntante 2y ago
    Not making an ethical/moral judgement here, just a practical one - is there any reason to believe that giving employees ownership of the projects will be any better than having "management" own it if all factors were truly considered ? If every decision an employee made on features/issues/quality/time was accompanied by how much their pay was affected, would the outcomes really be better ? The team could decide to fix all bugs before taking on a new feature, or that the 2 month allotment to a feature should really be three months to do it "right" without having to work nights/weekends, would the team really decide to do that if their paycheck was reduced by 10%, or delayed for that extra month for those new features were delivered ? If all factors were included in the employee decision process, including the real world effect of revenue/profit on individual compensation from those decisions, it is not clear to me that employees would make any "better" decisions. I would think that employees could be even more "short sighted" than senior management, as senior management likely has more at stake in terms of company reputation/equity/career than an employee who can change jobs easier, and an employee might choose not to "get those alerts to zero" if it meant they would have more immediate cash in their pocket. And how would disagreements between team members be worked out if some were willing to forgo compensation to "do it right', and others wanted to cut even more corners ? Truly having ownership means you have also financial risk.
    - rawgabbit 2y ago
      What I see is a movement where line employees have a say on who is retained at the director and VP level. The CEO reports to the board. But his immediate and second tier reports are also judged by the employees. The thought is that will give them pause before they embark on their next my way or the highway decision making. The most egregious directors who push out line employees in favor of their cronies will be fired under this evaluation.
    - OutOfHere 2y ago
      > is there any reason to believe that giving employees ownership of the projects will be any better than having "management" own it Non-technical management's skill level is almost always overrated. They're almost never qualified for it. Ultimately it still is management's decision, and always will be. If however management believes that employees are incapable of serving users, then it's management's fault for assigning mismatched employees. > how much their pay was affected Bringing pay into this discussion is a nonsensical distraction. If an employer misses two consecutive paychecks by even 1%, that's enough reason to stop showing up for work, and potentially to sue for severance+damages, and also claim unemployment wages. There is no room for any variation here. > Truly having ownership It should be obvious that ownership here refers to the ownership of the technical direction, not literal ownership in the way I own a backpack that I bring to work. If true financial ownership existed, the employee would be receiving substantial equity with a real tradable market value, with the risk of losing some of this equity if they were to lose their job. > how would disagreements between team members be worked out As noted, there would be just two employees per project, and this ought to minimize disagreements. If disagreements still exist, this is where management can assist with direction. There should always remain room for conducting diverse experiments without having to worry about which outcomes get discarded and which get used. --- In summary, if the suggested approach is not working, it's probably because there is significant unavoidable technical debt or the employees are mismatched to the task.
    - nostrademons 2y ago
      Been all of an engineer, a manager, and a founder/CEO, and I enjoy analyzing organizational dysfunction. The difference between an engineer and a manager's perspective usually comes down to their job description. An engineer is hired to get the engineering right; the reason the company pays them is for their ability to marry reality to organizational goals. The reason the company hires a manager is to set those organizational goals and ensure that everybody is marching toward them. This split is explicit for a reason: it ensures that when disagreements arise, they are explicitly negotiated. Most people are bad at making complex tradeoffs, and when they have to do so, their execution velocity suffers. Indeed, the job description for someone who is hired to make complex tradeoffs is called "executive", and they purposefully have to do no real work so that their decision-making functions only in terms of cost estimates that management bubbles up, not the personal pain that will result from those decisions. Dysfunction arises from a few major sources: 1. There's a power imbalance between management and engineering. An engineer usually only has one project; if it fails, it often means their job, even if the outcome reality dictates is that it should fail. That gives them a strong incentive to send good news up the chain even if the project is going to fail. Good management gets around this by never penalizing bad news or good-faith project failure, but good management is actually really counterintuitive, because your natural reaction is to react to negative news with negative emotions. 2. Information is lost with every explicit communication up the chain. The information an engineer provides to management is a summary of the actual state of reality; if they passed along everything, it'd require that management become an engineer. Likewise recursively along the management chain. It's not always possible to predict which information is critical to an executive's decision, and so sometimes this gets lost as the management chain plays telephone. 3. Executives and policy-makers, by definition, are the least reality-informed people in the system, but they have the final say on all the decisions. They naturally tend to overweight the things that they are informed on, like "Will we lose the contract?" or "Will we miss earnings this quarter?" All that said, the fact that most companies have a corporate hierarchy and they largely outcompete employee-owned or founder-owned cooperatives in the marketplace tends to suggest that even with the pitfalls, this is a more efficient system. The velocity penalty from having to both make the complex decisions and execute on them outweighs all the information loss. I experienced this with my startup: the failure mode was that I'd emotionally second-guess my executive decisions, which meant that I executed slowly on them, which meant that I didn't get enough iterations or enough feedback from the market to find product/market fit. This is also why startups that do succeed tend to be ones where the idea is obvious (to the founder at least, but not necessarily to the general public). They don't need to spend much time on complex positioning decisions, and can spend that time executing, and then eventually grow the company within the niche they know well.
  - lencastre 2y ago
    And groupthink
  - CSMastermind 2y ago
    > If employees had ownership over which changes they think are best, a good employee would act on bringing the alerts back to zero before they take on new features or a new project. You say this but as someone who's run a large platform organization that hasn't been my experience. Sure some employees, maybe you, care about things like bringing alerts back to zero but a large number are indifferent and a small number are outright dismissive. This is informed not just by individual personality but also by culture. Not too long ago I pointed out a bug in someone's code who I was reviewing and instead of fixing it they said, "Oh okay, I'll look out for bugs like that when I write code in the future" then proceeded to merge and deploy their unchanged code. And in that case I'm their manager not a peer or someone from another team, they have all the incentive in the world to stop and fix the problem. It was purely a cultural thing where in their mind their code worked 'good enough' so why not deploy it and just take the feedback as something that could be done better next time.
    - OutOfHere 2y ago
      With regard to alerts, I have written software that daytrades stocks, making a lot of trades over a lot of stocks. Let me assure you that not a single alert goes ignored, and if someone said it's okay to ignore said alerts, or to have persistent alerts that require no action, they would be losing money because in time, they will inevitably ignore a critical error. I stand by my claim that it's what sets apart good employees from those that don't care if the business lives or dies. I think a role of management is to ensure that employees understand the potential consequences to the business of the code being wrong.
      - steveBK123 2y ago
        Yes, there was a recent story about (yet another) Citi "fat finger" trade. The headlines mentioned things like "the trader ignored 700 error messages to put in the trade", but listening to a podcast about it.. its more like awful systems that are always half broken is what ultimately lead to it. The real punchline was this - the trader confused a field for entering shares quantity for notional quantity, but due to some European markets being closed, the system had a weird fallback logic that it sets the value of shares to $1, so the confirmation back to the trader was.. the correct number of dollars he expected. So awful system designs lead to useless and numerous alerts, false confirmations, and ultimately huge errors.
  - josefx 2y ago
    > If employees had ownership over which changes they think are best, a good employee would act on bringing the alerts back to zero before they take on new features or a new project That requires that you have good employees, which can be as rare as good management.
- simpaticoder 2y ago
  The more pernicious form of this, in my experience, are ignored compiler/linter/test warnings. Many codebases have a tremendous number of these warnings, devs learn to ignore them, and this important signal of code quality is effectively lost.
  - eschneider 2y ago
    It's almost always worth spending the time to either fix all warnings or, after determining it's a false positive, suppressing it with a #pragma. Once things are relatively clean, it's easy to see if new code/changes trip a warning. Often unexpected warnings are a sign of subtle bugs or at least use of undefined behaviors. Sorting those out when they come up is a heck of a lot easier than tracing a bug report back to the same warming.
- ChrisMarshallNY 2y ago
  I like to program with -wall. Doesn't win me fans, but I sleep well.
  - drited 2y ago
    Could you please expand on what that is?
    - vardump 2y ago
      Enable all warnings. https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wall https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#inde...
    - ChrisMarshallNY 2y ago
      It's a C/C++ compiler flag, saying all warnings on. Since I do Swift, these days, in Xcode, I use project settings, instead. I also like to treat warnings as errors. Forces me to be circumspect.
    - artificialLimbs 2y ago
      https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wall https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#inde...
    - diab0lic 2y ago
      It’s a CLI flag to the compiler that enables all warnings.
- rjmunro 2y ago
  In both Challenger and Columbia disasters, people noticed there might be a problem, tried to escalate it to get it fixed and failed to stop the launch, leading to disasters. Do we know how many times people noticed a problem, it launched anyway and everything was fine?

steveBK123 2y ago

Exactly the concept why you don't want to let whatever dashboards/alerts/etc you maintain on your systems have a "normal amount of reds/fails/spurious texts". At some point you become immune. It's a lot harder to notice theres 4 red lights today than the usual 2-3 vs noticing 1 when there are normally exactly 0.

OutOfHere 2y ago
Yes. The causative issue is the way in which projects are managed. Employees have no ownership of the project. If employees had ownership over which changes they think are best, a good employee would act on bringing the alerts back to zero before they take on new features or a new project. There are some obstacles: 1. Employees not having a say in which issues to work on. This pretty much leads to the death of a project in the medium term due to near-total disregard of maintenance issues and alerts. 2. Big-team ownership of a project. When everyone is in charge, no one is. This is why I advocate for a team size of exactly two for each corporate project. 3. Employees being unreasonably pressured for time. Perhaps the right framing for employees to think about it is: "If it were their own business or product, how would they do it?" This framing, combined with the backlog, should automatically help avoid spending more time than is necessary on an issue.
- whodidntante 2y ago
  Not making an ethical/moral judgement here, just a practical one - is there any reason to believe that giving employees ownership of the projects will be any better than having "management" own it if all factors were truly considered ? If every decision an employee made on features/issues/quality/time was accompanied by how much their pay was affected, would the outcomes really be better ? The team could decide to fix all bugs before taking on a new feature, or that the 2 month allotment to a feature should really be three months to do it "right" without having to work nights/weekends, would the team really decide to do that if their paycheck was reduced by 10%, or delayed for that extra month for those new features were delivered ? If all factors were included in the employee decision process, including the real world effect of revenue/profit on individual compensation from those decisions, it is not clear to me that employees would make any "better" decisions. I would think that employees could be even more "short sighted" than senior management, as senior management likely has more at stake in terms of company reputation/equity/career than an employee who can change jobs easier, and an employee might choose not to "get those alerts to zero" if it meant they would have more immediate cash in their pocket. And how would disagreements between team members be worked out if some were willing to forgo compensation to "do it right', and others wanted to cut even more corners ? Truly having ownership means you have also financial risk.
  - rawgabbit 2y ago
    What I see is a movement where line employees have a say on who is retained at the director and VP level. The CEO reports to the board. But his immediate and second tier reports are also judged by the employees. The thought is that will give them pause before they embark on their next my way or the highway decision making. The most egregious directors who push out line employees in favor of their cronies will be fired under this evaluation.
  - OutOfHere 2y ago
    > is there any reason to believe that giving employees ownership of the projects will be any better than having "management" own it Non-technical management's skill level is almost always overrated. They're almost never qualified for it. Ultimately it still is management's decision, and always will be. If however management believes that employees are incapable of serving users, then it's management's fault for assigning mismatched employees. > how much their pay was affected Bringing pay into this discussion is a nonsensical distraction. If an employer misses two consecutive paychecks by even 1%, that's enough reason to stop showing up for work, and potentially to sue for severance+damages, and also claim unemployment wages. There is no room for any variation here. > Truly having ownership It should be obvious that ownership here refers to the ownership of the technical direction, not literal ownership in the way I own a backpack that I bring to work. If true financial ownership existed, the employee would be receiving substantial equity with a real tradable market value, with the risk of losing some of this equity if they were to lose their job. > how would disagreements between team members be worked out As noted, there would be just two employees per project, and this ought to minimize disagreements. If disagreements still exist, this is where management can assist with direction. There should always remain room for conducting diverse experiments without having to worry about which outcomes get discarded and which get used. --- In summary, if the suggested approach is not working, it's probably because there is significant unavoidable technical debt or the employees are mismatched to the task.
  - nostrademons 2y ago
    Been all of an engineer, a manager, and a founder/CEO, and I enjoy analyzing organizational dysfunction. The difference between an engineer and a manager's perspective usually comes down to their job description. An engineer is hired to get the engineering right; the reason the company pays them is for their ability to marry reality to organizational goals. The reason the company hires a manager is to set those organizational goals and ensure that everybody is marching toward them. This split is explicit for a reason: it ensures that when disagreements arise, they are explicitly negotiated. Most people are bad at making complex tradeoffs, and when they have to do so, their execution velocity suffers. Indeed, the job description for someone who is hired to make complex tradeoffs is called "executive", and they purposefully have to do no real work so that their decision-making functions only in terms of cost estimates that management bubbles up, not the personal pain that will result from those decisions. Dysfunction arises from a few major sources: 1. There's a power imbalance between management and engineering. An engineer usually only has one project; if it fails, it often means their job, even if the outcome reality dictates is that it should fail. That gives them a strong incentive to send good news up the chain even if the project is going to fail. Good management gets around this by never penalizing bad news or good-faith project failure, but good management is actually really counterintuitive, because your natural reaction is to react to negative news with negative emotions. 2. Information is lost with every explicit communication up the chain. The information an engineer provides to management is a summary of the actual state of reality; if they passed along everything, it'd require that management become an engineer. Likewise recursively along the management chain. It's not always possible to predict which information is critical to an executive's decision, and so sometimes this gets lost as the management chain plays telephone. 3. Executives and policy-makers, by definition, are the least reality-informed people in the system, but they have the final say on all the decisions. They naturally tend to overweight the things that they are informed on, like "Will we lose the contract?" or "Will we miss earnings this quarter?" All that said, the fact that most companies have a corporate hierarchy and they largely outcompete employee-owned or founder-owned cooperatives in the marketplace tends to suggest that even with the pitfalls, this is a more efficient system. The velocity penalty from having to both make the complex decisions and execute on them outweighs all the information loss. I experienced this with my startup: the failure mode was that I'd emotionally second-guess my executive decisions, which meant that I executed slowly on them, which meant that I didn't get enough iterations or enough feedback from the market to find product/market fit. This is also why startups that do succeed tend to be ones where the idea is obvious (to the founder at least, but not necessarily to the general public). They don't need to spend much time on complex positioning decisions, and can spend that time executing, and then eventually grow the company within the niche they know well.
- lencastre 2y ago
  And groupthink
- CSMastermind 2y ago
  > If employees had ownership over which changes they think are best, a good employee would act on bringing the alerts back to zero before they take on new features or a new project. You say this but as someone who's run a large platform organization that hasn't been my experience. Sure some employees, maybe you, care about things like bringing alerts back to zero but a large number are indifferent and a small number are outright dismissive. This is informed not just by individual personality but also by culture. Not too long ago I pointed out a bug in someone's code who I was reviewing and instead of fixing it they said, "Oh okay, I'll look out for bugs like that when I write code in the future" then proceeded to merge and deploy their unchanged code. And in that case I'm their manager not a peer or someone from another team, they have all the incentive in the world to stop and fix the problem. It was purely a cultural thing where in their mind their code worked 'good enough' so why not deploy it and just take the feedback as something that could be done better next time.
  - OutOfHere 2y ago
    With regard to alerts, I have written software that daytrades stocks, making a lot of trades over a lot of stocks. Let me assure you that not a single alert goes ignored, and if someone said it's okay to ignore said alerts, or to have persistent alerts that require no action, they would be losing money because in time, they will inevitably ignore a critical error. I stand by my claim that it's what sets apart good employees from those that don't care if the business lives or dies. I think a role of management is to ensure that employees understand the potential consequences to the business of the code being wrong.
    - steveBK123 2y ago
      Yes, there was a recent story about (yet another) Citi "fat finger" trade. The headlines mentioned things like "the trader ignored 700 error messages to put in the trade", but listening to a podcast about it.. its more like awful systems that are always half broken is what ultimately lead to it. The real punchline was this - the trader confused a field for entering shares quantity for notional quantity, but due to some European markets being closed, the system had a weird fallback logic that it sets the value of shares to $1, so the confirmation back to the trader was.. the correct number of dollars he expected. So awful system designs lead to useless and numerous alerts, false confirmations, and ultimately huge errors.
- josefx 2y ago
  > If employees had ownership over which changes they think are best, a good employee would act on bringing the alerts back to zero before they take on new features or a new project That requires that you have good employees, which can be as rare as good management.
simpaticoder 2y ago
The more pernicious form of this, in my experience, are ignored compiler/linter/test warnings. Many codebases have a tremendous number of these warnings, devs learn to ignore them, and this important signal of code quality is effectively lost.
- eschneider 2y ago
  It's almost always worth spending the time to either fix all warnings or, after determining it's a false positive, suppressing it with a #pragma. Once things are relatively clean, it's easy to see if new code/changes trip a warning. Often unexpected warnings are a sign of subtle bugs or at least use of undefined behaviors. Sorting those out when they come up is a heck of a lot easier than tracing a bug report back to the same warming.
ChrisMarshallNY 2y ago
I like to program with -wall. Doesn't win me fans, but I sleep well.
- drited 2y ago
  Could you please expand on what that is?
  - vardump 2y ago
    Enable all warnings. https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wall https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#inde...
  - ChrisMarshallNY 2y ago
    It's a C/C++ compiler flag, saying all warnings on. Since I do Swift, these days, in Xcode, I use project settings, instead. I also like to treat warnings as errors. Forces me to be circumspect.
  - artificialLimbs 2y ago
    https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wall https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#inde...
  - diab0lic 2y ago
    It’s a CLI flag to the compiler that enables all warnings.
rjmunro 2y ago
In both Challenger and Columbia disasters, people noticed there might be a problem, tried to escalate it to get it fixed and failed to stop the launch, leading to disasters. Do we know how many times people noticed a problem, it launched anyway and everything was fine?