6 ms·

The findings from Salesforce example: 1. A lack of automation with safeguards for DNS changes 2. Insufficient guardrails to enforce the Change Management p

The findings from Salesforce example: 1. A lack of automation with safeguards for DNS changes 2. Insufficient guardrails to enforce the Change Management process 3. Subversion of the Emergency Break Fix (EBF) process It looks a lot like somebody tried to push their agenda (personal or organizational) and write points that suit their preferred solution rather than try to understand actual causes of the outage. These findings are essentially "Somebody made a mistake, let's remove accesses and/or put more controls on the process so that people are not even able to make a mistake". This seems to be very low trust environment. My experience tells me the most likely course of action is that even if the problem is fixed, it is going to happen at the cost of more overhead in the process and making it even more low-trust, causing more damage in the long run. You can apply this kind of flawed reasoning ("somebody made a mistake -- nobody can be trusted with it again") to any problem and soon nobody can do anything on their own. Productivity plummets. People are becoming disinterested in their work (try to be "engaged" when management takes all your tools away). Management blames people for not being able to do even basic things -- loosing even more trust and pushing even more solutions like that. Vicious cycle. Take for example point #3. How about figuring out WHY people need or feel they need to subvert EBF process? Isn't it rational to assume that if people have been subverting the process they might have actual reason to do so? Maybe their regular process is too onerous and they are using emergency process to meet deadlines? Maybe the effort should be directed at improving the regular process so that they don't have reason to subvert it?

blacksmith_tb 5y ago
I see that a lot in RCAs - it's generally easy to rush out how something went wrong, but it can be much more subtle why it did; but since heads will roll if it isn't sent in 24hr, we get useless recaps.
- deleted 5y ago
  [deleted]
kevin_thibedeau 5y ago
This isn't low trust but rather poka-yoke mistake proofing. If you have a fallible process you either replace it with an infallible one or put in controls to mitigate the risk. When you effectively run a factory for configuring software you have an interest in not letting a single production worker shut down the whole operation.
- lmilcin 5y ago
  That is exactly low trust environment. Trust requires that you accept that you can be hurt by the person you trust. If you work to remove that ability you are saying you do not trust that other person. Now, I am not saying that you should design processes so that employees can fail them. What I am saying is that when somebody is a developer and the company says it is ok to "burn" their time and effort because they do not trust the developer to make a right decision, it sucks and it makes it very difficult for that person to care to do a good job. There are usually different ways to prevent things from failing that do not require the organization to manifest that it does not trust their employees. And there are ways for the organization to show they trust their employees to make up for the cases when these types of solutions can't be implemented. I worked in Samsung. In the office I worked, there is a ban on having knives in the kitchen, even the cutlery that is basically unusable to cut anything more substantial than overcooked potatoes. I heard the same for all other offices. Apparently, two guys half a world away had a knife fight in the kitchen and so it was decided every single employee is not to be trusted handling cutlery. The funny thing is, people started bringing their own, really sharp knives. So... knives are still there but employees have one more reminder that they are not being treated as adults. I worked in a large number of companies, from small to very, very large ones. They all vary a lot in the level of permission any single employee has to do things. But one thing I noticed is that companies where employees have very little permission to do anything do not have less process failures. There is many reasons for this. Now that you made your employees impotent, you need a lot more of them to do the same work. Or maybe you need to shift the responsibilities from real thinking people to automation that can also make mistakes but with a much larger impact. People that are constantly reminded they are not trusted start behaving this way. They tend to lower standards and stop caring. Taking somebodys permission to do something can be done well but it usually requires much more care and thought than what typically happens at large corporations.
benchaney 5y ago
I think this is pretty misguided. If reliability is at all important, you need controls in place so that a single person making a single mistake won't cause an outage. Software engineers are human and humans make mistakes. That's just the reality. That isn't indicative of a lack of trust, but rather an acknowledgement of the constraints we are operating under. Its possible that people subverting these processes are doing so because there are some actual flaws in the processes, but generally it is just as likely that they are being lazy or just don't think there is any way they would ever make a mistake. That isn't to say that it isn't worth evaluating the processes and the tradeoffs they impose, just that I think you are definitely throwing the baby out with the bathwater.
- lmilcin 5y ago
  The most astonishing thing is that people actually believe that crap. That the solution to people making mistakes is taking away any power to do anything substantial on their own. No power to do anything == no power to do any damage. Think how a machine shop would work if we applied the same rules. "No, you can't use CNC machine because somebody broke an expensive part once. You need to fill this form, prepare this feasibility study and wait for director approval for every change to the program. Of course, director has absolutely no knowledge to validate your CNC routine but this is the process." I worked for those organizations, just left one. You get hundreds of people do work that couple smart engineers without artificial limitations could do in a fraction of time.
  - xboxnolifes 5y ago
    Unless the CNC machine is the entire business, breaking a part isn't an outage. It's just an financially expensive mistake. On the other hand, if you have a building running life-support machines, you probably don't want someone messing with the breaker box that connects to them without designing some protections and backups.

blacksmith_tb 5y ago

I see that a lot in RCAs - it's generally easy to rush out how something went wrong, but it can be much more subtle why it did; but since heads will roll if it isn't sent in 24hr, we get useless recaps.

deleted 5y ago
[deleted]

kevin_thibedeau 5y ago

This isn't low trust but rather poka-yoke mistake proofing. If you have a fallible process you either replace it with an infallible one or put in controls to mitigate the risk. When you effectively run a factory for configuring software you have an interest in not letting a single production worker shut down the whole operation.

lmilcin 5y ago
That is exactly low trust environment. Trust requires that you accept that you can be hurt by the person you trust. If you work to remove that ability you are saying you do not trust that other person. Now, I am not saying that you should design processes so that employees can fail them. What I am saying is that when somebody is a developer and the company says it is ok to "burn" their time and effort because they do not trust the developer to make a right decision, it sucks and it makes it very difficult for that person to care to do a good job. There are usually different ways to prevent things from failing that do not require the organization to manifest that it does not trust their employees. And there are ways for the organization to show they trust their employees to make up for the cases when these types of solutions can't be implemented. I worked in Samsung. In the office I worked, there is a ban on having knives in the kitchen, even the cutlery that is basically unusable to cut anything more substantial than overcooked potatoes. I heard the same for all other offices. Apparently, two guys half a world away had a knife fight in the kitchen and so it was decided every single employee is not to be trusted handling cutlery. The funny thing is, people started bringing their own, really sharp knives. So... knives are still there but employees have one more reminder that they are not being treated as adults. I worked in a large number of companies, from small to very, very large ones. They all vary a lot in the level of permission any single employee has to do things. But one thing I noticed is that companies where employees have very little permission to do anything do not have less process failures. There is many reasons for this. Now that you made your employees impotent, you need a lot more of them to do the same work. Or maybe you need to shift the responsibilities from real thinking people to automation that can also make mistakes but with a much larger impact. People that are constantly reminded they are not trusted start behaving this way. They tend to lower standards and stop caring. Taking somebodys permission to do something can be done well but it usually requires much more care and thought than what typically happens at large corporations.

benchaney 5y ago

I think this is pretty misguided. If reliability is at all important, you need controls in place so that a single person making a single mistake won't cause an outage. Software engineers are human and humans make mistakes. That's just the reality. That isn't indicative of a lack of trust, but rather an acknowledgement of the constraints we are operating under. Its possible that people subverting these processes are doing so because there are some actual flaws in the processes, but generally it is just as likely that they are being lazy or just don't think there is any way they would ever make a mistake. That isn't to say that it isn't worth evaluating the processes and the tradeoffs they impose, just that I think you are definitely throwing the baby out with the bathwater.

lmilcin 5y ago
The most astonishing thing is that people actually believe that crap. That the solution to people making mistakes is taking away any power to do anything substantial on their own. No power to do anything == no power to do any damage. Think how a machine shop would work if we applied the same rules. "No, you can't use CNC machine because somebody broke an expensive part once. You need to fill this form, prepare this feasibility study and wait for director approval for every change to the program. Of course, director has absolutely no knowledge to validate your CNC routine but this is the process." I worked for those organizations, just left one. You get hundreds of people do work that couple smart engineers without artificial limitations could do in a fraction of time.
- xboxnolifes 5y ago
  Unless the CNC machine is the entire business, breaking a part isn't an outage. It's just an financially expensive mistake. On the other hand, if you have a building running life-support machines, you probably don't want someone messing with the breaker box that connects to them without designing some protections and backups.