4 ms·
On the one hand, there are decent technical tools to deal with this kind of thing; I'm directly familiar with Ansible's support for controlled rollouts, most co
by yjftsjthsd-h 7y ago
On the one hand, there are decent technical tools to deal with this kind of thing; I'm directly familiar with Ansible's support for controlled rollouts, most container management layers happily support blue/green deploys, etc. On the other hand -
> Some people obviously couldn't be bothered to step their release along over the course of a couple of days, and we had to find some way to protect the production environment from them.
That's easy: HR. By all means start by being nice, but if people refuse to be careful then they can burn down someone else's prod.
- jameskraus 7y ago> That's easy: HR Unfortunately, that approach doesn't scale well. In a larger company, there are always new users discovering the systems and how they work (to varying degrees). If doing the right thing isn't by-and-far the easiest path, then it's highly likely someone will misuse the tool – no matter how smart they are. It's far easier to make the interface really easy to use than it is to educate users on proper usage.
- thanksforfish 7y ago> there are always new users discovering the systems Software engineering turnover rates can be high, like 13%/year [1]. Add to that engineers who internally switch teams. New users is a real concern. In this case, it sounds like a custom deployment tool, so even engineers with good experience at other companies may not find the feature. Good team on-boarding and documentation is important. I think code review should catch something like this. Even with the high rate of new users, there should be enough experienced code reviewers to catch the issue. [1] https://www.techrepublic.com/article/software-had-the-highest-job-turnover-rate-of-any-industry-in-2017/ https://www.techrepublic.com/article/software-had-the-highes...
- asveikau 7y agoThe problem is when the distance between the team writing the package manager and random team trying to work on some feature is high. That entire team might not be aware of the team that wants them to deploy gradually and in a specific way. To them, maybe even it's a lot of noise and useless complaining and they don't see it as at all connected to getting their feature done. A common large company problem. People who should be talking are not. New hires are also hired into the "local" subculture of their team and something very, very important elsewhere in the company is not on their radar.
- paxys 7y agoHR is exactly the worst approach to deal with organizational and process problems like this. What do you do? Fire the entire dev team involved with the decision? Fire their managers who failed to set the right expectations? Fire the product team for setting deadlines that were too aggressive? Fire the CEO because hey, ultimately they are the one responsible for everything? Even if you do all that, you're not going to go back in time and fix things for your customers, and you're definitely not going to prevent the replacement developers from making the exact same mistakes. It takes really good leadership to build an engineering culture where good practices are valued and people learn from mistakes, which is why filling those positions is so hard and expensive. A company whose solution for every outage is to fire people isn't going to last very long.
- deleted 7y ago[deleted]
- AnimalMuppet 7y agoGP said: >> That's easy: HR. By all means start by being nice, but if people refuse to be careful then they can burn down someone else's prod. If people refuse to be careful. If they make a mistake once, teach them. If they make a mistake again, that happens. But if they make a mistake regularly, warn them, then warn them sternly, then fire them.
- compiler-guy 7y agoMaybe so, but then you still have the problem, because in a big organization, some newbie is always present and going to make that first mistake. So you still need to make it hard for people to make mistakes.
- csours 7y agoTo amplify: think about road signs when you are trying to get somewhere new, especially in places with weird parking or entrances. After you've figured it out, it's easy, but everyone is a first timer at some point.
- 7y ago
- greenyoda 7y agoHR is not an organization that should be arbitrating technical issues that they don't know anything about. The solution is for technical managers - at all levels, from team leads up to the CEO - to actually provide leadership to their teams. This can consist of training people about why it's important to do things right and setting up incentives to do things right (e.g., don't promote people who get things done quickly by cutting corners). To me, this story sounds like a classical management failure (been there, done that myself).
- geofft 7y agoBut I don't want them to burn down someone else's prod, I want them to keep building features for my company. (Worse, if the other company has a misuse-resistant deployment system like the one the author alludes to, they might not burn down that company's prod and then beat us in the market while we're busy interviewing replacements!)
- triceratops 7y agoThat's dumb. If your system makes it possible for someone to screw up, someone will eventually screw up. And if screwing up (rather than being malicious or stupid) results in people getting fired, then no work will ever get done for fear of getting fired. You have to fix the system so it's really hard to screw up.
- csours 7y ago"A person is smart. People are dumb, panicky dangerous animals and you know it." When I was first hired into a manufacturing plant I wondered why it was so hard to get people to make a quality product. It turns out that people are people, and they will try to do the job you tell them to do, with the tools and instruction you give them. The better the tools are, the less instruction is needed. Output rate and quality is also improved. The poorer the tools are, more instruction is needed; output rate and quality suffer, and at some point safety as well. "A poor workman blames their tools"; A poor manager blames their workman after providing them with poor tools. The very first thing you need to do is assess if the worker is following process and if not why not. If they are not following process, is it because of a mistake or a slip-up? Did they intentionally decide not to follow a process because it was difficult? Disaster documentaries are quick to point out where workers failed to follow written procedures, but they VERY RARELY point out how common those contraventions are with no negative impact. If "it's stupid but it works" it may not be stupid, or you may have gotten lucky. https://www.youtube.com/watch?v=WPMMNvYTEyI https://www.youtube.com/watch?v=WPMMNvYTEyI