20 ms·

Nice little read. There is that little answer in the back of my head of: "It is a good question why I caused this problem. It is weird to think there is a comp

by scraft 10y ago

Nice little read. There is that little answer in the back of my head of: "It is a good question why I caused this problem. It is weird to think there is a competent company that has been around for so many years, yet they have no procedures in place to stop this from happening. You would think that any changes that could cause downtime on a clients website would go through an automatic test suite and only after passing all tests would it be tested by human QA and finally made live. In this particular case I am the one who has made a mistake, and being a sensible person who is eager to constantly better myself, I will try and learn from this error, however I work alongside dozens of other people on my team who like me, have every chance of making a mistake at some point. It seems like a bad policy for us as a company to say that we should expect every person on the team to break a clients site and then rely on overtime from other people to fix it. So of course I will try and do better, but this is not fixing the root of the problem, the root of the problem is something that needs to be fixed at a much higher level. Have you, as my boss, not considered this problem already? What has the company learned from this? I would be happy to be part of the team that solves this problem by creating unit tests and creating policy to avoid this." Of course I purposely make this slightly stand-off-ish, over the top, and written from a very specific perspective, to illustrate a point. But the reason I do this is I absolutely agree a single developer shouldn't feel bad about making a mistake (they should try not to, but these things happen), but the company should put measures in place to minimise the impact of mistakes. When the company fails to do this, the company has screwed up a lot more than the developer.

d0mine 10y ago
There is a cost in not having certain procedures in place. On the other hand the red tape has a cost too. A single mistake shouldn't lead unconditionally to a mandatory multi-step process that involves multiple people and may take several months to finish. The examples of the "cover your ass" decision making overpowering the common sense engineering are abundant.
- pwillia7 10y ago
  It depends on the project. Is it a web application for keeping track of daily tasks? You probably don't need that much rigor. Are you building something that if it fails will harm/kill/cost people a lot of money? More rigor is needed.
- godshatter 10y ago
  That's true, you can definitely over-correct. I guess if I was the supervisor in this case, I'd be doing some analysis with the developer after the fact. Were procedures correctly followed? If not, use this event as an example of why the procedures are in place. If they were followed, was it a brain-dead simple error? If so, then obviously there is something wrong with our test plans. If it was one of those errors that turned out to be a perfect storm of environment differences, unexpected inputs, code side-effects or what-have-you, then there probably is no quick solution and the imposition of some overbearing system to try to limit something that will probably not happen again would be overkill. It's always best to do an after action analysis to see where things broke down, I think. Changing procedures or adding protections might be warranted, as long as it's not the "get my permission before ever putting a Y in that field again" kind of enhancement.
overgard 10y ago
Think of what you're saying on a meta-level here. You would basically be telling this guy, "I fucked up, but I'm blaming you for treating me like an adult instead of a child. You screwed up by giving me enough autonomy that I could make a mistake." That might not exactly be a smart career move.. On a more personal level, if someone shows you kindness in the face of a mistake, the last thing you want to do is throw it back in their face and go on a tirade against them. That's a very quick way to make sure nobody ever wants to work with you again.
- edraferi 10y ago
  Good point. It's important to take responsibility even if you think there should have been guardrails.
- mundo 10y ago
  Reminds me of an old VP of mine who came thundering out of his office, absolutely livid, shouting, "What idiot gave me permissions to delete the source backups???"
  - kefka 10y ago
    From a security perspective, there is the least privilege concept. Why does the VP have that kind of access by default? I understand having a separate account if needs are there, of having some sort of priv-escalation. In my new job as network engineer, all the RH boxes have SElinux off. I don't get it, nor can anyone give a cogent answer. And up 4 floors, we have a selinux kernel Dev. I'd greatly like to even limit roots potential damage against users' data (in this case, DBs).
    - JoshTriplett 10y ago
      > Why does the VP have that kind of access by default? Two common reasons. First, because the VP or someone in a similar position resented not having such access; it's sometimes hard for people to accept that people who work "for" them hierarchically have permissions they don't. And second, because either they or the person who previously held their position had a legitimate reason for such access in the past, and didn't drop it when they no longer had such a reason. But yes, even if they have a legitimate reason for such access, they shouldn't have that permission all the time, only when they intentionally don that hat.
solatic 10y ago
I worked for a boss once who essentially engineered this kind of problem to exist: he knew not having automated testing etc. was a problem, and instead of asking his (too junior) programmers to build up a test suite, he decided to let them fail and discover for themselves why writing up test suites is a good idea. His reasoning (later revealed) was that you can't give people fish and expect them to become expert fishermen. They have to experience hunger and ask to be taught to fish. If people don't have the soul-crushing experience that teaches them why something is really important, they never really internalize why it's important. That the reason why we appreciate that this stuff is so important is BECAUSE we suffered through something that taught us that it was important, and that if we don't give junior folks that same kind of experience, then good practice just becomes something on the list of priorities and not a moral imperative. There's something to that, but at the same time, it's basically a defense of institutional hazing on the investor's dime. So take it for what it is.
- kbenson 10y ago
  > His reasoning (later revealed) was that you can't give people fish and expect them to become expert fishermen. I think you can, it's just not nearly as efficient. College and trade schools are essentially about mixing a very small amount of actual necessity (deadlines, tests, etc) with copious amounts of being told what's best and how to accomplish something. This works well because it's both more palatable to a wider audience of people, and it accounts for people's different interpretation of events. There may be multiple things to learn from any particular failure, and without some guidance you may come away having learned few or none of them the first time. Being told what to expect before hand, or what you could have done to mitigate problems afterward, goes a long way towards making sure you consider all the useful aspects of the problem. Will you learn any one single lesson as thoroughly or as fast as you would in the real world when it affects you so much? Likely not, but then again, that assumes you actually saw a solution to the problem, and that still might just be one aspect.
meesles 10y ago
I feel where you're coming from, but in reality it's not like any test suite can guarantee to prevent downtime caused by bad deploys. All it takes is a little networking related bug, something that differs ever so slightly between environments, whatever, and your site could be down. I agree with the other replies that while you may be the 'technically right dev' kind of person, it still comes of as being a dick and you will likely not get a positive response from something like this, especially if you caused it. What makes it more reasonable to say it's the company's responsibility to protect you from fucking up the site instead of you just being a good developer, and not fucking up the site?
- hilop 10y ago
  It's not about being a good cowboy developer, it's about beijg a professional engineer who builds systems that work. I don't expect the folks who built my house to detach and reattach the door pefectly every day, I expect them to build a house that works.
DenisM 10y ago
Exactly. If you're one mistake away from a disaster, you already lost. Ideally you would have either test gates, or a staged rollout with automated rollback.
deleted 10y ago
[deleted]

by scraft 10y ago

d0mine 10y ago

There is a cost in not having certain procedures in place. On the other hand the red tape has a cost too. A single mistake shouldn't lead unconditionally to a mandatory multi-step process that involves multiple people and may take several months to finish. The examples of the "cover your ass" decision making overpowering the common sense engineering are abundant.

pwillia7 10y ago
It depends on the project. Is it a web application for keeping track of daily tasks? You probably don't need that much rigor. Are you building something that if it fails will harm/kill/cost people a lot of money? More rigor is needed.
godshatter 10y ago
That's true, you can definitely over-correct. I guess if I was the supervisor in this case, I'd be doing some analysis with the developer after the fact. Were procedures correctly followed? If not, use this event as an example of why the procedures are in place. If they were followed, was it a brain-dead simple error? If so, then obviously there is something wrong with our test plans. If it was one of those errors that turned out to be a perfect storm of environment differences, unexpected inputs, code side-effects or what-have-you, then there probably is no quick solution and the imposition of some overbearing system to try to limit something that will probably not happen again would be overkill. It's always best to do an after action analysis to see where things broke down, I think. Changing procedures or adding protections might be warranted, as long as it's not the "get my permission before ever putting a Y in that field again" kind of enhancement.

overgard 10y ago

Think of what you're saying on a meta-level here. You would basically be telling this guy, "I fucked up, but I'm blaming you for treating me like an adult instead of a child. You screwed up by giving me enough autonomy that I could make a mistake." That might not exactly be a smart career move.. On a more personal level, if someone shows you kindness in the face of a mistake, the last thing you want to do is throw it back in their face and go on a tirade against them. That's a very quick way to make sure nobody ever wants to work with you again.

edraferi 10y ago
Good point. It's important to take responsibility even if you think there should have been guardrails.
mundo 10y ago
Reminds me of an old VP of mine who came thundering out of his office, absolutely livid, shouting, "What idiot gave me permissions to delete the source backups???"
- kefka 10y ago
  From a security perspective, there is the least privilege concept. Why does the VP have that kind of access by default? I understand having a separate account if needs are there, of having some sort of priv-escalation. In my new job as network engineer, all the RH boxes have SElinux off. I don't get it, nor can anyone give a cogent answer. And up 4 floors, we have a selinux kernel Dev. I'd greatly like to even limit roots potential damage against users' data (in this case, DBs).
  - JoshTriplett 10y ago
    > Why does the VP have that kind of access by default? Two common reasons. First, because the VP or someone in a similar position resented not having such access; it's sometimes hard for people to accept that people who work "for" them hierarchically have permissions they don't. And second, because either they or the person who previously held their position had a legitimate reason for such access in the past, and didn't drop it when they no longer had such a reason. But yes, even if they have a legitimate reason for such access, they shouldn't have that permission all the time, only when they intentionally don that hat.

solatic 10y ago

I worked for a boss once who essentially engineered this kind of problem to exist: he knew not having automated testing etc. was a problem, and instead of asking his (too junior) programmers to build up a test suite, he decided to let them fail and discover for themselves why writing up test suites is a good idea. His reasoning (later revealed) was that you can't give people fish and expect them to become expert fishermen. They have to experience hunger and ask to be taught to fish. If people don't have the soul-crushing experience that teaches them why something is really important, they never really internalize why it's important. That the reason why we appreciate that this stuff is so important is BECAUSE we suffered through something that taught us that it was important, and that if we don't give junior folks that same kind of experience, then good practice just becomes something on the list of priorities and not a moral imperative. There's something to that, but at the same time, it's basically a defense of institutional hazing on the investor's dime. So take it for what it is.

kbenson 10y ago
> His reasoning (later revealed) was that you can't give people fish and expect them to become expert fishermen. I think you can, it's just not nearly as efficient. College and trade schools are essentially about mixing a very small amount of actual necessity (deadlines, tests, etc) with copious amounts of being told what's best and how to accomplish something. This works well because it's both more palatable to a wider audience of people, and it accounts for people's different interpretation of events. There may be multiple things to learn from any particular failure, and without some guidance you may come away having learned few or none of them the first time. Being told what to expect before hand, or what you could have done to mitigate problems afterward, goes a long way towards making sure you consider all the useful aspects of the problem. Will you learn any one single lesson as thoroughly or as fast as you would in the real world when it affects you so much? Likely not, but then again, that assumes you actually saw a solution to the problem, and that still might just be one aspect.

meesles 10y ago

I feel where you're coming from, but in reality it's not like any test suite can guarantee to prevent downtime caused by bad deploys. All it takes is a little networking related bug, something that differs ever so slightly between environments, whatever, and your site could be down. I agree with the other replies that while you may be the 'technically right dev' kind of person, it still comes of as being a dick and you will likely not get a positive response from something like this, especially if you caused it. What makes it more reasonable to say it's the company's responsibility to protect you from fucking up the site instead of you just being a good developer, and not fucking up the site?

hilop 10y ago
It's not about being a good cowboy developer, it's about beijg a professional engineer who builds systems that work. I don't expect the folks who built my house to detach and reattach the door pefectly every day, I expect them to build a house that works.

DenisM 10y ago

Exactly. If you're one mistake away from a disaster, you already lost. Ideally you would have either test gates, or a staged rollout with automated rollback.

deleted 10y ago

[deleted]