7 ms·
That employee may be going to jail, but it's not his fault that this happened. If your fired employees have access to your servers, you fucked up the firing ho
by jrockway 16y ago
That employee may be going to jail, but it's not his fault that this happened. If your fired employees have access to your servers, you fucked up the firing horribly. Similarly, if one data center going offline can take out your entire season of work, perhaps you need to learn what redundancy is.
- uptown 16y agoWhat you said about redundancy is true, but it's still his fault this happened.
- jemfinch 16y agoI would not be surprised if their "data hosting service" claimed redundancy as a feature.
- ericd 16y agoHow is this not his fault?
- jrockway 16y agoWhat I'm saying is that relying on crazy people to not go crazy is not a good data loss prevention strategy. (In the end, putting someone in a situation where they'll do something bad is worse than being the person that does something bad. A good example is the Colgan Air crash from a few years ago. Anybody knows that when your plane is stalling you're supposed to push the control column forward and apply full engine power. The plane does the pushing the control column forward part automatically! But for some reason, the trained airline captain did the exact opposite, stalling the plane and killing everyone on board. Why? Because he got paid so little that he couldn't afford a hotel room to sleep in before his work that day, and had to commute across the country and then work a full shift. This is the airline's fault for impairing someone to the extent that he killed 50+ people by doing the exact opposite of what any trained pilot would do. $300 for a hotel room and everyone on that flight would still be alive today. Similarly, when you fire someone that works with critical data, you need to make sure the guy's access is revoked. It's a precaution that needs to be taken, as this case shows. When necessary precautions are omitted, bad things happen. Planes crash, and important data goes away forever.)
- tjmc 16y agoI understand your point but the difference in the Colgan case is that the pilot made a tragic mistake, whereas the disgruntled employee deliberately wiped the show. Yes, his employer is careless for not revoking his permissions, but the question remains - how is this not his fault?
- davidandgoliath 16y agoIt is. It's also the fault of everyone else involved who didn't back up.
- alexqgb 16y ago"What I'm saying is that relying on crazy people to not go crazy is not a good data loss prevention strategy." Upvoted for exactitude. Also, it's worth remembering that virtually no one hires already disgruntled employees. That's not to excuse anything that the truly disgruntled do to retaliate. But it does suggest that an abusive employer represents a risk to everyone relying on their 'teams'. EDIT: I don't think the pilot was retaliating, by the way. What he was subject to is a different kind of abuse. Separately, when two parties are in a fight, it's always worth remembering who started it. If blame needs to be shared, and it's unfair to share it evenly, the unprovoked aggressor deserves a special measure of approbation. Typically, this is the party with more power, not less, for the simple reason that it's a lot harder to abuse a lack of trust and authority.
- gaius 16y agoWhat I'm saying is that relying on crazy people to not go crazy is not a good data loss prevention strategy. Except companies that believe this are impossible to work for, there is so much process and policy to prevent anyone doing any harm that no-one can get any work done either. At the end of the day, you have to sort out all the trust issues before you give someone the root password, then you just have to trust them to get on with it.
- random42 16y agoBecause when the company has to man up, and tell the reason of the Fuck up to the stakeholders, the should not be pointing fingers to a person who was not even on their payroll. It was someone's job to avoid this from happening, or they do not have people covering such cases... either way, someone other (person or the company as a whole) than the fired employee messed up for the business/customers and they are liable/answerable. Because people fail... all the time. Good, Smart, Sincere people fail. (not in this particular case, though) This is just how things are. We all have heard numerous stories of rm -r / or drop database or something similar. Hell, the whole widely accept idea of "bugs" in software industry is based on the fact that people WILL do something wrong. Not because they are bad people, want to do bad things, but to err is human. So your process should not be designed around the idea of people doing the right things always. This is what I think he means (and I concur)
- deleted 16y ago[deleted]
- firemanx 16y agoI really don't get this philosophy. I've seen this a lot recently in various discussions on HN, and I'm honestly dumbfounded. How do you justify this line of thought, that negative action against someone, done deliberately, is not the perpetrator's fault? I mean this completely academically, so please don't take it as an insult (I'm genuinely curious) - but was this perspective something you were raised with or did this develop out of experience / general personal exploration of the world?
- palehose 16y agoA co-worker of mine once wiped out the entire CVS repository at the company we worked for because a symbolic link was placed in his home directory and he thought it was ok to 'rm -Rf' its contents. We were able to build a new repo based off of production and everyone who had changes not in production had to re-introduce those changes, but it was not the developer's fault for wiping out CVS. It was because the sysadmin had set the symbolic links for the primary repository. If you enable developers to do something bad, it isn't always their fault when they do it. Having intent to destroy something on purpose is a different matter though, and I'm not sure that there is a good argument for it.
- callmevlad 16y agoDid the sysadmin also forget to create a backup for the repo? I don't think it's a stretch to assume that the developers were somewhat negligent in working on a single repo that can disappear at any time.
- palehose 16y agoThere was no backup of the CVS repository. The production server saved us since we were able to treat it as the definitive backup. This was before Git and the concept that everyone had their own copy of the entire repository. There is a sort of half intent by the developer. Yes, the developer had intended to delete the repository that was in his home directory, but he had no idea that it was actually a link to the master repository.
- 16y ago
- swift 16y agoI wouldn't say that it's not the employee's fault, but if you are suggesting that it's other peoples' fault as well, I'd definitely agree. In an incident like this a lot of people have to have been very negligent.
- Confusion 16y agoYou judge without knowing enough details to do so. For all we know, the contract with the data center may have included regular backups, including off-site backups. In such a case, as a company, you've done everything you could and should have and the data center is entirely to blame, if the employee managed to destroy the backups as well.
- SeoxyS 16y agoSomewhat different situation, but after having quit one a Silicon Valley startup after its acquisition, I still have access to a lot of its network. I'm sure it's a common occurrence. Shutting people completely out of a network is really hard, where there are so many different accounts to close. (Unix access on all boxes, mysql, cms, email, bug-tracking, apple dev accts, irc, support forums, etc. etc.) Especially doing so without deleting data (which you might need in the future, to check on something that the employee worked on, or an old email containing documentation, etc…) Rather, I think the issue is the lack of a good backup strategy. Off-site, automated backups should have been in place. I've seen someone `rm -rf /var/mysql` and, after a heart attack, recover it. Your system should be rm-proof!
- davidandgoliath 16y agoYou may want to find a way of notifying them without alerting them it's you. Otherwise if something goes awry down the road and one of your accounts is involved (say an old username that gets brute-forced), you don't get something pinned on you.
- pyre 16y agoA couple of those things are incredibly easy: 1. Don't use per-machine unix accounts. Use something like NIS or LDAP. Now you can easily disable someone's account on all boxes. THe bonus is that their home directories still stick around (though you can't use aliases like ~username to access them). 2. Use an email system where you can disable a user without deleting all of their emails.
- ajb 16y ago"it's not his fault that this happened" This is wrong; and is a class of error which I see frequently, especially in political comment; both from the left and the right. The underlying incorrect assumption is that responsibility is mutually exclusive. In many cases, responsibilities are divided cleanly between people, but this is an organisational convenience, not a moral law. There are cases where it is inescapable that more than one person has the same responsibility. Consider an army sargeant: he may assign a task to a soldier, but still has the responsibility for seeing that it is done. This is true for all leadership positions. Much political discourse revolves around who has responsibility, and frequently one sees it argued that A is not responsible because B is, often in cases where this is incorrect. Eg, employer versus employee, individual versus government, etc etc. The error is made in both directions, depending on the political beliefs of the arguer: an individual is not responsibly because the organisation is, or the organisation is not responsible because the individual is.. A particularly dumb one, in my opinion, is the argument that government is not responsible for something, because individuals are. In my view, all the valid responsibilities of government are derived from responsibilities of the individual: the responsibility to defend the nation is derived from the individuals responsibility to protect him/herself and family, etc.
- astrofinch 16y agoResponsibility is one of those concepts that we humans made up and spend a lot of time uselessly arguing over. Other examples are whether an action was "justified", whether someone "deserves" something, who gets the credit for stuff, etc. Examples I'm less sure about: Who "owns" something, whether something is "clean" or "dirty", whether two things are the "same", whether something is "normal" or not. I collect these; if anyone has more to share I'd love to hear them.
- JonnieCache 16y agoThis is called Fundamental Attribution Error. As the name suggests it is a foundational part of human social psychology and it has been studied for years. Personally I'm convinced it is one of the cognitive flaws that are needed to stop us from going teeth-gnashingly insane after ten minutes exposure to reality. https://secure.wikimedia.org/wikipedia/en/wiki/Fundamental_attribution_error https://secure.wikimedia.org/wikipedia/en/wiki/Fundamental_a... See also: "She should have known what would happen to her, going out at night dressed like that." No.