26 ms·

Citibank IT guy deliberately wiped routers, shut down 90% of firm’s US networks

174 points by jackgavigan 10y ago

carlmcqueen 10y ago
I started on a help desk, from my experience the management knew which ones were smart enough to do real damage and triggered enough to actually do it. If I had any advice it would simply be to avoid letting a corporate environment allow you to forget you have real people as employees with feelings. "It's just business" doesn't translate to everyone.
- superswordfish 10y ago
  > triggered enough to actually do it. What does this mean? Do you mean "provoked"?
  - wccrawford 10y ago
    I think they mean that they have a lot of (or very strong) triggers, or things that will make them act irrationally.
    - basch 10y ago
      usually triggering leads to paralysis not retaliation, no?
      - daenney 10y ago
        A trigger is an event that precipitates other events, aka it sets things in motion. It could be paralysis, or it could be ejecting a bullet from a barrel or wiping a set of routers.
        deleted 10y ago
        [deleted]
        deleted 10y ago
        [deleted]
        jameskegel 10y ago
        The commenter above you is trying to make you aware that "triggered" is an internet SJW dog-whistle to attract attention to an event. His statement is forcing you to acknowledge the definition of the word "triggered" in its most literal sense.
        ZeroGravitas 10y ago
        Can you explain this more? It doesn't make any sense to me. Note, I am aware of the use of trigger as in "trigger warning" for potentially disturbing content. Even with that context I'm lost. Did you mean "literal" literally?
GrantSolar 10y ago
This is really off topic, but the inclusion of the man "icon" was bizarre.
- voltagex_ 10y ago
  It's an "administrator" apparently. http://www.iconarchive.com/show/windows-7-general-icons-by-iconshock/administrator-icon.html http://www.iconarchive.com/show/windows-7-general-icons-by-i...
893helios 10y ago
This is a total nightmare scenario for a lot of organizations, and is quiet doable for a lot of systems. Specifically on the router side (thanks Cisco) I haven't seen a lot of controls that would stop this. Network Engineers need full access to get things done in a timely fashion, and the limitations of control in TACACS. Recover from insider attacks of this nature are all based on how much to drill to recover from this style of attack.
- brianwawok 10y ago
  Someone has to have root at some level right? So why is the router attack any worse than a "Delete exchange server + all backups" command, or the "use ansible to remote format entire server farm in 1 click" command? You have to trust employees at some level. If people know that doing things like this will equal jailtime, I would assume that would stop most people. Now think if this employee lived in Russia and did this remote. What would the recourse be?
  - jon-wood 10y ago
    At the scale of companies like this one it should be physically impossible to bring down a critical server and all its backups. That's what offsite backups, preferably on write only media are for.
  - zeveb 10y ago
    > Someone has to have root at some level right? In Unix, yes, but there have been systems without a single ultimately-privileged user. One could imagine a system in which ultimate authority belongs to a 51% share of stockholders, whose keys delegate authority to the board of directors, who delegate authority to the CEO or CTO, who delegates authority on down the line. Each certifying party could revoke (or allow to expire) authority prior to firing a delegee. > If people know that doing things like this will equal jailtime, I would assume that would stop most people. What I want to know is how people grow to physical adulthood without realising that this is wrong. One simply doesn't destroy others' property.
    - 10y ago
typetypetype 10y ago
I would think the first and easiest thing to do to prevent situations like this is to make it so a single person cannot run certain commands. Sort of like the digital version of two keys on the opposite side of the room to launch a missile.
- zzzcpan 10y ago
  It's not the kind of problem, that can be solved with technology. There will always be people with access not loyal to the company and pissed at management.
  - kefka 10y ago
    I would disagree. Backups could be maintained on a regular schedule, including full router backups. Others in charge of this could be given create and append access to the backup systems, but not delete. Once you delineate split rights and controls, then you can mitigate the severity of these attacks. Yes, douchebag-router admin can still zero out the routers. But you can be in business in less than a day with those configs logged.
    - atmosx 10y ago
      And who's going to design the backup scheme, the management? One more click to erase all backup servers, I mean since we're here... Point in case, as parent commentator said, it's not the kind of problem that you solve with technology.
      - joenathan 10y ago
        Don't need to have the ability to delete, with create and append access one could also just corrupt all the backups.
      - typetypetype 10y ago
        Right - going down for a day can be damaging enough for some companies. Not allowing one person to click the "erase" button in the first place should be the focus. How hard could it be to implement a system where two or more people have to issue a command before it runs?
lagadu 10y ago
As someone who works as an identity and access governance consultant and developer, I endorse this article. I mean, there's very little you do about this particular scenario other than some small mitigations (at the end of the day you'll have to trust someone with something) but it's genuinely amazing how vulnerable the processes inside large, multi-million companies are to any internal attacker.
- atmosx 10y ago
  > I mean, there's very little you do about this particular scenario other than some small mitigations (at the end of the day you'll have to trust someone with something) but it's genuinely amazing how vulnerable the processes inside large, multi-million companies are to any internal attacker. Maybe the scenario is too rare for companies to care, so when it happens they just accept the cost of once in a while instead of setting up additional security policies which will turn procedures cumbersome.
  - coredog64 10y ago
    Maybe you're working at different firms than I am, but the typical reaction I see from management is that there is no problem that can't be solved with yet another layer of process.
- droopyEyelids 10y ago
  It's not actually amazing if you think about any aspect of human life and consider how it all hinges on continued cooperation, good will, and concern for your own future. What stops bad person in finance from making a huge wire transfer to a hidden account? What stops the CEO from plundering the company? What stops the average driver from taking out a bunch of pedestrians? What stops a doctor from purposefully poisoning one of her patients? What stops a parent from abusing their children? Only good will and care for the future. And any system in place to stop that stuff from ever happening would cost more than the benefit.
daenney 10y ago
What I find curious in this article is that they gloss over that this is also a people management problem. Sure you can put restrictive policies in place but apparently things went down at Citibank that caused so much ire for the person in question to take action like that. I like to think that even though people see sysadmins are grumpy rage-machines in a basement most are super-decent people caring deeply for the systems under their control. Perhaps I've got a too rosy view of my fellow engineers. I'd be very curious to hear what went on there on the people side of things. What led to this person feeling so angry that they would take an action like this. I'm also curious how it could even get so far. I have the benefit of being able to regularly talk to my manager so even if we did performance reviews in that way they wouldn't come as a surprise.
- reitoei 10y ago
  > What led to this person feeling so angry that they would take an action like this I'd be interested to hear it too, but in the grand scheme of things, it doesn't matter one iota.
  - consp 10y ago
    I disagree: If the employee is committed to his/her job and (ab)used by 'upper management', it is likely they reflect that on their job, up to the point of abuse. A healthy work environment is as needed as compartmentalization and other schemes to manage abuse. We do not know the extend and circumstances of the employee and his/her related story and the story of the company, so no judgement on that.
  - havetocharge 10y ago
    If it does not matter, why are you interested?
    - knodi123 10y ago
      who wouldn't be? humans love drama.
- deleted 10y ago
rm_-rf_slash 10y ago
It's a little unsettling to think about how the biggest threats to an organization can be from within. The inside hacker doesn't even have to be disgruntled: they could just download the wrong link or put in a flash stick from the parking lot and BAM! More hacking in a minute than any Hollywood effort could get you. The worst part about it is that to be completely on the ball with internal threats, you need to be constantly paranoid, assuming by default that any employee with enough access is a threat, or a threat waiting to happen. Such things do not add to team morale or cohesion, but in this case, a pound of prevention is still better than a metric ton of cure.
- mcbits 10y ago
  Regarding morale, there's always a "bad country" making headlines each week. A disgruntled employee and an employee coerced by Bad Country's operatives can do pretty much the same damage, so you might as well frame the risks and policies in terms of protecting employees from the latter.
sqldba 10y ago
I don't want to know about what he did unless there's some information on why he did it also. Was he pushed over the edge after years of mistreatment? Or was he mentally unstable and it was petty revenge? Most of us have the keys to the kingdom in one way or another and while I (and others) could never do this it doesn't mean I don't have sympathy for someone who does - for some justifiable reason. I can't imagine any that are justifiable... but I can ask. I found the other linked articles about, "The Malicious Insider", a bit fucked in the head. Unapproved hardware! Email misuse! Unapproved workaround! Unapproved software! Are you fucking kidding me? That's a pretty far cry from this guy.
- semi-extrinsic 10y ago
  From the SMS quoted in TFA, seems like he at least thought he and his colleagues were being continually mistreated by upper management: “They was firing me. I just beat them to it. Nothing personal, the upper management need to see what they guys on the floor is capable of doing when they keep getting mistreated. I took one for the team.” “Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.”
  - jameskegel 10y ago
    This may sound petty, but it is sincere nonetheless; I'm curious how someone with his command of the English language rose to such a position of power to be able to cause this much damage.
    - mfoy_ 10y ago
      I had an IT internship where my manager handed me a master key to the building and root logins for quite a few boxes. I also had the admin credentials to every user computer in the building. Literally the "keys to the kingdom". Sometimes the default level of trust is just really high...
      - tracker1 10y ago
atmosx 10y ago
What precautions can one take against their sysadmin? I can't think of many places that have stricter security policies than 3-letter agencies, still Snowden got away with a USB stick (man, can't wait to see the movie - I wonder if he really passed the USB stick out using the rubik's cube as in the movie trailer). The thing is that that you can't do much against someone who has root access for a living. There are other ways to make a point, especially when you're prepared to get fired. Damaging everyone else is selfish and stupid.
- hyperpape 10y ago
  Apparently the Rubix cube was Snowden's idea, but didn't actually happen: http://www.indiewire.com/2016/07/snowden-oliver-stone-joseph-gordon-levitt-comic-con-1201709654/ http://www.indiewire.com/2016/07/snowden-oliver-stone-joseph.... He did, however, hold a Rubix cube as a way for Greenwald and Poitras to identify him in an early meeting. http://www.cnet.com/news/edward-snowden-and-the-rubiks-cube-meet-up/# http://www.cnet.com/news/edward-snowden-and-the-rubiks-cube-...!
- vacri 10y ago
  Sysadmin is a position of trust, just like a CxO or a finance officer.
  - deleted 10y ago
    [deleted]
- lgieron 10y ago
  > The thing is that that you can't do much against someone who has root access for a living. In banks, significant decisions (like personally authorising multimillion dollar transfer) require two signatures (of two senior officers), so they can't be abused by just one person going rogue. It can be imagined that we apply the same approach for system administration - "pair-programming" work only, perhaps with proximity cards which detect when one of the admins is actually not at the desk.
  - mancerayder 10y ago
l3m0ndr0p 10y ago
I've heard, first hand that Citi is extremely toxic. People are mistreated, are abused (verbally) on a frequent basis. There's no accountability within the ranks and management doesn't care and encourages it I almost accepted an offer to work in their NY offices. When I was being interviewed, the Manger of the group was divulging his dislike for the person I was going to report to. It's a sick place & the IT guy in the article sounded like he was being abused. He shouldn't have done what he did, He should have documented the stuff he went through and maybe get a lawyer and sue for pain and suffering.
- st3v3r 10y ago
  One might say that these things happening are the market telling Citi to stop being a dick. Unfortunately it seems like they're not really taking the hint.
nstj 10y ago
> The truth is that the person hacking you may not be someone you’ve never met, wearing a hoody on the other side of the world. Take note - just in case, don't wear a 'hoody' as people may think you are a 'hacker'.
123456Seven 10y ago
I wonder if the fact that this was a bank had something to do with the short sentence. Back when I was a pr activist, it was known that the level of public anger - the public's trigger - on an issue would sway the decision of high-profile cases - that is, the public's perception would factor into the judge's decisionmaking. If crimes were committed inside certain "pathways", those crimes were easier to get away with under certain constraints. This man has sabotaged a bank using a computer and made pro-union statements. If he had made broadly anarchist statements I wonder how this sentence would have changed. I argue that these dynamics are relevant from a criminal psychology and a government-hacking standpoint. Being mostly derived from firsthand resources, Toffler, Bernays, et. al, along with some authors whose mention is straight-up dangerous, I am constantly seeking additional resources. Uh. Hack the planet. Edit: or don't. U du u.
- bryanlarsen 10y ago
  21 months in a federal prison is a short sentence?
  - DarkContinent 10y ago
    By comparison, Kevin Mitnick got 68 months.
    - jameskegel 10y ago
      Brian Salcedo got 9 years.
  - 123456Seven 10y ago
    Yes. The AT&T hacker 'weev' was sentenced to 41 months prior to the vacation of his conviction. As far as I know no systems were disrupted in his case. He expressed vaguely anti-authoritarian leanings without appealing to a stronger base. This is arguably what keeps shows like Mr. Robot on the air btw. We will tolerate hackers down certain pathways and within certain constraints. Even if they're "insane". Leave the pathway and God help you. Sometimes staying on the pathway is dangerous too, but my job is to motivate for great justice. Edit: Don't kill or torture anyone. This should be in your base operating code anyway; if it isn't, enforce it down to pith level.
sevenless 10y ago
Not totally different from, say, Edward Snowden. Down-trodden information technology workers becoming radicalized and committing acts of sabotage, like the factory floor workers also did a century ago. A society where IT workers have great power over corporate fortunes but little political representation is going to be unstable...
- tonysdg 10y ago
  I'd personally argue that there's quite a bit of a difference. Snowden, to my understanding, wasn't protesting a bad performance review and perceived neglect from upper management; he was protesting what he judged to be an illegal action on the part of his employer. That being said - I do agree with your sentiment that the more dependent society becomes on technology, the greater the risk posed by those who possess knowledge of how to operate and maintain that technology. This doesn't just apply to human beings too - electronic actors (e.g., automated systems or even AI) may someday if not already hold just as much power. The question in my mind then is how to "democratize" this power, so to speak.
ebbv 10y ago
This article is clickbait titled spam. There's a snippet of an interesting story there but not enough for a real story. The "analysis" at the end of the article is not really analysis at all. There's interesting details behind this; were they really firing him? Did Citibank make the horrible error of telling an IT worker that they are fired but not immediately revoking their privileges? Could management have handled his performance issues better, or were there red flags that he should have been let go sooner? All of that could be interesting but because this is just clickbait blog spam none of it is in the article.
agounaris 10y ago
You cannot control the human factor so easily but you can definitely keep you employees happy! I don't get why the article does not mention the quality of working environment. IT systems can be extremely stressful if there is not help from management, and most of us know what bad management means...
- computronus 10y ago
  I agree. There's a lot of focus on punishment, consequences, and all the lockdowns for frustrating insider threats, but not so much on the flip side: reaching out to people, valuing them and their needs and opinions, working together, defusing problems so that there are fewer threats. There are analogies to broader threats, say, to entire countries, and these days a lot of people are more interested in being antagonistic and hostile in dealing with the threats than working to understand them and their grievances. The two sides may never agree, but the effort makes all the difference.
- googletazer 10y ago
  Because its about the grunts. In 80% of the cases, happiness is for people at the top and the shareholders.
gypsy_boots 10y ago
> he has been sentenced to 21 months in a federal prison for transmitting a command that caused damage without authorisation to a protected computer I'm genuinely curious, is this the actual charge? What specific law did he break here?
- alt2319 10y ago
  US Code 1030(a)(5)(A) - http://www.law.cornell.edu/uscode/text/18/1030 http://www.law.cornell.edu/uscode/text/18/1030 Edit: He plead to I think a single count of the following 18:1030(a)(5)(A) and 18:1030(c)(4)(A)(i)(I) and (B) http://www.leagle.com/decision/In%20FDCO%2020160411834/U.S.%20v.%20BROWN http://www.leagle.com/decision/In%20FDCO%2020160411834/U.S.%...
- maxerickson 10y ago
  Yes. It's from the famous CFAA. https://www.law.cornell.edu/uscode/text/18/1030 https://www.law.cornell.edu/uscode/text/18/1030 It's a perfectly reasonable principle to write into law, a password or key or whatever doesn't give an employee the right to damage the systems of their employer.
  - merpnderp 10y ago
    What if it's and accident?
    - alt2319 10y ago
      Then there's no criminal intent (mens rea) and the law doesn't apply.
mc32 10y ago
This is reminiscent of the San Francisco network administrator who took it upon himself to "protect" the city from his supervisors and would only divulge his password to mayor Newsom.[1] IIRC it was also precipitated by perf reviews or discipline. Hope AWS has better systems in place to make something like this nearly impossible to carry out. [1]http://m.sfgate.com/bayarea/article/S-F-officials-locked-out-of-computer-network-3205200.php http://m.sfgate.com/bayarea/article/S-F-officials-locked-out...
ChemicalWarfare 10y ago
I worked for a large company that went through drastic downsizing at one point. Some ppl who were getting let go were "well positioned" to do some damage. The way the company handled this is there was no advance notice. Your manager and a couple of security dudes shows up at your desk with boxes and you're outta there in 5 min. There were things like managers getting punched, ppl screaming etc but no malicious hacking that I'm aware of. Somehow one dude (I'm assuming :)) managed to take a dump in the elevator on the way out :) They had EMS parked outside too. Wasn't pretty.
- seren 10y ago
  Sorry for the naive question, but in that kind of situation, how do you assure the continuity of the organization ? What if the people being let got were supporting customers, about to close a deal, and so on ? How do you make sure people are not leaving with a critical "know-how" ?
  - passivepinetree 10y ago
    I'd imagine that in the case of such a drastic downsizing, that sort of thing was the least of the company's worries.
  - MagnumOpus 10y ago
    People in the middle of a deal or with crucial know-how will be called to serve out their notice period for handover - which if the HR is switched-on is 1-3 months. Usually some extra severance money will be offered as a sweetener compared to running away immediately. However, potential to do damage usually supersedes potential to benefit the business in the notice period. A company with sane HR has preventative policies in place: - Never have single-person key-man risk in the first place; always have a backup/deputy designated well in advance because anyone can be hit by a bus. Helps with covering holidays too. - For highly sensitive jobs like sysops or people dealing with customer money, revoke access to systems first thing in the morning, then tell them, then immediately escort them out. - Don't have shared or master passwords anywhere, but have user-based ACL instead so access can be revoked.
  - 10y ago
riprowan 10y ago
Used to be there was "the guy" that knew the "secret formula." Like the guy that knew the Coke secret formula, or the guy that used to maintain Lay's potato chip fryers. Most everyone else was just a cog in a big machine with very little power individually. What this guy - and Snowden - show is that today's IT professionals have tremendous power that generally they never choose to exercise. If an IT shop in a major corporation decided to unionize and turn on its management, they would have the company totally by the balls. So many people in an IT shop are positioned to control so much, and management often treats its IT staff like disposable widgets. It would take two+ hands to count the number of major multinationals that I could have seriously damaged singlehandedly in a matter of hours if I had ever gone rogue back in the day. Few non-IT rank-and-file employees have such power. Of course the fact that I never did such a thing is one of the reasons I remain a well-paid consultant (trustworthiness and honesty is a consultant's #1 asset), but the fact is that many corporations are quite blind to the level of risk they undertake when they mistreat their IT staff.
- Retric 10y ago
  The critical difference is factory workers can shut everything down by not showing up. IT needs active sabotage which is a very different situation.
  - dragonwriter 10y ago
    > The critical difference is factory workers can shut everything down by not showing up. IT needs active sabotage which is a very different situation. If you have achieved a genuinely 100% ops-free setup, sure. Otherwise, an IT walkout will shut things down just as surely as a factory walkout.
    - Retric 10y ago
      You are assuming they can't get anyone in the doors. Strike breakers at an iron factory don't really work unless they get close to the old workforce. In IT 10% of the staff can often keep the bare bones systems operating.
hardwaresofton 10y ago
Yeah or don't be shitty to your employees, or don't hire managers that are so bad at giving negative performance reviews that the employee goes out for revenge. This is definitely a people management problem. The only way to rid yourself of the threat posed by other humans is to run a single-person company, and even then you might screw yourself over. You will never be able to completely neuter a threat that does any useful service for your company, and it's foolish to think you can. Any useful amount of delegated power or responsibility without the ability to misuse it is very rare, if not impossible.
- ceejayoz 10y ago
  > Yeah or don't be shitty to your employees, or don't hire managers that are so bad at giving negative performance reviews that the employee goes out for revenge. There are people unable to take even the nicest, most constructive of criticisms (or even perceive non-critical statements as such). Hard to say without knowing the people involved on a personal level.
  - pjmorris 10y ago
    Can you see that it's a people management problem to be careful about the roles you give the kind of people you describe?
    - ceejayoz 10y ago
      Some of these people have learned to hide it pretty well.
  - hardwaresofton 10y ago
    This is true absolutely true -- my post was a bit of a snap judgement of the management's skills, but it is indeed possible that the person was just as much to blame, though I think it's unlikely
throwaway11teen 10y ago
This happened when I was at Google, but it was an accident. On someone's last day they ran rm -rf / to wipe their desktop machine. I can't remember whether they typed it into the wrong terminal window or had mounted an external filesystem somewhere deep under their home directory and forgot about it, it was something like that. They wound up wiping some unfathomably large amount of data (I think most of a data center) before noticing what they'd done. There were backups, but it was still a huge hassle.
- vmp 10y ago
  I managed to do that one time but on a much smaller scale (home server). Forgot about my network mounts, rm -Rf, noticed that it was taking longer than expected to delete just a few dirs... "oh crap." :) Luckily I run zfs with zfs-auto-snap on my home server so everything was back the way it was after a quick 'zfs restore'.
- dba7dba 10y ago
  Oops. Wrong terminal. Always a fear of mine. I always try to pause for a second before I run a command like that.
  - tedunangst 10y ago
    Easy solution: start a new terminal.
  - darkarmani 10y ago
    You could change the color of your login prompt as a visual indicator.
- tracker1 10y ago
  I once left a "production" connection string in my local config while fixing a bug in my last few days on a job, this got checked into source control (yeah, configuration in source control is bad)... a couple weeks after I left, someone had used that connection string to connect to the database server, and truncated a table.... needless to say, there were some things that changed after that.
geggam 10y ago
The more automation we have the harder its going to be to figure out if the large scale outage was intentional or not.
cia48621793 10y ago
Did Citibank have backups so that they can recover partial works or do they have a workaround? I used to think all the banks would do that. Like UPS it is a very important insurance. Also the man who shuts down the bank will face serious consequences.
nickpsecurity 10y ago
Sounds like a management issue more than security. However, post-Snowden, I did some exploration of a strategy for countering sysadmin risk in organizations whose IT activities were pretty regular. https://www.schneier.com/blog/archives/2013/08/nsa_increasing.html#c1623226 https://www.schneier.com/blog/archives/2013/08/nsa_increasin...
deleted 10y ago
[deleted]
zeveb 10y ago
> “They was firing me. I just beat them to it. Nothing personal, the upper management need to see what they guys on the floor is capable of doing when they keep getting mistreated. I took one for the team.” > “Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.” Ah yes, the old double 'they are harming me by holding me accountable' and 'I am helping them by holding them accountable' defense. What I'd like to know is how mindsets like this are formed in life, and how to prevent them. One's employer doesn't owe one a job; one owes one's employer fair labour in return for one's wages. The world does not consist of shadowy forces plotting against one. Don't minimise one's own agency: be an active force for good, not a passive subject of whatever happens.
- kjs3 10y ago
  Narcissism?
- st3v3r 10y ago
  "One's employer doesn't owe one a job" That kind of thinking is exactly what breeds this kind of stuff, though.
- oxryly1 10y ago
  > One's employer doesn't owe one a job Labor laws disagree to a certain degree -- at least in that you can't fire someone for any reason you choose (see the EEOC poster in your breakroom/mailroom/kitchen for examples). Also employment contracts can stipulate various things that prevent a company from firing you, particularly in union jobs.
- tracker1 10y ago
  The daily beatings will continue until moral improves... if you don't like it, you may quit at any time.
- autotune 10y ago
  > and how to prevent them. I've found going by the motto "don't be a dick" tends to be a decent deterrent against doing something stupid like maliciously wiping company data and getting sued and/or going to jail for it. If it's your employer doing it to you, there is plenty enough opportunity in IT to find work at a place that doesn't have a hostile work environment.
ryanmarsh 10y ago
One of my clients, a well known fixture in American finance, has been hacked by employees or contractors several times over the past 10 years. These cases have been public and supposedly cost millions (it's so hard to know how accurate the assessed damages in these cases are). Because they now fear employees and contractors more than external threats it takes 30 tickets to different groups to set up a server. You can't chown a file in a directory you own without a ticket. You can use one of two old text editors. No new IDE's or modern text editors. You can't upgrade language runtimes without corp. approval. It's nuts. Ironically it leads to more attack vectors and it's ground their software dev to a halt.
- ElijahLynn 10y ago
  Sounds like that company is crippled by fear and is slowly dying.
  - ryanmarsh 10y ago
    It is. It should be nearly gone in 20 years, for other reasons as well.
- arcanus 10y ago
  I would also guess they have seen a flight of any talent to jobs at other companies.
  - ryanmarsh 10y ago
    Yes major brain drain.
- st3v3r 10y ago
  All because they decided that was easier than simply treating their employees like human beings.
  - pc86 10y ago
    > has been hacked by employees or contractors several times over the past 10 years It's simply an overreaction to risk. Just like insisting the company doesn't treat its employees like human beings is also an overreaction (and gross oversimplification of what is actually a complicated topic).
juandazapata 10y ago
His grammar and writing style are bad for a person with such level of access.
- deleted 10y ago
  [deleted]
Overtonwindow 10y ago
That is one pissed off IT guy.
rabboRubble 10y ago
I worked in Citi some years before, during, and after the 2008 crisis. Held a two positions, IT and a business aligned roles. I left IT because the treatment of staff was atrocious, and I could see the cost cutting hammer dropping on IT. In regards to the integrity of their employee evaluation process, I have an anecdote to share. We used a 360 evaluation process. My review set was my direct manager, plus 5 other people with whom I directly worked. That year, I received a 5 out of 5, the highest rating possible. During the official manager / employee review sign off meeting, my manager noticed that my 5 was no longer a 5, but a 3. He canceled the meeting to investigate what happened. Apparently, his manager with collusion with a senior HR manager, went into the employee review system and manually overwrote the ratings provided by my manager and colleagues. The reasoning provided was that there were too many high ratings within my employee pool and I was selected by "someone" to be downgraded. I can see going into a Citi employee review meeting, coming out, and wanting to nuke the entire site from orbit.
- overgryphon 10y ago
  Wow that is incredibly shady. I doubt that's even legal.
- pc86 10y ago
  Understanding the feeling and following through with it are two very different things.
  - wutangson1 10y ago
    underrated post.
- uremog 10y ago
  That's surprisingly common. Where I am, every department, no matter how big or small, has to be within x points of the average rating on a bell curve. If not, your whole dept gets scaled to it. Where my friend works, it's more like yours and a select few get the stick. But they have this feature where you get reviewed formally if you have 3 consecutive "bad ratings", so mysteriously, no one ever gets the stick if they got it last cycle. Just, for both of us, it's company-wide knowledge.
iamapatsy 10y ago
Throwaway. Obviously stupid on his part. He needed a patsy. He needed to get someone else to do this. Some kiss-ass golden boy whose only value to the company was unerring unthinking loyalty. I have seen these douchebags destroy so much infrastructure, even without a malicious shadow revenge plot.
rbc 10y ago
I'm kind of surprised that Citibank had such weak internal controls on their network configuration. On the other hand, internal controls are not free. Separation of duties and the two person rule are expensive in manpower and coordination of larger teams. It would appear that whoever accepted the risk of the network internal controls at Citibank, lost that gamble.
Theodores 10y ago
There are more constructive ways to handle workplace problems when one has root access to everything... If people from marketing, accounts, sales or wherever else are giving you problems then you can sometimes automate them out of existence. This is particularly so in ecommerce where the gig is with one of those regular small businesses that haven't quite realised that they need to be an internet company. What you can do is automate all the accounting, doing it right with decent database schemas and everything the server has to offer. The accounts can then be a live thing and not some dead Excel export from some 90's era accounts package. Then that guy in accounts who spends 48 hours a week to get to those reports is redundant. He does not bother you any more and the managers are pleased they can see their charts online whenever they want without someone having to prepare them specially. Meanwhile in customer service, let's automate all of those procedures and put the process in code. Put up some forms that work and make it so easy for everyone that the customer service team no longer need to be the size that they are. Then there is the non technical web team, the layer of cruft that collects with non-web companies after the 'webmaster' in the corner of the room is deemed to need 'help'. These people are like Chinese whisperers (no offence, just turn of phrase, sadly...) in that they liaise between those making content 90's style in places like the graphics/print department and then mangle those assets for the web. Again, put them out of a job, put procedures in place so the guys in graphics are aware of this web thing and create content first and foremost for that. Simple things like file naming procedures can be all that is required and the guy that collects 'assets' from one part of the company to then upload them is no longer needed, neither are those pointless meetings that were needed. Have a problem with the actual graphics department and their print media tendencies? Move the requirements on to responsive SVG instead of the legacy PSD/PDF exported to blurry jpeg they love. Automate all that processing of 'assets' with some imagemagick. Take all the design layout out of 'design' and into UX with everything 'aria' tagged. Then there is HR. Put in place the tools for them to manage recruitment in a modern online way and get rid of the need to have people copying and pasting stuff out of Word/Excel/Email. Do that right and you can erase the need for there to be an H.R. assistant running around doing this stuff. Distribution and warehouse stuff is also similar, why use those in-house legacy systems when you can go XML SOAP to the distribution center? Get rid of the heart of the existing company - the legacy IT systems - piece by piece and the people that go with it. If you dislike someone and you can automate them out of existence then that is a very good incentive to get on with that aspect of the overall project. Having done this, the company is even more reliant on IT as there is now no Excel spreadsheet solution. Note that the above is reverse empire building, it is doing the right thing and making a company a web company rather than a 90's SME built on Outlook and Excel. Also it is a different exercising of power to going for the jugular, 'rm -fr' stuff. You also don't have to strictly automate people out of existence, you could make their job massively easier with less repetition and difficult sums done and have it so that no managers know that a tedious hard job is now coastable, even making it so that managers don't have the shortcuts that the staff do.
kusmi 10y ago
A godamn hero if there ever was one.

174 points by jackgavigan 10y ago

I started on a help desk, from my experience the management knew which ones were smart enough to do real damage and triggered enough to actually do it. If I had any advice it would simply be to avoid letting a corporate environment allow you to forget you have real people as employees with feelings. "It's just business" doesn't translate to everyone.

superswordfish 10y ago
> triggered enough to actually do it. What does this mean? Do you mean "provoked"?
- wccrawford 10y ago
  I think they mean that they have a lot of (or very strong) triggers, or things that will make them act irrationally.
  - basch 10y ago
    usually triggering leads to paralysis not retaliation, no?
    - daenney 10y ago
      A trigger is an event that precipitates other events, aka it sets things in motion. It could be paralysis, or it could be ejecting a bullet from a barrel or wiping a set of routers.
      - deleted 10y ago
        [deleted]
      - deleted 10y ago
        [deleted]
      - jameskegel 10y ago
        The commenter above you is trying to make you aware that "triggered" is an internet SJW dog-whistle to attract attention to an event. His statement is forcing you to acknowledge the definition of the word "triggered" in its most literal sense.
        ZeroGravitas 10y ago
        Can you explain this more? It doesn't make any sense to me. Note, I am aware of the use of trigger as in "trigger warning" for potentially disturbing content. Even with that context I'm lost. Did you mean "literal" literally?

GrantSolar 10y ago

This is really off topic, but the inclusion of the man "icon" was bizarre.

voltagex_ 10y ago
It's an "administrator" apparently. http://www.iconarchive.com/show/windows-7-general-icons-by-iconshock/administrator-icon.html http://www.iconarchive.com/show/windows-7-general-icons-by-i...

893helios 10y ago

This is a total nightmare scenario for a lot of organizations, and is quiet doable for a lot of systems. Specifically on the router side (thanks Cisco) I haven't seen a lot of controls that would stop this. Network Engineers need full access to get things done in a timely fashion, and the limitations of control in TACACS. Recover from insider attacks of this nature are all based on how much to drill to recover from this style of attack.

brianwawok 10y ago
Someone has to have root at some level right? So why is the router attack any worse than a "Delete exchange server + all backups" command, or the "use ansible to remote format entire server farm in 1 click" command? You have to trust employees at some level. If people know that doing things like this will equal jailtime, I would assume that would stop most people. Now think if this employee lived in Russia and did this remote. What would the recourse be?
- jon-wood 10y ago
  At the scale of companies like this one it should be physically impossible to bring down a critical server and all its backups. That's what offsite backups, preferably on write only media are for.
- zeveb 10y ago
  > Someone has to have root at some level right? In Unix, yes, but there have been systems without a single ultimately-privileged user. One could imagine a system in which ultimate authority belongs to a 51% share of stockholders, whose keys delegate authority to the board of directors, who delegate authority to the CEO or CTO, who delegates authority on down the line. Each certifying party could revoke (or allow to expire) authority prior to firing a delegee. > If people know that doing things like this will equal jailtime, I would assume that would stop most people. What I want to know is how people grow to physical adulthood without realising that this is wrong. One simply doesn't destroy others' property.
  - 10y ago

typetypetype 10y ago

I would think the first and easiest thing to do to prevent situations like this is to make it so a single person cannot run certain commands. Sort of like the digital version of two keys on the opposite side of the room to launch a missile.

zzzcpan 10y ago
It's not the kind of problem, that can be solved with technology. There will always be people with access not loyal to the company and pissed at management.
- kefka 10y ago
  I would disagree. Backups could be maintained on a regular schedule, including full router backups. Others in charge of this could be given create and append access to the backup systems, but not delete. Once you delineate split rights and controls, then you can mitigate the severity of these attacks. Yes, douchebag-router admin can still zero out the routers. But you can be in business in less than a day with those configs logged.
  - atmosx 10y ago
    And who's going to design the backup scheme, the management? One more click to erase all backup servers, I mean since we're here... Point in case, as parent commentator said, it's not the kind of problem that you solve with technology.
    - joenathan 10y ago
      Don't need to have the ability to delete, with create and append access one could also just corrupt all the backups.
    - typetypetype 10y ago
      Right - going down for a day can be damaging enough for some companies. Not allowing one person to click the "erase" button in the first place should be the focus. How hard could it be to implement a system where two or more people have to issue a command before it runs?

lagadu 10y ago

As someone who works as an identity and access governance consultant and developer, I endorse this article. I mean, there's very little you do about this particular scenario other than some small mitigations (at the end of the day you'll have to trust someone with something) but it's genuinely amazing how vulnerable the processes inside large, multi-million companies are to any internal attacker.

atmosx 10y ago
> I mean, there's very little you do about this particular scenario other than some small mitigations (at the end of the day you'll have to trust someone with something) but it's genuinely amazing how vulnerable the processes inside large, multi-million companies are to any internal attacker. Maybe the scenario is too rare for companies to care, so when it happens they just accept the cost of once in a while instead of setting up additional security policies which will turn procedures cumbersome.
- coredog64 10y ago
  Maybe you're working at different firms than I am, but the typical reaction I see from management is that there is no problem that can't be solved with yet another layer of process.
droopyEyelids 10y ago
It's not actually amazing if you think about any aspect of human life and consider how it all hinges on continued cooperation, good will, and concern for your own future. What stops bad person in finance from making a huge wire transfer to a hidden account? What stops the CEO from plundering the company? What stops the average driver from taking out a bunch of pedestrians? What stops a doctor from purposefully poisoning one of her patients? What stops a parent from abusing their children? Only good will and care for the future. And any system in place to stop that stuff from ever happening would cost more than the benefit.

daenney 10y ago

What I find curious in this article is that they gloss over that this is also a people management problem. Sure you can put restrictive policies in place but apparently things went down at Citibank that caused so much ire for the person in question to take action like that. I like to think that even though people see sysadmins are grumpy rage-machines in a basement most are super-decent people caring deeply for the systems under their control. Perhaps I've got a too rosy view of my fellow engineers. I'd be very curious to hear what went on there on the people side of things. What led to this person feeling so angry that they would take an action like this. I'm also curious how it could even get so far. I have the benefit of being able to regularly talk to my manager so even if we did performance reviews in that way they wouldn't come as a surprise.

reitoei 10y ago
> What led to this person feeling so angry that they would take an action like this I'd be interested to hear it too, but in the grand scheme of things, it doesn't matter one iota.
- consp 10y ago
  I disagree: If the employee is committed to his/her job and (ab)used by 'upper management', it is likely they reflect that on their job, up to the point of abuse. A healthy work environment is as needed as compartmentalization and other schemes to manage abuse. We do not know the extend and circumstances of the employee and his/her related story and the story of the company, so no judgement on that.
- havetocharge 10y ago
  If it does not matter, why are you interested?
  - knodi123 10y ago
    who wouldn't be? humans love drama.
deleted 10y ago

rm_-rf_slash 10y ago

It's a little unsettling to think about how the biggest threats to an organization can be from within. The inside hacker doesn't even have to be disgruntled: they could just download the wrong link or put in a flash stick from the parking lot and BAM! More hacking in a minute than any Hollywood effort could get you. The worst part about it is that to be completely on the ball with internal threats, you need to be constantly paranoid, assuming by default that any employee with enough access is a threat, or a threat waiting to happen. Such things do not add to team morale or cohesion, but in this case, a pound of prevention is still better than a metric ton of cure.

mcbits 10y ago
Regarding morale, there's always a "bad country" making headlines each week. A disgruntled employee and an employee coerced by Bad Country's operatives can do pretty much the same damage, so you might as well frame the risks and policies in terms of protecting employees from the latter.

sqldba 10y ago

I don't want to know about what he did unless there's some information on why he did it also. Was he pushed over the edge after years of mistreatment? Or was he mentally unstable and it was petty revenge? Most of us have the keys to the kingdom in one way or another and while I (and others) could never do this it doesn't mean I don't have sympathy for someone who does - for some justifiable reason. I can't imagine any that are justifiable... but I can ask. I found the other linked articles about, "The Malicious Insider", a bit fucked in the head. Unapproved hardware! Email misuse! Unapproved workaround! Unapproved software! Are you fucking kidding me? That's a pretty far cry from this guy.

semi-extrinsic 10y ago
From the SMS quoted in TFA, seems like he at least thought he and his colleagues were being continually mistreated by upper management: “They was firing me. I just beat them to it. Nothing personal, the upper management need to see what they guys on the floor is capable of doing when they keep getting mistreated. I took one for the team.” “Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.”
- jameskegel 10y ago
  This may sound petty, but it is sincere nonetheless; I'm curious how someone with his command of the English language rose to such a position of power to be able to cause this much damage.
  - mfoy_ 10y ago
    I had an IT internship where my manager handed me a master key to the building and root logins for quite a few boxes. I also had the admin credentials to every user computer in the building. Literally the "keys to the kingdom". Sometimes the default level of trust is just really high...
    - tracker1 10y ago

atmosx 10y ago

What precautions can one take against their sysadmin? I can't think of many places that have stricter security policies than 3-letter agencies, still Snowden got away with a USB stick (man, can't wait to see the movie - I wonder if he really passed the USB stick out using the rubik's cube as in the movie trailer). The thing is that that you can't do much against someone who has root access for a living. There are other ways to make a point, especially when you're prepared to get fired. Damaging everyone else is selfish and stupid.

hyperpape 10y ago
Apparently the Rubix cube was Snowden's idea, but didn't actually happen: http://www.indiewire.com/2016/07/snowden-oliver-stone-joseph-gordon-levitt-comic-con-1201709654/ http://www.indiewire.com/2016/07/snowden-oliver-stone-joseph.... He did, however, hold a Rubix cube as a way for Greenwald and Poitras to identify him in an early meeting. http://www.cnet.com/news/edward-snowden-and-the-rubiks-cube-meet-up/# http://www.cnet.com/news/edward-snowden-and-the-rubiks-cube-...!
vacri 10y ago
Sysadmin is a position of trust, just like a CxO or a finance officer.
- deleted 10y ago
  [deleted]
lgieron 10y ago
> The thing is that that you can't do much against someone who has root access for a living. In banks, significant decisions (like personally authorising multimillion dollar transfer) require two signatures (of two senior officers), so they can't be abused by just one person going rogue. It can be imagined that we apply the same approach for system administration - "pair-programming" work only, perhaps with proximity cards which detect when one of the admins is actually not at the desk.
- mancerayder 10y ago

l3m0ndr0p 10y ago

I've heard, first hand that Citi is extremely toxic. People are mistreated, are abused (verbally) on a frequent basis. There's no accountability within the ranks and management doesn't care and encourages it I almost accepted an offer to work in their NY offices. When I was being interviewed, the Manger of the group was divulging his dislike for the person I was going to report to. It's a sick place & the IT guy in the article sounded like he was being abused. He shouldn't have done what he did, He should have documented the stuff he went through and maybe get a lawyer and sue for pain and suffering.

st3v3r 10y ago
One might say that these things happening are the market telling Citi to stop being a dick. Unfortunately it seems like they're not really taking the hint.

nstj 10y ago

> The truth is that the person hacking you may not be someone you’ve never met, wearing a hoody on the other side of the world. Take note - just in case, don't wear a 'hoody' as people may think you are a 'hacker'.

123456Seven 10y ago

I wonder if the fact that this was a bank had something to do with the short sentence. Back when I was a pr activist, it was known that the level of public anger - the public's trigger - on an issue would sway the decision of high-profile cases - that is, the public's perception would factor into the judge's decisionmaking. If crimes were committed inside certain "pathways", those crimes were easier to get away with under certain constraints. This man has sabotaged a bank using a computer and made pro-union statements. If he had made broadly anarchist statements I wonder how this sentence would have changed. I argue that these dynamics are relevant from a criminal psychology and a government-hacking standpoint. Being mostly derived from firsthand resources, Toffler, Bernays, et. al, along with some authors whose mention is straight-up dangerous, I am constantly seeking additional resources. Uh. Hack the planet. Edit: or don't. U du u.

bryanlarsen 10y ago
21 months in a federal prison is a short sentence?
- DarkContinent 10y ago
  By comparison, Kevin Mitnick got 68 months.
  - jameskegel 10y ago
    Brian Salcedo got 9 years.
- 123456Seven 10y ago
  Yes. The AT&T hacker 'weev' was sentenced to 41 months prior to the vacation of his conviction. As far as I know no systems were disrupted in his case. He expressed vaguely anti-authoritarian leanings without appealing to a stronger base. This is arguably what keeps shows like Mr. Robot on the air btw. We will tolerate hackers down certain pathways and within certain constraints. Even if they're "insane". Leave the pathway and God help you. Sometimes staying on the pathway is dangerous too, but my job is to motivate for great justice. Edit: Don't kill or torture anyone. This should be in your base operating code anyway; if it isn't, enforce it down to pith level.

sevenless 10y ago

Not totally different from, say, Edward Snowden. Down-trodden information technology workers becoming radicalized and committing acts of sabotage, like the factory floor workers also did a century ago. A society where IT workers have great power over corporate fortunes but little political representation is going to be unstable...

tonysdg 10y ago
I'd personally argue that there's quite a bit of a difference. Snowden, to my understanding, wasn't protesting a bad performance review and perceived neglect from upper management; he was protesting what he judged to be an illegal action on the part of his employer. That being said - I do agree with your sentiment that the more dependent society becomes on technology, the greater the risk posed by those who possess knowledge of how to operate and maintain that technology. This doesn't just apply to human beings too - electronic actors (e.g., automated systems or even AI) may someday if not already hold just as much power. The question in my mind then is how to "democratize" this power, so to speak.

ebbv 10y ago

This article is clickbait titled spam. There's a snippet of an interesting story there but not enough for a real story. The "analysis" at the end of the article is not really analysis at all. There's interesting details behind this; were they really firing him? Did Citibank make the horrible error of telling an IT worker that they are fired but not immediately revoking their privileges? Could management have handled his performance issues better, or were there red flags that he should have been let go sooner? All of that could be interesting but because this is just clickbait blog spam none of it is in the article.

agounaris 10y ago

You cannot control the human factor so easily but you can definitely keep you employees happy! I don't get why the article does not mention the quality of working environment. IT systems can be extremely stressful if there is not help from management, and most of us know what bad management means...

computronus 10y ago
I agree. There's a lot of focus on punishment, consequences, and all the lockdowns for frustrating insider threats, but not so much on the flip side: reaching out to people, valuing them and their needs and opinions, working together, defusing problems so that there are fewer threats. There are analogies to broader threats, say, to entire countries, and these days a lot of people are more interested in being antagonistic and hostile in dealing with the threats than working to understand them and their grievances. The two sides may never agree, but the effort makes all the difference.
googletazer 10y ago
Because its about the grunts. In 80% of the cases, happiness is for people at the top and the shareholders.

gypsy_boots 10y ago

> he has been sentenced to 21 months in a federal prison for transmitting a command that caused damage without authorisation to a protected computer I'm genuinely curious, is this the actual charge? What specific law did he break here?

alt2319 10y ago
US Code 1030(a)(5)(A) - http://www.law.cornell.edu/uscode/text/18/1030 http://www.law.cornell.edu/uscode/text/18/1030 Edit: He plead to I think a single count of the following 18:1030(a)(5)(A) and 18:1030(c)(4)(A)(i)(I) and (B) http://www.leagle.com/decision/In%20FDCO%2020160411834/U.S.%20v.%20BROWN http://www.leagle.com/decision/In%20FDCO%2020160411834/U.S.%...
maxerickson 10y ago
Yes. It's from the famous CFAA. https://www.law.cornell.edu/uscode/text/18/1030 https://www.law.cornell.edu/uscode/text/18/1030 It's a perfectly reasonable principle to write into law, a password or key or whatever doesn't give an employee the right to damage the systems of their employer.
- merpnderp 10y ago
  What if it's and accident?
  - alt2319 10y ago
    Then there's no criminal intent (mens rea) and the law doesn't apply.

mc32 10y ago

This is reminiscent of the San Francisco network administrator who took it upon himself to "protect" the city from his supervisors and would only divulge his password to mayor Newsom.[1] IIRC it was also precipitated by perf reviews or discipline. Hope AWS has better systems in place to make something like this nearly impossible to carry out. [1]http://m.sfgate.com/bayarea/article/S-F-officials-locked-out-of-computer-network-3205200.php http://m.sfgate.com/bayarea/article/S-F-officials-locked-out...

ChemicalWarfare 10y ago

I worked for a large company that went through drastic downsizing at one point. Some ppl who were getting let go were "well positioned" to do some damage. The way the company handled this is there was no advance notice. Your manager and a couple of security dudes shows up at your desk with boxes and you're outta there in 5 min. There were things like managers getting punched, ppl screaming etc but no malicious hacking that I'm aware of. Somehow one dude (I'm assuming :)) managed to take a dump in the elevator on the way out :) They had EMS parked outside too. Wasn't pretty.

seren 10y ago
Sorry for the naive question, but in that kind of situation, how do you assure the continuity of the organization ? What if the people being let got were supporting customers, about to close a deal, and so on ? How do you make sure people are not leaving with a critical "know-how" ?
- passivepinetree 10y ago
  I'd imagine that in the case of such a drastic downsizing, that sort of thing was the least of the company's worries.
- MagnumOpus 10y ago
  People in the middle of a deal or with crucial know-how will be called to serve out their notice period for handover - which if the HR is switched-on is 1-3 months. Usually some extra severance money will be offered as a sweetener compared to running away immediately. However, potential to do damage usually supersedes potential to benefit the business in the notice period. A company with sane HR has preventative policies in place: - Never have single-person key-man risk in the first place; always have a backup/deputy designated well in advance because anyone can be hit by a bus. Helps with covering holidays too. - For highly sensitive jobs like sysops or people dealing with customer money, revoke access to systems first thing in the morning, then tell them, then immediately escort them out. - Don't have shared or master passwords anywhere, but have user-based ACL instead so access can be revoked.
- 10y ago

riprowan 10y ago

Used to be there was "the guy" that knew the "secret formula." Like the guy that knew the Coke secret formula, or the guy that used to maintain Lay's potato chip fryers. Most everyone else was just a cog in a big machine with very little power individually. What this guy - and Snowden - show is that today's IT professionals have tremendous power that generally they never choose to exercise. If an IT shop in a major corporation decided to unionize and turn on its management, they would have the company totally by the balls. So many people in an IT shop are positioned to control so much, and management often treats its IT staff like disposable widgets. It would take two+ hands to count the number of major multinationals that I could have seriously damaged singlehandedly in a matter of hours if I had ever gone rogue back in the day. Few non-IT rank-and-file employees have such power. Of course the fact that I never did such a thing is one of the reasons I remain a well-paid consultant (trustworthiness and honesty is a consultant's #1 asset), but the fact is that many corporations are quite blind to the level of risk they undertake when they mistreat their IT staff.

Retric 10y ago
The critical difference is factory workers can shut everything down by not showing up. IT needs active sabotage which is a very different situation.
- dragonwriter 10y ago
  > The critical difference is factory workers can shut everything down by not showing up. IT needs active sabotage which is a very different situation. If you have achieved a genuinely 100% ops-free setup, sure. Otherwise, an IT walkout will shut things down just as surely as a factory walkout.
  - Retric 10y ago
    You are assuming they can't get anyone in the doors. Strike breakers at an iron factory don't really work unless they get close to the old workforce. In IT 10% of the staff can often keep the bare bones systems operating.

hardwaresofton 10y ago

Yeah or don't be shitty to your employees, or don't hire managers that are so bad at giving negative performance reviews that the employee goes out for revenge. This is definitely a people management problem. The only way to rid yourself of the threat posed by other humans is to run a single-person company, and even then you might screw yourself over. You will never be able to completely neuter a threat that does any useful service for your company, and it's foolish to think you can. Any useful amount of delegated power or responsibility without the ability to misuse it is very rare, if not impossible.

ceejayoz 10y ago
> Yeah or don't be shitty to your employees, or don't hire managers that are so bad at giving negative performance reviews that the employee goes out for revenge. There are people unable to take even the nicest, most constructive of criticisms (or even perceive non-critical statements as such). Hard to say without knowing the people involved on a personal level.
- pjmorris 10y ago
  Can you see that it's a people management problem to be careful about the roles you give the kind of people you describe?
  - ceejayoz 10y ago
    Some of these people have learned to hide it pretty well.
- hardwaresofton 10y ago
  This is true absolutely true -- my post was a bit of a snap judgement of the management's skills, but it is indeed possible that the person was just as much to blame, though I think it's unlikely

throwaway11teen 10y ago

This happened when I was at Google, but it was an accident. On someone's last day they ran rm -rf / to wipe their desktop machine. I can't remember whether they typed it into the wrong terminal window or had mounted an external filesystem somewhere deep under their home directory and forgot about it, it was something like that. They wound up wiping some unfathomably large amount of data (I think most of a data center) before noticing what they'd done. There were backups, but it was still a huge hassle.

vmp 10y ago
I managed to do that one time but on a much smaller scale (home server). Forgot about my network mounts, rm -Rf, noticed that it was taking longer than expected to delete just a few dirs... "oh crap." :) Luckily I run zfs with zfs-auto-snap on my home server so everything was back the way it was after a quick 'zfs restore'.
dba7dba 10y ago
Oops. Wrong terminal. Always a fear of mine. I always try to pause for a second before I run a command like that.
- tedunangst 10y ago
  Easy solution: start a new terminal.
- darkarmani 10y ago
  You could change the color of your login prompt as a visual indicator.
tracker1 10y ago
I once left a "production" connection string in my local config while fixing a bug in my last few days on a job, this got checked into source control (yeah, configuration in source control is bad)... a couple weeks after I left, someone had used that connection string to connect to the database server, and truncated a table.... needless to say, there were some things that changed after that.

geggam 10y ago

The more automation we have the harder its going to be to figure out if the large scale outage was intentional or not.

cia48621793 10y ago

Did Citibank have backups so that they can recover partial works or do they have a workaround? I used to think all the banks would do that. Like UPS it is a very important insurance. Also the man who shuts down the bank will face serious consequences.

nickpsecurity 10y ago

Sounds like a management issue more than security. However, post-Snowden, I did some exploration of a strategy for countering sysadmin risk in organizations whose IT activities were pretty regular. https://www.schneier.com/blog/archives/2013/08/nsa_increasing.html#c1623226 https://www.schneier.com/blog/archives/2013/08/nsa_increasin...

deleted 10y ago

[deleted]

zeveb 10y ago

> “They was firing me. I just beat them to it. Nothing personal, the upper management need to see what they guys on the floor is capable of doing when they keep getting mistreated. I took one for the team.” > “Sorry if I made my peers look bad, but sometimes it take something like what I did to wake the upper management up.” Ah yes, the old double 'they are harming me by holding me accountable' and 'I am helping them by holding them accountable' defense. What I'd like to know is how mindsets like this are formed in life, and how to prevent them. One's employer doesn't owe one a job; one owes one's employer fair labour in return for one's wages. The world does not consist of shadowy forces plotting against one. Don't minimise one's own agency: be an active force for good, not a passive subject of whatever happens.

kjs3 10y ago
Narcissism?
st3v3r 10y ago
"One's employer doesn't owe one a job" That kind of thinking is exactly what breeds this kind of stuff, though.
oxryly1 10y ago
> One's employer doesn't owe one a job Labor laws disagree to a certain degree -- at least in that you can't fire someone for any reason you choose (see the EEOC poster in your breakroom/mailroom/kitchen for examples). Also employment contracts can stipulate various things that prevent a company from firing you, particularly in union jobs.
tracker1 10y ago
The daily beatings will continue until moral improves... if you don't like it, you may quit at any time.
autotune 10y ago
> and how to prevent them. I've found going by the motto "don't be a dick" tends to be a decent deterrent against doing something stupid like maliciously wiping company data and getting sued and/or going to jail for it. If it's your employer doing it to you, there is plenty enough opportunity in IT to find work at a place that doesn't have a hostile work environment.

ryanmarsh 10y ago

One of my clients, a well known fixture in American finance, has been hacked by employees or contractors several times over the past 10 years. These cases have been public and supposedly cost millions (it's so hard to know how accurate the assessed damages in these cases are). Because they now fear employees and contractors more than external threats it takes 30 tickets to different groups to set up a server. You can't chown a file in a directory you own without a ticket. You can use one of two old text editors. No new IDE's or modern text editors. You can't upgrade language runtimes without corp. approval. It's nuts. Ironically it leads to more attack vectors and it's ground their software dev to a halt.

ElijahLynn 10y ago
Sounds like that company is crippled by fear and is slowly dying.
- ryanmarsh 10y ago
  It is. It should be nearly gone in 20 years, for other reasons as well.
arcanus 10y ago
I would also guess they have seen a flight of any talent to jobs at other companies.
- ryanmarsh 10y ago
  Yes major brain drain.
st3v3r 10y ago
All because they decided that was easier than simply treating their employees like human beings.
- pc86 10y ago
  > has been hacked by employees or contractors several times over the past 10 years It's simply an overreaction to risk. Just like insisting the company doesn't treat its employees like human beings is also an overreaction (and gross oversimplification of what is actually a complicated topic).

juandazapata 10y ago

His grammar and writing style are bad for a person with such level of access.

deleted 10y ago
[deleted]

Overtonwindow 10y ago

That is one pissed off IT guy.

rabboRubble 10y ago

I worked in Citi some years before, during, and after the 2008 crisis. Held a two positions, IT and a business aligned roles. I left IT because the treatment of staff was atrocious, and I could see the cost cutting hammer dropping on IT. In regards to the integrity of their employee evaluation process, I have an anecdote to share. We used a 360 evaluation process. My review set was my direct manager, plus 5 other people with whom I directly worked. That year, I received a 5 out of 5, the highest rating possible. During the official manager / employee review sign off meeting, my manager noticed that my 5 was no longer a 5, but a 3. He canceled the meeting to investigate what happened. Apparently, his manager with collusion with a senior HR manager, went into the employee review system and manually overwrote the ratings provided by my manager and colleagues. The reasoning provided was that there were too many high ratings within my employee pool and I was selected by "someone" to be downgraded. I can see going into a Citi employee review meeting, coming out, and wanting to nuke the entire site from orbit.

overgryphon 10y ago
Wow that is incredibly shady. I doubt that's even legal.
pc86 10y ago
Understanding the feeling and following through with it are two very different things.
- wutangson1 10y ago
  underrated post.
uremog 10y ago
That's surprisingly common. Where I am, every department, no matter how big or small, has to be within x points of the average rating on a bell curve. If not, your whole dept gets scaled to it. Where my friend works, it's more like yours and a select few get the stick. But they have this feature where you get reviewed formally if you have 3 consecutive "bad ratings", so mysteriously, no one ever gets the stick if they got it last cycle. Just, for both of us, it's company-wide knowledge.

iamapatsy 10y ago

Throwaway. Obviously stupid on his part. He needed a patsy. He needed to get someone else to do this. Some kiss-ass golden boy whose only value to the company was unerring unthinking loyalty. I have seen these douchebags destroy so much infrastructure, even without a malicious shadow revenge plot.

rbc 10y ago

I'm kind of surprised that Citibank had such weak internal controls on their network configuration. On the other hand, internal controls are not free. Separation of duties and the two person rule are expensive in manpower and coordination of larger teams. It would appear that whoever accepted the risk of the network internal controls at Citibank, lost that gamble.

Theodores 10y ago

There are more constructive ways to handle workplace problems when one has root access to everything... If people from marketing, accounts, sales or wherever else are giving you problems then you can sometimes automate them out of existence. This is particularly so in ecommerce where the gig is with one of those regular small businesses that haven't quite realised that they need to be an internet company. What you can do is automate all the accounting, doing it right with decent database schemas and everything the server has to offer. The accounts can then be a live thing and not some dead Excel export from some 90's era accounts package. Then that guy in accounts who spends 48 hours a week to get to those reports is redundant. He does not bother you any more and the managers are pleased they can see their charts online whenever they want without someone having to prepare them specially. Meanwhile in customer service, let's automate all of those procedures and put the process in code. Put up some forms that work and make it so easy for everyone that the customer service team no longer need to be the size that they are. Then there is the non technical web team, the layer of cruft that collects with non-web companies after the 'webmaster' in the corner of the room is deemed to need 'help'. These people are like Chinese whisperers (no offence, just turn of phrase, sadly...) in that they liaise between those making content 90's style in places like the graphics/print department and then mangle those assets for the web. Again, put them out of a job, put procedures in place so the guys in graphics are aware of this web thing and create content first and foremost for that. Simple things like file naming procedures can be all that is required and the guy that collects 'assets' from one part of the company to then upload them is no longer needed, neither are those pointless meetings that were needed. Have a problem with the actual graphics department and their print media tendencies? Move the requirements on to responsive SVG instead of the legacy PSD/PDF exported to blurry jpeg they love. Automate all that processing of 'assets' with some imagemagick. Take all the design layout out of 'design' and into UX with everything 'aria' tagged. Then there is HR. Put in place the tools for them to manage recruitment in a modern online way and get rid of the need to have people copying and pasting stuff out of Word/Excel/Email. Do that right and you can erase the need for there to be an H.R. assistant running around doing this stuff. Distribution and warehouse stuff is also similar, why use those in-house legacy systems when you can go XML SOAP to the distribution center? Get rid of the heart of the existing company - the legacy IT systems - piece by piece and the people that go with it. If you dislike someone and you can automate them out of existence then that is a very good incentive to get on with that aspect of the overall project. Having done this, the company is even more reliant on IT as there is now no Excel spreadsheet solution. Note that the above is reverse empire building, it is doing the right thing and making a company a web company rather than a 90's SME built on Outlook and Excel. Also it is a different exercising of power to going for the jugular, 'rm -fr' stuff. You also don't have to strictly automate people out of existence, you could make their job massively easier with less repetition and difficult sums done and have it so that no managers know that a tedious hard job is now coastable, even making it so that managers don't have the shortcuts that the staff do.

kusmi 10y ago

A godamn hero if there ever was one.