7 ms·
Former Hostgator employee arrested, charged with rooting 2,700 servers
- ghshephard 13y agoThis all seemed like a pretty run of the mill story about an insider violating company trust, and then getting caught - until the final sentence: "Among other things, a desktop monitoring system that took screenshots of employee workstations in one-minute increments helped Hostgator officials quickly zero in on Gisse." Not something I'd want on my personal system, but it's exactly the sort of thing that I think every NOC/Secure environment should have for post-mortem assessments.
- freehunter 13y agoIt's fairly expensive (processing and storage), but it's well worth it for secure environments. I've worked for companies who have this set up on their Windows Server environment (since they were administered through the remote GUI) and SSH logging for the Unix/Linux servers when running as root.
- jonknee 13y agoExpensive? 1 image a minute for a 40 hour work week is 2,400 images or 120k a year. We're talking screenshots so they aren't large files (you don't need a high bit rate), probably in the range of 500KB which would mean a whole year is less than 60GB. 3TB hard drives cost ~$130 these days and have room to store 50 employee years of screenshots. If money is that tight you can always compress the images and get double or more for your cash. As for processing, you need to find a new computer if you can notice a screenshot being taken.
- freehunter 13y agoIf you're talking bare minimum, yeah. The software solution this company had wasn't quite that barebones. It was a robust suite, and actually did take some planning on the desktop and server side to set aside resources for it. The software was part of a package that managed software installs as well.
- bigiain 13y ago"3TB hard drives cost ~$130 these days and have room to store … " No trying to call out jinknee specifically, but I often see this argument: "But storage only costs ~$50/TB!", and I read: "here's someone who's got no idea - who's never seen what 'enterprise' pays for proper secured/redundant/backed-up/auditable/managed storage." Does anyone _really_ think sending the IT department junior round to BestBuy to grab ~$400 worth of external USB drives would then let you say "Right, that's then next 5 years worth of storage and archive of high-security and potentially-lawsuit-relevant employee data sorted!"?
- deleted 13y ago[deleted]
- gav 13y agoA previous employeer used Spector 360[1] on the majority of workstations. It would monitor everything including taking a screengrab every 5 seconds that you could then watch later. They'd sit down employees and playback fast-forwarded video showing how much time was wasted on Facebook, personal email, shopping, etc. It's horribly invasive but it meant everyone was too scared to use work computers for personal things. [1] http://www.spector360.com/ http://www.spector360.com/
- nephesh 13y agoI'm sure productivity skyrocketed /s
- venomsnake 13y agoIt would be fun to see the people's productivity on that system. And who of the tech people will try to break it as a sport.
- hfsktr 13y agoMy old job used spector. I'm pretty sure everyone knew it but people would still be on FB playing games when you walked past. I believe IT was the last to get it and for about a week after they installed it on our machines it didn't work because Microsoft Essentials disabled it as malware. I don't think they ever looked at it unless they wanted to fire someone and didn't want to pay unemployment and needed proof that they weren't doing their job.
- stoic 13y agoThis is pretty much why they had this at HG
- tekacs 13y agoGiven the way that hovers over employees, I think there's a fair case in there to unofficially rename the product 'Spectre'. -.-' shudder
- davidandgoliath 13y agoNot nearly as exciting as the microphones they had installed all around the building to monitor discussion amongst teammates. :)
- homosaur 13y agoI don't think this is appropriate for general office use simply because you might leach private data from employees (bank accounts, retirement accounts, medical records, whatever), but in secure, restricted environments I think it's totally appropriate and probably needed.
- pc86 13y agoThere is an argument to made that you shouldn't be looking at your personal bank accounts or medical records on a work computer, particularly if you work at a company that cares so little about their employees that they'd implement a system like this.
- ghshephard 13y agoIt's not so much that the company cares so little about their employees, but that they have so many responsibilities for the data and systems they are entrusted with. Seriously, though - what everyone I know does in this situation does is just bring their own laptop into work for personal stuff, and treats the work console for precisely that - work activity.
- pc86 13y agoIs this common? I have a few coworkers that bring in personal laptops and have them sitting right next to the work laptop. It absolutely blew my mind the first time I saw it. It would have been unheard of at my previous employers and probably gotten you called into an office.
- lawnchair_larry 13y agoI'll never work for any employer who does that, so I'd have trouble calling it a reasonable thing to do.
- davorak 13y agoWhat alternative do you think is reasonable to help track down problems like this after the fact?
- andrewljohnson 13y agoAny powerful IT guy should be monitored and have his power checked. IT personnel have almost unmatched power in an organization to cause damage without detection. Anything with lots of confidential information, or anything financial, and you are going to want to monitor all the people with access constantly. You may not want to snoop real-time, but you are going to want to be able to find and fix breaches after the fact, and do root-cause analysis. It's not a matter of trust in the IT people, it's a matter of people go crazy sometimes, and people make bad hiring decisions sometimes.
- ghshephard 13y agoThe areas where it would be important are on security/NOC systems, where there is almost unlimited power given the proper credentials. The Runbooks that NOC teams have, quite often have them connecting to a lot of systems with greatly heightened privileges - It's not unusual for a NOC employee to have expansive sudo privileges on many of the unix hosts they manage. They are also often on privileged VLANs, with direct IP routing to a lot of hosts that normally wouldn't be reachable. Most of our NOC guys have their own personal laptops, and they can hop onto the (unprivileged) wireless system and do their own thing when they aren't working an incident. I'd have no problem having my screen captured once a minute when I was working in that type of environment.
- incision 13y ago>Not something I'd want on my personal system, but it's exactly the sort of thing that I think every NOC/Secure environment should have for post-mortem assessments. Tools of that sort [1][2] are pretty standard in call center environments. I'm not a fan. First, the software tends to be incredibly expensive. Second, in my experience, it's primarily used by managers looking for reasons to bludgeon their $30K, entry-level call takers over trivial infractions. 1: http://www.nice.com/contact-center-interaction-recording http://www.nice.com/contact-center-interaction-recording 2: http://www.callcopy.com/products/screen-capture http://www.callcopy.com/products/screen-capture
- jamescun 13y ago> While his root access gave Gisse access to private data stored on a large number of customer websites, there's no evidence he used it, the Hostgator executive said. I think the article is quick to jump to the conclusion that he was attempting to be malicious with his actions however this could be a case of Hanlon's razor. His actions could easily be attributed to a less-than-aware sysadmin developing his own solution to get around often arduous security restrictions. Stupid, yes. Malicious, no.
- gvb 13y agoHe attempted to access the HostGator from outside computers: Hetzner Data Center in Nuremberg, Germany, and efnet.pe (Peru) are mentioned in the article. The Hetzner access was the day after he was dismissed. Malicious, yes. Stupid, absolutely.
- tracker1 13y agoIf he hasn't accessed any of these systems since he was terminated, he could state that it was for "emergency" access to remote systems upon other compromises. Since most of these systems are likely headless, then remote access is the only way to get in. A lot of remote exploits will nuke SSH, and other access tools, so having a "backdoor" is often a good idea. That said, it's still likely that this guy is just a douche with a bad attitude, and deserves everything he has coming. Big difference between this, and "stealing" a bunch of reports that were government funded, and open to any and all users on the school network they were accessed from.
- homosaur 13y agoI don't think this would work. Installing your own solution on customer facing hardware? Might get you out of prosecution, but I think you'd have a nearly impossible time explaining that.
- roel_v 13y ago%| You serious? A dude backdooring several system utilities, giving him access to 2700 customers' data and applications, and you're defending him with 'it's just all a mistake' nonsense?
- dotmanish 13y agoDid these 2700 servers play a role in any DDoS attacks as well? It would be quite a lucrative stance for the employee to sell access to these servers to one or more groups who could potentially make more use of them.
- davidandgoliath 13y agoDoubt it, based on the timeline. Doesn't sound like he accessed the systems at all beyond implementing the rootkit and/or patching things up.
- cperciva 13y ago2700 servers all on the same network makes for far less of a DDoS attack then 2700 similar servers on different networks - and it's far easier to detect and block too. They would be more valuable for bitcoin mining most likely.
- nwh 13y ago2700 servers probably wouldn't be worth much for that. It would be noticed fairly quickly, given that only CPU mining would be available, and how monitored servers usually are.
- ultimoo 13y agoWow, he actually patched 'netstat' and 'ps'. It must have been to hide certain processes and port numbers from showing up. I wonder how may one go about understanding if they are using a 'hacked' version of something as non-trivial and comprehensive as 'ps' and 'netstat'. I mean, if I'm not suspicious, I wouldn't give it a second though and trust the output of these commands.
- w1ntermute 13y agoReminds me of Ken Thompson's Reflections on Trusting Trust: http://cm.bell-labs.com/who/ken/trust.html http://cm.bell-labs.com/who/ken/trust.html
- saurik 13y agoWhile this isn't guaranteed (all tools, including the compiler, may be patched), you can use checks and balances: verify /proc doesn't contain phantom processes, compile your own copy of ps, try more-obscure tools like top. If by "understand" you just mean "notice"... well, you don't, until one day you accidentally stumble across one of the above and start digging. (Maybe, for example, you install some kind of server monitoring tool, and when you log in to the web portal it provides you see a process that you find very suspicious; when you use ps, it doesn't show.) In my case, I've noticed this kind of thing twice: once, when the tool was binary pacthed to death (and just crashed), and once when the "patch" was "replace binary entirely", and the replacement was older and did not support a command line argument I knew that it should.
- ultimoo 13y agoNice. So both times that you noticed this, was it malice on someone's part due to which the tools were patched? I have just never heard or encountered such a situation and am frankly paranoid about something like this happening to one of the tools I use.
- saurik 13y agoI've been pwned with 0-days in various email servers: sendmail over a decade ago, and exim4 more recently (still many years ago, though). The patched copy of ssh on one of my boxes was then distributing passwords to someone, and which then was used to gain access to another machine. What I'm always paranoid about is that I work in a community of security researchers that sit on and occasionally drop 0-days: I have very little trust that much software is actually remotely "secure". Meanwhile, the only reason I had noticed those other attacks is just how sloppy they were... a more targeted-to-me run by a more careful attacker would have maybe never been noticed. It has drastically changed the way I think about security, FWIW; as one example: I don't every store logs on a box being logged anymore. Instead, logs are immediately transported to another machine whose only purpose is to accept and store logs (and so is listening for incoming log packets, OpenSSH, and nothing else. The first thing anyone does is attempt to patch themselves out of logs (one attack I noticed because wtmp was mysteriously damaged).
- efnx 13y agoWould be way cooler if he rooted 2600 servers ;)
- Shadow_Death 13y agoI'm not surprised with that company actually, hostgator is a joke and the employees in Austin, Texas are too. I think the one thing that bothers me how did he get the SSH key? The fact that he had it tells me that someone higher up dropped the ball somewhere.
- aptwebapps 13y ago"Gisse didn't return a voicemail and e-mail seeking comment for this report. A Court docket shows he is scheduled to be arraigned next month and gives no indication he has entered a plea in the case. He's being held at the Harris County Jail on $20,000 bond, a spokeswoman at the district attorney's office said." Wait, wait, just because the guy's in jail is no reason not to return voicemail and emails!
- nnq 13y agoThis smells really funny. They could have buried this instead of going to court (with a 1+ year delay!) and committing PR seppuku by making this public and giving their clients a reason of distrust. Now, this is indeed the right thing to do, the guy shouldn't go unpunished and they should disclose their security breach, but if they are doing it for the "right" reasons, why is it 1 year later?
- mikeurbanski 13y agoSounds like he knows Linux...
- Shadow_Death 13y agoIt's funny you say that because I remember their add. I made a jab at the company and - 2 rep.. If they knew the company they would have +2 rep me haha
- d23 13y ago> "He did not access customer content," Pelanne told Ars. "We caught it well before he had any chance to do any of that." > Given the rapid discovery, the malware was on Hostgator systems for less than a month. Then yes, he did. If the malware was on there for more than a few days, I find it extremely unlikely that at least some data wasn't compromised.