8 ms·
A blackhat social engineered me by posing as an early adopter of my startup
- lincolnq 16y agoHuh? I don't see how this person is a "blackhat". Just because they are playing with your server and trying to find holes doesn't mean they are intending to maliciously exploit them. I also don't see how it is social engineering. What did the attacker convince you of?
- kranner 16y agoThey didn't mention it on the emails at all. It was the contrast between 'my son is named x' and 'i kill u' that scared me. What I should have mentioned in the blog post was that there were other messages they left as comments in the code they submitted. So at the moment you have to leave an email address to take an open test. (This will change soon to no address required). The guy consistently signed in with a fake email address, email to which was bouncing. I blocked that email address, which amounts to not accepting it to let someone sign in for an open test. This is just a soft warning, of course. So the guy appended a character to the address, and left me the message: "Why have you blocked test@example.com? Too bad codeboffin!" I just don't see why they couldn't have discussed it on email on which they were already in mid-conversation with me. I presume the social engineering was coming. I would probably have told the person anything he wanted to know about my security setup by then. I don't think I would have been as open to anyone else off the bat.
- mceachen 16y ago(cross-posted to the blog): If all the "attacker" did was a #include /dev/random, how is that not something someone in your own QA team would do? Heck, if this was my company, that would be one of the first things I'd do (albeit only on my laptop or my staging environment)... There may be other details that he hasn't shared, but if this is the "attack," it's pretty benign.
- kranner 16y agoThe #include /dev/random hosed the code evaluation task queue and everyone else's code evals started piling up behind it. This behaviour has been fixed now; every submission is tried just once and then the evaluator moves on. The attacker might not have known they could break my system in this way, but they threw in some system("rm -rf /")'s in there which is fairly malicious. I'd yell at my QA team if I had one...
- poet 16y agoIf I could give one piece of advice to every startup who is worried about server security, it would be this: read the NSA's two page pamphlet for hardening Red Hat Enterprise Linux [1] for a quick overview and then move on to the ~200 page complete guide [2]. The principles are generally applicable to most Linux systems. The NSA's guide is good enough for low value targets. Besides, these are your tax dollars at work, so you really should make use of them. :-p [1] http://www.nsa.gov/ia/_files/factsheets/rhel5-pamphlet-i731.pdf http://www.nsa.gov/ia/_files/factsheets/rhel5-pamphlet-i731.... [2] http://www.nsa.gov/ia/_files/os/redhat/rhel5-guide-i731.pdf http://www.nsa.gov/ia/_files/os/redhat/rhel5-guide-i731.pdf On the web application side, you're going to want OWASP. http://www.owasp.org/index.php/Main_Page http://www.owasp.org/index.php/Main_Page EDIT: Just thought of something.... There were several privilege escalation exploits for the kernel made public around this time frame [3]. I'm guessing that's the reason why his kernel was silently upgraded. :-p [3] http://www.ubuntu.com/usn/usn-988-1 http://www.ubuntu.com/usn/usn-988-1
- kranner 16y agoThanks for that! More reading for me.
- cosgroveb 16y agoWhen I started managing my own VPS, I found the Linode Library's Linux Security Basics to be a fantastic primer. http://library.linode.com/using-linux/security-basics/ http://library.linode.com/using-linux/security-basics/
- kranner 16y agoThanks, I checked that list. All clear!
- jseifer 16y agoSummary: The author of the article has a web app that allows you to administer programming tests to job applicants. The two languages currently supported are C and C++. As part of that process, (I'm guessing) the code needs to go through some sort of compilation process. The author of the article had left a default whitelist on the production box that whitelisted /dev/random as being readable. The attacker compiled a program (took a test I'm guessing) that included /dev/random. The author found out based on ip and sheer luck that the attacker was a user of the system that he had corresponded with previously. The reason I summarized this is because I wasn't totally clear on what codeboff.in was from the article. It makes sense how an attack like this could happen. The author seems to really know his stuff and the attack vector really was a small oversight. It's pretty unsettling to me, though, how persistent/insane this attacker seems.
- kranner 16y agoThank you, that's a perfect summary.
- RBerenguel 16y agoPerfect, indeed. And sure, the read was disturbing.
- ax0n 16y agoFrom the perspective of an information security researcher with plenty of time spent working on both sides of the issue, I can tell you that any attacker worth their salt will be persistent and use multiple attack vectors to get what they want. This holds true for bored teens, corporate spies and real penetration testers (not just dorks that get paid to run scanning tools on your network) alike. I've spent the last decade doing mostly IT Security defense work. I don't worry when I see someone port-scan my servers. Those kiddos are almost always benign. I worry when I see that our IDS starts to correlate many subtle attacks over a long period of time from similar places (such as the same school, ISP, etc)
- kranner 16y agoThat's interesting. I've never admin'ed a server before so I hadn't thought about this at all. It'll probably be useful to me to keep more verbose access logs and hash IP addresses to track what individual users (at least those not behind a group firewall) are up to. Thanks for the tip!
- deleted 16y ago[deleted]
- strebler 16y agoIt's never too early to educate your employees about social engineering. My startup is small but has been targeted a few times by people trying to social engineer their way into more information. An employee of one of our direct competitors even called one of my developers to try and extract information - good thing she caught on quick. We eventually tracked a number of attack vectors to either him or his company. They weren't the brightest attacks though (calling her from his work #, emailing her and other employees from email addresses which he'd used online, running port scanners and SQL Injection tests (etc) on our servers from company IP addresses). Other times people call pretending to be customers, but they don't pass the smell test. We've had a few people call with tons of questions about our tech, capabilities, existing customers, etc, but won't divulge much of their own information or can't properly answer some of the basic questions that our real customers would know. Early customers are kind of like a job interview, it has to feel right for both sides.
- kranner 16y ago> Early customers are kind of like a job interview, it has to feel right for both sides. So right. I consulted for a while as a programmer. It took a few times being burnt before I could distinguish between the good and the bad clients, but after that the difference was night and day.
- edparcell 16y agoInteresting. Could either of you expand on that? What qualities do you now look for in a client/customer?
- kranner 16y agoThe biggest thing I look for is to be clued in to the feasibility and difficulty of their commission. If someone wants a Facebook clone for $500 and/or finished in 3 weeks, I'd walk away immediately. I guess that's an exaggerated example but if a client is completely non-technical it's almost always too painful to work with them directly. I don't agree with 'never say no to a customer' in software development work.
- Aegean 16y agoWhat surprised me is how this one evil guy took the effort to hack his server. Some strange motive is in place. Usually they scan ip addresses and ports and attack servers randomly.
- adn37 16y agoThe former case happens when the attacker challenges himself to just do it. Motivation may increase with time. This is the attacker mindset. The latter is when any host would do the trick (relay, host platform, whatever).
- parfe 16y agoNo social engineering took place unless you gave up something you shouldn't have. Someone was friendly in email and was playing around with your code submission system seeing how far he could push it. rm -fr * is a nice joke on you for letting it slide and the #include /dev/random is just funny.
- phaedrus 16y agoI fell out of my chair laughing when I read this: "So of course, the blackhat compiled a C program with #include “/dev/random” in it, and gcc was hosed."
- eru 16y agoJust a technical question: Shouldn't you use a virtual machine (or similar) whenever you let anything user created run (or compile) on your machine?
- kranner 16y agoI do! It's a separate (and smaller) linode VPS from the one that houses the app servers.
- eru 16y agoWould it be feasible to have one (short-lived) VM per user?
- kranner 16y agoNot if they're running linux. It's an interesting idea though! Food for thought.
- eru 16y agoIf you do something like copy-on-write (for storage, and possibly for memory, too), it could be fairly light-weight. Disclosure: I am working on XenServer.
- kranner 16y agoThat's neat. What I'd worry about is boot-up time. I let people compile and test their program with an AJAX button in the middle of taking a test. It's already 3-4 seconds before they get the result back, more delay would hurt. With tasks queuing up as they already are, I might not be able to keep a fresh VM pre-warmed every time, either. edit: The other factor is of course the prospective cost of a brand new configuration. Unless I find I can't prevent the current config from being breached and ephemeral VMs will fix it.
- 16y ago