8 ms·
Kids, this is story of How I Met... my VPS hacked
- chippy 13y agoAlways useful to see attack vectors. Nice well written blog post.
- tinco 13y agoThe attack vector is some random part of the application server stack (JBoss) demarshalling and running commands on strings received directly from web clients. A rather horrible practice to begin with. There's always so much that can go wrong when parsing tainted strings to native data structures, let alone to full objects using the builtin marshalling functions, they're not meant to be used on objects from untrusted sources at all!
- slig 13y agoNot long time ago Rails had an attack vector very similar to this, IIRC.
- meowface 13y agoYep, same thing but with YAML deserialization. Deserialization vulnerabilities are common for Java, Python, Ruby, and PHP web apps, because deserializing an untrusted input is nearly akin to running eval() on an untrusted input.
- ambrop7 13y ago1. Define exactly what the deserialization output should be. 2. Implement it that exactly that way. Now it's simple. Does the definition in part (1) include execution of arbitrary commands?
- Robin_Message 13y agoA deserializer might be able to instantiate arbitrary classes, so any class with a constructor that could execute an arbitrary command makes the deserializer vulnerable. Of course, the correct answer is not to use the deserializer that can instantiate arbitrary classes when you have a well-defined list of classes that can be instantiated.
- nogridbag 13y agoNot loading for me. Here's Google's cached version: http://webcache.googleusercontent.com/search?q=cache:ZbXfxz_SkcYJ:www.corrspt.com/blog/2014/01/18/tale-vps-hacked/+&cd=1&hl=en&ct=clnk&gl=us http://webcache.googleusercontent.com/search?q=cache:ZbXfxz_...
- joshbaptiste 13y agoAnd kids, here's another story of how my VPS got DDoS'd by Hackernews/Reddit.
- sp332 13y agoThis is more permanent http://web.archive.org/web/20140120182655/http://www.corrspt.com/blog/2014/01/18/tale-vps-hacked/ http://web.archive.org/web/20140120182655/http://www.corrspt...
- zedpm 13y agoI'd think twice before declaring victory; I don't see any reason to believe that the server is now "clean". If this had happened to me, I'd spin up a new VPS, configure it appropriately, then install my app and migrate any needed data. What I wouldn't do was continue running on an instance that had been exploited and assume I'd successfully cleaned it up.
- laumars 13y agoAgreed, particularly when he's already admitted this was the second time. However I still praise him for not only taking time to investigate how the breach happened, but also to blog about it so we can all benefit from the experience.
- zedpm 13y agoOh yeah, I agree. I guess I failed to address that in my original comment, but the analysis is actually cool and it's a valuable submission.
- jared314 13y agoThe author answered that criticism in the comments to the Reddit post. > And you're right, I don't know and it's probable that they left something else. I'll probably need to nuke the server and rebuild, but I wanted to have a better idea of how I could improve my situation in the future. And I believe that exposing what I did and how I did, the community would be of a assistance. http://www.reddit.com/r/programming/comments/1vo7zv/kids_this_is_story_of_how_i_met_my_vps_hacked/ceu82v2 http://www.reddit.com/r/programming/comments/1vo7zv/kids_thi...
- moystard 13y agoI praise his curiosity for actually digging the problem and trying to understand how he got his server hacked. It makes a much better post than: I nuked my installation, and installed everything from scratched crossing my fingers that this will never happen again in the future.
- pwnna 13y agoUgh. I totally thought that someone met his/her spouse after some his/her VPS was hacked. Now I'm slightly disappointed. Am I the only one? :\
- ItendToDisagree 13y agoI was really hoping for this as well. There was some good cyberpunky storyline to be had I thought.
- shill 13y agoIt's a very odd title.
- systematical 13y agoI'd go one step further, its a crappy title.
- ableal 13y agoI'd guess something may have got lost in (mental) translation: in Portuguese, the same verb may be used to mean meeting people and finding things.
- Buge 13y agoThe title reminded me a little of this. https://www.youtube.com/watch?v=YDW7kobM6Ik https://www.youtube.com/watch?v=YDW7kobM6Ik
- agumonkey 13y agoMe too, and everybody should watch that talk, the hackery build up is quite something.
- joshbaptiste 13y agoLong story short, he was running an instance of JBoss that had a vulnerability that allows an attacker to execute commands as the Jboss user and ended up Bitcoin mining on his VPS. Isn't Jboss one of these bloated “enterprisey” JVM container frameworks? I would think something like Play/vert.x would suit the likes of a small VPS better.
- drdaeman 13y agoChoice of framework is not that important. The real problem was (and, judging from the article it's still not fixed) that his server has write access to directories it can execute code from. I mean, this is a classic example of badly configured filesystem permissions.
- deckiedan 13y agoThis is one of the things that scares me about deploying PHP apps. I'm being asked to take over hosting a couple of Joomla & Wordpress sites at work, and find it terrifying that they both ask for permission to install php scripts. I much prefer having a clear separation, but it seems that that isn't really an option.
- dclara 13y agoYes, PHP has weaker security but fast implementation because it allows direct database access from the front pages. There are a lot of discuss about it. I always disable PHP on the HTTPD to avoid potential issues.
- jsight 13y agoWell, his choice of runtime was well over 5 years old (JBoss 4), and apparently not being kept up to date. I bet there are plenty of frameworks with security holes when not updated for five years. It also sounds like he may have been exposing the entire app server via his frontend proxy. Both mistakes are amazingly easy to fall into, unfortunately.
- stonemetal 13y ago
- drdaeman 13y agoAnd the real issue was (is?) the fact that web application process had write access to a location it was able to execute code from. It is well-known for decades (at least since PHP got popular, which was in '90s!) that such locations should only be writeable by the user(s) who maintain the code and not anyone else. Sometimes, such setup can't be done on dirt-cheap FTP-only shared hosting services (shrugs), but certainly not on VPSes. Ignorance is bliss.
- jlgaddis 13y agoYep, this can be a lesson to developers who read this. If your web application must write to files, keep them outside of the DocumentRoot! edit: I know a lot of HN'ers are fans of Ubuntu but, FWIW, SELinux (default on RHEL and derivatives) would have prevented this.
- drdaeman 13y agoOutside of any executable location, not only DocumentRoot. Latter is just an easier and more popular target. I mean, all writeable directories (if any) must not only be not runnable by some server (like Apache with mod_php), but also be on devices mounted with noexec, like this is the case with /tmp.
- jlgaddis 13y agoI'm not sure that'll help like you think it will. I can't verify it at the moment, but create a /tmp/phpinfo.php file with "<?php phpinfo(); ?>" in it and then run "php /tmp/phpinfo.php". Regardless of whether /tmp is mounted noexec I think you'll find that it executes (because the php binary is under /usr which almost certainly isn't mounted noexec).
- drdaeman 13y agoWhoops, seems that you're right, I was wrong about interpreters taking noexec into account - either I misremembered something or things changed, but the fact is they don't. Sorry. Anyway, it's also better to have noexec than not. :) At least, it'll help if there's no complete shell access but only ability to run some binary by name, without args, or modify environment variables when running some executable.
- xhrpost 13y agoI feel an often overlooked prevention for these semi-random attacks is a change in the SSH default port. I've posted this idea elsewhere and people seem convinced that this is pointless because an attacker can just port-scan your machine. While true, this generally only happens when you are being specifically targeted by an attacker. More random attacks like the one mentioned here are likely just people scanning all the IPv4 space looking for open 22 ports and then testing known exploits. Since I don't run a super-popular site, I'm more likely to be the victim of the second kind of attack. I used to have bad-logins hitting my box on a regular basis, after switching ports for SSH, the log-in attempts went to zero.
- jsight 13y agoThis was my first though as well, although the attack this time wasn't really SSH-based. It really does greatly reduce the frequency of attacks.
- moreentropy 13y agoJust disable password authentication, problem solved. Changing the SSH port does not add security against people who are determined to break into your system, but as you said, it helps to keep the noise down so you're more likely to notice suspicious activity early.
- sliverstorm 13y agoI stick with iptables IP whitelists. We're talking /31 or /32 whitelists. I really don't need SSH access 24/7, so it's fine having just a couple chairs permitted. I don't need to SSH from the airport. That's naturally not the only layer of security, but I figure it's a nicer option than non-default port.
- middleclick 13y agoThat's what I use. Since I access my VPS from just two places (home, work), I have only those two address allowed. Rest everything is dropped.
- cortesoft 13y ago
- agumonkey 13y agoWas the attacker gathering results through remote logrotate communications ? Reminds me of wordpress attacks, you quickly wish for FS diffs in order to identify any change in your code/data ...
- bjg 13y agoMiners report progress directly to the configured pool. It's pretty much a set it and forget it mining.
- jlgaddis 13y agoNo, there was no real connection to the logrotate application. The miner that was running simply replaced argv[0] with "logrotate", presumably to look benign if/when the administrator took a look at the list of running processes.
- spydum 13y agoTripwire can be used, you just need to add your webroot and update it. These tools aren't new, just not often utilized with the new kids on the block
- agumonkey 13y agoYou're right, I found about it afterwards. I wonder why this hosting company didn't provide it (or an equivalent) out of the box.
- belluchan 13y agoI pasted a gist of the backdoor script the hacker downloaded onto that server: https://gist.github.com/anonymous/8527149 https://gist.github.com/anonymous/8527149 Original URL is http://pdd-nos.info/.tmp/back.conn.txt http://pdd-nos.info/.tmp/back.conn.txt
- marquis 13y agoAnyone running php, move the location from /cgi-bin/php. We've seen a few takeovers with unpatched systems to run miners. Not everyone can take their server down and changing the location according to apache/nginx at least stops the bots from finding you immediately.
- this_user 13y agoThe problem with old JBoss versions is that their configurations were insecure by default with several potential attack vectors available. If you didn't know about this and just deployed your JBoss out of the box, chances were good you'd get "hacked". Newer versions (>= 7) fortunately solved that issue and are not that easy to take over.
- Zirro 13y agoI'm interested in setting up a Linux-based server at home, but one of the things that have kept me from doing it is security concerns. Are there any server distributions which are very strict in regards to which packages are included/enabled, and configured with as little remote access as possible by default?
- orthecreedence 13y agoYeah, Slackware =]. People tend to laugh when you use it ("LOL WHO USES SLACKWARE!?!??!111") but its packages are hand-picked to be stable and rock solid. On top of this, it's very easy to get a bare-bones install that doesn't enable a bunch of useless shit (and open ports) at startup. Beware, all package management is by hand for the most part and doesn't support dependencies. So if you install a package, you have to know what other packages it depends on. Also, be ready to compile some of your favorite services by hand since a lot of them aren't in the distributed packages (think things like NginX, HAProxy, etc). Things like Apache, MariaDB, Python, Ruby, PHP, are included though. It's one the oldest linux distros and doesn't really hold your hand that much. You'll be editing a lot of config files by hand. However once you get used to it you'll definitely feel a much stronger connection to your box than with a lot of distros that try to make things easier. /r/slackware is a good resource, there are a good amount of lurkers that jump at any chance to help someone out, and linuxquestions.org is a good forum as well. EDIT: forgot to mention: don't rule out FreeBSD/OpenBSD as well. They are known to be pretty solid for hosting as well.
- agwa 13y agoSlackware is OK if you want to play around with Linux on a non-public-facing system, but it's bad advice if you want to run a secure public-facing server. It's too difficult to do automatic package updates on Slackware. Since so little software is packaged, you end up having to install most software manually, which means you become responsible for monitoring that software for security advisories and upgrading it in a timely manner when there's an advisory. Here's my advice for a secure public-facing server: use Debian Stable, set up automatic upgrades every night, install as much as possible from the official Debian repository, and be sure to upgrade to the next Debian release before your current release loses security support. This way, the Debian Security Team is responsible for monitoring security advisories and rebuilding packages instead of you having to do it yourself. Source: I just finished transitioning to Debian after 10 years of Slackware use, in large part because I found it too difficult to keep my Slackware installations secure.
- jliptzin 13y agoBased on that pool address the attacker seems to be mining Batcoin, not Bitcoin, which is appears to be an scrypt-based altcoin, far more profitable to CPU mine than Bitcoin. (In case anyone was wondering why an attacker was bothering to mine Bitcoin with a CPU).
- phaed 13y agoCame to say this. Indeed this has nothing to do with Bitcoin.
- antsam 13y agoI hope he flattened that VM and reimaged...
- asdffdsajkl 13y ago> Things could have been worse If the attacker found a way to upgrade the privileges of the user running jboss (it’s a sudoer, but the password is really hard) NEVER GIVE THE USER YOUR APPLICATION SERVER RUNS UNDER SUDO PERMISSIONS!