6 ms·

Hacked: Investigating an Intrusion on My Server

100 points by frantzmiccoli 10y ago

efoto 10y ago
In short: an old server running un-updated software was broken into not once but at least four times since the start of the year. The post describes some steps the author took to investigate and block the attack. It's an entertaining reading, but I'd strongly advise against trying this at home: it is not worth the risk. Reinstalling the system from scratch instead would have been much more prudent.
- voltagex_ 10y ago
  Yep. "Treat servers like cattle, not pets". I won't recommend Docker but something like Ansible can be used to make reproducible servers.
  - majewsky 10y ago
    Also, remember that while no one likes backups, everyone wants restore. ;)
  - redbeard0x0a 10y ago
    I think running a Docker container for Wordpress or Joomla installs. If setup correctly with the right folders setup as volumes (i.e. wp-content for WP) and the rest would be part of the docker image. If WP gets corrupted/infected, then starting a brand new image would remove most of the corrupted files.
fake-name 10y ago
Ahhh PHP, where `assert()` `eval()`s strings passed to it.
Hogg 10y ago
Does }__ appear in your logs? All versions of all branches of Joomla prior to I think 3.4.6 had a problem with serialization that allowed arbitrary PHP execution.
viraptor 10y ago
This article was absolutely painful to read. Let me try a different ending: > Conclusion: 1. Ensure the files that php is running are not writable by the same process. Different app -> different user. 2. Unless you're planning to send emails from the server, firewall output on those ports. If you do plan to, firewall everything apart from that server (you can setup alerts when DNS changes, it's not going to happen often for email hosts) 3. Disable most unnecessary functions and modules. Anything touching eval, str_rot13, exec, and many others should be killed right away. 4. Enable basedir. 5. If you can't handle regular system updates, don't run your own server. If you can't handle wordpress updates, host it with someone else. 6. Wherever you host, make sure your app can be easily redeployed. You can't rely on dates to see when anything changed. 7. If you can't handle system upgrades, don't think that docker or any software is going to solve any of your problems.
- wtracy 10y ago
  I've never worked with PHP before, so basedir meant nothing to me. A minute with Google turned this up: http://php.net/manual/en/ini.core.php#ini.open-basedir http://php.net/manual/en/ini.core.php#ini.open-basedir I have to say, for all the general brain-deadness we hear about from the PHP community, the core developers have made a lot of progress on cleaning up their mess. I don't know of anything equivalent to this functionality from Python or Ruby.
  - malux85 10y ago
    I think basedir like functionality is a good idea, but this is not the languages responsibility, this is an operational concern. There's too many ways around it if you're trying to do this in a language ... for example, limiting checks on the open() call would be relatively straight forward, but what about people who use syscalls directly from the language? Or what about shelling out to another process? Chroot jails get kind of close of this in a very broad/blunt way, but back to my original point: This is not the languages responsibility
    - viraptor 10y ago
rmdoss 10y ago
Fun read. If you ever get hacked, I recommend just destroying the server and starting from scratch and a clean backup. If you try to find all backdoors and left over rootkits, you will end up forgetting one and being re-compromised.
- deleted 10y ago
  [deleted]
- z3t4 10y ago
  You can learn a lot by investigating how the attacker got into your system.
  - startling 10y ago
    Sure, but you shouldn't continue using the system.
AReallyGoodName 10y ago
Question about this: It's not possible for me to track all the 0 days for every piece of software and library that my servers run. One of my long running servers could have been backdoor-ed by a 0 day 6 months ago and i probably wouldn't know it. The servers are kept updated but 0 days don't care about that by definition. What's the best practice here? Should we pre-emptively have our servers rebuild daily just in case a 0-day backdoored them?
- viraptor 10y ago
  > What's the best practice here? Defense in depth. Remove the unneeded code so it never executes. If it's needed, block features you don't need. For the rest go one level higher and block operations you don't need. (php.ini controls, file permissions, LSM, etc.) For operations that are allowed, ensure data/code separation to prevent persistence. When that fails, have a very restrictive firewall so that data extraction and attacking other systems is difficult. Have network restrictions again one level higher. (switches / cloud network) If everything fails, have detailed logs to know what failed and when. If even that fails, have backups. If that fails, have company insurance prepared. > Should we pre-emptively have our servers rebuild daily just in case a 0-day backdoored them? Depends on your threat model and available money. Are you trying to stop automated malware or state actors? In the first case a good configuration should be enough. In the second you may want to start thinking about how you're going to verify the firmware your harddrives are running. And there's a whole spectrum in between. I mean, to block majority of php blog/cms malware it's enough to make sure your code is owned by user A, php server runs as user B, and B cannot write to A's files. 0-day or not, lots of stupid automated malware stopped. > It's not possible for me to track all the 0 days for every piece of software and library that my servers run. That's why it's nice to run unmodified distro software. Because these guys can track it for you: http://www.ubuntu.com/usn/ http://www.ubuntu.com/usn/ (for example, any reasonable distro has similar announcements)
schwede 10y ago
You should consider running this malware detection script[0]. That script is designed to catch malware php scripts. I've used it before with lots of success. You will also want to check the access logs for all php scripts hit in the last few months. Your hacker's spam script is likely being activated/ran by a GET or a POST to that script. That's a pretty cheap way to screen for other compromised files. [0] - https://www.rfxn.com/projects/linux-malware-detect/ https://www.rfxn.com/projects/linux-malware-detect/
0xmohit 10y ago
One may find the following resources useful for measures on how to secure one's infrastructure: - Security Guide: How to Protect Your Infrastructure Against the Basic Attacker [0] - 7 Security Measures to Protect Your Servers [1] [0] http://blog.mailgun.com/security-guide-basic-infrastructure-security/ http://blog.mailgun.com/security-guide-basic-infrastructure-... [1] https://www.digitalocean.com/community/tutorials/7-security-measures-to-protect-your-servers https://www.digitalocean.com/community/tutorials/7-security-...
cm2187 10y ago
I did get one of these OVH spam warning emails too once. It was for a failover mail server. It turned out that the software I was using (smartermail) wasn't using TLS by default so the link between the failover and the primary server was unencrypted. Spammers often target failover over primary servers. And OVH was interpreting the failover passing on the spam back to the primary server as me sending spam.
chmike 10y ago
Here are my advice: 1. once compromized, start with a fresh copy. Use ansible to define automatic installation scripts so that the server can be fully reinstalled in a "snap". Use backups of the user data to restore. 2. Automate upgrades. It's a piece of cake with ansible. 3. Stay away from php. There are too many potential loopholes. I now start using go to implement my web servers. Especially if the web site is static. Check Hugo or tools like that. I implement dynamic servers with go and Iris that uses fasthttp.
- noinsight 10y ago
  > I now start using go to implement my web servers. Then again, you can write buggy/insecure code in any language. I don't think I've seen any articles about potential security pitfalls when writing web services (or code in general) in Go. I'm actually interested if there's anything surprising out there.
  - chmike 10y ago
    The difference is that we have a better control on what is being done when using a statically typed language. Of course it is always possible to create a vulnerability too.
Neil44 10y ago
Don't delete the outbound mail queue as they have a header in them which tells you which php script sent them. Also in the httpd logs look for POST 200's to scripts that shouldn't be getting them, and look for long query strings esp with base64 in them. Check those scripts and also scripts messed with by the same user-agent and ip's. Never just delete anything, always check the contents and write a script that can search and nuke it en masse. e.g. Find | xargs | sed etc. When I had issues earlier this year I had postfix outbound throttled right down, and a script to shut it down completely if the queue got big. So if I got the monitoring email about smtp going down I'd know I'd missed something ;) Ps, wp-cli, you need it in your life :)
hwh 10y ago
I wish it was easier to convert "standard" dynamic web pages into static HTML easier. There's one or two Wordpress sites I'd really like to get rid of updating over and over again (which is still less work than actually reading mostly meaningless Changelogs and making a decision if there are security issues that actually apply to me). Probably, my solution to this problem will be "wget -r", plus a bit of sanitizing.
- unkoman 10y ago
  There are plugins for this - Simply static, staticpress and more.
mboelen 10y ago
Thanks for sharing your intrusion story. Although it is always painful for the person to experience, it gives good insights on what happens in the wild. As the author of security tools rkhunter and Lynis, it is still a sign that basic (and advanced) system hardening remains needed for a long time. The other comments show that there are so many things you could do. But some are more relevant than others. That is something we want to test for in Lynis with upcoming updates. One focus area will be detecting Drupal/Joomla/WordPress installations. If anyone wants to help out the open source project and make the web a safer place, this is your chance to help: https://github.com/CISOfy/lynis https://github.com/CISOfy/lynis Thanks again for sharing!
waltwalther 10y ago
There is always something to be learned. Thank you for sharing.

100 points by frantzmiccoli 10y ago

efoto 10y ago

In short: an old server running un-updated software was broken into not once but at least four times since the start of the year. The post describes some steps the author took to investigate and block the attack. It's an entertaining reading, but I'd strongly advise against trying this at home: it is not worth the risk. Reinstalling the system from scratch instead would have been much more prudent.

voltagex_ 10y ago
Yep. "Treat servers like cattle, not pets". I won't recommend Docker but something like Ansible can be used to make reproducible servers.
- majewsky 10y ago
  Also, remember that while no one likes backups, everyone wants restore. ;)
- redbeard0x0a 10y ago
  I think running a Docker container for Wordpress or Joomla installs. If setup correctly with the right folders setup as volumes (i.e. wp-content for WP) and the rest would be part of the docker image. If WP gets corrupted/infected, then starting a brand new image would remove most of the corrupted files.

fake-name 10y ago

Ahhh PHP, where `assert()` `eval()`s strings passed to it.

Hogg 10y ago

Does }__ appear in your logs? All versions of all branches of Joomla prior to I think 3.4.6 had a problem with serialization that allowed arbitrary PHP execution.

viraptor 10y ago

This article was absolutely painful to read. Let me try a different ending: > Conclusion: 1. Ensure the files that php is running are not writable by the same process. Different app -> different user. 2. Unless you're planning to send emails from the server, firewall output on those ports. If you do plan to, firewall everything apart from that server (you can setup alerts when DNS changes, it's not going to happen often for email hosts) 3. Disable most unnecessary functions and modules. Anything touching eval, str_rot13, exec, and many others should be killed right away. 4. Enable basedir. 5. If you can't handle regular system updates, don't run your own server. If you can't handle wordpress updates, host it with someone else. 6. Wherever you host, make sure your app can be easily redeployed. You can't rely on dates to see when anything changed. 7. If you can't handle system upgrades, don't think that docker or any software is going to solve any of your problems.

wtracy 10y ago
I've never worked with PHP before, so basedir meant nothing to me. A minute with Google turned this up: http://php.net/manual/en/ini.core.php#ini.open-basedir http://php.net/manual/en/ini.core.php#ini.open-basedir I have to say, for all the general brain-deadness we hear about from the PHP community, the core developers have made a lot of progress on cleaning up their mess. I don't know of anything equivalent to this functionality from Python or Ruby.
- malux85 10y ago
  I think basedir like functionality is a good idea, but this is not the languages responsibility, this is an operational concern. There's too many ways around it if you're trying to do this in a language ... for example, limiting checks on the open() call would be relatively straight forward, but what about people who use syscalls directly from the language? Or what about shelling out to another process? Chroot jails get kind of close of this in a very broad/blunt way, but back to my original point: This is not the languages responsibility
  - viraptor 10y ago

rmdoss 10y ago

Fun read. If you ever get hacked, I recommend just destroying the server and starting from scratch and a clean backup. If you try to find all backdoors and left over rootkits, you will end up forgetting one and being re-compromised.

deleted 10y ago
[deleted]
z3t4 10y ago
You can learn a lot by investigating how the attacker got into your system.
- startling 10y ago
  Sure, but you shouldn't continue using the system.

AReallyGoodName 10y ago

Question about this: It's not possible for me to track all the 0 days for every piece of software and library that my servers run. One of my long running servers could have been backdoor-ed by a 0 day 6 months ago and i probably wouldn't know it. The servers are kept updated but 0 days don't care about that by definition. What's the best practice here? Should we pre-emptively have our servers rebuild daily just in case a 0-day backdoored them?

viraptor 10y ago
> What's the best practice here? Defense in depth. Remove the unneeded code so it never executes. If it's needed, block features you don't need. For the rest go one level higher and block operations you don't need. (php.ini controls, file permissions, LSM, etc.) For operations that are allowed, ensure data/code separation to prevent persistence. When that fails, have a very restrictive firewall so that data extraction and attacking other systems is difficult. Have network restrictions again one level higher. (switches / cloud network) If everything fails, have detailed logs to know what failed and when. If even that fails, have backups. If that fails, have company insurance prepared. > Should we pre-emptively have our servers rebuild daily just in case a 0-day backdoored them? Depends on your threat model and available money. Are you trying to stop automated malware or state actors? In the first case a good configuration should be enough. In the second you may want to start thinking about how you're going to verify the firmware your harddrives are running. And there's a whole spectrum in between. I mean, to block majority of php blog/cms malware it's enough to make sure your code is owned by user A, php server runs as user B, and B cannot write to A's files. 0-day or not, lots of stupid automated malware stopped. > It's not possible for me to track all the 0 days for every piece of software and library that my servers run. That's why it's nice to run unmodified distro software. Because these guys can track it for you: http://www.ubuntu.com/usn/ http://www.ubuntu.com/usn/ (for example, any reasonable distro has similar announcements)

schwede 10y ago

You should consider running this malware detection script[0]. That script is designed to catch malware php scripts. I've used it before with lots of success. You will also want to check the access logs for all php scripts hit in the last few months. Your hacker's spam script is likely being activated/ran by a GET or a POST to that script. That's a pretty cheap way to screen for other compromised files. [0] - https://www.rfxn.com/projects/linux-malware-detect/ https://www.rfxn.com/projects/linux-malware-detect/

0xmohit 10y ago

One may find the following resources useful for measures on how to secure one's infrastructure: - Security Guide: How to Protect Your Infrastructure Against the Basic Attacker [0] - 7 Security Measures to Protect Your Servers [1] [0] http://blog.mailgun.com/security-guide-basic-infrastructure-security/ http://blog.mailgun.com/security-guide-basic-infrastructure-... [1] https://www.digitalocean.com/community/tutorials/7-security-measures-to-protect-your-servers https://www.digitalocean.com/community/tutorials/7-security-...

cm2187 10y ago

I did get one of these OVH spam warning emails too once. It was for a failover mail server. It turned out that the software I was using (smartermail) wasn't using TLS by default so the link between the failover and the primary server was unencrypted. Spammers often target failover over primary servers. And OVH was interpreting the failover passing on the spam back to the primary server as me sending spam.

chmike 10y ago

Here are my advice: 1. once compromized, start with a fresh copy. Use ansible to define automatic installation scripts so that the server can be fully reinstalled in a "snap". Use backups of the user data to restore. 2. Automate upgrades. It's a piece of cake with ansible. 3. Stay away from php. There are too many potential loopholes. I now start using go to implement my web servers. Especially if the web site is static. Check Hugo or tools like that. I implement dynamic servers with go and Iris that uses fasthttp.

noinsight 10y ago
> I now start using go to implement my web servers. Then again, you can write buggy/insecure code in any language. I don't think I've seen any articles about potential security pitfalls when writing web services (or code in general) in Go. I'm actually interested if there's anything surprising out there.
- chmike 10y ago
  The difference is that we have a better control on what is being done when using a statically typed language. Of course it is always possible to create a vulnerability too.

Neil44 10y ago

Don't delete the outbound mail queue as they have a header in them which tells you which php script sent them. Also in the httpd logs look for POST 200's to scripts that shouldn't be getting them, and look for long query strings esp with base64 in them. Check those scripts and also scripts messed with by the same user-agent and ip's. Never just delete anything, always check the contents and write a script that can search and nuke it en masse. e.g. Find | xargs | sed etc. When I had issues earlier this year I had postfix outbound throttled right down, and a script to shut it down completely if the queue got big. So if I got the monitoring email about smtp going down I'd know I'd missed something ;) Ps, wp-cli, you need it in your life :)

hwh 10y ago

I wish it was easier to convert "standard" dynamic web pages into static HTML easier. There's one or two Wordpress sites I'd really like to get rid of updating over and over again (which is still less work than actually reading mostly meaningless Changelogs and making a decision if there are security issues that actually apply to me). Probably, my solution to this problem will be "wget -r", plus a bit of sanitizing.

unkoman 10y ago
There are plugins for this - Simply static, staticpress and more.

mboelen 10y ago

Thanks for sharing your intrusion story. Although it is always painful for the person to experience, it gives good insights on what happens in the wild. As the author of security tools rkhunter and Lynis, it is still a sign that basic (and advanced) system hardening remains needed for a long time. The other comments show that there are so many things you could do. But some are more relevant than others. That is something we want to test for in Lynis with upcoming updates. One focus area will be detecting Drupal/Joomla/WordPress installations. If anyone wants to help out the open source project and make the web a safer place, this is your chance to help: https://github.com/CISOfy/lynis https://github.com/CISOfy/lynis Thanks again for sharing!

waltwalther 10y ago

There is always something to be learned. Thank you for sharing.