3 ms·

What would your monitoring strategy of choice be to notice these things, for a small "hobby" server? I've apprenticed as a junior sysadmin in the past, and I c

What would your monitoring strategy of choice be to notice these things, for a small "hobby" server? I've apprenticed as a junior sysadmin in the past, and I can easily see how a professional operation would detect or prevent this by design. But a lot of those methods are more than a little over the top for a one-of, single-user, low-end VPS.

noinsight 12y ago
I agree that this is a problem and I've been researching this for some time now. Ideally you would want a HIDS system for small deployments (i.e. a single server) but there doesn't seem to exist too many of those and there's little information about them online (as far as I can find). OSSEC is one example but I'm not convinced it installs too cleanly. At least FreeBSD has a package for it however. Actual configuration would be a mystery. I think information about this sort of stuff (intrusion detection) seems to be hard to come by in general. At least stuff that is written by knowledgeable people. I haven't seen much about it from major vendors either - it seems to be a black box kind of technology. I'd love to have a "lightweight" simple daemon that does this. I guess ideally you would monitor logs for stuff, user and group modifications, cron additions etc.
- sliverstorm 12y ago
  I tried using AIDE for a while, but it was a mild PITA and I was never convinced an attacker with a brain couldn't just patch the database to hide his changes.
- detaro 12y ago
  tripwire, rkhunter, ... are relatively simple options. And then hope that your configuration prevents the attacker from getting root or if they do, then they/their automated scripts are not prepared to break them.
  - noinsight 12y ago
    RE: hashing files and whatnot: FreeBSD with a high enough securelevel allows you to set files append-only or immutable so that not even root can touch them. But then updating the index becomes problematic, you would need to create delta files (which themselves are also immutable) that contain the changed information and combine the data. Even then, I'm not convinced you could get that much valuable information from a full filesystem hash (or whatever) scan. The amount of information would probably be overwhelming. You would need to apply heuristics or filters. Checking filetypes with libmagic would work, mostly. You could only alert on new executables or whatever, etc..
- koide 12y ago
  OSSEC requires you to understand how it works. That's good, but means that you have to devote 10 or 15 hours reading docs and testing before being productive with it. I recommend it though
cpncrunch 12y ago
Simply logging into your server once a day and typing "top" is a good start. You can usually spot suspicious processes associated with these spamkits. Also, check /var/log/messages, and check root's email. Also if you run anything like phpbb or wordpress, make sure you keep it updated. I also like to disable Apache's suexec (as I once had a rootkit that exploited it, and I don't require it for my systems), limit inbound connections with iptables, and limit ssh access to certain ip addresses. Also, get rid of or disable access to any services you don't really need (like ftp, bind, etc). It's also very important to check all your php/perl scripts for file/sql injection vulnerabilities, as that is one of the most common methods of infiltration.
- dTal 12y ago
  I'd like to add that it's a good idea to routinely check /var/log/auth.log for successful SSH logins. If you find that onerous because of the volume of breakin attempts, that's a good sign you need something like fail2ban. Fail2ban isn't foolproof, incidentally. I once nearly caught a worm despite a limit of 6 login attempts per IP, because I had created an account with username and password both "test" that I had since forgotten about, well before I exposed that particular machine to the internet. Oops! An automated script managed to guess that in well under 6 attempts! Fortunately I was physically present at the time and noticed the abnormal load within a few minutes, and killed the offending process (doubtless a local password cracker for privilege escalation) before it could do any damage. Lesson learned there, and it also taught me the value of knowing your machine well enough to sniff when it's not behaving right. On a Windows machine I would have written off the abnormal load, as abnormal load is normal on Windows.
  - LukeShu 12y ago
    I'd like to add that in my experience simply moving SSH to a port other than 22 reduces the volume of breaking attempts low enough that it's viable to see things in the log.
- raziel2p 12y ago
  Adding to this, I've found `lsof -i` more useful for spotting suspicious activity - it shows processes listening on ports as well as established connections. Enabling and checking cronjob logs is also something I wasn't doing before.

noinsight 12y ago

I agree that this is a problem and I've been researching this for some time now. Ideally you would want a HIDS system for small deployments (i.e. a single server) but there doesn't seem to exist too many of those and there's little information about them online (as far as I can find). OSSEC is one example but I'm not convinced it installs too cleanly. At least FreeBSD has a package for it however. Actual configuration would be a mystery. I think information about this sort of stuff (intrusion detection) seems to be hard to come by in general. At least stuff that is written by knowledgeable people. I haven't seen much about it from major vendors either - it seems to be a black box kind of technology. I'd love to have a "lightweight" simple daemon that does this. I guess ideally you would monitor logs for stuff, user and group modifications, cron additions etc.

sliverstorm 12y ago
I tried using AIDE for a while, but it was a mild PITA and I was never convinced an attacker with a brain couldn't just patch the database to hide his changes.
detaro 12y ago
tripwire, rkhunter, ... are relatively simple options. And then hope that your configuration prevents the attacker from getting root or if they do, then they/their automated scripts are not prepared to break them.
- noinsight 12y ago
  RE: hashing files and whatnot: FreeBSD with a high enough securelevel allows you to set files append-only or immutable so that not even root can touch them. But then updating the index becomes problematic, you would need to create delta files (which themselves are also immutable) that contain the changed information and combine the data. Even then, I'm not convinced you could get that much valuable information from a full filesystem hash (or whatever) scan. The amount of information would probably be overwhelming. You would need to apply heuristics or filters. Checking filetypes with libmagic would work, mostly. You could only alert on new executables or whatever, etc..
koide 12y ago
OSSEC requires you to understand how it works. That's good, but means that you have to devote 10 or 15 hours reading docs and testing before being productive with it. I recommend it though

cpncrunch 12y ago

Simply logging into your server once a day and typing "top" is a good start. You can usually spot suspicious processes associated with these spamkits. Also, check /var/log/messages, and check root's email. Also if you run anything like phpbb or wordpress, make sure you keep it updated. I also like to disable Apache's suexec (as I once had a rootkit that exploited it, and I don't require it for my systems), limit inbound connections with iptables, and limit ssh access to certain ip addresses. Also, get rid of or disable access to any services you don't really need (like ftp, bind, etc). It's also very important to check all your php/perl scripts for file/sql injection vulnerabilities, as that is one of the most common methods of infiltration.

dTal 12y ago
I'd like to add that it's a good idea to routinely check /var/log/auth.log for successful SSH logins. If you find that onerous because of the volume of breakin attempts, that's a good sign you need something like fail2ban. Fail2ban isn't foolproof, incidentally. I once nearly caught a worm despite a limit of 6 login attempts per IP, because I had created an account with username and password both "test" that I had since forgotten about, well before I exposed that particular machine to the internet. Oops! An automated script managed to guess that in well under 6 attempts! Fortunately I was physically present at the time and noticed the abnormal load within a few minutes, and killed the offending process (doubtless a local password cracker for privilege escalation) before it could do any damage. Lesson learned there, and it also taught me the value of knowing your machine well enough to sniff when it's not behaving right. On a Windows machine I would have written off the abnormal load, as abnormal load is normal on Windows.
- LukeShu 12y ago
  I'd like to add that in my experience simply moving SSH to a port other than 22 reduces the volume of breaking attempts low enough that it's viable to see things in the log.
raziel2p 12y ago
Adding to this, I've found `lsof -i` more useful for spotting suspicious activity - it shows processes listening on ports as well as established connections. Enabling and checking cronjob logs is also something I wasn't doing before.