6 ms·

This has been covered elsewhere (like on serverfault: http://serverfault.com/a/107346/2557 http://serverfault.com/a/107346/2557 ) But it comes down to: - Take

This has been covered elsewhere (like on serverfault: http://serverfault.com/a/107346/2557 http://serverfault.com/a/107346/2557 ) But it comes down to: - Take existing server down immediately. I'm assuming it is not on an isolated network -- so this should really be a priority. - Prep a new patched server (with a smaller attack surface and updated security credentials) - Postmortem the old box on an isolated network. Try to understand how the attacker got in. If necessary, get security professionals involved.

LoSboccacc 10y ago
also remember to do the same on every server that could be reached by sharing the same credentials or having the same vulnerability once discovered, and at the end of the process trigger password change for every user if applicable
dpina 10y ago
Also, wipe clean the existing server after the Postmortem. Even if fairly sure that the server can be cleaned it's never safe to re-use a box which was compromised.
- joobus 10y ago
  Use one of these: http://www.wnd.com/files/2015/08/hillary-clinton-secret-server-wiper-600.png http://www.wnd.com/files/2015/08/hillary-clinton-secret-serv...
  - zaroth 10y ago
    "You mean like, with a cloth or something?"
    - BraveNewCurency 10y ago
      No, using a towel could lead to contamination. Use a disposable paper towel. Also, Security Experts (TM) have not proven that Windex helps, but I'd use it just in case.
talideon 10y ago
Also, you thank the reporter profusely for doing the right thing.
- dleslie 10y ago
  Ask for a BTC address and send a tip.
  - NTripleOne 10y ago
    Alternatively, ask them for their preferred payment method and don't force your FOTMcoin on them.
    - Mahn 10y ago
      I assume he suggested Bitcoin because it's still a good way to handle money anonymously. If the person that breached the server wants to remain anonymous but get paid, that's pretty much the only way to go.
      - ChristianBundy 10y ago
        I assume he suggested asking them because Bitcoin isn't the only cryptocurrency.
        um_ya 10y ago
        I'm pretty sure the hacker won't want dogecoin or forked coin.
        raverbashing 10y ago
        Oh I don't know, depending on the person they might want some fringe criptocurrency that they swear is the best (for non-economic reasons)
      - bduerst 10y ago
ascendantlogic 10y ago
If the box is on AWS, make sure to make an AMI snapshot of it for later postmortem. Also create a security group just for boxes built using that AMI that specifically only allows the minimum amount of access required to do the postmortem.
deleted 10y ago
[deleted]
emcrazyone 10y ago
100% agree. Disclaimer I own a data center and have dealt with customer collocated equipment breaches. In addition to the above steps: -> disable root being able to login inside your sshd_config file. Make sure PermitRootLogin no -> rename the root account too so if they are using an exploit based on user authentication then perhaps they won't be able to elevate to root. -> disable password based logins and go to cert based auth. This will shutdown brute force attacks. -> lower MaxAuthTries in sshd_config to something like 1 or 2 to help slow down attackers. -> change the port you listen on. A little security through obscurity while not very effective might slow future casual port scanners that are testing a single port. In practice I've seen this really eliminate a lot of reconnaissance or farming type activities. -> Make sure you're openssl and openssh are at the latest stable releases. -> If the perp is coming from the same IP, you can use an iptables rule to block the netblock. Again, not perfect but may help slow things down. -> grab shell history of all the user accounts on the box. Example ~/.bash_history. This is more reconnaissance but may be helpful if they are sloppy - you might see what they were doing on the box. -> look for any modified files or new ones. Obviously logs will show up in the list but you're looking for things that should not be there. Example: find / * -mtime -60 -print where 60 is how many days ago you want to go back. -> look at chron file to see if any timed delay bombs exist. -> look at ps -aux output for any running processes that don't make sense -> look at iptraf for any suspicious traffic to IPs you can't reconcile. Good luck!
- VikingCoder 10y ago
  ...it's a Windows box. (Just kidding.)
  - sredfern2 10y ago
    :-)
- eonw 10y ago
  just a note: intelligent exploiters hide their files inside of yours, so the -mtime is useless in many cases, they will set the mtime of their upload to match the rest of the folder they hide in. command history is also easy to alter if you know what you are doing.
drzaiusapelord 10y ago
Great advice, I'd also add a couple more things about checking the desktop environment because its much more likely a windows or osx box was compromised and keys stolen or keystrokes captured than someone brute-forcing sshd or finding some sshd zero-day. The few times we've been in this position, its always been someone's desktop that's been compromised. If you don't have a strong desktop security policy, a UTM, traffic inspection, key encryption policy, etc and you store keys or passwords locally, then this becomes a risk.
zpallin 10y ago
You still don't know how they got in, and since they're not doing anything nefarious (that OP knows of) it's probably easier to pay them off to inform OP how they got in.

LoSboccacc 10y ago

also remember to do the same on every server that could be reached by sharing the same credentials or having the same vulnerability once discovered, and at the end of the process trigger password change for every user if applicable

dpina 10y ago

Also, wipe clean the existing server after the Postmortem. Even if fairly sure that the server can be cleaned it's never safe to re-use a box which was compromised.

joobus 10y ago
Use one of these: http://www.wnd.com/files/2015/08/hillary-clinton-secret-server-wiper-600.png http://www.wnd.com/files/2015/08/hillary-clinton-secret-serv...
- zaroth 10y ago
  "You mean like, with a cloth or something?"
  - BraveNewCurency 10y ago
    No, using a towel could lead to contamination. Use a disposable paper towel. Also, Security Experts (TM) have not proven that Windex helps, but I'd use it just in case.

talideon 10y ago

Also, you thank the reporter profusely for doing the right thing.

dleslie 10y ago
Ask for a BTC address and send a tip.
- NTripleOne 10y ago
  Alternatively, ask them for their preferred payment method and don't force your FOTMcoin on them.
  - Mahn 10y ago
    I assume he suggested Bitcoin because it's still a good way to handle money anonymously. If the person that breached the server wants to remain anonymous but get paid, that's pretty much the only way to go.
    - ChristianBundy 10y ago
      I assume he suggested asking them because Bitcoin isn't the only cryptocurrency.
      - um_ya 10y ago
        I'm pretty sure the hacker won't want dogecoin or forked coin.
        raverbashing 10y ago
        Oh I don't know, depending on the person they might want some fringe criptocurrency that they swear is the best (for non-economic reasons)
    - bduerst 10y ago

ascendantlogic 10y ago

If the box is on AWS, make sure to make an AMI snapshot of it for later postmortem. Also create a security group just for boxes built using that AMI that specifically only allows the minimum amount of access required to do the postmortem.

deleted 10y ago

[deleted]

emcrazyone 10y ago

100% agree. Disclaimer I own a data center and have dealt with customer collocated equipment breaches. In addition to the above steps: -> disable root being able to login inside your sshd_config file. Make sure PermitRootLogin no -> rename the root account too so if they are using an exploit based on user authentication then perhaps they won't be able to elevate to root. -> disable password based logins and go to cert based auth. This will shutdown brute force attacks. -> lower MaxAuthTries in sshd_config to something like 1 or 2 to help slow down attackers. -> change the port you listen on. A little security through obscurity while not very effective might slow future casual port scanners that are testing a single port. In practice I've seen this really eliminate a lot of reconnaissance or farming type activities. -> Make sure you're openssl and openssh are at the latest stable releases. -> If the perp is coming from the same IP, you can use an iptables rule to block the netblock. Again, not perfect but may help slow things down. -> grab shell history of all the user accounts on the box. Example ~/.bash_history. This is more reconnaissance but may be helpful if they are sloppy - you might see what they were doing on the box. -> look for any modified files or new ones. Obviously logs will show up in the list but you're looking for things that should not be there. Example: find / * -mtime -60 -print where 60 is how many days ago you want to go back. -> look at chron file to see if any timed delay bombs exist. -> look at ps -aux output for any running processes that don't make sense -> look at iptraf for any suspicious traffic to IPs you can't reconcile. Good luck!

VikingCoder 10y ago
...it's a Windows box. (Just kidding.)
- sredfern2 10y ago
  :-)
eonw 10y ago
just a note: intelligent exploiters hide their files inside of yours, so the -mtime is useless in many cases, they will set the mtime of their upload to match the rest of the folder they hide in. command history is also easy to alter if you know what you are doing.

drzaiusapelord 10y ago

Great advice, I'd also add a couple more things about checking the desktop environment because its much more likely a windows or osx box was compromised and keys stolen or keystrokes captured than someone brute-forcing sshd or finding some sshd zero-day. The few times we've been in this position, its always been someone's desktop that's been compromised. If you don't have a strong desktop security policy, a UTM, traffic inspection, key encryption policy, etc and you store keys or passwords locally, then this becomes a risk.

zpallin 10y ago

You still don't know how they got in, and since they're not doing anything nefarious (that OP knows of) it's probably easier to pay them off to inform OP how they got in.