14 ms·

My logs show increased break-in attempts after moving to Digital Ocean. Their IPs are hot targets for hackers, making it all the more important to properly secu

My logs show increased break-in attempts after moving to Digital Ocean. Their IPs are hot targets for hackers, making it all the more important to properly secure my droplets, something I'm having a hard time doing in Linux. With Linux, it's lot of beating around the bush. Overcomplicated config files, a lot of disabling and removing of things one don't need, having to deal with a messy and outdated firewall. With OpenBSD, lo and behold I'd have one of the simplest, most secure and well-configured droplets the world has ever seen.

bananas 13y ago
Yeah noticed that as well. First thing I do is (on debian), update packages and the kernel, install ufw (allow only 22), fail2ban, postfix and logwatch. Then setup a non privileged user, sudo, turn of root ssh and password auth and move to key based auth. Then I can sleep/configure the rest of the system. Been meaning to package this all up in an ansible playbook but I can't be bothered :( I'd be using FreeBSD if they offered it. Much like OpenBSD it's a lot more pleasant for sysadmins!
- hiphopyo 13y ago
  Awesome, please do! Don't forget https://github.com/kickstarter/rack-attack https://github.com/kickstarter/rack-attack as well though.
- dmix 13y ago
  I'm amazed noone ever mentions GRSecurity as must have. It will reduce the risks of an actual comprise significantly. Non-patched Linux is always full of rootkit potential.
  - bananas 13y ago
    Well they don't issue GRSecurity kernels. To be fair I'd just use FreeBSD which has a little better kernel architecture with respect to security.
  - hiphopyo 13y ago
    Some more regarding GRSecurity: https://mail-archive.com/misc@openbsd.org/msg127100.html https://mail-archive.com/misc@openbsd.org/msg127100.html
- jnsaff2 13y ago
  It took me about an hour to put all of this into an ansible playbook. I think there are couple of examples out on blogs about this as well. If there is interest I'll share my playbook. Writing this playbook paid off already for the second installation of the box.
  - nikhilalmeida 13y ago
    Please do.
  - radioact1ve 13y ago
    Don't mind me, I'll just wait here till you're ready.
  - OWaz 13y ago
    The most I know about securing a Linux vps I picked up from this: http://feross.org/how-to-setup-your-linode/ http://feross.org/how-to-setup-your-linode/ . If there is an ultimate resource on how to start securing a *nix vps then please do let me know. If I knew what to search for I could but a lot of this is unknown unknowns to me.
    - mverwijs 13y ago
      After having fixed a sudo user (and confirm it works!), please also disable PermitRootLogin by setting it to 'no'. Restart ssh after editing sshd_conf. If you're squeemish, run dropbear on another port so you can log into your machine in case you made a boo-boo configuring openssh. Also, more resources for security settings: https://benchmarks.cisecurity.org/downloads/multiform/index.cfm https://benchmarks.cisecurity.org/downloads/multiform/index....
  - flexd 13y ago
    I'd like a copy of that playbook too :-), publish it somewhere?
  - mimiflynn 13y ago
    yes please
- iamtew 13y ago
  Moving your SSH server away from port 22 and disabling password logins is also something you should consider doing. Small thing but at least then it doesn't shout "Hey I'm a *nix box with remote login enabled"
  - api 13y ago
    Is it actually likely that someone would brute force a password? I always install fail2ban, but it's more just to stop bots from filling up my logs with junk and eating my bandwidth/CPU. Unless you use dumb passwords, the odds are phenomenally low.
    - mindcrime 13y ago
      I'm not sure, but I've come around to realize that it's always worth installing and configuring fail2ban just for the reasons you mentioned, if nothing else. I have a demo server that, for a while, was being hammered so hard by ssh brute-force bots that it kept getting knocked offline. I installed fail2ban and no more problems since. I personally can't recommend fail2ban highly enough.
      - MertsA 13y ago
        fail2ban is based on the horribly flawed premise of correctly parsing arbitrary text logs from tons of different programs that wrote their log format without any concerns about parsing that output later. Not surprisingly there have been numerous vulnerabilities in fail2ban that let an attacker ban arbitrary hosts such as your DNS server, database, etc. https://www.google.com/search?q=fail2ban+dos+bug https://www.google.com/search?q=fail2ban+dos+bug A better approach is something like pam_abl which is a pam module that will accomplish mostly the same thing but only for login attempts and without the crappy plain text log parsing.
      - flexd 13y ago
        Have you tried denyhosts? I've always been installing denyhosts but I have not compared the two.
- phaker 13y ago
  > update packages and the kernel PSA: If you just update the kernel using system tools within the instance, it'll still boot the old one. (At least it did few months ago.) The grub (or other bootloader) and kernels that are installed inside the instance's disk image are silently ignored and the vm uses a kernel stored outside of your instance's image. You can select which kernel image you want to boot in the control panel. The problem is that the kernels available in there are updated very, very rarely. For popular distributions like Ubuntu they can lag several months, for others it can get up to, say, _two years_ [1]. Since digitalocean "doesn't support custom kernels" it appears that the only way to use something up to date (and stay secure) is to write a script that loads a custom kernel using kexec. :S edit: The fact that they use kvm (and that kexec works) which means real visualization but can't boot user's kernels is just very, very weird. The only technical reason i can think of is that they can't manage to make it work with their control panel. If that's the reason, then it's very worrying. (The "right" way is easier to code and it's the solution that you'd think of first and it makes custom kernels a non-problem, but they picked one that both has obvious problems and is harder to setup, i can't think of a reason why.) [1]: https://www.digitalocean.com/community/questions/how-can-i-boot-my-compiled-kernel-image-on-gentoo https://www.digitalocean.com/community/questions/how-can-i-b...
  - bananas 13y ago
    It does still boot the old one. You have to power it off, change the kernel and power it up again.
  - sn 13y ago
    Second shameless plug on this page: if running your own linux kernel or BSD is really important to you, http://prgmr.com http://prgmr.com can do that and we have out of band console access so you can debug why it's not working (I was having problems with that when I was checking whether some linux+xen bugs were also present on ec2.) At this time we only support xen paravirtualization(PV) though, which almost, but not quite, all major distributions have. Notably we can't run 64-bit freebsd right now because they only have HVM support. I'd like to change that but it needs more testing first. Here is a partial list of distributions supporting xen: http://wiki.xen.org/wiki/DomU_Support_for_Xen http://wiki.xen.org/wiki/DomU_Support_for_Xen Really any linux distribution with a modern kernel can be made to support it though as support has been in mainline for a long time now.
eli 13y ago
I think that's a little unfair... Your server needs to be hardened against drive-by SSH password guessing and similar nonsense regardless of how or where it's connected to the internet.
- bananas 13y ago
  That is true but as per the poster above, I've seen 10x the attack traffic on a DO IP address so any zero-days and you're already herded into a predictable net block ready to be poked.
  - aroch 13y ago
    This is true for pretty much any VPS-only hoster, they all have well defined ranges and tons of targets to attack. I see thousands of attempts a day at Linode and hundreds at Leaseweb (dedi and VPSes) and DO. It's just part of doing business.
    - eli 13y ago
      It's certainly true for dedicated-only hosters too.
      - aroch 13y ago
        I guess I should have said 'personal server', not like AWS or Google's Cloud platforms.
phaemon 13y ago
Do: sudo ufw allow proto tcp from any to any port 22 sudo ufw enable Then you can just edit /etc/ssh/sshd_config the same as you would on OpenBSD.
- rschmitty 13y ago
  This is why I love having a static IP. sudo ufw allow from your.ip.addr to tcp port 22 Rest of the world doesnt know I have ssh up and if I'm ever in a hotel or something I can tunnel through my home or login to DO's panel and allow another ip
  - aroch 13y ago
    I have essentially a static IP (A 'dynamic' IP with a 6month lease that stays the same on renewal and follows my account regardless of where I live), but you're super fucked if for some reason your IP changes. I usually set it to the /24 I'm assigned from.
    - rschmitty 13y ago
      How are you screwed? You can always login to the console in the web admin and change it
  - fredsted 13y ago
    I could imagine this being quite dangerous, like being on holiday, there's a power outage, fire, ISP error, etc at home and no way to control your server.
    - deleted 13y ago
      [deleted]
  - bananas 13y ago
    Don't do that. I did that once and our static ip changed suddenly despite lots of money being spent on it. That was a painful day driving from London to Manchester for me to get to the console.
bad_user 13y ago
Just install ufw. It's very simple to configure the rules for a basic server. Here's how: https://www.digitalocean.com/community/articles/how-to-setup-a-firewall-with-ufw-on-an-ubuntu-and-debian-cloud-server https://www.digitalocean.com/community/articles/how-to-setup...
- oinksoft 13y ago
  Why would you use that over iptables? It's simple and doesn't require installing a package. > There is a lot of functionality built into these > utilities, iptables being the most popular nowadays, but > they require a decent effort on behalf of the user to > learn and understand them. Is that a corner worth cutting?
  - bananas 13y ago
    Iptables is a crime against humanity from a config and documentation perspective. ufw makes it bearable. I'd rather use pf on BSD though.
    - AznHisoka 13y ago
      I agree. iptables is horrid.
  - bad_user 13y ago
    iptables is powerful but definitely not simple. Simple is being able to simply say "allow all outgoing traffic and incoming traffic should only be allowed for HTTP(S) and SSH" and being able to figure out how to do it by just invoking "ufw --help". Maybe someday I'll learn about iptables, I'm sure it's going to be worth it, but for now ufw does the job for me.
    - nknighthb 13y ago
      > iptables is powerful but definitely not simple. > Maybe someday I'll learn about iptables Hmm... iptables -A INPUT -p tcp --dport 22 -j ALLOW iptables -A INPUT -p tcp --dport 80 -j ALLOW iptables -A INPUT -p tcp --dport 443 -j ALLOW iptables -A INPUT -p icmp -j ALLOW iptables -A INPUT -i eth0 -j DROP So complicated...

bananas 13y ago

Yeah noticed that as well. First thing I do is (on debian), update packages and the kernel, install ufw (allow only 22), fail2ban, postfix and logwatch. Then setup a non privileged user, sudo, turn of root ssh and password auth and move to key based auth. Then I can sleep/configure the rest of the system. Been meaning to package this all up in an ansible playbook but I can't be bothered :( I'd be using FreeBSD if they offered it. Much like OpenBSD it's a lot more pleasant for sysadmins!

hiphopyo 13y ago
Awesome, please do! Don't forget https://github.com/kickstarter/rack-attack https://github.com/kickstarter/rack-attack as well though.
dmix 13y ago
I'm amazed noone ever mentions GRSecurity as must have. It will reduce the risks of an actual comprise significantly. Non-patched Linux is always full of rootkit potential.
- bananas 13y ago
  Well they don't issue GRSecurity kernels. To be fair I'd just use FreeBSD which has a little better kernel architecture with respect to security.
- hiphopyo 13y ago
  Some more regarding GRSecurity: https://mail-archive.com/misc@openbsd.org/msg127100.html https://mail-archive.com/misc@openbsd.org/msg127100.html
jnsaff2 13y ago
It took me about an hour to put all of this into an ansible playbook. I think there are couple of examples out on blogs about this as well. If there is interest I'll share my playbook. Writing this playbook paid off already for the second installation of the box.
- nikhilalmeida 13y ago
  Please do.
- radioact1ve 13y ago
  Don't mind me, I'll just wait here till you're ready.
- OWaz 13y ago
  The most I know about securing a Linux vps I picked up from this: http://feross.org/how-to-setup-your-linode/ http://feross.org/how-to-setup-your-linode/ . If there is an ultimate resource on how to start securing a *nix vps then please do let me know. If I knew what to search for I could but a lot of this is unknown unknowns to me.
  - mverwijs 13y ago
    After having fixed a sudo user (and confirm it works!), please also disable PermitRootLogin by setting it to 'no'. Restart ssh after editing sshd_conf. If you're squeemish, run dropbear on another port so you can log into your machine in case you made a boo-boo configuring openssh. Also, more resources for security settings: https://benchmarks.cisecurity.org/downloads/multiform/index.cfm https://benchmarks.cisecurity.org/downloads/multiform/index....
- flexd 13y ago
  I'd like a copy of that playbook too :-), publish it somewhere?
- mimiflynn 13y ago
  yes please
iamtew 13y ago
Moving your SSH server away from port 22 and disabling password logins is also something you should consider doing. Small thing but at least then it doesn't shout "Hey I'm a *nix box with remote login enabled"
- api 13y ago
  Is it actually likely that someone would brute force a password? I always install fail2ban, but it's more just to stop bots from filling up my logs with junk and eating my bandwidth/CPU. Unless you use dumb passwords, the odds are phenomenally low.
  - mindcrime 13y ago
    I'm not sure, but I've come around to realize that it's always worth installing and configuring fail2ban just for the reasons you mentioned, if nothing else. I have a demo server that, for a while, was being hammered so hard by ssh brute-force bots that it kept getting knocked offline. I installed fail2ban and no more problems since. I personally can't recommend fail2ban highly enough.
    - MertsA 13y ago
      fail2ban is based on the horribly flawed premise of correctly parsing arbitrary text logs from tons of different programs that wrote their log format without any concerns about parsing that output later. Not surprisingly there have been numerous vulnerabilities in fail2ban that let an attacker ban arbitrary hosts such as your DNS server, database, etc. https://www.google.com/search?q=fail2ban+dos+bug https://www.google.com/search?q=fail2ban+dos+bug A better approach is something like pam_abl which is a pam module that will accomplish mostly the same thing but only for login attempts and without the crappy plain text log parsing.
    - flexd 13y ago
      Have you tried denyhosts? I've always been installing denyhosts but I have not compared the two.
phaker 13y ago
> update packages and the kernel PSA: If you just update the kernel using system tools within the instance, it'll still boot the old one. (At least it did few months ago.) The grub (or other bootloader) and kernels that are installed inside the instance's disk image are silently ignored and the vm uses a kernel stored outside of your instance's image. You can select which kernel image you want to boot in the control panel. The problem is that the kernels available in there are updated very, very rarely. For popular distributions like Ubuntu they can lag several months, for others it can get up to, say, _two years_ [1]. Since digitalocean "doesn't support custom kernels" it appears that the only way to use something up to date (and stay secure) is to write a script that loads a custom kernel using kexec. :S edit: The fact that they use kvm (and that kexec works) which means real visualization but can't boot user's kernels is just very, very weird. The only technical reason i can think of is that they can't manage to make it work with their control panel. If that's the reason, then it's very worrying. (The "right" way is easier to code and it's the solution that you'd think of first and it makes custom kernels a non-problem, but they picked one that both has obvious problems and is harder to setup, i can't think of a reason why.) [1]: https://www.digitalocean.com/community/questions/how-can-i-boot-my-compiled-kernel-image-on-gentoo https://www.digitalocean.com/community/questions/how-can-i-b...
- bananas 13y ago
  It does still boot the old one. You have to power it off, change the kernel and power it up again.
- sn 13y ago
  Second shameless plug on this page: if running your own linux kernel or BSD is really important to you, http://prgmr.com http://prgmr.com can do that and we have out of band console access so you can debug why it's not working (I was having problems with that when I was checking whether some linux+xen bugs were also present on ec2.) At this time we only support xen paravirtualization(PV) though, which almost, but not quite, all major distributions have. Notably we can't run 64-bit freebsd right now because they only have HVM support. I'd like to change that but it needs more testing first. Here is a partial list of distributions supporting xen: http://wiki.xen.org/wiki/DomU_Support_for_Xen http://wiki.xen.org/wiki/DomU_Support_for_Xen Really any linux distribution with a modern kernel can be made to support it though as support has been in mainline for a long time now.

eli 13y ago

I think that's a little unfair... Your server needs to be hardened against drive-by SSH password guessing and similar nonsense regardless of how or where it's connected to the internet.

bananas 13y ago
That is true but as per the poster above, I've seen 10x the attack traffic on a DO IP address so any zero-days and you're already herded into a predictable net block ready to be poked.
- aroch 13y ago
  This is true for pretty much any VPS-only hoster, they all have well defined ranges and tons of targets to attack. I see thousands of attempts a day at Linode and hundreds at Leaseweb (dedi and VPSes) and DO. It's just part of doing business.
  - eli 13y ago
    It's certainly true for dedicated-only hosters too.
    - aroch 13y ago
      I guess I should have said 'personal server', not like AWS or Google's Cloud platforms.

phaemon 13y ago

Do: sudo ufw allow proto tcp from any to any port 22 sudo ufw enable Then you can just edit /etc/ssh/sshd_config the same as you would on OpenBSD.

rschmitty 13y ago
This is why I love having a static IP. sudo ufw allow from your.ip.addr to tcp port 22 Rest of the world doesnt know I have ssh up and if I'm ever in a hotel or something I can tunnel through my home or login to DO's panel and allow another ip
- aroch 13y ago
  I have essentially a static IP (A 'dynamic' IP with a 6month lease that stays the same on renewal and follows my account regardless of where I live), but you're super fucked if for some reason your IP changes. I usually set it to the /24 I'm assigned from.
  - rschmitty 13y ago
    How are you screwed? You can always login to the console in the web admin and change it
- fredsted 13y ago
  I could imagine this being quite dangerous, like being on holiday, there's a power outage, fire, ISP error, etc at home and no way to control your server.
  - deleted 13y ago
    [deleted]
- bananas 13y ago
  Don't do that. I did that once and our static ip changed suddenly despite lots of money being spent on it. That was a painful day driving from London to Manchester for me to get to the console.

bad_user 13y ago

Just install ufw. It's very simple to configure the rules for a basic server. Here's how: https://www.digitalocean.com/community/articles/how-to-setup-a-firewall-with-ufw-on-an-ubuntu-and-debian-cloud-server https://www.digitalocean.com/community/articles/how-to-setup...

oinksoft 13y ago
Why would you use that over iptables? It's simple and doesn't require installing a package. > There is a lot of functionality built into these > utilities, iptables being the most popular nowadays, but > they require a decent effort on behalf of the user to > learn and understand them. Is that a corner worth cutting?
- bananas 13y ago
  Iptables is a crime against humanity from a config and documentation perspective. ufw makes it bearable. I'd rather use pf on BSD though.
  - AznHisoka 13y ago
    I agree. iptables is horrid.
- bad_user 13y ago
  iptables is powerful but definitely not simple. Simple is being able to simply say "allow all outgoing traffic and incoming traffic should only be allowed for HTTP(S) and SSH" and being able to figure out how to do it by just invoking "ufw --help". Maybe someday I'll learn about iptables, I'm sure it's going to be worth it, but for now ufw does the job for me.
  - nknighthb 13y ago
    > iptables is powerful but definitely not simple. > Maybe someday I'll learn about iptables Hmm... iptables -A INPUT -p tcp --dport 22 -j ALLOW iptables -A INPUT -p tcp --dport 80 -j ALLOW iptables -A INPUT -p tcp --dport 443 -j ALLOW iptables -A INPUT -p icmp -j ALLOW iptables -A INPUT -i eth0 -j DROP So complicated...