4 ms·

Although you didn't state your OS, I'm going to assume we're talking about Linux. As others mentioned, reading hardening guides is a first step. Those will tell

by 4oo4 5y ago

Although you didn't state your OS, I'm going to assume we're talking about Linux. As others mentioned, reading hardening guides is a first step. Those will tell you how to avoid the most common configuration footguns and reduce attack surface. Obviously a good firewall (ufw suffices) is a must. Using a reverse proxy web server back to your web apps (I prefer nginx, but caddy is also another good one). In your reverse proxy web server, also setting up web application firewall config to flag suspicious things (for example, anyone going to a URL with `../` or `/etc/passwd` is clearly up to no good, as is a user-agent that's a known scraping tool). In particular, I like using the custom HTTP 444 response code with nginx, so I can instantly flag the worst offenders. Then, use fail2ban to blacklist hosts that are up to no good. If you can create a regex for something in a logfile, you can automatically ban almost anything. However, fail2ban is a very powerful footgun, and if misconfigured you can easily ban yourself from your own server! However, fail2ban is probably the best hardening tool, since you greatly reduce the number of tries that someone has to exploit something, and severely slow them down. Finally, regular monitoring and patching. I swear by check_mk for monitoring, so I can see every suspicious query that's coming through, and instantly identify most ongoing attacks. Fail2ban takes care of 99% of the work if it's configured correctly, but the worst attacks are the ones that you don't know are happening. I've been self-hosting most of the webapps I depend on for almost 10 years now, and I can say that self-hosting is extremely fun, but does require a a decent time commitment to maintain your infrastructure. However, if you are willing to invest time in automation with some basic shell scripting, you can get this down to less than an hour per week, which is mainly just checking your monitoring console and scheduled jobs. If this sounds too daunting but you still want to go this route, check out yunohost: https://yunohost.org/#/ https://yunohost.org/#/

SahAssar 5y ago
> Obviously a good firewall (ufw suffices) is a must I keep hearing this, but I don't get why. I make sure my services just bind to localhost if they should not be public and I make sure to not run services I don't need. I portscan my server after everything is setup to make sure I didn't miss anything. Does a firewall really buy me anything in that case?
- 4oo4 5y ago
  It gives you a little bit more control over what happens when someone hits a closed port (do you want to drop or reject a packet?), this affects how different nmap scans interpret the port being open/closed/filtered. Also, it's a failsafe in case you don't configure something correctly and something is listening on more than just the loopack (localhost) network interface that you intended. Validating your config with portscanning is great. Most importantly, you can rate limit and log. For example, when I have a ufw default deny rule, I get these in my kern.log, and then I can apply a fail2ban rule if someone hits too many closed ports in a certain period (to detect someone portscanning): kern.log: [ 494.229628] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=1064 TOS=0x00 PREC=0x00 TTL=59 ID=62111 DF PROTO=TCP SPT=443 DPT=53202 WINDOW=72 RES=0x00 ACK URGP=0 [ 498.140717] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=135 TOS=0x00 PREC=0x00 TTL=48 ID=8474 DF PROTO=TCP SPT=443 DPT=50851 WINDOW=16 RES=0x00 ACK PSH URGP=0 [ 500.091643] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=88 TOS=0x00 PREC=0x00 TTL=246 ID=32348 PROTO=TCP SPT=443 DPT=53209 WINDOW=136 RES=0x00 ACK URGP=0 [ 507.055805] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=135 TOS=0x00 PREC=0x00 TTL=48 ID=8475 DF PROTO=TCP SPT=443 DPT=50851 WINDOW=16 RES=0x00 ACK PSH URGP=0 [ 524.907273] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=135 TOS=0x00 PREC=0x00 TTL=48 ID=8476 DF PROTO=TCP SPT=443 DPT=50851 WINDOW=16 RES=0x00 ACK PSH URGP=0 [ 560.619488] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=135 TOS=0x00 PREC=0x00 TTL=48 ID=8477 DF PROTO=TCP SPT=443 DPT=50851 WINDOW=16 RES=0x00 ACK PSH URGP=0 [ 595.045967] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=79 TOS=0x00 PREC=0x00 TTL=59 ID=14064 DF PROTO=TCP SPT=443 DPT=50131 WINDOW=73 RES=0x00 ACK PSH URGP=0 [ 595.045985] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=64 TOS=0x00 PREC=0x00 TTL=59 ID=14065 DF PROTO=TCP SPT=443 DPT=50131 WINDOW=73 RES=0x00 ACK PSH URGP=0 /etc/fail2ban/filter.d/portscan.conf [Definition] # Option: failregex # Notes: Looks for attempts on ports not open in your firewall. Expects the # iptables logging utility to be used. Add the following to your iptables # config, as the last item before you DROP or REJECT: # -A <chain_name> -j LOG --log-prefix "PORT DENIED: " --log-level 5 --log-ip-options --log-tcp-options --log-tcp-sequence # This will place a notice in /var/log/messages about any attempt on a port that isn't open. failregex = \[UFW BLOCK\] .\* SRC=<HOST> ignoreregex = Then if you want to rate limit something (to help prevent DOS), you can do an iptables rule like this (not sure what the equivalent ufw config is): -A INPUT -m limit --limit 1/sec -j LOG --log-prefix "INPUT:REJECT:"
speedgoose 5y ago
Try to use a firewall outside the server if possible. You can easily mess up the firewall configuration on Linux. Or your users can. For example Docker will ignore the firewall by default when you expose ports.
arunnanda 5y ago
It might be slightly easier to use sshguard instead of fail2ban for protecting against ssh attacks. Using passwordless (key only) login is a given. As soon as your server is provisioned, log in with the password, first setup ssh, and disable password login. You can use fail2ban jails for different services (like nginx). You need to decide how strict you need to be. On FreeBSD, using blacklistd might also be a better idea than using fail2ban. To quote from the internet - "fail2ban and sshguard are both log scrapers. Log scrapers are gross hacks. blacklistd as an integrated solution is what should have happened many years ago." FreeBSD jails also provide excellent protection. It can be a good idea to run each service in its own jail. E.g. a separate jail for nginx, one for your webserver, another for your db servers. This way you can also limit the resources that are allocated/dedicated to each jail. Also, while running pf (or whatever firewall you have), you can limit the number of requests (rate limiting) to somewhat protect yourself. Using Cloudflare or something else on the front can help against ddos. Also, some providers like OVH and Hetzner have ddos protection built in for free. Some like Vultr have it as a paid service, iirc.
- 4oo4 5y ago
  SSHGuard looks cool, wasn't aware that existed. Haven't messed with FreeBSD jails, but I use unprivileged lxc containers on Linux, iirc that's the closest Linux equivalent? Those help me sleep better at night.
  - arunnanda 5y ago
    lxc was rather new when I started looking into this stuff. Since jails were much older, I just went with FreeBSD. Also because I was a little biased against Linux because on my personal computers Ubuntu and Fedora had crashed occasionally. Now I think lxd is supposed to be a better user experience than lxc. Same backend better frontend. Like ezjail or iocage for FreeBSD makes the management easier than doing it all directly.

by 4oo4 5y ago

SahAssar 5y ago

> Obviously a good firewall (ufw suffices) is a must I keep hearing this, but I don't get why. I make sure my services just bind to localhost if they should not be public and I make sure to not run services I don't need. I portscan my server after everything is setup to make sure I didn't miss anything. Does a firewall really buy me anything in that case?

4oo4 5y ago
It gives you a little bit more control over what happens when someone hits a closed port (do you want to drop or reject a packet?), this affects how different nmap scans interpret the port being open/closed/filtered. Also, it's a failsafe in case you don't configure something correctly and something is listening on more than just the loopack (localhost) network interface that you intended. Validating your config with portscanning is great. Most importantly, you can rate limit and log. For example, when I have a ufw default deny rule, I get these in my kern.log, and then I can apply a fail2ban rule if someone hits too many closed ports in a certain period (to detect someone portscanning): kern.log: [ 494.229628] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=1064 TOS=0x00 PREC=0x00 TTL=59 ID=62111 DF PROTO=TCP SPT=443 DPT=53202 WINDOW=72 RES=0x00 ACK URGP=0 [ 498.140717] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=135 TOS=0x00 PREC=0x00 TTL=48 ID=8474 DF PROTO=TCP SPT=443 DPT=50851 WINDOW=16 RES=0x00 ACK PSH URGP=0 [ 500.091643] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=88 TOS=0x00 PREC=0x00 TTL=246 ID=32348 PROTO=TCP SPT=443 DPT=53209 WINDOW=136 RES=0x00 ACK URGP=0 [ 507.055805] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=135 TOS=0x00 PREC=0x00 TTL=48 ID=8475 DF PROTO=TCP SPT=443 DPT=50851 WINDOW=16 RES=0x00 ACK PSH URGP=0 [ 524.907273] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=135 TOS=0x00 PREC=0x00 TTL=48 ID=8476 DF PROTO=TCP SPT=443 DPT=50851 WINDOW=16 RES=0x00 ACK PSH URGP=0 [ 560.619488] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=135 TOS=0x00 PREC=0x00 TTL=48 ID=8477 DF PROTO=TCP SPT=443 DPT=50851 WINDOW=16 RES=0x00 ACK PSH URGP=0 [ 595.045967] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=79 TOS=0x00 PREC=0x00 TTL=59 ID=14064 DF PROTO=TCP SPT=443 DPT=50131 WINDOW=73 RES=0x00 ACK PSH URGP=0 [ 595.045985] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=64 TOS=0x00 PREC=0x00 TTL=59 ID=14065 DF PROTO=TCP SPT=443 DPT=50131 WINDOW=73 RES=0x00 ACK PSH URGP=0 /etc/fail2ban/filter.d/portscan.conf [Definition] # Option: failregex # Notes: Looks for attempts on ports not open in your firewall. Expects the # iptables logging utility to be used. Add the following to your iptables # config, as the last item before you DROP or REJECT: # -A <chain_name> -j LOG --log-prefix "PORT DENIED: " --log-level 5 --log-ip-options --log-tcp-options --log-tcp-sequence # This will place a notice in /var/log/messages about any attempt on a port that isn't open. failregex = \[UFW BLOCK\] .\* SRC=<HOST> ignoreregex = Then if you want to rate limit something (to help prevent DOS), you can do an iptables rule like this (not sure what the equivalent ufw config is): -A INPUT -m limit --limit 1/sec -j LOG --log-prefix "INPUT:REJECT:"

speedgoose 5y ago

Try to use a firewall outside the server if possible. You can easily mess up the firewall configuration on Linux. Or your users can. For example Docker will ignore the firewall by default when you expose ports.

arunnanda 5y ago

It might be slightly easier to use sshguard instead of fail2ban for protecting against ssh attacks. Using passwordless (key only) login is a given. As soon as your server is provisioned, log in with the password, first setup ssh, and disable password login. You can use fail2ban jails for different services (like nginx). You need to decide how strict you need to be. On FreeBSD, using blacklistd might also be a better idea than using fail2ban. To quote from the internet - "fail2ban and sshguard are both log scrapers. Log scrapers are gross hacks. blacklistd as an integrated solution is what should have happened many years ago." FreeBSD jails also provide excellent protection. It can be a good idea to run each service in its own jail. E.g. a separate jail for nginx, one for your webserver, another for your db servers. This way you can also limit the resources that are allocated/dedicated to each jail. Also, while running pf (or whatever firewall you have), you can limit the number of requests (rate limiting) to somewhat protect yourself. Using Cloudflare or something else on the front can help against ddos. Also, some providers like OVH and Hetzner have ddos protection built in for free. Some like Vultr have it as a paid service, iirc.

4oo4 5y ago
SSHGuard looks cool, wasn't aware that existed. Haven't messed with FreeBSD jails, but I use unprivileged lxc containers on Linux, iirc that's the closest Linux equivalent? Those help me sleep better at night.
- arunnanda 5y ago
  lxc was rather new when I started looking into this stuff. Since jails were much older, I just went with FreeBSD. Also because I was a little biased against Linux because on my personal computers Ubuntu and Fedora had crashed occasionally. Now I think lxd is supposed to be a better user experience than lxc. Same backend better frontend. Like ezjail or iocage for FreeBSD makes the management easier than doing it all directly.