13 ms·

Essential security for Linux servers (2013)

199 points by jcgoette 5y ago

Justin_K 5y ago
Is there an up to date version of this?
- vbezhenar 5y ago
  While I don't agree to this article, from quick glance it does not seem to be obsolete and everything should work on modern Ubuntu. One thing that's not immediately obvious is that docker does not care about your firewall.
- solfox 5y ago
  Not by me (author) :)
- xcambar 5y ago
  AFAICT, the contents is still up to date and solid advice (except IP locks, but YMMV)
- shric 5y ago
  One thing I'd update for today is use Ed25519 keys instead of RSA
- 3np 5y ago
  Two changes I would make: * use -t ed25519 to generate keys, much more efficient for same security compared to RSA * don’t use ufw. It easily becomes a big mess and is a pain to manage with ansible. firewalld is a much better high-lever firewall. Preferably with nftables backend. If you have a bit bigger fleet and manage a CA you could look into using signed SSH certificates instead of public keys. That way you can provision access centrally without adding individual keys to individual servers.
  - cheshire_cat 5y ago
    Do you switch Debian based systems to firewalld or do you just prefer RedHat based systems?
    - 3np 5y ago
      I'm running mostly Debian, actually. No issues.
- hamburglar 5y ago
  It’s all still what I’d call “good advice if you refuse to take some better advice”. The caveat at the beginning acknowledges that this is a pragmatic approach rather than the best approach, and I think in the intervening time I’ve become more convinced that the better approach is the only approach: namely to automate a lot more of these things, which is alluded to at the end. I’d also ditch the use of any shared credential other than the emergency root password, which should be locked away and not actually known by any people. Your mechanism for syncing ssh pubkeys (which, btw, isn’t specified in the article, which in my experience means it doesn’t really exist :D) on the shared account should instead populate the user keys directory and there should be one logon per user.
abagheri43 5y ago
Very beautiful and excellent
rspoerri 5y ago
maybe read the last discussion on hn: https://news.ycombinator.com/item?id=5316093 https://news.ycombinator.com/item?id=5316093
- jerrya 5y ago
  Is there a library of well-designed, well-tested configuration scripts for various servers implementing various tasks?
  - egberts1 5y ago
    Well-designed? Well-tested? There are lots of scripts out there, notably in Git repositories but none attained the dual-status that you seek.
abdusco 5y ago
What has helped me more than fail2ban with reducing login attempts by many orders of magnitude is changing default SSH port from 22 to something in 10000-30000 range.
- atoav 5y ago
  Additionally it might be a good idea to forbid password logins altogether by adding this line in /etc/ssh/sshd_config: PasswordAuthentication no Of course you should make really sure to actually have a working public key in your users "~/.ssh/authorized_keys" file and/or in "/root/.ssh/authorized_keys" otherwise you might lock yourself out of the server. But the point here is: given the choice, you should never log in regularily with a ssh password if you can also use a key.
  - Rygian 5y ago
    Newbie question here: how is a private key stored in a device I can easily lose more secure than a (long and sufficiently random) password that I've memorized and can type down only when intended?
    - hansel_der 5y ago
      passwords are a symetric key, hence if the server is compromised, so is the password. with asymetric keys, a compromise of the public key is no problem. but you are right, key-files on a disk are more vulnerable to theft than secrets in your head. keyfiles with a password ontop are most secure but also most uncomfortable.
      - zamfi 5y ago
        > passwords are a symetric key, hence if the server is compromised, so is the password Pretty sure that’s not how it works, iirc passwords are stored one-way encrypted. And if it were true, then anyone with root access to a box could comprise every other (Unix) user’s key, which seems like a potentially bigger problem…
simonmales 5y ago
TIL about logwatch. Seems handy as I run a couple of VPSs.
- pabs3 5y ago
  Sounds similar to logcheck. There is also journalcheck for systemd based systems: https://github.com/trentbuck/journalcheck https://github.com/trentbuck/journalcheck
ncmncm 5y ago
That is not the whole job on /etc/ssh/sshd_config. You also need ChallengeResponseAuthentication no
- vaylian 5y ago
  why?
  - yosamino 5y ago
    Because in the most common use-case it allows for the the same functionality as PasswordAuthentication, so if you want to disallow password-based logins, it also has to go. Note that newer openssh version (don't remember how new) renamed this to KbdInteractiveAuthentication. So check your documentation, and double-check everything you read on the internet. This article is from 2013 after all.
devwastaken 5y ago
Do not lock down to your IP address. Home IP's change all the time, and in proper security there should be no other way to access the server other than proper ssh authentication or physical access. There is no good reason to be doing it in this context. If SSH turns out to have a massive vulnerability that bypasses keyauth then every service on the net will be torn down.
- hansel_der 5y ago
  > Do not lock down to your IP address. > If SSH turns out to have a massive vulnerability that bypasses keyauth then every service on the net will be torn down those seem to contradict each other. i agree that black-/whitelisting should not be the center of you security architecture but it sure helps in the scenario of authentication bypass vuln.
- formerly_proven 5y ago
  Some people recommend running a VPN server and then using SSH over VPN for "improved security", but pretty much every VPN apart from WireGuard has a pretty poor track record there. SSH is in all likelihood the most secure server software that you can have on a Linux box. Everything else you put in front of it is likely to be a downgrade.
  - hn_throwaway_69 5y ago
    As you essentially say, WireGuard is great. I firewall off direct SSH and first use WireGuard to connect to the server instead. One advantage is that if your firewall is setup right it's completely invisible, as unauthenticated UDP packets are dropped, as is the case with any other, unused, UDP port. I still configure SSH to best practices just in case a configuration blunder inadvertently causes the firewall to accept connections.
- hvgk 5y ago
  That usually doesn’t matter that much. You can get into the console of the node from whichever cloud company you’re renting it off and change it there.
INTPenis 5y ago
First step is actually passwd -l root for me, I alwasy lock the root password. After ensuring I have a working admin account of course. Using the root account at all is obsolete imho. Fedora, CentOS and RHEL all allow me to skip setting a root password and just use my admin user.
- hansel_der 5y ago
  imho a distinct admin account is better than elevating a useraccount, which also runs a browser.
  - INTPenis 5y ago
    Yeah sure but it's still better to lock the root account and create a sudo admin account for all root tasks.
    - hansel_der 5y ago
      sure, security throu obscurity is somewhat valid as the attacker then has to discover the 'sudo admin account' instead of going for root directly
      - INTPenis 5y ago
        Heh yeah but we don't call it security by obscurity because those are dirty words, we call it "best practice" instead. ;)
        hansel_der 5y ago
        true
- maxk42 5y ago
  That was explicitly mentioned in the article, as was disabling password-based logins entirely.
formerly_proven 5y ago
fail2ban is unnecessary if a non-standard port is used. Even a sub-1024 SSH port gets extremely little traffic with spurious login attempts just once per day or every few days and most of these aren't going anywhere (admin:admin). Similarly I don't think for personal servers and the like there is much point in disabling root login, though I disable password auth in SSH as a general rule. A firewall on a server itself should not be necessary in most cases, because unneeded "listen everywhere for everything" services should not be running in the first place. If this is managed by multiple people, the firewall should be external to the server so that the same person who "just wants to run this service for a test real quick" can't "change firewall policy real quick".
- rnestler 5y ago
  My experience is that even non-standard ssh ports gets hundreds of login attempts on ssh per day.
  - formerly_proven 5y ago
    I suppose that depends on what you think of as an "login attempt". Is opening a connection a login attempt? I would say it isn't. Is sending some random protocol header a login attempt? Doubtful. Is failing to negotiate a login attempt? Again, I'd say no (most likely a port scanner looking for old/vulnerable servers). Is SSH-1.5-Nmap a login attempt? I don't think so. As we have disabled password authentication, a client can't actually try to do a user/pass login, so what can't happen, isn't. These things show up, but are completely irrelevant to security.
  - usr1106 5y ago
    Yes, it has increased quite a bit in recent years. I think still 5 years ago there was basically no traffic.
ahepp 5y ago
Who cares about fail2ban? Unless you're worried about DOS, it seems pretty silly. Similarly with a firewall, assuming you have NAT and a firewall on your router.
acatton 5y ago
Even for 2013, this is not very good advice. fail2ban makes your iptables dynamic, which is a nightmare to audit. This is what I do in 2021: * Set up spiped[1] in front of SSH * Install and setup nftables[2]. * Lock down every service as much as possible in systemd[3]. (If the service is built-in the distro, just use drop in files[4]) [1] https://www.tarsnap.com/spiped.html https://www.tarsnap.com/spiped.html [2] https://wiki.archlinux.org/title/nftables https://wiki.archlinux.org/title/nftables [3] https://ruderich.org/simon/notes/systemd-service-hardening https://ruderich.org/simon/notes/systemd-service-hardening [4] https://wiki.archlinux.org/index.php?title=Systemd&oldid=704440#Drop-in_files https://wiki.archlinux.org/index.php?title=Systemd&oldid=704...
- iso1631 5y ago
  Why use spiped rather than wireguard?
  - acatton 5y ago
    Wireguard is also a good alternative. But you need to connect to a VPN every time you want to SSH. Here is my personal reasons why I use spiped: * spiped is transparent by using ProxyCommand[1]. This allow me to do "ssh host" and thanks to my ssh_config, it just connects. * spiped can be run in a very hardened way.[2] It just needs to listen() to a socket, connect to another one, and access a key file. Wireguard needs complex network access, it needs to create interfaces and open raw sockets. * spiped is much simple to manage, just run a daemon. With wireguards there are two possibilites: ** Every host runs wireguard, you might need to connect to multiple hosts at the time, you need to manage internal IP conflicts, etc... ** One central wireguard server, you have a single point of failure, and can't ssh anywhere if this host is down. Don't get me wrong, I love wireguard, use it all the time as a VPN, but I don't think it's appropriate as a layer of protection in front of my SSH server. Both Wireguard and Spiped are written by very smart people. [1] https://man.openbsd.org/ssh_config#ProxyCommand https://man.openbsd.org/ssh_config#ProxyCommand [2] https://ruderich.org/simon/notes/systemd-service-hardening https://ruderich.org/simon/notes/systemd-service-hardening
    - 5y ago
makach 5y ago
In a professional setting, today, there is much much more that needs to be taken into consideration. Anyone else found it sus that the OP didn't mention backups?
- paradaux 5y ago
  I usually wait until I have content to backup prior to setting up backups, and definitely wouldn't be in my mind in the "first 5 minutes." If you're concerned you will lose 5 minutes of work make use of snapshotting if available. Of course the first-5-minute title is hyperbolic, but backups are besides the point when you're first setting up and securing a machine.
  - Sebb767 5y ago
    On the other hand, backups as an afterthought is what leaves you paying ransoms. I prefer to think about backups directly when setting up the machine, since grouping data directories can help a lot with backup strategies later on. Of course, making sure you're the only one on the machine is step 1, but at least I like to set up backups before placing any serious data on the machine, it's a part of the initial setup for me.
  - garmaine 5y ago
    > Of course the first-5-minute title is hyperbolic I don't think it is. I've managed a server directly connected to the internet with a US government IP, and it was being port scanned from a Chinese IP within minutes of being turned on. If you are a target, then there is an adversary out there that is patiently waiting for the opportunity to exploit an unpatched vulnerability in new installs, as if your security is otherwise good it might be how they get their foot in the door on your network. (In our case I really did have a "5 minute plan" to login as soon as the fresh install was booted, setup a firewall, lockdown the ssh server, and install fail2ban ASAP. I'd then check system logs to see if anyone got in before proceeding. Time was of the essence.)
    - tmottabr 5y ago
      no one in that scenario would not do things manually like in the article. but if doing it, then at minimum you should use an custom install media with latest packages bundled and all the configuration already backed so you hit the ground with sane defaults and cover the first 5 minutes from this articles during install time. also in any install i would always do a netinstall to get any updates between media generation and install time, so you should always have the latest and greats at install time.
marvinblum 5y ago
Everytime I read something about Linux server hardening, I get more confused. We're lacking a clear and simple, modern guide on how to do things. I know, every setup is different, but there should at least be consensus for a fresh installation. Also, do I really HAVE to change something so that it is secure? Isn't a Ubuntu server secure out of the box? With a strong, unique root password of course.
- sh4un 5y ago
  Everyone has an opinion on doing it better, that is why.
  - eddieroger 5y ago
    In addition to this, everyone has different requirements. A mix of opinions and a mix of requirements make a real variety of ways of doing it.
- DyslexicAtheist 5y ago
  > Isn't a Ubuntu server secure out of the box? I think people working at RedHat are more competent in moving security forward on Linux than what Ubuntu does. Ubuntu hardly innovates at all. Its target market seems to be desktop users (or server admins that are only familiar with the Desktop version). Personally I wouldn't put Ubuntu (or any other distribution) on a server without an elaborate playbook to tailor it to my needs (on Ubuntu that playbook is always more complex from my experience). This is where Ubuntu fails for me because it makes some weird assumptions as to what I want in terms of security (which are absent in Debian). YMMV. Although I think that a distribution's goal should be accessibility and configurability - in that regard all of them don't prioritize security features as much as I'd like to see (but knowing myself I probably would complain the second these features become too opinionated - which they most certainly would - which is why I think Debian does the right thing with not making opinionated assumptions). Ubuntu compared to Debian standard install is more bloated, interim releases are much buggier, and Ubuntu LTS is less stable than Debian stable. Ubuntu's root certificate store is constantly outdated (though the same issue might also be on Debian). Their apparmor configuration lags behind, ... whatever is good they usually inherit from Debian. All distributions could do more to lock down processes with seccomp-filters in systemd. Would be interesting to see what lynis⁰ discovers when comparing a fresh server install between Ubuntu and others. In over 20 years I have seen some real shit-shows in production with all distributions except Debian (again ymmv). Jason Donenfeld, the creator of Wireguard said about Ubuntu on the latest¹ SCW podcast: > Ubuntu is always, a horrible distribution to work with, ... > Well, they [Ubuntu] sort of inherit from Debian, but they're like not super tuned in to what's going on and like not really on top of things. And so it was just always, it's still a pain to like make sure Ubuntu is working well. but I don't know, it's not too much interesting to say about the distro story, just open source politics as usual. while somewhat anecdotal I trust that Jason knows what he is talking about having been on the linux security kernel team for ages and familiar with the quirks of various downstream vendors. His development cycle for WG is: implement -> decompile -> formal-verification -> rinse/repeat :-/ All of Linux security is a shit show. This is why grsecurity is charging money for it's service. ⁰ https://cisofy.com/lynis/ https://cisofy.com/lynis/ ¹ https://securitycryptographywhatever.buzzsprout.com/1822302/9667632-wireguard-feat-jason-donenfeld https://securitycryptographywhatever.buzzsprout.com/1822302/...
belter 5y ago
After Security...on Performance, Brendan Gregg First 60 Seconds: https://netflixtechblog.com/linux-performance-analysis-in-60-000-milliseconds-accc10403c55 https://netflixtechblog.com/linux-performance-analysis-in-60...
clay-dreidels 5y ago
Here is my setup, but I’m sure it could use som work. https://lamda-chops.bearblog.dev/steps-i-take-when-setting-up-a-vpn-server-on-digital-ocean/ https://lamda-chops.bearblog.dev/steps-i-take-when-setting-u...
- onehair 5y ago
  > Enable Automatic Upgrades I can't count with my fingers alone how many times things broke because I `apt upgrade`d without thoroughly reading the change logs. Or even when reading the change logs something still gets past me. Having auto upgrades on a server is not a good idea.
  - deadbunny 5y ago
    This is why automatic updates are usually only for security updates, not everything.
Sosh101 5y ago
Do you really need fail2ban if you require keys for ssh login?
- rascul 5y ago
  It can help cut down on log sizes.
  - deadbunny 5y ago
    So does logrotate.
heipei 5y ago
What I'd like to see is a modern guide for setting up and operating a cluster of hosts that does not rely on any specific provider settings. Say you want to run a cluster of Ubuntu servers, maybe exclusively with a workload scheduler like k8s, maybe with a mixture of nodes, how do you set it up securely and consistently, how to you apply updates, provision users, deploy applications, and how do you centrally log and alert on events (systemd logs, docker logs, auditd). Bonus points if you there are pointers about how compliant that setup is wrt to modern compliance requirements. I know it's a lot to ask, but maybe there is such a guide available that does not just fall back to talking about provider-specific features (e.g. IAM).
- idoubtit 5y ago
  ansible. And similar tools, but lack experience outside of ansible. You can see Ansible a a tool to automate the install/configure/update process that you would do manually on a single server. Then you can apply this "playbook" to any server. You just need an ssh connection to the target servers, with python installed on them. Of course, you have to write rules for setting up a server, provisioning users, monitoring (deploying Prometheus, pushing logs to a central server...). There various plugins for integrating with providers, but the basic features are provider independent. Ansible is far from perfect (dependency on python, inconsistent syntax, abuse of aliases, missing a strict mode...), but it's rather easy to learn and I've used it successfully (at a small scale).
errcorrectcode 5y ago
Naïve, black-&-white, over-simplified, amateurish advice. Security measures to deploy depend on the risk level involved, e.g., potential costs of being hacked. Measures like SELinux, grsec, fwknop, snort, IDS (tripwire or Samhain), HSMs, hw entropy sources, split SSH/TLS, and microservices compartmentalized into VM containers rather than Docker have their place.

199 points by jcgoette 5y ago

Justin_K 5y ago

Is there an up to date version of this?

vbezhenar 5y ago
While I don't agree to this article, from quick glance it does not seem to be obsolete and everything should work on modern Ubuntu. One thing that's not immediately obvious is that docker does not care about your firewall.
solfox 5y ago
Not by me (author) :)
xcambar 5y ago
AFAICT, the contents is still up to date and solid advice (except IP locks, but YMMV)
shric 5y ago
One thing I'd update for today is use Ed25519 keys instead of RSA
3np 5y ago
Two changes I would make: * use -t ed25519 to generate keys, much more efficient for same security compared to RSA * don’t use ufw. It easily becomes a big mess and is a pain to manage with ansible. firewalld is a much better high-lever firewall. Preferably with nftables backend. If you have a bit bigger fleet and manage a CA you could look into using signed SSH certificates instead of public keys. That way you can provision access centrally without adding individual keys to individual servers.
- cheshire_cat 5y ago
  Do you switch Debian based systems to firewalld or do you just prefer RedHat based systems?
  - 3np 5y ago
    I'm running mostly Debian, actually. No issues.
hamburglar 5y ago
It’s all still what I’d call “good advice if you refuse to take some better advice”. The caveat at the beginning acknowledges that this is a pragmatic approach rather than the best approach, and I think in the intervening time I’ve become more convinced that the better approach is the only approach: namely to automate a lot more of these things, which is alluded to at the end. I’d also ditch the use of any shared credential other than the emergency root password, which should be locked away and not actually known by any people. Your mechanism for syncing ssh pubkeys (which, btw, isn’t specified in the article, which in my experience means it doesn’t really exist :D) on the shared account should instead populate the user keys directory and there should be one logon per user.

abagheri43 5y ago

Very beautiful and excellent

rspoerri 5y ago

maybe read the last discussion on hn: https://news.ycombinator.com/item?id=5316093 https://news.ycombinator.com/item?id=5316093

jerrya 5y ago
Is there a library of well-designed, well-tested configuration scripts for various servers implementing various tasks?
- egberts1 5y ago
  Well-designed? Well-tested? There are lots of scripts out there, notably in Git repositories but none attained the dual-status that you seek.

abdusco 5y ago

What has helped me more than fail2ban with reducing login attempts by many orders of magnitude is changing default SSH port from 22 to something in 10000-30000 range.

atoav 5y ago
Additionally it might be a good idea to forbid password logins altogether by adding this line in /etc/ssh/sshd_config: PasswordAuthentication no Of course you should make really sure to actually have a working public key in your users "~/.ssh/authorized_keys" file and/or in "/root/.ssh/authorized_keys" otherwise you might lock yourself out of the server. But the point here is: given the choice, you should never log in regularily with a ssh password if you can also use a key.
- Rygian 5y ago
  Newbie question here: how is a private key stored in a device I can easily lose more secure than a (long and sufficiently random) password that I've memorized and can type down only when intended?
  - hansel_der 5y ago
    passwords are a symetric key, hence if the server is compromised, so is the password. with asymetric keys, a compromise of the public key is no problem. but you are right, key-files on a disk are more vulnerable to theft than secrets in your head. keyfiles with a password ontop are most secure but also most uncomfortable.
    - zamfi 5y ago
      > passwords are a symetric key, hence if the server is compromised, so is the password Pretty sure that’s not how it works, iirc passwords are stored one-way encrypted. And if it were true, then anyone with root access to a box could comprise every other (Unix) user’s key, which seems like a potentially bigger problem…

simonmales 5y ago

TIL about logwatch. Seems handy as I run a couple of VPSs.

pabs3 5y ago
Sounds similar to logcheck. There is also journalcheck for systemd based systems: https://github.com/trentbuck/journalcheck https://github.com/trentbuck/journalcheck

ncmncm 5y ago

That is not the whole job on /etc/ssh/sshd_config. You also need ChallengeResponseAuthentication no

vaylian 5y ago
why?
- yosamino 5y ago
  Because in the most common use-case it allows for the the same functionality as PasswordAuthentication, so if you want to disallow password-based logins, it also has to go. Note that newer openssh version (don't remember how new) renamed this to KbdInteractiveAuthentication. So check your documentation, and double-check everything you read on the internet. This article is from 2013 after all.

devwastaken 5y ago

Do not lock down to your IP address. Home IP's change all the time, and in proper security there should be no other way to access the server other than proper ssh authentication or physical access. There is no good reason to be doing it in this context. If SSH turns out to have a massive vulnerability that bypasses keyauth then every service on the net will be torn down.

hansel_der 5y ago
> Do not lock down to your IP address. > If SSH turns out to have a massive vulnerability that bypasses keyauth then every service on the net will be torn down those seem to contradict each other. i agree that black-/whitelisting should not be the center of you security architecture but it sure helps in the scenario of authentication bypass vuln.
formerly_proven 5y ago
Some people recommend running a VPN server and then using SSH over VPN for "improved security", but pretty much every VPN apart from WireGuard has a pretty poor track record there. SSH is in all likelihood the most secure server software that you can have on a Linux box. Everything else you put in front of it is likely to be a downgrade.
- hn_throwaway_69 5y ago
  As you essentially say, WireGuard is great. I firewall off direct SSH and first use WireGuard to connect to the server instead. One advantage is that if your firewall is setup right it's completely invisible, as unauthenticated UDP packets are dropped, as is the case with any other, unused, UDP port. I still configure SSH to best practices just in case a configuration blunder inadvertently causes the firewall to accept connections.
hvgk 5y ago
That usually doesn’t matter that much. You can get into the console of the node from whichever cloud company you’re renting it off and change it there.

INTPenis 5y ago

First step is actually passwd -l root for me, I alwasy lock the root password. After ensuring I have a working admin account of course. Using the root account at all is obsolete imho. Fedora, CentOS and RHEL all allow me to skip setting a root password and just use my admin user.

hansel_der 5y ago
imho a distinct admin account is better than elevating a useraccount, which also runs a browser.
- INTPenis 5y ago
  Yeah sure but it's still better to lock the root account and create a sudo admin account for all root tasks.
  - hansel_der 5y ago
    sure, security throu obscurity is somewhat valid as the attacker then has to discover the 'sudo admin account' instead of going for root directly
    - INTPenis 5y ago
      Heh yeah but we don't call it security by obscurity because those are dirty words, we call it "best practice" instead. ;)
      - hansel_der 5y ago
        true
maxk42 5y ago
That was explicitly mentioned in the article, as was disabling password-based logins entirely.

formerly_proven 5y ago

fail2ban is unnecessary if a non-standard port is used. Even a sub-1024 SSH port gets extremely little traffic with spurious login attempts just once per day or every few days and most of these aren't going anywhere (admin:admin). Similarly I don't think for personal servers and the like there is much point in disabling root login, though I disable password auth in SSH as a general rule. A firewall on a server itself should not be necessary in most cases, because unneeded "listen everywhere for everything" services should not be running in the first place. If this is managed by multiple people, the firewall should be external to the server so that the same person who "just wants to run this service for a test real quick" can't "change firewall policy real quick".

rnestler 5y ago
My experience is that even non-standard ssh ports gets hundreds of login attempts on ssh per day.
- formerly_proven 5y ago
  I suppose that depends on what you think of as an "login attempt". Is opening a connection a login attempt? I would say it isn't. Is sending some random protocol header a login attempt? Doubtful. Is failing to negotiate a login attempt? Again, I'd say no (most likely a port scanner looking for old/vulnerable servers). Is SSH-1.5-Nmap a login attempt? I don't think so. As we have disabled password authentication, a client can't actually try to do a user/pass login, so what can't happen, isn't. These things show up, but are completely irrelevant to security.
- usr1106 5y ago
  Yes, it has increased quite a bit in recent years. I think still 5 years ago there was basically no traffic.

ahepp 5y ago

Who cares about fail2ban? Unless you're worried about DOS, it seems pretty silly. Similarly with a firewall, assuming you have NAT and a firewall on your router.

acatton 5y ago

Even for 2013, this is not very good advice. fail2ban makes your iptables dynamic, which is a nightmare to audit. This is what I do in 2021: * Set up spiped[1] in front of SSH * Install and setup nftables[2]. * Lock down every service as much as possible in systemd[3]. (If the service is built-in the distro, just use drop in files[4]) [1] https://www.tarsnap.com/spiped.html https://www.tarsnap.com/spiped.html [2] https://wiki.archlinux.org/title/nftables https://wiki.archlinux.org/title/nftables [3] https://ruderich.org/simon/notes/systemd-service-hardening https://ruderich.org/simon/notes/systemd-service-hardening [4] https://wiki.archlinux.org/index.php?title=Systemd&oldid=704440#Drop-in_files https://wiki.archlinux.org/index.php?title=Systemd&oldid=704...

iso1631 5y ago
Why use spiped rather than wireguard?
- acatton 5y ago
  Wireguard is also a good alternative. But you need to connect to a VPN every time you want to SSH. Here is my personal reasons why I use spiped: * spiped is transparent by using ProxyCommand[1]. This allow me to do "ssh host" and thanks to my ssh_config, it just connects. * spiped can be run in a very hardened way.[2] It just needs to listen() to a socket, connect to another one, and access a key file. Wireguard needs complex network access, it needs to create interfaces and open raw sockets. * spiped is much simple to manage, just run a daemon. With wireguards there are two possibilites: ** Every host runs wireguard, you might need to connect to multiple hosts at the time, you need to manage internal IP conflicts, etc... ** One central wireguard server, you have a single point of failure, and can't ssh anywhere if this host is down. Don't get me wrong, I love wireguard, use it all the time as a VPN, but I don't think it's appropriate as a layer of protection in front of my SSH server. Both Wireguard and Spiped are written by very smart people. [1] https://man.openbsd.org/ssh_config#ProxyCommand https://man.openbsd.org/ssh_config#ProxyCommand [2] https://ruderich.org/simon/notes/systemd-service-hardening https://ruderich.org/simon/notes/systemd-service-hardening
  - 5y ago

makach 5y ago

In a professional setting, today, there is much much more that needs to be taken into consideration. Anyone else found it sus that the OP didn't mention backups?

paradaux 5y ago
I usually wait until I have content to backup prior to setting up backups, and definitely wouldn't be in my mind in the "first 5 minutes." If you're concerned you will lose 5 minutes of work make use of snapshotting if available. Of course the first-5-minute title is hyperbolic, but backups are besides the point when you're first setting up and securing a machine.
- Sebb767 5y ago
  On the other hand, backups as an afterthought is what leaves you paying ransoms. I prefer to think about backups directly when setting up the machine, since grouping data directories can help a lot with backup strategies later on. Of course, making sure you're the only one on the machine is step 1, but at least I like to set up backups before placing any serious data on the machine, it's a part of the initial setup for me.
- garmaine 5y ago
  > Of course the first-5-minute title is hyperbolic I don't think it is. I've managed a server directly connected to the internet with a US government IP, and it was being port scanned from a Chinese IP within minutes of being turned on. If you are a target, then there is an adversary out there that is patiently waiting for the opportunity to exploit an unpatched vulnerability in new installs, as if your security is otherwise good it might be how they get their foot in the door on your network. (In our case I really did have a "5 minute plan" to login as soon as the fresh install was booted, setup a firewall, lockdown the ssh server, and install fail2ban ASAP. I'd then check system logs to see if anyone got in before proceeding. Time was of the essence.)
  - tmottabr 5y ago
    no one in that scenario would not do things manually like in the article. but if doing it, then at minimum you should use an custom install media with latest packages bundled and all the configuration already backed so you hit the ground with sane defaults and cover the first 5 minutes from this articles during install time. also in any install i would always do a netinstall to get any updates between media generation and install time, so you should always have the latest and greats at install time.

marvinblum 5y ago

Everytime I read something about Linux server hardening, I get more confused. We're lacking a clear and simple, modern guide on how to do things. I know, every setup is different, but there should at least be consensus for a fresh installation. Also, do I really HAVE to change something so that it is secure? Isn't a Ubuntu server secure out of the box? With a strong, unique root password of course.

sh4un 5y ago
Everyone has an opinion on doing it better, that is why.
- eddieroger 5y ago
  In addition to this, everyone has different requirements. A mix of opinions and a mix of requirements make a real variety of ways of doing it.
DyslexicAtheist 5y ago
> Isn't a Ubuntu server secure out of the box? I think people working at RedHat are more competent in moving security forward on Linux than what Ubuntu does. Ubuntu hardly innovates at all. Its target market seems to be desktop users (or server admins that are only familiar with the Desktop version). Personally I wouldn't put Ubuntu (or any other distribution) on a server without an elaborate playbook to tailor it to my needs (on Ubuntu that playbook is always more complex from my experience). This is where Ubuntu fails for me because it makes some weird assumptions as to what I want in terms of security (which are absent in Debian). YMMV. Although I think that a distribution's goal should be accessibility and configurability - in that regard all of them don't prioritize security features as much as I'd like to see (but knowing myself I probably would complain the second these features become too opinionated - which they most certainly would - which is why I think Debian does the right thing with not making opinionated assumptions). Ubuntu compared to Debian standard install is more bloated, interim releases are much buggier, and Ubuntu LTS is less stable than Debian stable. Ubuntu's root certificate store is constantly outdated (though the same issue might also be on Debian). Their apparmor configuration lags behind, ... whatever is good they usually inherit from Debian. All distributions could do more to lock down processes with seccomp-filters in systemd. Would be interesting to see what lynis⁰ discovers when comparing a fresh server install between Ubuntu and others. In over 20 years I have seen some real shit-shows in production with all distributions except Debian (again ymmv). Jason Donenfeld, the creator of Wireguard said about Ubuntu on the latest¹ SCW podcast: > Ubuntu is always, a horrible distribution to work with, ... > Well, they [Ubuntu] sort of inherit from Debian, but they're like not super tuned in to what's going on and like not really on top of things. And so it was just always, it's still a pain to like make sure Ubuntu is working well. but I don't know, it's not too much interesting to say about the distro story, just open source politics as usual. while somewhat anecdotal I trust that Jason knows what he is talking about having been on the linux security kernel team for ages and familiar with the quirks of various downstream vendors. His development cycle for WG is: implement -> decompile -> formal-verification -> rinse/repeat :-/ All of Linux security is a shit show. This is why grsecurity is charging money for it's service. ⁰ https://cisofy.com/lynis/ https://cisofy.com/lynis/ ¹ https://securitycryptographywhatever.buzzsprout.com/1822302/9667632-wireguard-feat-jason-donenfeld https://securitycryptographywhatever.buzzsprout.com/1822302/...

belter 5y ago

After Security...on Performance, Brendan Gregg First 60 Seconds: https://netflixtechblog.com/linux-performance-analysis-in-60-000-milliseconds-accc10403c55 https://netflixtechblog.com/linux-performance-analysis-in-60...

clay-dreidels 5y ago

Here is my setup, but I’m sure it could use som work. https://lamda-chops.bearblog.dev/steps-i-take-when-setting-up-a-vpn-server-on-digital-ocean/ https://lamda-chops.bearblog.dev/steps-i-take-when-setting-u...

onehair 5y ago
> Enable Automatic Upgrades I can't count with my fingers alone how many times things broke because I `apt upgrade`d without thoroughly reading the change logs. Or even when reading the change logs something still gets past me. Having auto upgrades on a server is not a good idea.
- deadbunny 5y ago
  This is why automatic updates are usually only for security updates, not everything.

Sosh101 5y ago

Do you really need fail2ban if you require keys for ssh login?

rascul 5y ago
It can help cut down on log sizes.
- deadbunny 5y ago
  So does logrotate.

heipei 5y ago

What I'd like to see is a modern guide for setting up and operating a cluster of hosts that does not rely on any specific provider settings. Say you want to run a cluster of Ubuntu servers, maybe exclusively with a workload scheduler like k8s, maybe with a mixture of nodes, how do you set it up securely and consistently, how to you apply updates, provision users, deploy applications, and how do you centrally log and alert on events (systemd logs, docker logs, auditd). Bonus points if you there are pointers about how compliant that setup is wrt to modern compliance requirements. I know it's a lot to ask, but maybe there is such a guide available that does not just fall back to talking about provider-specific features (e.g. IAM).

idoubtit 5y ago
ansible. And similar tools, but lack experience outside of ansible. You can see Ansible a a tool to automate the install/configure/update process that you would do manually on a single server. Then you can apply this "playbook" to any server. You just need an ssh connection to the target servers, with python installed on them. Of course, you have to write rules for setting up a server, provisioning users, monitoring (deploying Prometheus, pushing logs to a central server...). There various plugins for integrating with providers, but the basic features are provider independent. Ansible is far from perfect (dependency on python, inconsistent syntax, abuse of aliases, missing a strict mode...), but it's rather easy to learn and I've used it successfully (at a small scale).

errcorrectcode 5y ago

Naïve, black-&-white, over-simplified, amateurish advice. Security measures to deploy depend on the risk level involved, e.g., potential costs of being hacked. Measures like SELinux, grsec, fwknop, snort, IDS (tripwire or Samhain), HSMs, hw entropy sources, split SSH/TLS, and microservices compartmentalized into VM containers rather than Docker have their place.