30 ms·

As a *nix sysadmin since before Google was a thing, I find articles like this frustrating and arguably dangerous. They lay out a set of "rules" the author has

As a *nix sysadmin since before Google was a thing, I find articles like this frustrating and arguably dangerous. They lay out a set of "rules" the author has collated as a dogmatic doctrine that only a fool would not follow. But they provide no "why", take absolutely no context in to account, and talk only about their perceived upsides. Anyone reading this should take each point as "here's a possible thing you should go and research yourself", because there are consequences to most of these rules. I'm not saying anything they've written is wrong (though there is a lot of unsubstantiated opinion there), just that you should learn what any of these changes truly do before implementing them AND that your particular context is really important. Edit: There is a disclaimer at the top which mustn't be forgotten while reading. > DISCLAIMER: Do not attempt to apply anything in this article if you do not know exactly what you are doing. This guide is focused purely on security and privacy, not performance, usability, or anything else. I do appreciate people putting work into producing this sort of content, but I think the article could be improved by phrasing the various steps as suggestions and perhaps linking out to more detailed documentation on the "why" elsewhere.

quijoteuniv 6y ago
Maybe Simonjgreen can help improve the article? Seems like a relevant topic to me. Glad people put work and share on hjis subject.
pjmlp 6y ago
Before Google was a thing, UNIX security was already something to worry about, hence why Tru64 was created, or HP-UX introduced vaults.
- 867-5309 6y ago
  and just before that cavemen invented shift rotation to defend the entrance to their dwelling
devwastaken 6y ago
The problem I find is that those who know this tend to not spend the time going into detail on what you should actually do.
schoen 6y ago
Yes, and there's a clear mix of threat models (for instance, sometimes the guide seems to be emphasizing Whonix-style user anonymity, but other times something else entirely). I see some very interesting things to go learn about in this guide, but as you note, it's not really presented that way. I would find it more useful if it either picked a more specific threat model and stuck to it, or suggested a particular system that has already made opinionated choices on some of these protection mechanisms (maybe like Whonix, Tails, or Qubes), or invited a discussion of "for each of these options, why isn't it already the default everywhere? did some people find the associated threats too exotic, unlikely, or irrelevant? are there harsh tradeoffs here?". One example for me is the ptrace restrictions. It's clearly true that attackers can use ptrace to attach to a running process to steal secrets (if you are relying on processes as a security boundary, this feature will tend to pierce it), which was the main motivation for the kernel having an option to restrict it. On the other hand, if you're a software developer or security researcher dealing with binary executables, ptrace is very helpful to you and you may be irritated if you turn it off. That might be a sign that there should be different OS default profiles for "R&D" and for "production server" or "dedicated administrative workstation" (or maybe not), and perhaps if you don't know what ptrace is you should default to disabling it, but it still strikes me the wrong way not to see any context about this.
Erlich_Bachman 6y ago
Yeah, that's to put it mildly. This gem is right up there in the first page: Use a distribution with an init system other than systemd. systemd contains a lot of unnecessary attack surface; it attempts to do far more things than necessary and goes beyond what an init system should do. An init system should not need many lines of code to function properly. Wow! To put such a controversial feature, and present it as "this is the truth, just how it is", without even mentioning that it's just an opinion and a lot of people have a completely opposite one, and both for good reasons. That just makes me distrust this guide because it is so opinionated and subjective.
- sliken 6y ago
  Systemd is a security nightmare, a huge amount of code, taking over various network services ... poorly. Doesn't seem controversial at all to me. Gives me the heebie jeebies to think of systemd talking to the network before I even have the client firewall up. Generally if security is your top priority I'd disable as much of systemd as possible. Things like find a good local resolver (like unbound) and disable systemd's implementation, etc.
  - tinco 6y ago
    I'd like to read a guide on hardening systemd. Systemd is really convenient and I hadn't realized it poses a large attack surface. A good list with possible attack vectors and how to mitigate them would be epic. Is it really an established idea in the security community that systemd is a security nightmare?
    - tarruda 6y ago
      > Is it really an established idea in the security community that systemd is a security nightmare? Here's a list of systemd-specific CVEs over the years: https://www.cvedetails.com/product/38088/Freedesktop-Systemd.html?vendor_id=7971 https://www.cvedetails.com/product/38088/Freedesktop-Systemd... Note that these apply to all programs in systemd distribution (not sure how many of these are specific to systemd init program). It seems that a lot of criticism is due systemd doing more than a classic init process does, but some of it seem unjustified since you it is not an "all or nothing" package. For example, this specific CVE affected systemd-resolved (systemd's DNS daemon), but IIRC Debian was not affected since it did not use systemd-resolved by default: https://www.cvedetails.com/cve/CVE-2017-15908/ https://www.cvedetails.com/cve/CVE-2017-15908/ So it is perfectly fine for a distribution use systemd init system but use other network daemons.
someRandoJunk 6y ago
To be fair I can't blame the author. The why part probably doesn't exist because he wrote it for himself, or, he wrote it for people who he knows and would be able to seperate what advice to follow for their use-case. This is just an assumption on my part, but I feel like he just wrote down a list of what he has followed down in the past. He didn't write down the "why" because he wrote it in context of his own experiences. Like that systemd portion looks very icky, but when you have the disclaimer in context. You'd want to go to something with the least amount of attack surface. Systemd is a giant project, and a very useful one at that, and for context I am firm supporter of systemd, but I can see why he'd put that in the guide, since his disclaimer says the guide isn't really about usability but minimizing the attack surface. In turn it got submitted to hackernews by someone who read their article, and unfortunately some people would treat it as gospel and turn off their brains, since as you mentioned there's no "Why?", but it makes sense to not include it if he didn't really write it for the masses but just to document something for himself and his friends/colleagues.
- z3t4 6y ago
  We need to be better at being suspicious about any information. And do our own research and experimentation. But who has the time for that in this day and era when information is expected to be received by push notification and most people can't comprehend messages longer then an SMS.
- p410n3 6y ago
  Can confirm. So many people asked him to write a hardening guide he just ended up doing it.
asymptotically4 6y ago
The recommendation for using LibreSSL is even worse. The fork started off okay, but is now far behind and totally useless.
- tjoff 6y ago
  Do you know of any resource to read up on this?
  - asymptotically4 6y ago
    https://archives.gentoo.org/gentoo-dev/message/9a92320c599e63c8c18b2ed29050f22f https://archives.gentoo.org/gentoo-dev/message/9a92320c599e6...
- voltagex_ 6y ago
  Microsoft is using LibreSSL in the OpenSSH Windows port - got any more info?
  - anonunivgrad 6y ago
    Do you have a source for that?
    - voltagex_ 6y ago
      https://github.com/PowerShell/Win32-OpenSSH/releases https://github.com/PowerShell/Win32-OpenSSH/releases
TheOperator 6y ago
My first mistake as a green "jack of all trades" admin was getting scared and feeling like I had to apply a "Hardening guide". Wasted so much time and energy because I didn't know how to determine what was a priority, both when it came to business needs and security. Guides like this presume you have generous amounts of time to put into a Linux hardening project with a questionable ROI when the person looking at it might be a solo overwhelmed admin in a security insensitive line of work. Fact of the matter is that you can't take a broad checklist approach to security because you can't secure everything, you have to have an idea of where your biggest risks are and focus on those. If the computers in question aren't dealing with PII which one has some moral duty to protect, security risks can be outweighed by other business risks, and going on a security crusade can INCREASE business risk. Sometimes things that might be security risks are mitigated by other aspects of IT security or non-IT controls making them very low priority. Sometimes hardening certain things will have an especially adverse not worth it effect on certain users/applications (which is why these hardening settings aren't defaults). Maybe worry about phishing attacks, that your authentication strategy makes sense, that your firewall is intact before worrying about arcane aspects of Linux Hardening. A guide like this is fairly useless in the real world without some discussion of the risk/benefits of hardening something and some mechanism to prioritize. To me this guide serves as an interesting starting point to constructing a practical hardening strategy, but I would advise against following it blindly. If you don't know what you're doing I would trust that Linux has fairly sane defaults and focus elsewhere to start with. Configuring your systems in non-standard ways is a Pandora's box of headaches.
- voltagex_ 6y ago
  I mean hell, in a corporate network where you might want to look at hardening the configuration you roll out to your machines, you're more likely to have a hardware firewall and a VPN requirement before you even get to your "squishy" Linux box.
- zorked 6y ago
  A lot of this hardening is based on a fairly obsolete Unix view of the world. E.g. preventing users from seeing other users' PIDs, changing /etc/securetty, non-root X server... In real life, nowadays, a single system is likely to be running a single application, and once that's owned there's very little of the rest of the system to take over. Owned a Web app with access to user accounts? That's enough to steal all the credit cards flowing through it, and spy/scam all the users in it. Becoming root, or peeking into other users, is usually unimportant because there's nothing special happening there. I guess Kubernetes is the one thing that sort-of brings back these concerns, but even that has its own more modern particularities (e.g. the guide doesn't even mention cgroups).
hxtk 6y ago
A lot of what they talk about also conflates security with anonymity. While it may be good advice for connecting to coffee shop WiFi on your laptop, for a secure network, being able to tell who performed an action is less of a security hazard and more like a security requirement.
LinuxBender 6y ago
I agree with this and would add that some of the hardening they are suggesting will make debugging problems much harder. This article assumes that if you are doing this in a corporate or datacenter environment, that all of your engineers have vast experience. Even then, some of the tweaks mentioned will require making changes and rebooting to debug some problems. Not only should people read up on and understand the implications of each change, but also do "battle testing" before doing this in production. That means going beyond functional testing you might get from a CI/CD environment and instead letting this bake in a staging environment for a few months first. It needs to be an environment with real traffic, real workloads, but low service level agreement and low revenue impact for an extended period of time. As an aside, I can tell they have not tested some of the things they are suggesting. The first obvious one that pops out is the changes to /proc in fstab. That won't do anything on 100% of linux distros, as /proc gets mounted by the init process. That has to be done in a init script as a "remount". Also, the "ipv6.disable=1" in the kernel options is not advised. Do this in sysctl and in the network scripts, or some things may break depending on how your system is being used. The ipv6 module is used for more than ipv6. There are also some kernel options that change in kernel 5.x that this article does not mention. The ulimit settings will not apply to systemd processes and will not apply to root. They mention disabling debugfs, but not binfmt. If I had to pick a more dangerous thing to disable it would be binfmt, which brings in dangerous concepts from Windows 95/98. I need to stop reading this or I will spend all day writing notes on the gotchas.
- blodorn 6y ago
  > The ipv6 module is used for more than ipv6. Would you tell me why the ipv6 module used for more than ipv6? If you don't want something running, disabling it at the kernel level makes sense to me, so I am curious as to why that is not the case. If things don't work with it disabled in the kernel, but DO work with it disabled via sysctl, what is actually being disabled?
  - LinuxBender 6y ago
    One example would be other kernel modules that require it as a dependency. So if you disable loading the ipv6 module, you can no longer load any module that references it, even if you would not have been using ipv6 in that module. The first one that pops into mind is the sctp kernel module. This also used to prevent loading the bonding module but I am not sure if that is still the case. You might try testing that. Newer kernels have the ipv6 code built into the kernel, so the kernel boot option to disable ipv6 won't do anything regardless. The supported way to disable it is sysctl and the networking scripts.
zests 6y ago
Your disclaimer applies to every article ever. I agree 100% but there’s nothing really specific to this article that makes me worry more. Its just common sense.
fit2rule 6y ago
One old sysop to another: the cargo cults rise and fall.

quijoteuniv 6y ago

Maybe Simonjgreen can help improve the article? Seems like a relevant topic to me. Glad people put work and share on hjis subject.

pjmlp 6y ago

Before Google was a thing, UNIX security was already something to worry about, hence why Tru64 was created, or HP-UX introduced vaults.

867-5309 6y ago
and just before that cavemen invented shift rotation to defend the entrance to their dwelling

devwastaken 6y ago

The problem I find is that those who know this tend to not spend the time going into detail on what you should actually do.

schoen 6y ago

Yes, and there's a clear mix of threat models (for instance, sometimes the guide seems to be emphasizing Whonix-style user anonymity, but other times something else entirely). I see some very interesting things to go learn about in this guide, but as you note, it's not really presented that way. I would find it more useful if it either picked a more specific threat model and stuck to it, or suggested a particular system that has already made opinionated choices on some of these protection mechanisms (maybe like Whonix, Tails, or Qubes), or invited a discussion of "for each of these options, why isn't it already the default everywhere? did some people find the associated threats too exotic, unlikely, or irrelevant? are there harsh tradeoffs here?". One example for me is the ptrace restrictions. It's clearly true that attackers can use ptrace to attach to a running process to steal secrets (if you are relying on processes as a security boundary, this feature will tend to pierce it), which was the main motivation for the kernel having an option to restrict it. On the other hand, if you're a software developer or security researcher dealing with binary executables, ptrace is very helpful to you and you may be irritated if you turn it off. That might be a sign that there should be different OS default profiles for "R&D" and for "production server" or "dedicated administrative workstation" (or maybe not), and perhaps if you don't know what ptrace is you should default to disabling it, but it still strikes me the wrong way not to see any context about this.

Erlich_Bachman 6y ago

Yeah, that's to put it mildly. This gem is right up there in the first page: Use a distribution with an init system other than systemd. systemd contains a lot of unnecessary attack surface; it attempts to do far more things than necessary and goes beyond what an init system should do. An init system should not need many lines of code to function properly. Wow! To put such a controversial feature, and present it as "this is the truth, just how it is", without even mentioning that it's just an opinion and a lot of people have a completely opposite one, and both for good reasons. That just makes me distrust this guide because it is so opinionated and subjective.

sliken 6y ago
Systemd is a security nightmare, a huge amount of code, taking over various network services ... poorly. Doesn't seem controversial at all to me. Gives me the heebie jeebies to think of systemd talking to the network before I even have the client firewall up. Generally if security is your top priority I'd disable as much of systemd as possible. Things like find a good local resolver (like unbound) and disable systemd's implementation, etc.
- tinco 6y ago
  I'd like to read a guide on hardening systemd. Systemd is really convenient and I hadn't realized it poses a large attack surface. A good list with possible attack vectors and how to mitigate them would be epic. Is it really an established idea in the security community that systemd is a security nightmare?
  - tarruda 6y ago
    > Is it really an established idea in the security community that systemd is a security nightmare? Here's a list of systemd-specific CVEs over the years: https://www.cvedetails.com/product/38088/Freedesktop-Systemd.html?vendor_id=7971 https://www.cvedetails.com/product/38088/Freedesktop-Systemd... Note that these apply to all programs in systemd distribution (not sure how many of these are specific to systemd init program). It seems that a lot of criticism is due systemd doing more than a classic init process does, but some of it seem unjustified since you it is not an "all or nothing" package. For example, this specific CVE affected systemd-resolved (systemd's DNS daemon), but IIRC Debian was not affected since it did not use systemd-resolved by default: https://www.cvedetails.com/cve/CVE-2017-15908/ https://www.cvedetails.com/cve/CVE-2017-15908/ So it is perfectly fine for a distribution use systemd init system but use other network daemons.

someRandoJunk 6y ago

To be fair I can't blame the author. The why part probably doesn't exist because he wrote it for himself, or, he wrote it for people who he knows and would be able to seperate what advice to follow for their use-case. This is just an assumption on my part, but I feel like he just wrote down a list of what he has followed down in the past. He didn't write down the "why" because he wrote it in context of his own experiences. Like that systemd portion looks very icky, but when you have the disclaimer in context. You'd want to go to something with the least amount of attack surface. Systemd is a giant project, and a very useful one at that, and for context I am firm supporter of systemd, but I can see why he'd put that in the guide, since his disclaimer says the guide isn't really about usability but minimizing the attack surface. In turn it got submitted to hackernews by someone who read their article, and unfortunately some people would treat it as gospel and turn off their brains, since as you mentioned there's no "Why?", but it makes sense to not include it if he didn't really write it for the masses but just to document something for himself and his friends/colleagues.

z3t4 6y ago
We need to be better at being suspicious about any information. And do our own research and experimentation. But who has the time for that in this day and era when information is expected to be received by push notification and most people can't comprehend messages longer then an SMS.
p410n3 6y ago
Can confirm. So many people asked him to write a hardening guide he just ended up doing it.

asymptotically4 6y ago

The recommendation for using LibreSSL is even worse. The fork started off okay, but is now far behind and totally useless.

tjoff 6y ago
Do you know of any resource to read up on this?
- asymptotically4 6y ago
  https://archives.gentoo.org/gentoo-dev/message/9a92320c599e63c8c18b2ed29050f22f https://archives.gentoo.org/gentoo-dev/message/9a92320c599e6...
voltagex_ 6y ago
Microsoft is using LibreSSL in the OpenSSH Windows port - got any more info?
- anonunivgrad 6y ago
  Do you have a source for that?
  - voltagex_ 6y ago
    https://github.com/PowerShell/Win32-OpenSSH/releases https://github.com/PowerShell/Win32-OpenSSH/releases

TheOperator 6y ago

My first mistake as a green "jack of all trades" admin was getting scared and feeling like I had to apply a "Hardening guide". Wasted so much time and energy because I didn't know how to determine what was a priority, both when it came to business needs and security. Guides like this presume you have generous amounts of time to put into a Linux hardening project with a questionable ROI when the person looking at it might be a solo overwhelmed admin in a security insensitive line of work. Fact of the matter is that you can't take a broad checklist approach to security because you can't secure everything, you have to have an idea of where your biggest risks are and focus on those. If the computers in question aren't dealing with PII which one has some moral duty to protect, security risks can be outweighed by other business risks, and going on a security crusade can INCREASE business risk. Sometimes things that might be security risks are mitigated by other aspects of IT security or non-IT controls making them very low priority. Sometimes hardening certain things will have an especially adverse not worth it effect on certain users/applications (which is why these hardening settings aren't defaults). Maybe worry about phishing attacks, that your authentication strategy makes sense, that your firewall is intact before worrying about arcane aspects of Linux Hardening. A guide like this is fairly useless in the real world without some discussion of the risk/benefits of hardening something and some mechanism to prioritize. To me this guide serves as an interesting starting point to constructing a practical hardening strategy, but I would advise against following it blindly. If you don't know what you're doing I would trust that Linux has fairly sane defaults and focus elsewhere to start with. Configuring your systems in non-standard ways is a Pandora's box of headaches.

voltagex_ 6y ago
I mean hell, in a corporate network where you might want to look at hardening the configuration you roll out to your machines, you're more likely to have a hardware firewall and a VPN requirement before you even get to your "squishy" Linux box.
zorked 6y ago
A lot of this hardening is based on a fairly obsolete Unix view of the world. E.g. preventing users from seeing other users' PIDs, changing /etc/securetty, non-root X server... In real life, nowadays, a single system is likely to be running a single application, and once that's owned there's very little of the rest of the system to take over. Owned a Web app with access to user accounts? That's enough to steal all the credit cards flowing through it, and spy/scam all the users in it. Becoming root, or peeking into other users, is usually unimportant because there's nothing special happening there. I guess Kubernetes is the one thing that sort-of brings back these concerns, but even that has its own more modern particularities (e.g. the guide doesn't even mention cgroups).

hxtk 6y ago

A lot of what they talk about also conflates security with anonymity. While it may be good advice for connecting to coffee shop WiFi on your laptop, for a secure network, being able to tell who performed an action is less of a security hazard and more like a security requirement.

LinuxBender 6y ago

I agree with this and would add that some of the hardening they are suggesting will make debugging problems much harder. This article assumes that if you are doing this in a corporate or datacenter environment, that all of your engineers have vast experience. Even then, some of the tweaks mentioned will require making changes and rebooting to debug some problems. Not only should people read up on and understand the implications of each change, but also do "battle testing" before doing this in production. That means going beyond functional testing you might get from a CI/CD environment and instead letting this bake in a staging environment for a few months first. It needs to be an environment with real traffic, real workloads, but low service level agreement and low revenue impact for an extended period of time. As an aside, I can tell they have not tested some of the things they are suggesting. The first obvious one that pops out is the changes to /proc in fstab. That won't do anything on 100% of linux distros, as /proc gets mounted by the init process. That has to be done in a init script as a "remount". Also, the "ipv6.disable=1" in the kernel options is not advised. Do this in sysctl and in the network scripts, or some things may break depending on how your system is being used. The ipv6 module is used for more than ipv6. There are also some kernel options that change in kernel 5.x that this article does not mention. The ulimit settings will not apply to systemd processes and will not apply to root. They mention disabling debugfs, but not binfmt. If I had to pick a more dangerous thing to disable it would be binfmt, which brings in dangerous concepts from Windows 95/98. I need to stop reading this or I will spend all day writing notes on the gotchas.

blodorn 6y ago
> The ipv6 module is used for more than ipv6. Would you tell me why the ipv6 module used for more than ipv6? If you don't want something running, disabling it at the kernel level makes sense to me, so I am curious as to why that is not the case. If things don't work with it disabled in the kernel, but DO work with it disabled via sysctl, what is actually being disabled?
- LinuxBender 6y ago
  One example would be other kernel modules that require it as a dependency. So if you disable loading the ipv6 module, you can no longer load any module that references it, even if you would not have been using ipv6 in that module. The first one that pops into mind is the sctp kernel module. This also used to prevent loading the bonding module but I am not sure if that is still the case. You might try testing that. Newer kernels have the ipv6 code built into the kernel, so the kernel boot option to disable ipv6 won't do anything regardless. The supported way to disable it is sysctl and the networking scripts.

zests 6y ago

Your disclaimer applies to every article ever. I agree 100% but there’s nothing really specific to this article that makes me worry more. Its just common sense.

fit2rule 6y ago

One old sysop to another: the cargo cults rise and fall.