5 ms·

I still wonder (and this one of many reason I'm not using CoreOS, because I don't now how to secure it as good as Ubuntu/Debian) how do you install a simple Fa

I still wonder (and this one of many reason I'm not using CoreOS, because I don't now how to secure it as good as Ubuntu/Debian) how do you install a simple Fail2Ban (ssh) implementation on CoreOS? Are there any practices how to make the server which runs all containers more secure? Or do I simply rely on CoreOS team for all that ssh securing and firewall stuff (it seams it is this way now!) ? I have the feeling that hardening a CoreOS server is not the main priority for CoreOS.

the_angry_angel 10y ago
I believe the point is that you have CoreOS hidden behind a firewall you can control, therefore you only have ssh open to sources you permit, you use a bastion host, etc.
- therealmarv 10y ago
  So a Ubuntu server in front of CoreOS works... :D ;) I'm thinking of a small cloud setup like on DigitalOcean were you need ssh into the CoreOS server.
lima 10y ago
There's no point in using fail2ban if you use public key authentication. It's a kludge which helps against brute force attacks, which are limited to password login. Since CoreOS enforces public key authentication, and the SSH server is the only external attack surface, they're doing just fine. Security is definitely a top priority for CoreOS, and they're doing a great job at it - image signing, Docker container signing, ASLR binaries, a clean build system (based on Gentoo)...
- vidarh 10y ago
  I tend to agree. One reason to use something like it, though, is that it clogs up logs. But I'd tend to want to tie it down with iptables and limited IP ranges and/or just move ssh to another port (the latter as a quick and dirty way to avoid the steady annoying log nonsense if you for whatever reason can't limit ip ranges, not because I have any illusions it does much for security)
  - ultramancool 10y ago
    When filtering your logs, filter for only successful attempts. You really don't care how many people failed to login on a server with only pubkey enabled. It's the ones who succeeded you want to know about.
    - vidarh 10y ago
      Exactly, but it's nice to be able to follow your logs without all that clutter. Though with journald I mostly look at logs on a per-unit basis anyway (or via e.g. an ELK setup), so it's a slight nuisance rather than something essential.
- therealmarv 10y ago
  After rethinking... I also tend to agree. Actually I never use password login, maybe it felt safer without brute-force logging of ssh logins and I also think that a rate limit would not hurt, especially when there is a SSH security issue like here ;) Answering my own question (it was a longer time ago I used CoreOS) here two interesting links: Securing CoreOS with iptables rules (Firewall like): https://www.jimmycuadra.com/posts/securing-coreos-with-iptables/ https://www.jimmycuadra.com/posts/securing-coreos-with-iptab... IP tables rule to reduce ssh attack spam: http://serverfault.com/a/217066 http://serverfault.com/a/217066 Maybe this advices should be integrated into CoreOS official docs? Something like "How to harden CoreOS servers". When I first looked into CoreOS I wanted to use familiar tools like fail2ban or ufw to run there (which is not possible, at least not easy). I'm no iptables expert (who is? I think it's not a trivial to use program) and this lack of admin tools made me think it's not good to secure. It's for sure more complicated than in Ubuntu (from an admin perspective).
- ams6110 10y ago
  fail2ban can protect other services besides ssh.
- xorcist 10y ago
  The purpose is also to be able to keep a sane amount of logging.
- dbalan 10y ago
  As another used pointed out, fail2ban is a generic tool, I don't necessarily agree with the method its implemented though there is a recent module in iptables that should be enough for most services. Also, about SSH - it is still better to block the bad actors before you process the request. I've seen occurances where the network was saturated bots trying to brute force.
vidarh 10y ago
CoreOS is basically a trimmed down, customized Gentoo. It's not hard installing custom stuff on CoreOS, just be aware that CoreOS blows away a lot of the system directories on update, so you don't want to rely on putting things in /bin, /sbin etc. /etc/ survives, though, so you can easily find examples of systemd service files to apply iptables rules, sysctl hardening etc, and I'm sure you can find fail2ban setups too.. It is also fairly trivial to put together a service file that'll run some arbitrary script you can put wherever you like to apply additional changes on boot. Systemd will let you apply capability based restrictions etc. to fail2ban too. CoreOS service files do get replaced, but you can override values in them using the systemd dropin mechanism to e.g. override settings for the ssh server. Your main limitation is that CoreOS itself is intentionally very sparse. You have two options there: Deploy whatever dependencies you like or package up the code you need in a Docker or Rocket container, and run it with sufficient privileges if you want it to be able to apply changes to the host (this will work for /etc/hosts.deny based restrictions but not for applying changes to iptables, though it'd be easy enough to output changes to a file on the host and use a script on the host to apply those to iptables). To be clear: CoreOS is not effort free. It makes very explicit value judgements, such as aiming for as much as possible to run in containers, that requires extra effort initially. If you are not willing to take extra effort to keep the "outside" host as clean as possible, CoreOS may not be the right choice for you. Most of the return on that investment first becomes noticeable when you're deploying larger numbers of servers/instances, and want to update them. With respect to keeping it secure, note that the default behaviour for CoreOS is that all installed servers will check in with the CoreOS update servers regularly, and will apply the updates and will reboot once the updates have been applied. This may or may not be what you want (it's easy to change - check the docs), but it means that if you're not paying attention, security issues like in this article will get patched for you. Doesn't mean you shouldn't pay attention, but the window for anything bad to happen is smaller. If you are paying attention you can change the reboot policies so you can control when they happen, but if possible I'd recommend not to, as you can also make the system take a lock in etcd so only a limited number of your machines will reboot at any one point in time (or you can reduce the number of available locks to explicitly prevent reboots at times when it might interfere with something else, or set specific machines to manual updates only). At least for anything running the alpha I'd very much suggest leaving it set to automatic reboot/updates to ensure stuff gets patched quickly when they find anything serious (not just security stuff). Since you mentioned DigitalOcean elsewhere, DO's CoreOS images do retain the default of automatic reboots.
blorgle 10y ago
I personally don't think iptables or switching ports is a viable solution and always use the following policy: All the internal services should never be exposed to the internet and only accept connections from signed packets using IPsec or OpenVPN with TLS auth. Yes, this means more key management.
- kchoudhu 10y ago
  Oh, hooray. I'm not the only person who runs openvpn to create internal overlay networks.
- therealmarv 10y ago
  Interesting approach. I only think that OpenVPN is not more secure than SSH, somewhere needs to be access point/loophole and SSH is pretty good tested for this. It also does not solve the problem when using a single CoreOS server where you need to run OpenVPN in a container (and I'm also not sure if you can access the host from this container).
  - vidarh 10y ago
    You can certainly give the container access to the host a number of ways. For something like OpenVPN --net=host passed to Docker will give OpenVPN full access to the host network. I agree with you - ssh is fine. If you have multiple CoreOS boxes somewhere without a secure private network, though, OpenVPN, PeerVPN or similar solution works fine. If you couple it with Flannel set to use host routing, you can give all containers their own IP addresses on non-colliding IP ranges (Flannel takes are of coordinating that via etcd) and Flannel doesn't add (in the host routing variant) extra overhead as it just adds suitable routes on each server. You can set this up a in few different ways: CoreOS provides Flannel coupled with an "early" Docker daemon (so you'll have two) to run stuff that needs to run before the "real" Docker daemon, such as to set up a VPN etc. You could also use Rocket/ACI containers, or run it outside a container. Alternatively newer versions of Docker supports network plugins, though I've not yet had time to test this with CoreOS as I already have working VPN setups based on Flannel + early-docker.
  - blorgle 10y ago
robszumski 10y ago
There is a hardening guide in the docs [1] and it would be great if you have ideas/contributions on how to make it more complete. Right now it is mostly focused on reducing remote services. [1] https://coreos.com/os/docs/latest/coreos-hardening-guide.html https://coreos.com/os/docs/latest/coreos-hardening-guide.htm...
meta_AU 10y ago
Have a look at ssh-guard, there is a Fleet example for deploying it that I've been using.

the_angry_angel 10y ago

I believe the point is that you have CoreOS hidden behind a firewall you can control, therefore you only have ssh open to sources you permit, you use a bastion host, etc.

therealmarv 10y ago
So a Ubuntu server in front of CoreOS works... :D ;) I'm thinking of a small cloud setup like on DigitalOcean were you need ssh into the CoreOS server.

lima 10y ago

There's no point in using fail2ban if you use public key authentication. It's a kludge which helps against brute force attacks, which are limited to password login. Since CoreOS enforces public key authentication, and the SSH server is the only external attack surface, they're doing just fine. Security is definitely a top priority for CoreOS, and they're doing a great job at it - image signing, Docker container signing, ASLR binaries, a clean build system (based on Gentoo)...

vidarh 10y ago
I tend to agree. One reason to use something like it, though, is that it clogs up logs. But I'd tend to want to tie it down with iptables and limited IP ranges and/or just move ssh to another port (the latter as a quick and dirty way to avoid the steady annoying log nonsense if you for whatever reason can't limit ip ranges, not because I have any illusions it does much for security)
- ultramancool 10y ago
  When filtering your logs, filter for only successful attempts. You really don't care how many people failed to login on a server with only pubkey enabled. It's the ones who succeeded you want to know about.
  - vidarh 10y ago
    Exactly, but it's nice to be able to follow your logs without all that clutter. Though with journald I mostly look at logs on a per-unit basis anyway (or via e.g. an ELK setup), so it's a slight nuisance rather than something essential.
therealmarv 10y ago
After rethinking... I also tend to agree. Actually I never use password login, maybe it felt safer without brute-force logging of ssh logins and I also think that a rate limit would not hurt, especially when there is a SSH security issue like here ;) Answering my own question (it was a longer time ago I used CoreOS) here two interesting links: Securing CoreOS with iptables rules (Firewall like): https://www.jimmycuadra.com/posts/securing-coreos-with-iptables/ https://www.jimmycuadra.com/posts/securing-coreos-with-iptab... IP tables rule to reduce ssh attack spam: http://serverfault.com/a/217066 http://serverfault.com/a/217066 Maybe this advices should be integrated into CoreOS official docs? Something like "How to harden CoreOS servers". When I first looked into CoreOS I wanted to use familiar tools like fail2ban or ufw to run there (which is not possible, at least not easy). I'm no iptables expert (who is? I think it's not a trivial to use program) and this lack of admin tools made me think it's not good to secure. It's for sure more complicated than in Ubuntu (from an admin perspective).
ams6110 10y ago
fail2ban can protect other services besides ssh.
xorcist 10y ago
The purpose is also to be able to keep a sane amount of logging.
dbalan 10y ago
As another used pointed out, fail2ban is a generic tool, I don't necessarily agree with the method its implemented though there is a recent module in iptables that should be enough for most services. Also, about SSH - it is still better to block the bad actors before you process the request. I've seen occurances where the network was saturated bots trying to brute force.

vidarh 10y ago

CoreOS is basically a trimmed down, customized Gentoo. It's not hard installing custom stuff on CoreOS, just be aware that CoreOS blows away a lot of the system directories on update, so you don't want to rely on putting things in /bin, /sbin etc. /etc/ survives, though, so you can easily find examples of systemd service files to apply iptables rules, sysctl hardening etc, and I'm sure you can find fail2ban setups too.. It is also fairly trivial to put together a service file that'll run some arbitrary script you can put wherever you like to apply additional changes on boot. Systemd will let you apply capability based restrictions etc. to fail2ban too. CoreOS service files do get replaced, but you can override values in them using the systemd dropin mechanism to e.g. override settings for the ssh server. Your main limitation is that CoreOS itself is intentionally very sparse. You have two options there: Deploy whatever dependencies you like or package up the code you need in a Docker or Rocket container, and run it with sufficient privileges if you want it to be able to apply changes to the host (this will work for /etc/hosts.deny based restrictions but not for applying changes to iptables, though it'd be easy enough to output changes to a file on the host and use a script on the host to apply those to iptables). To be clear: CoreOS is not effort free. It makes very explicit value judgements, such as aiming for as much as possible to run in containers, that requires extra effort initially. If you are not willing to take extra effort to keep the "outside" host as clean as possible, CoreOS may not be the right choice for you. Most of the return on that investment first becomes noticeable when you're deploying larger numbers of servers/instances, and want to update them. With respect to keeping it secure, note that the default behaviour for CoreOS is that all installed servers will check in with the CoreOS update servers regularly, and will apply the updates and will reboot once the updates have been applied. This may or may not be what you want (it's easy to change - check the docs), but it means that if you're not paying attention, security issues like in this article will get patched for you. Doesn't mean you shouldn't pay attention, but the window for anything bad to happen is smaller. If you are paying attention you can change the reboot policies so you can control when they happen, but if possible I'd recommend not to, as you can also make the system take a lock in etcd so only a limited number of your machines will reboot at any one point in time (or you can reduce the number of available locks to explicitly prevent reboots at times when it might interfere with something else, or set specific machines to manual updates only). At least for anything running the alpha I'd very much suggest leaving it set to automatic reboot/updates to ensure stuff gets patched quickly when they find anything serious (not just security stuff). Since you mentioned DigitalOcean elsewhere, DO's CoreOS images do retain the default of automatic reboots.

blorgle 10y ago

I personally don't think iptables or switching ports is a viable solution and always use the following policy: All the internal services should never be exposed to the internet and only accept connections from signed packets using IPsec or OpenVPN with TLS auth. Yes, this means more key management.

kchoudhu 10y ago
Oh, hooray. I'm not the only person who runs openvpn to create internal overlay networks.
therealmarv 10y ago
Interesting approach. I only think that OpenVPN is not more secure than SSH, somewhere needs to be access point/loophole and SSH is pretty good tested for this. It also does not solve the problem when using a single CoreOS server where you need to run OpenVPN in a container (and I'm also not sure if you can access the host from this container).
- vidarh 10y ago
  You can certainly give the container access to the host a number of ways. For something like OpenVPN --net=host passed to Docker will give OpenVPN full access to the host network. I agree with you - ssh is fine. If you have multiple CoreOS boxes somewhere without a secure private network, though, OpenVPN, PeerVPN or similar solution works fine. If you couple it with Flannel set to use host routing, you can give all containers their own IP addresses on non-colliding IP ranges (Flannel takes are of coordinating that via etcd) and Flannel doesn't add (in the host routing variant) extra overhead as it just adds suitable routes on each server. You can set this up a in few different ways: CoreOS provides Flannel coupled with an "early" Docker daemon (so you'll have two) to run stuff that needs to run before the "real" Docker daemon, such as to set up a VPN etc. You could also use Rocket/ACI containers, or run it outside a container. Alternatively newer versions of Docker supports network plugins, though I've not yet had time to test this with CoreOS as I already have working VPN setups based on Flannel + early-docker.
- blorgle 10y ago

robszumski 10y ago

There is a hardening guide in the docs [1] and it would be great if you have ideas/contributions on how to make it more complete. Right now it is mostly focused on reducing remote services. [1] https://coreos.com/os/docs/latest/coreos-hardening-guide.html https://coreos.com/os/docs/latest/coreos-hardening-guide.htm...

meta_AU 10y ago

Have a look at ssh-guard, there is a Fleet example for deploying it that I've been using.