12 ms·

Show HN: A proactive way of securing your Linux server

gregod 6y ago
If you do not need the more granular firewall configuration options there is also classic port knocking (https://en.wikipedia.org/wiki/Port_knocking https://en.wikipedia.org/wiki/Port_knocking) where the daemon sits behind the firewall so all ports can be closed by default.
geocrasher 6y ago
This is interesting in that it combines the concept of port knocking with a REST interface, which I'm assuming is up to the user to create a front end for. Unfotunately it also relies on Kubernetes which means that using it for a single system isn't practical. At least, not for this server owner. My own approach is simply security by obscurity (a non standard port) with APF/BFD doing the needful for locking bots out if they figure out the port. I've had to change ports only once in 6 years, so it's working to keep bots out rather nicely. And really that's all these things are- a way to keep bots out. A determined attacker will figure this stuff out anyway.
CamJN 6y ago
6. Possible enhancements Rate limiting the number of requests that can be made to the application So this just moves the brute forcing target from ssh to a web app. A lot of work for no added security.
nyx_ 6y ago
I like to run fail2ban in conjunction with a non-standard SSH port on which only public key auth is available. This way, most of the junkware that does rude things to port 22 is banging on a closed door; the slightly more effective junkware that actually finds the SSH port gets banned immediately, because I know anyone trying to log in with a password is full of it.
- dmos62 6y ago
  Wouldn't it make more sense then to keep SSH on 22? Think of it like a honeypot. The more accessible you make it, the more bans you get. There was a guy on here saying that fail2ban banning SSH bruteforcers also reduced the number of HTTP bruteforcers, because they overlap.
  - greenie_beans 6y ago
    This is a noob question, but could you not setup fail2ban to ban bad IPs on both ports, even if you're not using SSH on one? I'm wondering if it's possible to close port 22 and set fail2ban rules to ban any IPs that try to SSH there, while changing your SSH port and also setting up fail2ban on that port, too. Does a closed port work like that?
    - ses1984 6y ago
      You could have the firewall log connection attempts on port 22 and plug that into fail2ban.
    - kelnos 6y ago
      fail2ban works by tailing authentication logs and matching regexes to find failed attempts and the IP address of the source. So if nothing is listening on the port, there won't be anything to spit out logs that fail2ban can read. You could potentially create a fake daemon that listens on port 22 and just logs every access attempt, and set up fail2ban to block any IP that even opens a connection to the port, regardless of what they try to send over it. Actually, I think iptables has a LOG target that can log connection attempts even without something listening, so that could work too and be much simpler.
      - Legogris 6y ago
        Just make sure to exclude that port from service checks ;)
  - netsectoday 6y ago
    Collecting bans isn't a good thing today with the scale of background noise malicious behavior. You will very quickly collect thousands of IP addresses doing this and need to implement ipset - an iptables plugin that allows O(log n) lookup time on a list of IP addresses. Another issue; the overlap between SSH scanners also running HTTP/S attacks is negligible. From experience; what makes sense is shifting your SSH port away from 22, disabling password based authentication, whitelisting your IP address from your cloud provider's firewall, and still aggressively auto-banning incorrect logins with fail2ban. Then, for good measure, implement a WAF to protect your HTTP/S traffic as well. Do not turn your production system into a honeypot. Only do this with a separate system that contains no valuable data.
- mjoin 6y ago
  > anyone trying to log in with a password is full of it Dang! But so true
ryanjkirk 6y ago
Setting this up makes much less sense than setting up a tested vpn, such as wireguard or open, or even a persistent ssh tunnel using autossh to your home rpi. I would never allow my prod systems to be potentially exposed by an api that runs as root. (And the documentation is incorrect on that; it should run as an unprivileged user with sudo privs to only run a wrapper script that runs firewall-cmd). This also makes little sense in the context of configuration management, which should be enforcing a static set of iptables rules.
- stqism 6y ago
  Personally I’d make a binary run as a non-root user with just CAP_NET_ADMIN and CAP_NET_RAW permissions to cut the scope down even more.
  - ryanjkirk 6y ago
    Having not used setcap, would that also require the setuid bit to be set with ownership as root (like sudo)? And reside on a filesystem without the nosuid mount option?
    - Reelin 6y ago
      > would that also require the setuid bit to be set with ownership as root (like sudo)? Doesn't setuid just change to whichever user owns the file? So with setuid root only needs to own it if you want full administrative privileges. I also haven't played with setcap myself. My understanding is that capabilities don't rely on file ownership but instead on extended file attributes (which can't be changed by ordinary users on a correctly configured system). So root doesn't need to own the file, but of course granting any capabilities to a binary for which a non-root user has write permission seems like it would be a really bad idea in general. > And reside on a filesystem without the nosuid mount option? Yes, it appears the nosuid mount option disables file capabilities. (As one would hope!) But I'm not really seeing how that's an issue since (for example) bind mounts and btrfs subvolume mounts both allow changing the nosuid option. (man 7 capabilities) > 1. For all privileged operations, the kernel must check whether the thread has the required capability in its effective set. > 2. The kernel must provide system calls allowing a thread's capability sets to be changed and retrieved. > 3. The file system must support attaching capabilities to an executable file, so that a process gains those capabilities when the file is executed. (https://wiki.archlinux.org/index.php/Capabilities https://wiki.archlinux.org/index.php/Capabilities) > Capabilities are implemented on Linux using extended attributes (xattr(7)) in the security namespace. (man 2 execve) > The capabilities of the program file (see capabilities(7)) are also ignored if any of the above are true.
johnghanks 6y ago
Wow - why didn't I think of just setting up a firewall instead of running fail2ban. Great idea.
- _joel 6y ago
  eh, fail2ban uses iptables to ban?
  - joana035 6y ago
    Fail2ban uses whatever one wants to do whatever one need based on logs pattern matching :-)
  - seized 6y ago
    Fail2Ban can execute scripts etc to do things. I have mine making API calls to OPNsense to block IPs there.
ff7c11 6y ago
Have you tried knockd https://linux.die.net/man/1/knockd https://linux.die.net/man/1/knockd ? You send a special sequence of "knocks" to the server (packets to different ports) and it executes a command such as allowing your IP for a time period. No JWTs.
- yearoflinux 6y ago
  This is very nice and renders the OP useless. OP itself is another attack vector. Do you know what syscall API knockd uses to listen to "link" ?
- thedanbob 6y ago
  My go-to is fwnop (https://www.cipherdyne.org/fwknop/ https://www.cipherdyne.org/fwknop/) which is actually similar to the OP’s thing but battle tested. Only downside is it’s not available on iOS, so recently I setup WireGuard for my iPad. And yes, ssh pubkey + fwnop + WireGuard + fail2ban is ridiculous overkill, but hey, it’s my homelab server. That’s how I learn this stuff.
thatha7777 6y ago
This is very similar to port knocking, although more complicated. Although a proper JWT token is “more secure” than a sequence of 0-65535 integers, I contend that having complicated and/or unvetted logic as your first line of defense is more problematic than secure.
- kbenson 6y ago
  On the positive side, it's probably pretty easy to encrypt the RPC traffic. On the negative side, you've traded the fairly well known risk of running SSH with some extra set of software (whoops: s/hardware/software/) with less well known vetting, and it's own possible security problems. This is basically relying on security through obscurity because of that. SSH is almost definitely handled by your OS package management, so any problems with SSH will be patched immediately as found, and automatically distributed to you if you opt in to that (or you will be notified there are updated to apply, and you can do so). If this system is not packaged and distributed by your OS, you're going to have to keep up to date on any security problems with it yourself. It may have less permissions by default because it's running through a container, but that doesn't always help (kernel exploits are a thing, a lot of successful hacks are first getting the ability to run unprivileged arbitrary code, then using a kernel exploit to gain privileged access. There's something to be said for well engineered, well vetted simplicity. This is an interesting solution, but I would much rather just lock down the set of IPs it allows connections from to SSH to a very small subset of known good ones, including the IP address of a $5 DigitalOcean VM that I keep off but can turn on and connect to and use to jump to that box if at a location I can't connect from directly (and you can set a timed firewalld rules to allow access from your current IP if it's not permanent and shut of the VM again). Of use the free tier of AWS if you can get it to have a specific IP for free as well (I don't recall what you can get in the free tier other than the smallest VM). Edit: Oh, and disable password auth. That's SSH hardening action #1, I kind of just assume everyone does that (because they should, if they can).
- jrochkind1 6y ago
  Yeah, I'd want to see a mention of port-knocking in the README, indicating that the author was familiar with this technique, and the known critiques of it, and why they think this method (or port-knocking in general?) withstands it. Cause it's definitely in the vicinity. With no mention, it makes me wonder if the author is familiar with the history of this approach, which is not a good sign. And yeah, I'm skeptical. Is this new, uncommon/unvetted, and somewhat complex "access" app just another place for potential vulnerabilities that give someone the keys to the kingdom?
oehtXRwMkIs 6y ago
If I have public key authentication set up for ssh, should I even bother with fail2ban, firewalld-rest, port knocking, etc.? There's no way anyone is brute forcing my ed25519 key, so what's the point? Sure security should be layered and all that but it seems like public key auth is so strong by itself anything on top seems unnecessary.
- avtar 6y ago
  > If I have public key authentication set up for ssh, should I even bother with fail2ban If someone's bruteforcing their way in via ssh it's probably worthwhile blocking them before they start trying other services.
- jooize 6y ago
  Might protect against some ludicrous attack that we're not considering. Limiting attempts seems to me to always be a good idea.
- RL_Quine 6y ago
  There's no point to this at all, it's just making services more fragile. If "brute force" attacks are a concern, you are a fool that has password authentication turned on. If you have password authentication turned off, why does it matter?
  - calvinmorrison 6y ago
    Given some other attacks to ssh over the years, isn't it safer to use fail2ban
- fmajid 6y ago
  It's very easy for brute-force attacks from botnets to consume all your SSH process slots and then you are locked out.
  - roveo 6y ago
    Can confirm, had this problem the other day. CI started failing and after a short investigation I figured out sshd was just dropping my connections. Installed fail2ban first, but then just used a firewall since all valid IPs are known.
yabones 6y ago
I don't like this approach very much because it's much more complicated than it would have to be. Security is about layers, and this is essentially one layer that acts as the sole guardian for sshd. The way that I like to do this is to have a common 'entry point' for all my cloud systems. Instead of whitelisting IPs on every VPS or cluster I build, I just add them to the ACL on my management server. All the other systems only allow SSH connections in from the bastion server. In practice, it works like this: * Add IP to the whitelist file in my change control * Run the Terraform script to update the DigitalOcean ACL * Start an SSH agent locally and add the key for the bastion, as well as the key for the destination * Connect to the destination server by using ProxyJump So, connecting to a box would always route through my bastion system first, like this: ssh -J mgmt.mydomain.net cool-app-server.mydomain.net I've been doing this for a couple years, and it works great. I practically never have attempted login attempts on my systems. And, since I use an ssh agent to forward the keys through the bastion without ever actually storing them, a compromise of that system doesn't really give the attacker anything other than access to port 22 on a bunch of systems that they wouldn't know where to find. Only the most sophisticated attack (https://xkcd.com/538/ https://xkcd.com/538/) would lead to a real compromise.
- jiggawatts 6y ago
  I love that XKCD, but this particular cartoon gives a false impression in order to be humorous. (You can't fault a comic artist for this of course.) In my experience the #1 real attack vector is the shouty middle manager demanding that: "Everything be opened right now because some outsourced admin says he needed it". It's nigh impossible to use technological measures to defeat a CxO or a department head.
wrkronmiller 6y ago
I usually just disable password auth and update regularly, when I have an SSH server open to the internet. Short of an 0day in the SSH service, I expect brute forcing the private key(s) to take longer than I have years to live.
- oceanghost 6y ago
  There are bots out there that will attack from thousands if not 10's of thousands of IPs... It's impossible to ban the individual IPs byh hand, so I let them tell me who they are by leaving the SSH port open and have fail2ban ban any IP that fails to authenticate for a month. This had an unintended effect... most of my automated HTTP(s) attacks stopped as well-- presumably because the IPs of compromised machines were already blocked.
  - Libbum 6y ago
    That's interesting! Care to share your fail2ban config for this?
    - avian 6y ago
      Sounds like just a vanilla sshd filter with bantime cranked up to 1 month instead of the default (and possibly maxretry=1 or something low like that)
- tambourine_man 6y ago
  But it takes CPU, fills up log files, etc. fail2ban is the first thing install. It’s also easy to understand, which I love. It’s basically regex on log files.
  - necheffa 6y ago
    It really doesn't make that big a difference. I have single core, 515MiB VMs that idle at <1% utilization with background noise. Most distros come with logrotate anyways.
Tepix 6y ago
Running SSH on a non-standard port gets rid of the brute force login attempts, too.
- zootboy 6y ago
  In my experience, a non-standard port reduces the number of brute force attempts, but it doesn't eliminate them. I just do a simple rate limit on syn packets to the port and use SSH keys for auth, and that feels like plenty of security.
john-shaffer 6y ago
On EC2, I've had a great experience using AWS Systems Manager [1]. I don't need any ports open, and it works great with normal shell tools and emacs given a .ssh/config like this: host i-* mi-* ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'" [1] https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html https://docs.aws.amazon.com/systems-manager/latest/userguide...
hitpointdrew 6y ago
Why is SSH open to the public internet at all, ever? Use WireGuard as VPN tunnel to your servers network, then SSH via internal IP.
- kbenson 6y ago
  Sometimes you literally just have a server on the internet, and there's not a network to connect to with wireguard. You could connect directly to the server with wireguard, but I actually trust OpenSSH more than VPN software built into the kernel (not to mention I can update it easily without rebooting).
- joana035 6y ago
  > Why is SSH open to the public internet at all, ever? Wasn't SSH made for that?
  - fiddlerwoaroof 6y ago
    Yeah, I trust the OpenSSH developers a lot more than almost anyone else I can think of: if I’m going to trust any services open to the public internet, I’ll trust OpenSSH
- floatboth 6y ago
  Why not avoid VPN and fully embrace the end-to-end ideology – only authenticate against services, never against networks. Abolish private networks! :P
koolba 6y ago
Adding a JWT authenticated API layer to something is not a first choice for adding additional security. If you want something like this look into platform level firewalls (ex: AWS security groups) or run spiped in front of your SSH server. I’d trust that a hell of a lot more than a REST API.
deeblering4 6y ago
Requiring ssh keys, disabling password auth, and using ssh bastion hosts is my preferred approach.
- robomc 6y ago
  Yeah it's not clear to me why exposing the ability to configure your firewall, protected by a keypair, to the internet, is safer than just letting someone bang their head against sshd, protected by a keypair, and presumably having had a lot more eyes on it for security.
  - donor20 6y ago
    And the distributions can also do the SElinux stuff that I don't use day to day.
fl0wenol 6y ago
The idea is better than port knocking, in the sense that you take active action to associate your host with the server, but there's a few issues. * Something this simple shouldn't need k8s, unless it was intended as an exercise by the developer for that reason * It combines the idea of using a non-standard port with certificate based authentication, which you can already do with SSH-- it's functionally the same with more steps Hypothetically, this approach can be more powerful as a centralized service running elsewhere, (i.e. cloud, your remote DC) and used for a whole bunch of jump boxes and end-users. End-users could run a "check-in" script wrapping around SSH that notified the service that a user was imminent, and then could check the server to see if 1) The bearer token is accepted for the destination server 2) That the destination server checked in, saw the new incoming request, and has already opened the port -- And then proceeds to run the SSH command if all is well, or fails with appropriate error.
kelnos 6y ago
This just feels like moving ssh auth to another port, using an obscure "authentication format" that nearly no one uses. If you're using pubkey or certificate auth, have disabled password auth, and restrict what users are allowed to ssh in, this just feels like added complexity and points of failure (not to mention a possible source of crypto vulnerabilities) with not much added benefit. Having said that, it's a cool project in general, and I feel like it could be useful to dynamically manage access to large groups of servers (perhaps not just ssh; you could use it to manage access to https interfaces or other things). Then again, if you have a large group of servers, you should have those ports blocked to the world and only allow access through a VPN and/or jump boxes.
- jackric 6y ago
  Agreed - it looks like security through obscurity to me. It has value by shifting the goalposts away from common mass scanning, but to a motivated attacker the security is equivalent.
  - treve 6y ago
    Relying on security through obscurity alone is a terrible idea, but there is absolutely a place for it. Most of us do not specifically get targeted, but will be affected by automated attacks that look at a large block of ips.
    - daveidol 6y ago
      Yeah, just moving your SSH daemon from port 22 to some other port number alone can help with that.
      - foxylad 6y ago
        We have continuous distributed dictionary attacks on our SSH servers, with attempts from literally thousands of IP addresses at a cadence designed not to trigger rate alerts. We moved from port 22, and within days all ports were scanned and the attacks started again on the new port. We are working on a different scheme to thwart the attackers.
joana035 6y ago
Nope nope nope. This has serious quality issues and any competent and nice sysadmin should say _nope_ to run this in any serious environment, for your own good ;-) IP addresses are just strings? At least parse/validate for IPv4/IPv6. Why yet another database? Can't the rules be loaded from the running system? Why not just ipset-persistent + knockd + portsentry? I know it is easy to get overexcited with a new pet project, but just be careful to not put this kind of stuff in a production system kidos.
- prashantgupta24 6y ago
  Thank you for the feedback!
  - joana035 6y ago
    Anytime. I saw you are already implementing some ideas from my inputs, that is great, please keep it going. Eventually, in notorious systems, you may face issues of having too many /32 rules, a good idea is to implement some mechanism to defragment them into single/bigger contiguous cidr blocks. There are other firewall implementations that can serve as inspiration. OR I recommend using ipset with a single iptables rule to match/block from the set. This way you don't need to reload stuff everytime. (Just add the ips into the set) PS: I hope you don't mind my snark comments, HN is fun.
    - prashantgupta24 6y ago
      I do not mind at all. I appreciate constructive criticism. Plus you gave me points to work on, I really appreciate that. So much to learn!
swinglock 6y ago
A proactive way of increasing your attack surface no doubt.
- joana035 6y ago
  A proactive way to shoot yourself in the foot for sure.
manexploitsman 6y ago
I like this "proactive" solution :) Endlessh: an SSH Tarpit https://nullprogram.com/blog/2019/03/22/ https://nullprogram.com/blog/2019/03/22/
Jnr 6y ago
This sounds like a complex port knocking setup with bigger overhead. If you want better security and UDP is not a problem, consider use of Wireguard VPN. It is passive and silent, random attackers won’t even know it is there.
cperciva 6y ago
I still think that running ssh over spiped is more secure. Every byte being passed to sshd is cryptographically verified (and encrypted).
ed25519FUUU 6y ago
Secure SSH and you won't have to worry about rogue login attempts, which happen to everyone. If it really bothers you then move it to another port where it will happen less. But install a new firewall management system? Sounds like it will definitely introduce more risks than the problems it solves, which for the SSH example isn't really a problem at all.
nine_k 6y ago
(1) The auth interface is per IP address, as visible by the listening server. (2) The word "NAT" is found 0 times in the text. (3) There is no point 3 from the practical standpoint, please move along :(
jlgaddis 6y ago
If you're gonna go through all of this work -- including creating and maintaining private keys -- why not just restrict the SSH server to only permit key-based authentication (optionally, signed by your CA)? If having 22/TCP open to the world is an issue, then set up Wireguard on the host and only allow SSH connections that are coming in over the Wireguard interface. Got a bunch of machines to deal with? Set up a jumpbox or two running OpenBSD, lock it down, give your users access to it via SSH (optionally, over a Wireguard connection) and then only allow SSH access to all of those other hosts from the jumpbox(es). Then there's the fact that I have a whole lot more trust in the security of OpenSSH than I do some random web application! To me, this just seems kinda pointless -- there's a bunch of other, better (IMO) ways to deal with this -- but I guess if it fits your needs ...
- MrStonedOne 6y ago
  Step 1: assume every service listening on your network has an auth exploit that has not been found. Step 2: deploy this. Step 3: ... Step 4: profit! The line between something like this and a locking the service behind a VPN is very narrow. They both achieve the same thing, protect high risk/value targets by not even allowing unauthorized people to talk to them at the network level.
MrStonedOne 6y ago
Documentation and idea framing is too snowflaked. This a JWT based port knocking over https framework that would be useful when used in a much broader sense. Framing it as proactive fail2ban is technically correct, it also masks the other and more powerful use-cases. I could see this in use as a vpn bypass for a prosumer production system, where normal operational commands go over a secured vpn but in a pinch, you can disable that restriction for direct control during a partial failure.
abpavel 6y ago
This looks like reincarnation of "Lock & Key" feature of Cisco routers available since the late 90s. There were two major issues that led to abandonment and hindered adoption. The first is that it's an extra step. Extra steps for installation, for complexity, for single point of failure, for availability of key services. The second is that instead of thwarting attackers, you're thwarting yourself, every single time. It breaks so many usecases, for example if you have a new machine, or if you want to access through a jump host, or from ssh client on phone, etc.
deathanatos 6y ago
> you can go to jwt.io and generate a valid JWT using RS256 algorithm (the payload doesn't matter). You will be using that JWT to make calls to the REST application, so keep the JWT safe. The JWT you got after you plugged the private key into a random website is going to protect access to your machine?
peterwwillis 6y ago
A dynamic IP filtering list is still reactive because you haven't actually secured the box or its services. You've just made it slightly inconvenient to brute force. You might as well use port knocking, because even with a fancy schmancy authentication system (and the attack surface of a custom web app...) I can MITM your active connections just fine either way, or spoof your whitelisted IP. I know everybody likes Fail2ban, but these two iptables rules (or something just like them) actually work better and don't fill up your logs: iptables -t mangle -A PREROUTING -p tcp --dport ssh -m conntrack --ctstate NEW -m recent --set iptables -t mangle -A PREROUTING -p tcp --dport ssh -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 4 -j DROP To actually protect the network layer, use some kind of VPN with MFA.
johnisgood 6y ago
I just whitelist IPs. You can do this easily using iptables or sshd config.
netsectoday 6y ago
Please don't use this software. From a security perspective this is horrible. Option A) Expose all firewall rules via some hacky web-based API. Option B) https://nvlpubs.nist.gov/nistpubs/ir/2015/NIST.IR.7966.pdf https://nvlpubs.nist.gov/nistpubs/ir/2015/NIST.IR.7966.pdf
solatic 6y ago
If you're going to take this approach, a bigger question is why bother running the SSH jumpbox 24/7 in the first place? Shut down the SSH jumpbox when you don't need it (thus achieving SSH isolation) and start it up with your public IP in user-data to enable access to you and only you when you do need it.
wruza 6y ago
Can anyone explain what's wrong with "pubkey only, ignore attempts" method?
gitgud 6y ago
Much simpler to stick with SSH and an IP Address whitelist... Then you can't even get login attempts, unless they're from the same IP... am I missing something?

If you do not need the more granular firewall configuration options there is also classic port knocking (https://en.wikipedia.org/wiki/Port_knocking https://en.wikipedia.org/wiki/Port_knocking) where the daemon sits behind the firewall so all ports can be closed by default.

geocrasher 6y ago

This is interesting in that it combines the concept of port knocking with a REST interface, which I'm assuming is up to the user to create a front end for. Unfotunately it also relies on Kubernetes which means that using it for a single system isn't practical. At least, not for this server owner. My own approach is simply security by obscurity (a non standard port) with APF/BFD doing the needful for locking bots out if they figure out the port. I've had to change ports only once in 6 years, so it's working to keep bots out rather nicely. And really that's all these things are- a way to keep bots out. A determined attacker will figure this stuff out anyway.

CamJN 6y ago

6. Possible enhancements Rate limiting the number of requests that can be made to the application So this just moves the brute forcing target from ssh to a web app. A lot of work for no added security.

nyx_ 6y ago

I like to run fail2ban in conjunction with a non-standard SSH port on which only public key auth is available. This way, most of the junkware that does rude things to port 22 is banging on a closed door; the slightly more effective junkware that actually finds the SSH port gets banned immediately, because I know anyone trying to log in with a password is full of it.

dmos62 6y ago
Wouldn't it make more sense then to keep SSH on 22? Think of it like a honeypot. The more accessible you make it, the more bans you get. There was a guy on here saying that fail2ban banning SSH bruteforcers also reduced the number of HTTP bruteforcers, because they overlap.
- greenie_beans 6y ago
  This is a noob question, but could you not setup fail2ban to ban bad IPs on both ports, even if you're not using SSH on one? I'm wondering if it's possible to close port 22 and set fail2ban rules to ban any IPs that try to SSH there, while changing your SSH port and also setting up fail2ban on that port, too. Does a closed port work like that?
  - ses1984 6y ago
    You could have the firewall log connection attempts on port 22 and plug that into fail2ban.
  - kelnos 6y ago
    fail2ban works by tailing authentication logs and matching regexes to find failed attempts and the IP address of the source. So if nothing is listening on the port, there won't be anything to spit out logs that fail2ban can read. You could potentially create a fake daemon that listens on port 22 and just logs every access attempt, and set up fail2ban to block any IP that even opens a connection to the port, regardless of what they try to send over it. Actually, I think iptables has a LOG target that can log connection attempts even without something listening, so that could work too and be much simpler.
    - Legogris 6y ago
      Just make sure to exclude that port from service checks ;)
- netsectoday 6y ago
  Collecting bans isn't a good thing today with the scale of background noise malicious behavior. You will very quickly collect thousands of IP addresses doing this and need to implement ipset - an iptables plugin that allows O(log n) lookup time on a list of IP addresses. Another issue; the overlap between SSH scanners also running HTTP/S attacks is negligible. From experience; what makes sense is shifting your SSH port away from 22, disabling password based authentication, whitelisting your IP address from your cloud provider's firewall, and still aggressively auto-banning incorrect logins with fail2ban. Then, for good measure, implement a WAF to protect your HTTP/S traffic as well. Do not turn your production system into a honeypot. Only do this with a separate system that contains no valuable data.
mjoin 6y ago
> anyone trying to log in with a password is full of it Dang! But so true

ryanjkirk 6y ago

Setting this up makes much less sense than setting up a tested vpn, such as wireguard or open, or even a persistent ssh tunnel using autossh to your home rpi. I would never allow my prod systems to be potentially exposed by an api that runs as root. (And the documentation is incorrect on that; it should run as an unprivileged user with sudo privs to only run a wrapper script that runs firewall-cmd). This also makes little sense in the context of configuration management, which should be enforcing a static set of iptables rules.

stqism 6y ago
Personally I’d make a binary run as a non-root user with just CAP_NET_ADMIN and CAP_NET_RAW permissions to cut the scope down even more.
- ryanjkirk 6y ago
  Having not used setcap, would that also require the setuid bit to be set with ownership as root (like sudo)? And reside on a filesystem without the nosuid mount option?
  - Reelin 6y ago
    > would that also require the setuid bit to be set with ownership as root (like sudo)? Doesn't setuid just change to whichever user owns the file? So with setuid root only needs to own it if you want full administrative privileges. I also haven't played with setcap myself. My understanding is that capabilities don't rely on file ownership but instead on extended file attributes (which can't be changed by ordinary users on a correctly configured system). So root doesn't need to own the file, but of course granting any capabilities to a binary for which a non-root user has write permission seems like it would be a really bad idea in general. > And reside on a filesystem without the nosuid mount option? Yes, it appears the nosuid mount option disables file capabilities. (As one would hope!) But I'm not really seeing how that's an issue since (for example) bind mounts and btrfs subvolume mounts both allow changing the nosuid option. (man 7 capabilities) > 1. For all privileged operations, the kernel must check whether the thread has the required capability in its effective set. > 2. The kernel must provide system calls allowing a thread's capability sets to be changed and retrieved. > 3. The file system must support attaching capabilities to an executable file, so that a process gains those capabilities when the file is executed. (https://wiki.archlinux.org/index.php/Capabilities https://wiki.archlinux.org/index.php/Capabilities) > Capabilities are implemented on Linux using extended attributes (xattr(7)) in the security namespace. (man 2 execve) > The capabilities of the program file (see capabilities(7)) are also ignored if any of the above are true.

johnghanks 6y ago

Wow - why didn't I think of just setting up a firewall instead of running fail2ban. Great idea.

_joel 6y ago
eh, fail2ban uses iptables to ban?
- joana035 6y ago
  Fail2ban uses whatever one wants to do whatever one need based on logs pattern matching :-)
- seized 6y ago
  Fail2Ban can execute scripts etc to do things. I have mine making API calls to OPNsense to block IPs there.

ff7c11 6y ago

Have you tried knockd https://linux.die.net/man/1/knockd https://linux.die.net/man/1/knockd ? You send a special sequence of "knocks" to the server (packets to different ports) and it executes a command such as allowing your IP for a time period. No JWTs.

yearoflinux 6y ago
This is very nice and renders the OP useless. OP itself is another attack vector. Do you know what syscall API knockd uses to listen to "link" ?
thedanbob 6y ago
My go-to is fwnop (https://www.cipherdyne.org/fwknop/ https://www.cipherdyne.org/fwknop/) which is actually similar to the OP’s thing but battle tested. Only downside is it’s not available on iOS, so recently I setup WireGuard for my iPad. And yes, ssh pubkey + fwnop + WireGuard + fail2ban is ridiculous overkill, but hey, it’s my homelab server. That’s how I learn this stuff.

thatha7777 6y ago

This is very similar to port knocking, although more complicated. Although a proper JWT token is “more secure” than a sequence of 0-65535 integers, I contend that having complicated and/or unvetted logic as your first line of defense is more problematic than secure.

kbenson 6y ago
On the positive side, it's probably pretty easy to encrypt the RPC traffic. On the negative side, you've traded the fairly well known risk of running SSH with some extra set of software (whoops: s/hardware/software/) with less well known vetting, and it's own possible security problems. This is basically relying on security through obscurity because of that. SSH is almost definitely handled by your OS package management, so any problems with SSH will be patched immediately as found, and automatically distributed to you if you opt in to that (or you will be notified there are updated to apply, and you can do so). If this system is not packaged and distributed by your OS, you're going to have to keep up to date on any security problems with it yourself. It may have less permissions by default because it's running through a container, but that doesn't always help (kernel exploits are a thing, a lot of successful hacks are first getting the ability to run unprivileged arbitrary code, then using a kernel exploit to gain privileged access. There's something to be said for well engineered, well vetted simplicity. This is an interesting solution, but I would much rather just lock down the set of IPs it allows connections from to SSH to a very small subset of known good ones, including the IP address of a $5 DigitalOcean VM that I keep off but can turn on and connect to and use to jump to that box if at a location I can't connect from directly (and you can set a timed firewalld rules to allow access from your current IP if it's not permanent and shut of the VM again). Of use the free tier of AWS if you can get it to have a specific IP for free as well (I don't recall what you can get in the free tier other than the smallest VM). Edit: Oh, and disable password auth. That's SSH hardening action #1, I kind of just assume everyone does that (because they should, if they can).
jrochkind1 6y ago
Yeah, I'd want to see a mention of port-knocking in the README, indicating that the author was familiar with this technique, and the known critiques of it, and why they think this method (or port-knocking in general?) withstands it. Cause it's definitely in the vicinity. With no mention, it makes me wonder if the author is familiar with the history of this approach, which is not a good sign. And yeah, I'm skeptical. Is this new, uncommon/unvetted, and somewhat complex "access" app just another place for potential vulnerabilities that give someone the keys to the kingdom?

oehtXRwMkIs 6y ago

If I have public key authentication set up for ssh, should I even bother with fail2ban, firewalld-rest, port knocking, etc.? There's no way anyone is brute forcing my ed25519 key, so what's the point? Sure security should be layered and all that but it seems like public key auth is so strong by itself anything on top seems unnecessary.

avtar 6y ago
> If I have public key authentication set up for ssh, should I even bother with fail2ban If someone's bruteforcing their way in via ssh it's probably worthwhile blocking them before they start trying other services.
jooize 6y ago
Might protect against some ludicrous attack that we're not considering. Limiting attempts seems to me to always be a good idea.
RL_Quine 6y ago
There's no point to this at all, it's just making services more fragile. If "brute force" attacks are a concern, you are a fool that has password authentication turned on. If you have password authentication turned off, why does it matter?
- calvinmorrison 6y ago
  Given some other attacks to ssh over the years, isn't it safer to use fail2ban
fmajid 6y ago
It's very easy for brute-force attacks from botnets to consume all your SSH process slots and then you are locked out.
- roveo 6y ago
  Can confirm, had this problem the other day. CI started failing and after a short investigation I figured out sshd was just dropping my connections. Installed fail2ban first, but then just used a firewall since all valid IPs are known.

yabones 6y ago

I don't like this approach very much because it's much more complicated than it would have to be. Security is about layers, and this is essentially one layer that acts as the sole guardian for sshd. The way that I like to do this is to have a common 'entry point' for all my cloud systems. Instead of whitelisting IPs on every VPS or cluster I build, I just add them to the ACL on my management server. All the other systems only allow SSH connections in from the bastion server. In practice, it works like this: * Add IP to the whitelist file in my change control * Run the Terraform script to update the DigitalOcean ACL * Start an SSH agent locally and add the key for the bastion, as well as the key for the destination * Connect to the destination server by using ProxyJump So, connecting to a box would always route through my bastion system first, like this: ssh -J mgmt.mydomain.net cool-app-server.mydomain.net I've been doing this for a couple years, and it works great. I practically never have attempted login attempts on my systems. And, since I use an ssh agent to forward the keys through the bastion without ever actually storing them, a compromise of that system doesn't really give the attacker anything other than access to port 22 on a bunch of systems that they wouldn't know where to find. Only the most sophisticated attack (https://xkcd.com/538/ https://xkcd.com/538/) would lead to a real compromise.

jiggawatts 6y ago
I love that XKCD, but this particular cartoon gives a false impression in order to be humorous. (You can't fault a comic artist for this of course.) In my experience the #1 real attack vector is the shouty middle manager demanding that: "Everything be opened right now because some outsourced admin says he needed it". It's nigh impossible to use technological measures to defeat a CxO or a department head.

wrkronmiller 6y ago

I usually just disable password auth and update regularly, when I have an SSH server open to the internet. Short of an 0day in the SSH service, I expect brute forcing the private key(s) to take longer than I have years to live.

oceanghost 6y ago
There are bots out there that will attack from thousands if not 10's of thousands of IPs... It's impossible to ban the individual IPs byh hand, so I let them tell me who they are by leaving the SSH port open and have fail2ban ban any IP that fails to authenticate for a month. This had an unintended effect... most of my automated HTTP(s) attacks stopped as well-- presumably because the IPs of compromised machines were already blocked.
- Libbum 6y ago
  That's interesting! Care to share your fail2ban config for this?
  - avian 6y ago
    Sounds like just a vanilla sshd filter with bantime cranked up to 1 month instead of the default (and possibly maxretry=1 or something low like that)
tambourine_man 6y ago
But it takes CPU, fills up log files, etc. fail2ban is the first thing install. It’s also easy to understand, which I love. It’s basically regex on log files.
- necheffa 6y ago
  It really doesn't make that big a difference. I have single core, 515MiB VMs that idle at <1% utilization with background noise. Most distros come with logrotate anyways.

Tepix 6y ago

Running SSH on a non-standard port gets rid of the brute force login attempts, too.

zootboy 6y ago
In my experience, a non-standard port reduces the number of brute force attempts, but it doesn't eliminate them. I just do a simple rate limit on syn packets to the port and use SSH keys for auth, and that feels like plenty of security.

john-shaffer 6y ago

On EC2, I've had a great experience using AWS Systems Manager [1]. I don't need any ports open, and it works great with normal shell tools and emacs given a .ssh/config like this: host i-* mi-* ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'" [1] https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html https://docs.aws.amazon.com/systems-manager/latest/userguide...

hitpointdrew 6y ago

Why is SSH open to the public internet at all, ever? Use WireGuard as VPN tunnel to your servers network, then SSH via internal IP.

kbenson 6y ago
Sometimes you literally just have a server on the internet, and there's not a network to connect to with wireguard. You could connect directly to the server with wireguard, but I actually trust OpenSSH more than VPN software built into the kernel (not to mention I can update it easily without rebooting).
joana035 6y ago
> Why is SSH open to the public internet at all, ever? Wasn't SSH made for that?
- fiddlerwoaroof 6y ago
  Yeah, I trust the OpenSSH developers a lot more than almost anyone else I can think of: if I’m going to trust any services open to the public internet, I’ll trust OpenSSH
floatboth 6y ago
Why not avoid VPN and fully embrace the end-to-end ideology – only authenticate against services, never against networks. Abolish private networks! :P

koolba 6y ago

Adding a JWT authenticated API layer to something is not a first choice for adding additional security. If you want something like this look into platform level firewalls (ex: AWS security groups) or run spiped in front of your SSH server. I’d trust that a hell of a lot more than a REST API.

deeblering4 6y ago

Requiring ssh keys, disabling password auth, and using ssh bastion hosts is my preferred approach.

robomc 6y ago
Yeah it's not clear to me why exposing the ability to configure your firewall, protected by a keypair, to the internet, is safer than just letting someone bang their head against sshd, protected by a keypair, and presumably having had a lot more eyes on it for security.
- donor20 6y ago
  And the distributions can also do the SElinux stuff that I don't use day to day.

fl0wenol 6y ago

The idea is better than port knocking, in the sense that you take active action to associate your host with the server, but there's a few issues. * Something this simple shouldn't need k8s, unless it was intended as an exercise by the developer for that reason * It combines the idea of using a non-standard port with certificate based authentication, which you can already do with SSH-- it's functionally the same with more steps Hypothetically, this approach can be more powerful as a centralized service running elsewhere, (i.e. cloud, your remote DC) and used for a whole bunch of jump boxes and end-users. End-users could run a "check-in" script wrapping around SSH that notified the service that a user was imminent, and then could check the server to see if 1) The bearer token is accepted for the destination server 2) That the destination server checked in, saw the new incoming request, and has already opened the port -- And then proceeds to run the SSH command if all is well, or fails with appropriate error.

kelnos 6y ago

This just feels like moving ssh auth to another port, using an obscure "authentication format" that nearly no one uses. If you're using pubkey or certificate auth, have disabled password auth, and restrict what users are allowed to ssh in, this just feels like added complexity and points of failure (not to mention a possible source of crypto vulnerabilities) with not much added benefit. Having said that, it's a cool project in general, and I feel like it could be useful to dynamically manage access to large groups of servers (perhaps not just ssh; you could use it to manage access to https interfaces or other things). Then again, if you have a large group of servers, you should have those ports blocked to the world and only allow access through a VPN and/or jump boxes.

jackric 6y ago
Agreed - it looks like security through obscurity to me. It has value by shifting the goalposts away from common mass scanning, but to a motivated attacker the security is equivalent.
- treve 6y ago
  Relying on security through obscurity alone is a terrible idea, but there is absolutely a place for it. Most of us do not specifically get targeted, but will be affected by automated attacks that look at a large block of ips.
  - daveidol 6y ago
    Yeah, just moving your SSH daemon from port 22 to some other port number alone can help with that.
    - foxylad 6y ago
      We have continuous distributed dictionary attacks on our SSH servers, with attempts from literally thousands of IP addresses at a cadence designed not to trigger rate alerts. We moved from port 22, and within days all ports were scanned and the attacks started again on the new port. We are working on a different scheme to thwart the attackers.

joana035 6y ago

Nope nope nope. This has serious quality issues and any competent and nice sysadmin should say _nope_ to run this in any serious environment, for your own good ;-) IP addresses are just strings? At least parse/validate for IPv4/IPv6. Why yet another database? Can't the rules be loaded from the running system? Why not just ipset-persistent + knockd + portsentry? I know it is easy to get overexcited with a new pet project, but just be careful to not put this kind of stuff in a production system kidos.

prashantgupta24 6y ago
Thank you for the feedback!
- joana035 6y ago
  Anytime. I saw you are already implementing some ideas from my inputs, that is great, please keep it going. Eventually, in notorious systems, you may face issues of having too many /32 rules, a good idea is to implement some mechanism to defragment them into single/bigger contiguous cidr blocks. There are other firewall implementations that can serve as inspiration. OR I recommend using ipset with a single iptables rule to match/block from the set. This way you don't need to reload stuff everytime. (Just add the ips into the set) PS: I hope you don't mind my snark comments, HN is fun.
  - prashantgupta24 6y ago
    I do not mind at all. I appreciate constructive criticism. Plus you gave me points to work on, I really appreciate that. So much to learn!

swinglock 6y ago

A proactive way of increasing your attack surface no doubt.

joana035 6y ago
A proactive way to shoot yourself in the foot for sure.

manexploitsman 6y ago

I like this "proactive" solution :) Endlessh: an SSH Tarpit https://nullprogram.com/blog/2019/03/22/ https://nullprogram.com/blog/2019/03/22/

Jnr 6y ago

This sounds like a complex port knocking setup with bigger overhead. If you want better security and UDP is not a problem, consider use of Wireguard VPN. It is passive and silent, random attackers won’t even know it is there.

cperciva 6y ago

I still think that running ssh over spiped is more secure. Every byte being passed to sshd is cryptographically verified (and encrypted).

ed25519FUUU 6y ago

Secure SSH and you won't have to worry about rogue login attempts, which happen to everyone. If it really bothers you then move it to another port where it will happen less. But install a new firewall management system? Sounds like it will definitely introduce more risks than the problems it solves, which for the SSH example isn't really a problem at all.

nine_k 6y ago

(1) The auth interface is per IP address, as visible by the listening server. (2) The word "NAT" is found 0 times in the text. (3) There is no point 3 from the practical standpoint, please move along :(

jlgaddis 6y ago

If you're gonna go through all of this work -- including creating and maintaining private keys -- why not just restrict the SSH server to only permit key-based authentication (optionally, signed by your CA)? If having 22/TCP open to the world is an issue, then set up Wireguard on the host and only allow SSH connections that are coming in over the Wireguard interface. Got a bunch of machines to deal with? Set up a jumpbox or two running OpenBSD, lock it down, give your users access to it via SSH (optionally, over a Wireguard connection) and then only allow SSH access to all of those other hosts from the jumpbox(es). Then there's the fact that I have a whole lot more trust in the security of OpenSSH than I do some random web application! To me, this just seems kinda pointless -- there's a bunch of other, better (IMO) ways to deal with this -- but I guess if it fits your needs ...

MrStonedOne 6y ago
Step 1: assume every service listening on your network has an auth exploit that has not been found. Step 2: deploy this. Step 3: ... Step 4: profit! The line between something like this and a locking the service behind a VPN is very narrow. They both achieve the same thing, protect high risk/value targets by not even allowing unauthorized people to talk to them at the network level.

MrStonedOne 6y ago

Documentation and idea framing is too snowflaked. This a JWT based port knocking over https framework that would be useful when used in a much broader sense. Framing it as proactive fail2ban is technically correct, it also masks the other and more powerful use-cases. I could see this in use as a vpn bypass for a prosumer production system, where normal operational commands go over a secured vpn but in a pinch, you can disable that restriction for direct control during a partial failure.

abpavel 6y ago

This looks like reincarnation of "Lock & Key" feature of Cisco routers available since the late 90s. There were two major issues that led to abandonment and hindered adoption. The first is that it's an extra step. Extra steps for installation, for complexity, for single point of failure, for availability of key services. The second is that instead of thwarting attackers, you're thwarting yourself, every single time. It breaks so many usecases, for example if you have a new machine, or if you want to access through a jump host, or from ssh client on phone, etc.

deathanatos 6y ago

> you can go to jwt.io and generate a valid JWT using RS256 algorithm (the payload doesn't matter). You will be using that JWT to make calls to the REST application, so keep the JWT safe. The JWT you got after you plugged the private key into a random website is going to protect access to your machine?

peterwwillis 6y ago

A dynamic IP filtering list is still reactive because you haven't actually secured the box or its services. You've just made it slightly inconvenient to brute force. You might as well use port knocking, because even with a fancy schmancy authentication system (and the attack surface of a custom web app...) I can MITM your active connections just fine either way, or spoof your whitelisted IP. I know everybody likes Fail2ban, but these two iptables rules (or something just like them) actually work better and don't fill up your logs: iptables -t mangle -A PREROUTING -p tcp --dport ssh -m conntrack --ctstate NEW -m recent --set iptables -t mangle -A PREROUTING -p tcp --dport ssh -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 4 -j DROP To actually protect the network layer, use some kind of VPN with MFA.

johnisgood 6y ago

I just whitelist IPs. You can do this easily using iptables or sshd config.

netsectoday 6y ago

Please don't use this software. From a security perspective this is horrible. Option A) Expose all firewall rules via some hacky web-based API. Option B) https://nvlpubs.nist.gov/nistpubs/ir/2015/NIST.IR.7966.pdf https://nvlpubs.nist.gov/nistpubs/ir/2015/NIST.IR.7966.pdf

solatic 6y ago

If you're going to take this approach, a bigger question is why bother running the SSH jumpbox 24/7 in the first place? Shut down the SSH jumpbox when you don't need it (thus achieving SSH isolation) and start it up with your public IP in user-data to enable access to you and only you when you do need it.

wruza 6y ago

Can anyone explain what's wrong with "pubkey only, ignore attempts" method?

gitgud 6y ago

Much simpler to stick with SSH and an IP Address whitelist... Then you can't even get login attempts, unless they're from the same IP... am I missing something?