5 ms·

No firewall! Wow that's brave. Hetzner will let you configure one that runs outside of the box so you might want to add that too, as part of your defense in dep

No firewall! Wow that's brave. Hetzner will let you configure one that runs outside of the box so you might want to add that too, as part of your defense in depth - that will cover you if you make a mistake with ufw. Personally I keep SSH firewalled only to my home address in this way; if I'm out and about and need access, I can just log into Hetzner's website and change it temporarily.

Nextgrid 10mo ago
But the firewall wouldn't have saved them if they're running a public web service or need to interact with external services. I guess you can have the appserver fully firewalled and have another bastion host acting as an HTTP proxy, both for inbound as well as outbound connections. But it's not trivial to set up especially for the outbound scenario.
- danparsonson 10mo ago
  No you're right, I didn't mean the firewall would have saved them, but just as a general point of advice. And yes a second VPS running opnSense or similar makes a nice cheap proxy and then you can firewall off the main server completely. Although that wouldn't have saved them either - they'd still need to forward HTTP/S to the main box.
  - Nextgrid 10mo ago
    A firewall blocking outgoing connections (except those whitelisted through the proxy) would’ve likely prevented the download of the malware (as it’s usually done by using the RCE to call a curl/wget command rather than uploading the binary through the RCE) and/or its connection to the mining server.
    - denkmoon 10mo ago
      How many people do proper egress filtering though, even when running a firewall
    - drnick1 10mo ago
      In practice, this is basically impossible to implement. As a user behind a firewall you normally expect to be able to open connections with any remote host.
      - metafunctor 10mo ago
        Not impossible at all with a policy-filtering HTTPS proxy. See https://laurikari.github.io/exfilguard/ https://laurikari.github.io/exfilguard/ In this model, hosts don’t need any direct internet connectivity or access to public DNS. All outbound traffic is forced through the proxy, giving you full control over where each host is allowed to connect. It’s not painless: you must maintain a whitelist of allowed URLs and HTTP methods, distribute a trusted CA certificate, and ensure all software is configured to use the proxy.
jwrallie 10mo ago
Password auth being enabled is also very brave. I don’t think fail2ban is necessary personally, but it’s popular enough that it always come up.
tete 10mo ago
Firewalls in the majority of cases don't get you much. Yes it's a last line of defense if you do something really stupid and don't even know where or what you configure your services to listen on, but if you don't the difference between running firewalls and not is minuscule. There are way more important things like actually knowing that you are running software with widely known RCE that don't even use established mechanisms to sandbox themselves it seems. The way the author describes docker being the savior appears to be sheer luck.
- danparsonson 10mo ago
  The author mentioned they had other services exposed to the internet (Postgres, RabbitMQ) which increases their attack surface area. There may be vulnerabilities or misconfigurations in those services for example. Good security is layered.
  - seszett 10mo ago
    But if they have to be exposed then a firewall won't help, and if they don't have to be exposed to the internet then a firewall isn't needed either, just configure them not to listen on non-local interfaces.
    - spoaceman7777 10mo ago
      This sounds like an extremely effective foot gun. Just use a firewall.
      - seszett 10mo ago
        I'm not sure what you mean, what sounds dangerous to me is not caring about what services are listening to on a server. The firewall is there as a safeguard in case a service is temporarily misconfigured, it should certainly not be the only thing standing between your services and the internet.
        10mo ago
3abiton 10mo ago
Honestly fail2ban is amazing. I might doa write up on the countless of attempts on my servers.
- dizhn 10mo ago
  The only way I've envisioned fail2ban to be of any use at all is if you gather IPs from one server and use them on your whole fleet and I got it running like this for a while. Ultimately I decided that all it does is give you a cleaner log file since by definition its working on logs for attacks/attempts that did not succeed. We need to stop worrying about attempts we see in the logs and let software do its job.
figassis 10mo ago
Yup. All my servers are behind Tailscale. The only thing I expose is a load balancer that routes tcp (email) and http. That balancer is running docker, fully firewalled (incl docker bypasses). Every server is behind herzner’s firewall in addition to the internal firewall. App servers run docker, with images that run a single executable (no os, no shell), strict cpu and memory limits. Most of my apps only require very limited temporary storage so usually no need to mount anything. So good luck executing anything in there. I used, way back in the day, to run Wordpress sites. Would get hacked monthly every possible way. Learned so much, including the fact that often your app is your threat. With Wordpress, every plugin is a vector. Also the ability to easily hop into an instance and rewrite running code (looking at you scripting languages incl JS) is terrible. This motivated my move to Go. The code I compiled is what will run. Period.
dizhn 10mo ago
I have SSH blocked altogether and use wireguard to access the server. If something goes wrong I can always go to the dashboard and reenable SSH for my IP. But ultimately your setup is just as secure. Perhaps a tiny bit less convenient.
danw1979 10mo ago
The only time I have ever had a machine compromised in 30 years of running Linux is when I ran something exposed to the internet on a well known port. I know port scanners are a thing but the act of using non-default ports seems unreasonably effective at preventing most security problems.
- jraph 10mo ago
  I do this too, but I think it should only be a defense in depth thing, you still need the other measures.
- rainonmoon 10mo ago
  This is very, very, very bad advice. A non-standard port is not a defence. It’s not even slightly a defence.
  - bostik 10mo ago
    Correct. From what I understand, Shodan has had for years a search feature in their paid plans to query for "service X listening on non-standard port". The only sane assumption is that any half-decent internet-census[tm] tool has the same as standard by now.
  - danw1979 10mo ago
    Did I at any point in my previous comment say that using non-standard ports was my only line of defence ? Its security through obscurity, which puts you out of view of the vast majority of the chaos of the internet. It by no means protects you from all threats.
- tonyplee 10mo ago
  If you do any npm install, pip install ..., docker pull ... / docker run ... , etc in linux. It is very easy to get compromise. I did docker pull a few times base on some webpost (looks reasonable) and detect app/scripts from inside the docker connect to some .ru sites immediately or a few days later....
oakhaven 10mo ago
I don't whitelist IPs for ssh anymore, but I always run sshd on randomly selected port, in order to not get noticed by port scanners. I do it for a really long time already, and until now I am not sure if it has any benefit or it's just umbrella in a sideways storm.
- lordnacho 10mo ago
  As long as you understand it's security by obscurity, rather than by cryptography. I don't think it's wrong, it's just not the same as eg using a yubikey.
- forbiddenlake 10mo ago
  This won't hide you completely, but it will reduce log spam. My sshd only listens on the VPN interface
xetera 10mo ago
For the record this is only available for their VPS offering and not dedis. If you rent a dedi through their server auction you still need to configure your own firewall.
- danparsonson 10mo ago
  Dedicated servers can configure external firewalls too; there's a tab for it on the server config. It's basic but functional.

Nextgrid 10mo ago

But the firewall wouldn't have saved them if they're running a public web service or need to interact with external services. I guess you can have the appserver fully firewalled and have another bastion host acting as an HTTP proxy, both for inbound as well as outbound connections. But it's not trivial to set up especially for the outbound scenario.

danparsonson 10mo ago
No you're right, I didn't mean the firewall would have saved them, but just as a general point of advice. And yes a second VPS running opnSense or similar makes a nice cheap proxy and then you can firewall off the main server completely. Although that wouldn't have saved them either - they'd still need to forward HTTP/S to the main box.
- Nextgrid 10mo ago
  A firewall blocking outgoing connections (except those whitelisted through the proxy) would’ve likely prevented the download of the malware (as it’s usually done by using the RCE to call a curl/wget command rather than uploading the binary through the RCE) and/or its connection to the mining server.
  - denkmoon 10mo ago
    How many people do proper egress filtering though, even when running a firewall
  - drnick1 10mo ago
    In practice, this is basically impossible to implement. As a user behind a firewall you normally expect to be able to open connections with any remote host.
    - metafunctor 10mo ago
      Not impossible at all with a policy-filtering HTTPS proxy. See https://laurikari.github.io/exfilguard/ https://laurikari.github.io/exfilguard/ In this model, hosts don’t need any direct internet connectivity or access to public DNS. All outbound traffic is forced through the proxy, giving you full control over where each host is allowed to connect. It’s not painless: you must maintain a whitelist of allowed URLs and HTTP methods, distribute a trusted CA certificate, and ensure all software is configured to use the proxy.

jwrallie 10mo ago

Password auth being enabled is also very brave. I don’t think fail2ban is necessary personally, but it’s popular enough that it always come up.

tete 10mo ago

Firewalls in the majority of cases don't get you much. Yes it's a last line of defense if you do something really stupid and don't even know where or what you configure your services to listen on, but if you don't the difference between running firewalls and not is minuscule. There are way more important things like actually knowing that you are running software with widely known RCE that don't even use established mechanisms to sandbox themselves it seems. The way the author describes docker being the savior appears to be sheer luck.

danparsonson 10mo ago
The author mentioned they had other services exposed to the internet (Postgres, RabbitMQ) which increases their attack surface area. There may be vulnerabilities or misconfigurations in those services for example. Good security is layered.
- seszett 10mo ago
  But if they have to be exposed then a firewall won't help, and if they don't have to be exposed to the internet then a firewall isn't needed either, just configure them not to listen on non-local interfaces.
  - spoaceman7777 10mo ago
    This sounds like an extremely effective foot gun. Just use a firewall.
    - seszett 10mo ago
      I'm not sure what you mean, what sounds dangerous to me is not caring about what services are listening to on a server. The firewall is there as a safeguard in case a service is temporarily misconfigured, it should certainly not be the only thing standing between your services and the internet.
      - 10mo ago

3abiton 10mo ago

Honestly fail2ban is amazing. I might doa write up on the countless of attempts on my servers.

dizhn 10mo ago
The only way I've envisioned fail2ban to be of any use at all is if you gather IPs from one server and use them on your whole fleet and I got it running like this for a while. Ultimately I decided that all it does is give you a cleaner log file since by definition its working on logs for attacks/attempts that did not succeed. We need to stop worrying about attempts we see in the logs and let software do its job.

figassis 10mo ago

Yup. All my servers are behind Tailscale. The only thing I expose is a load balancer that routes tcp (email) and http. That balancer is running docker, fully firewalled (incl docker bypasses). Every server is behind herzner’s firewall in addition to the internal firewall. App servers run docker, with images that run a single executable (no os, no shell), strict cpu and memory limits. Most of my apps only require very limited temporary storage so usually no need to mount anything. So good luck executing anything in there. I used, way back in the day, to run Wordpress sites. Would get hacked monthly every possible way. Learned so much, including the fact that often your app is your threat. With Wordpress, every plugin is a vector. Also the ability to easily hop into an instance and rewrite running code (looking at you scripting languages incl JS) is terrible. This motivated my move to Go. The code I compiled is what will run. Period.

dizhn 10mo ago

I have SSH blocked altogether and use wireguard to access the server. If something goes wrong I can always go to the dashboard and reenable SSH for my IP. But ultimately your setup is just as secure. Perhaps a tiny bit less convenient.

danw1979 10mo ago

The only time I have ever had a machine compromised in 30 years of running Linux is when I ran something exposed to the internet on a well known port. I know port scanners are a thing but the act of using non-default ports seems unreasonably effective at preventing most security problems.

jraph 10mo ago
I do this too, but I think it should only be a defense in depth thing, you still need the other measures.
rainonmoon 10mo ago
This is very, very, very bad advice. A non-standard port is not a defence. It’s not even slightly a defence.
- bostik 10mo ago
  Correct. From what I understand, Shodan has had for years a search feature in their paid plans to query for "service X listening on non-standard port". The only sane assumption is that any half-decent internet-census[tm] tool has the same as standard by now.
- danw1979 10mo ago
  Did I at any point in my previous comment say that using non-standard ports was my only line of defence ? Its security through obscurity, which puts you out of view of the vast majority of the chaos of the internet. It by no means protects you from all threats.
tonyplee 10mo ago
If you do any npm install, pip install ..., docker pull ... / docker run ... , etc in linux. It is very easy to get compromise. I did docker pull a few times base on some webpost (looks reasonable) and detect app/scripts from inside the docker connect to some .ru sites immediately or a few days later....

oakhaven 10mo ago

I don't whitelist IPs for ssh anymore, but I always run sshd on randomly selected port, in order to not get noticed by port scanners. I do it for a really long time already, and until now I am not sure if it has any benefit or it's just umbrella in a sideways storm.

lordnacho 10mo ago
As long as you understand it's security by obscurity, rather than by cryptography. I don't think it's wrong, it's just not the same as eg using a yubikey.
forbiddenlake 10mo ago
This won't hide you completely, but it will reduce log spam. My sshd only listens on the VPN interface

xetera 10mo ago

For the record this is only available for their VPS offering and not dedis. If you rent a dedi through their server auction you still need to configure your own firewall.

danparsonson 10mo ago
Dedicated servers can configure external firewalls too; there's a tab for it on the server config. It's basic but functional.