11 ms·
A story on home server security
- cynicalsecurity 2y ago> None of the database guides I followed had warned me about the dangers of exposing a docker containerized database to the internet. This is like IT security 101. > A quick IP search revealed that it was registed with a Chinese ISP and located in the Russian Federation. Welcome to the Internet. Follow me. Always block China and Russia from any of your services. You are never going to need them and the only thing that can come from them is either a remote attack or unauthorized data sent to them.
- euroderf 2y ago> Always block China and Russia from any of your services. But does this add much security if bad actors are using VPNs ?
- philipwhiuk 2y agoThey'd need to run a VPN with end point in a less hostile country and so there's an element of enforcement on those endpoints. At any rate, blocking China and Russia isn't ever presented as a foolproof defence, it just raises the barrier to attack.
- Fnoord 2y agoThey extensively use these VPNs, in NL. They can also use Tor exit node. Either way, nothing is full proof. It is part of a solid defense in depth [1]. [1] https://en.m.wikipedia.org/wiki/Defense_in_depth_(computing) https://en.m.wikipedia.org/wiki/Defense_in_depth_(computing)
- philipwhiuk 2y agoYeah, the problem today is that there are many guides on MVPing software which don't tell you basic security. The guy doesn't have a software background. This is basically the problem with the 'everyone-should-code' approach - a little knowledge can be dangerous.
- immibis 2y agoBlocking China and Russia adds no security. Attacks can come from anywhere. And when you start blocking IPs, you may accidentally block yourself.
- ocdtrekkie 2y agoYes, but go further: Are you ever using your home server outside your own country? If the answer is no, block everything but your country. You can remove the vast majority of your attack surface by just understanding where you need to be exposed.
- wrren 2y agoI’d recommend using something like Tailscale for these use cases and general access, there’s no need to expose services to the internet much of the time.
- sweca 2y agoTailscale also has funnel which would be more secure for public exposure.
- rpadovani 2y ago> "None of the database guides I followed had warned me about the dangers of exposing a docker containerized database to the internet." This prompts a reflection about, as an industry, we should make a better job in providing solid foundations. When I check tutorials on how to drill in the wall, there is (almost) no warning about how I could lose a finger doing so. It is expected that I know I should be careful around power tools. How do we make some information part of the common sense? "Minimize the surface of exposure on the Internet" should be drilled in everyone, but we are clearly not there yet
- tossandthrow 2y agoJust like people shouldn't just buy industrial welding machines, SCUBA equipment or a parachute and "wing it" I think the same can be said here. As a society we already have the structures setup: The author had been more than welcome to attend a course or a study programme in server administration that would prepare them to run their own server. I myself even wouldn't venture into exposing a server to the internet to maintain it in my freetime, and that is with a post graduate degree in an engineering field and more than 20 years of experience.
- WaxProlix 2y agoYou can't just click a few buttons and have industrial machinery - and when you DO get it there's a ton of safety warnings on and around it. And I don't agree with your fundamental premise; self owned computing should be for everyone. It shouldn't be - at least for some subset of basics - arcane or onerous.
- fullspectrumdev 2y agoYou absolutely can. Have you a credit card and a web browser? You can buy all sorts of heavy machinery and have it shipped to your door!
- WaxProlix 2y agoYou've introduced a new element here - the credit card. And if you did have the money and whimsy it'd still show up with (regulated, mandatory, industry-standardized) safety documentation.
- AlgebraFox 2y agoTailscale is a great solution for this problem. I too run homeserver with Nextcloud and other stuff, but protected behind Tailscale (Wireguard) VPN. I can't even imagine exposing something like my family's personal data over internet, no matter how convenient it is. But I sympathize with OP. He is not a developer and it is sad that whatever software engineers produce is vulnerable to script kiddies. Exposing database or any server with a good password should not be exploitable in any way. C and C++ has been failing us for decades yet we continue to use such unsafe stacks.
- mattrighetti 2y ago> C and C++ has been failing us for decades yet we continue to use such unsafe stacks. I'm not sure — what do C and C++ have to do with this?
- timcambrant 2y agoThey are not memory safe by design. See: https://xeiaso.net/blog/series/no-way-to-prevent-this/ https://xeiaso.net/blog/series/no-way-to-prevent-this/ Of course all languages can produce insecure binaries, but C/C++ buffer overflows and similar vulnerabilities are likely what AlgebraFox refers to.
- mattrighetti 2y ago> They are not memory safe by design I'm aware of that, but the C/C++ thing seemed more like a rant, hence my question. I've searched up the malware and it doesn't seem to use memory exploitation. Rust is not going to magically protect you against any security issue caused by cloud misconfiguration.
- lopkeny12ko 2y agoWhat is the point you're trying to make here? Are you waiting for some malware that exploits a buffer overrun to infect you before conceding that C/C++ is a terrible choice for memory-safe code?
- matharmin 2y agoI think I'm missing something here - what is specific about Docker in the exploit? Nowhere is it mentioned what the actual exploit was, and whether for example a non-containerized postgres would have avoided it. Should the recommendation rather be "don't expose anything from your home network publically unless it's properly secured"?
- phoronixrly 2y agoFrom TFA: > This was somewhat releiving, as the latest change I made was spinning up a postgres_alpine container in Docker right before the holidays. Spinning it up was done in a hurry, as I wanted to have it available remotely for a personal project while I was away from home. This also meant that it was exposed to the internet, with open ports in the router firewall and everything. Considering the process had been running for 8 days, this means that the infection occured just a day after creating the database. None of the database guides I followed had warned me about the dangers of exposing a docker containerized database to the internet. Ofcourse I password protected it, but seeing as it was meant to be temporary, I didn't dive into securing it properly. Seems like they opened up a postgres container to the Internet (IIRC docker does this whether you want to or not, it punches holes in iptables without asking you). Possibly misconfigured authentication or left a default postgres password?
- globular-toast 2y ago> Seems like they opened up a postgres container to the Internet Yes, but so what? Getting access to a postgres instance shouldn't allow arbitrary execution on the host. > IIRC docker does this whether you want to or not, it punches holes in iptables without asking you Which is only relevant if you run your computer directly connected to the internet. That's a dumb thing to do regardless. The author probably also opened their firewall or forwarded a port to the host, which Docker cannot do.
- echelon_musk 2y agoAlso from TFA: > it was exposed to the internet, with open ports in the router firewall Upvoted because you're right that the comments in this thread have nothing to do with what happened here. The story would have been no different if OP had created an Alpine Linux container and exposed SSH to the internet with SSH password authentication enabled and a weak password. It's nothing to do with Docker's firewalling.
- acidburnNSA 2y agoI really like the "VPN into home first" philosophy of remote access to my home IT. I was doing openvpn into my ddwrt router fortunately years, and now it's wireguard into openwrt. It's quite easy for me to vpn in first and then do whatever: check security cams, control house via home assistant, print stuff, access my zfs shared drive, run big scientific simulations or whatever on big computer, etc. The router VPN endpoint is open to attack but I think it's a relatively small attack surface.
- rsolva 2y agoI use WireGuard to access all in-home stuff as well, but there is one missing feature and one bug with the official WireGuard app for android that is inconvenient: - Missing feature; do not connect when on certain SSIDs. - Bug: When the WG connection is active and I put my phone in Flightmode (which I do every night), it drains the battery from full to almost empty during the night.
- Mister_Snuggles 2y ago> - Missing feature; do not connect when on certain SSIDs. I'm very surprised by this omission as this feature exists on the official iOS client.
- 6ak74rfy 2y ago> I think it's a relatively small attack surface. Plus, you can obfuscate that too by using a random port for Wireguard (instead of the default 51820): if Wireguard isn't able to authenticate (or pre-authenticate?) a client, it'll act as if the port is closed. So, a malicious actor/bot wouldn't even know you have a port open that it can exploit.
- Mister_Snuggles 2y agoI've taken this approach as well. The WireGuard clients can be configured to make this basically transparent based on what SSID I'm connected to. I used to do similar with IPSec/IKEv2, but WireGuard is so much easier to manage. The only thing missing on the client is Split DNS. With my IPSec/IKEv2 setup, I used a configuration profile created with Apple Configurator plus some manual modifications to make DNS requests for my internal stuff go through the tunnel and DNS requests for everything else go to the normal DNS server. My compromise for WireGuard is that all DNS does to my home network but only packets destined for my home subnets go through the tunnel.
- joshghent 2y agoDespite people slating the author, I think this is a reasonable oversight. On the surface, spinning up a Postgres instance in Docker seems secure because it’s contained. I know many articles claim “Docker= Secure”. Whilst easy to point to common sense needed, perhaps we need to have better defaults. In this case, the Postgres images should only permit the cli, and nothing else.
- Fnoord 2y agoEvery guide out there says to link Postgres to the application (the one using Postgres). So the Postgres network is not reachable. Then, even if it were exposed, a firewall would need to be configured to allow access. Then, another thing every guide does is suggesting a reverse proxy, decreasing attack service. Then, such reverse proxy would need some kind of authentication. Instead, I simply run it behind Wireguard. There's still plenty to go wrong, such as backdoor in Postgres database image (you used docker pull), not upgrading it while it contains serious vulnerabilities, or a backdoor in some other image.
- lopkeny12ko 2y ago> spinning up a Postgres instance in Docker seems secure because it’s contained This doesn't make any sense. Running something in a container doesn't magically make it "secure." Where does this misconception come from?
- diggan 2y ago> Where does this misconception come from? When docker first appeared, a lot of people explaining docker to others said something along the lines "It's like a fast VM you can create with a Dockerfile", leading a bunch of people to believe it's actually not just another process + some more stuff, but instead an actual barrier between host/guest like in a proper VM. I remember talking about this a lot when explaining docker to people in the beginning, and how they shouldn't use it for isolation, but now after more than a decade with that misconception still being popular, I've lost energy about it...
- f00l82 2y agoWith the advent of wireguard. I no longer see a reason to expose anything outside of that to the outside world. Just VPN back in and work like usual.
- smarx007 2y agoDocker has a known security issue with port exposure in that it punches holes through the firewall without asking your permission, see https://github.com/moby/moby/issues/4737 https://github.com/moby/moby/issues/4737 I usually expose ports like `127.0.0.1:1234:1234` instead of `1234:1234`. As far as I understand, it still punches holes this way but to access the container, an attacker would need to get a packet routed to the host with a spoofed IP SRC set to `127.0.0.1`. All other solutions that are better seem to be much more involved.
- dawnerd 2y agoWe’ve found this out a few times when someone inexperienced with docker would expose a redis port and run docker compose up on a public accessible machine. Would only be minutes until that redis would be infected. Also blame redis for having the ability to run arbitrary code without auth by default.
- bluedino 2y agoContainers are widely used at our company, by developers who don't understand underlying concepts, and they often expose services on all interfaces, or to all hosts. You can explain this to them, they don't care, you can even demonstrate how you can access their data without permission, and they don't get it. Their app "works" and that's the end of it. Ironically enough even cybersecurity doesn't catch them for it, they are too busy harassing other teams about out of date versions of services that are either not vulnerable, or already patched but their scanning tools don't understand that.
- dijit 2y agoThis is pretty common, developers are focused on making things that work. Sysadmins were always the ones who focused on making things secure, and for a bunch of reasons they basically don’t exist anymore. EDIT: what guidelines did I break?
- ocdtrekkie 2y agoI suspect you'll find a lot of intersection between the move to "devops" outfits who "don't need IT anymore" and "there's a lot more security breaches now", but hey, everyone's making money so who cares?
- immibis 2y agoEvery server gets constantly probed for SSH. Since half the servers on the Internet haven't been taken over yet, it doesn't seem like SSH has significant exploits (well, there was that one signal handler race condition). Unless you're trying to do one of those designs that cloud vendors push to fully protect every single traffic flow, most people have some kind of very secure entry point into their private network and that's sufficient to stop any random internet attacks (doesn't stop trojans, phishing, etc). You have something like OpenSSH or Wireguard and then it doesn't matter how insecure the stuff behind that is, because the attacker can't get past it.
- Fnoord 2y agoOpenSSH has no currently known flaws but in past it contained a couple. For example, the xz backdoor utilized OpenSSH and it has contained a remote vulnerability in past (around 2003). Furthermore, some people use password auth as well as weak (low entropy or reused, breached) passwords. Instead, only use public key authentication. And tarpit the mofos brute forcing SSH (e.g. with fail2ban). They always do it on IPv4, not IPv6. So another useful layer (aside from not using IPv4) is whitelist IPv4 addresses who require access to SSH server. There is no reason for the entire world to need access to your home network's SSH server. Or, at the very least, don't use port 22. When in doubt: check your logs.
- wl 2y agoAlso, if you’re running SSH on a non-standard port, block Censys’ IP ranges. They port scan the entire internet constantly and bad actors use their scans to target their attacks. Once I did that, the number of attempted SSH logins went to zero pretty quickly.
- Fnoord 2y agoSolid advice! I've had certain countries in my blocklist thus far, and now I have added Censys (I did not know that was the company behind Shodan). Now, I've also added the Tor exit node list as my blocklist. Since nothing good comes from any of these. I used this blocklist for the latter [1] (the Censys ranges I just did manually, as it is only 12 entries in total). [1] https://github.com/7c/torfilter https://github.com/7c/torfilter
- elashri 2y agoThere usual route that people would take is either use VPN/tailscale/Clouflare Tunnels ..etc and only expose things locally and you will need to be on VPN network to access services. The other route is not to expose any ports and rely on reverse proxy. Actually you can combine the two approaches and it is relativity easy for non SWE homelab hobbyists.
- dawnerd 2y agoIf anyone is looking for one, https://nginxproxymanager.com/ https://nginxproxymanager.com/ Been using it for years and it’s been solid.
- zenoprax 2y agoI use HAProxy on PFSense to expose a home media server (among other services) for friends to access. It runs on a privileged LXC (because NFS) but as an unprivileged user. Is this reckless? Reading through all this makes me wonder if SSHFS (instead of NFS) with limited scope might be necessary.
- packtreefly 2y agoThat's a popular architecture, but I personally wouldn't run part of the application stack (HAProxy) on my network firewall, and would instead opt to move it to the media server. Suppose you have the media server in its own VLAN/Subnet, chances are good that the firewall is instrumental in enforcing that security boundary. If any part of the layer-7 attack surface is running on the firewall... you probably get the idea.
- zenoprax 2y agoInteresting, I never considered it part of the application stack. It routes a dozen or so separate services so it feels at home next to Wireguard, DHCP, and DNS.
- paulnpace 2y agoI had ~500 kbps sustained attacks against my residential ISP for years. I learned how to do everything through SSH - it is really an incredible Swiss Army knife of a tool.
- malfist 2y agoI remember working at a non-tech company and their firewall blocked anything not on port 22, 80 and 443. I got real good at port forwarding with ssh.
- _jplc 2y agoIm curious about what the password for postgres looked like. This exact thing happened to a friend, but there was no need to exploit obscure memory safety vulnerabilities, the password was “password”. My friend learnt that day that you can start processes in the machine from Postgres.
- fleetside72 2y agoThe exploit is not known in this case. The claim that it was password protected seems like an unverified statement. No pg_hba.conf content provided, this docker must have a very open default config for postgres. `Ofcourse I password protected it, but seeing as it was meant to be temporary, I didn't dive into securing it properly.`
- daneel_w 2y ago"None of the database guides I followed had warned me about the dangers of exposing a docker containerized database to the internet." Really? Nor any Docker guides warning in general about exposing a container to the Internet?
- phartenfeller 2y agoA lot of people comment about Docker firewall issues. But it still doesn't answer how an exposed postgres instance leads to arbitrary code execution. My guess is that the attacker figured out or used the default password for the superuser. A quick lookup reveals that a pg superuser can create extension and run some system commands. I think the takeaway here is that the pg image should autogenerate a strong password or not start unless the user defines a strong one. Currently it just runs with "postgres" as the default username and password.
- diggan 2y ago> I think the takeaway here is that the pg image should autogenerate a strong password or not start unless the user defines a strong one. Currently it just runs with "postgres" as the default username and password. Takeaway for beginner application hosters (aka "webmasters") is to never expose something on the open internet unless you're 100% sure you absolutely have to. Everything should default to using a private network, and if you need to accept external connections, do so via some bastion host that isn't actually hosted on your network, which reaches into your private network via proper connections.
- phartenfeller 2y agoThere are ton of hobby VPSes with a simple website, CMS, email server, or maybe a Plex. Do you really think that these kind of scenarios should use a bastion host?
- diggan 2y agoSince we're talking about a home network, definitely. Don't accept connections from public internet straight into your home unless you're OK with the consequences of that.
- Shank 2y agoFor all intents and purposes, the only ports you should ever forward are ones that are explicitly designed for being public facing, like TLS, HTTP, and SSH. All other ports should be closed. If you’re ever reaching for DMZ, port forwarding, etc., think long and hard about what you’re doing. This is a perfect problem for Tailscale or WireGuard. You want a remote database? Tailscale. I even get a weird feeling these days with SSH listening on a public interface. A database server, even with a good password/ACLs, just isn’t a great safe idea unless you can truly keep on top of all security patches.
- downrightmike 2y agoGood time to make sure UPnP is not enabled. Its an authenticationless protocol. Yeah, you read that right, no auth needed.
- skankhunt44 2y agoIt is more likely that the container image was already pwned, as other users had the same issue, see: https://unix.stackexchange.com/questions/757120/how-was-my-postgres-docker-container-hacked-infected https://unix.stackexchange.com/questions/757120/how-was-my-p.... Edit: It can also happen with the official image: https://www.postgresql.org/about/news/cve-2019-9193-not-a-security-vulnerability-1935/ https://www.postgresql.org/about/news/cve-2019-9193-not-a-se...
- gobblegobble2 2y ago> Fortunately, despite the scary log entries showing attempts to change privileges and delete critical folders, it seemed that all the malicious activity was contained within the container. OP can't prove that. The only way is to scrap the server completely and start with a fresh OS image. If OP has no backup and ansible repo (or anything similar) to configure a new home server quickly, then I guess another valuable lesson was learned here.
- diggan 2y agoNot 100% what you mean with "scrapping" the server, you suggest just a re-install OS? I'd default to assuming the hardware itself is compromised somehow, if I'm assuming someone had root access. If you were doing automated backups from something you assume was compromised, I'm not sure restoring from backups is a great idea either.
- sgarland 2y agoI think it’s reasonable to take a measured view of attacks. I doubt someone installing crypto miners has a hardware rootkit.
- diggan 2y agoI'm guessing it's an automated attack, where it found running services and then threw payloads at it until it got something. Once they're there, since docker isn't a real security barrier, I'd consider it all bets off. Especially when it comes to my home network, I would rather be safe than sorry. How would you even begin to investigate a rootkit since it can clean up after itself and basically make itself invisible? Particularly when it comes to Kinsing attacks, as there seem to been rootkits detected in tandem with it, which is exactly what OP got hit by it seems (although they could only see the coinminer).
- sgarland 2y agoFor crypto miners, it’s pretty easy to tell if your servers are in your house. Even if they aren’t, if you have any kind of metrics collection, you’ll notice the CPU spike. My general feeling is that if someone wants to install a hardware rootkit on my extremely boring home servers, it’s highly unlikely that I’ll be able to stop them. I can do best practices (like not exposing things publicly), but ultimately I can’t stop Mossad; on the other hand, I am an unlikely target for anything other than script kiddies and crypto miners.
- salmon 2y agoAlmost the same thing happened to me 2 years ago[1]. I didn’t understand the docker/ufw footgun at that time. [1] https://danthesalmon.com/ufw-docker-tailscale/ https://danthesalmon.com/ufw-docker-tailscale/
- Quizzical4230 2y agoMy self-hosting setup uses Cloudflare tunnels to host the website without opening any ports. And Tailscale VPN to directly access the machine. You may want to look at it!
- ocdtrekkie 2y agoDocker is a terrible self-hosting tool reason #4056. Let's give everyone a footgun and recommend they use it at work and their house. Especially with a tool you don't have an enterprise class firewall in front of, security needs to be automatic, not afterthought.
- 2OEH8eoCRo0 2y agoDocker's poor design strikes again!
- bennythomsson 2y ago> None of the database guides I followed had warned me about the dangers of exposing a docker containerized database to the internet. Um that's pretty ... naïve? Closing even your VPN port goes a little far. Wireguard for example doesn't even reply on the port if no acceptable signature is presented. Assuming the VPN software is recent, that's as close as you can come to having everything closed.
- j_bum 2y agoOk - curious if anyone can provide some feedback for me on this. I am running Immich on my home server and want to be able to access it remotely. I’ve seen the options of using wireguard or using a reverse proxy (nginx) with Cloudflare CDN, on top of properly configured router firewalls, while also blocking most other countries. Lots of this understanding comes from a YouTube guide I watched [0]. From what I understand, people say reverse proxy/Cloudflare is faster for my use case, and if everything is configured correctly (which it seems like OP totally missed the mark on here), the threat of breaches into to my server should be minimal. Am I misunderstanding the “minimal” nature of the risk when exposing the server via a reverse proxy/CDN? Should I just host a VPN instead even if it’s slower? Obviously I don’t know much about this topic. So any help or pointing to resources would be greatly appreciated. [0] https://youtu.be/Cs8yOmTJNYQ?si=Mwv8YlEf934Y3ZQk https://youtu.be/Cs8yOmTJNYQ?si=Mwv8YlEf934Y3ZQk
- depaulagu 2y agoPiggybacking on your request, I would also like feedback. I also run some services on my home computer. The setup I'm currently using is a VPN (Wireguard) redirecting a UDP port from my router to my PC. Although I am a Software Engineer, I don't know much about networks/infra, so I chose what seemed to me the most conservative approach.
- bennythomsson 2y agoTo both of you, wireguard is the way to go. So, parent poster: yes, you are doing it right. Grandparent: Use a VPN, close everything else.
- j_bum 2y agoThanks, Benny!
- RajT88 2y agoReverse proxy is pretty good - you've isolated the machine from direct access so that is something. I'm in the same boat. I've got a few services exposed from a home service via NGINX with a LetsEncrypt cert. That removes direct network access to your machine. Ways I would improve my security: - Adding a WAF (ModSecurity) to NGINX - big time investment! - Switching from public facing access to Tailscale only (Overlay network, not VPN, so ostensibly faster). Lots of guys on here do this - AFAIK, this is pretty secure. Reverse proxy vs. Overlay network - the proxy itself can have exploitable vulnerabilities. You should invest some time in seeing how nmap can identify NGINX services, and see if those methods can be locked down. Good debate on it here: https://security.stackexchange.com/questions/252480/blocking-nginx-from-nmap-version-detection https://security.stackexchange.com/questions/252480/blocking...
- LelouBil 2y agoI was exposing my services the same way for a long time, now I only expose web services via cloudflare, with an iptable configuration to reject everything on port 443 not coming from them. I also use knockd for port knocking to allow the ssh port, just in case I need to log in to my server without having access to one of my devices with Wireguard, but I may drop this since it doesn't seem very useful.
- OutOfHere 2y agoHow is it that your router allowing the incoming connection? It doesn't make sense.
- deleted 2y ago[deleted]
- rr808 2y agoI'd kinda like a VPN with my own storage but gave up this is one of the main reasons. I just dont have time to keep an eye on all this stuff.
- gus_ 2y agoThe article misses one critical point in these attacks: practically all these attacks require downloading remote files to the server once they gain access, using curl, wget or bash. Restricting arbitrary downloads from curl, wget or bash (or better, any binary) makes these attacks pretty much useless. Also these cryptominers are usually dropped to /tmp, /var/tmp or /dev/shm. They need internet access to work, so again, restricting outbound connections per binary usually mitigates these issues. https://www.aquasec.com/blog/kinsing-malware-exploits-novel-openfire-vulnerability/ https://www.aquasec.com/blog/kinsing-malware-exploits-novel-...
- packtreefly 2y ago> Restricting arbitrary downloads from curl, wget or bash (or better, any binary) makes these attacks pretty much useless. Any advice what that looks like for a docker container? My border firewall isn't going to know what binary made the request, and I'm not aware of per-process restrictions of that kind
- justinl33 2y agoa simple docker run --memory=512m --cpus=1 could have at least limited the damage here.
- Pooge 2y agoI don't think so; it would have made the malware harder to spot.
- behringer 2y agoYour first mistake is allowing chinese/russian traffic to your server...
- lmaoguy 2y ago“ None of the database guides I followed had warned me about the dangers of exposing a docker containerized database to the internet.” Unless you want it publicly available and expect people to try and illegitimately access it, never expose it to the internet.
- deleted 2y ago[deleted]