17 ms·

I got hacked: My Hetzner server started mining Monero

• whalesalad 10mo ago
  [flagged]
  • mrkeen 10mo ago
    Someone mined Monero on my server a few years ago. I was running Jenkins.
    • whalesalad 10mo ago
      that was a canon event
• guerrilla 10mo ago
  Whew, load average of 0 here.
• V__ 10mo ago
  > The Reddit post I’d seen earlier? That guy got completely owned because his container was running as root. The malware could: [...] Is that the case, though? My understanding was, that even if I run a docker container as root and the container is 100% compromised, there still would need to be a vulnerability in docker for it to “attack” the host, or am I missing something?
  • Onavo 10mo ago
    Either docker or a kernel level exploit. With non-VM containers, you are sharing a kernel.
  • ronsor 10mo ago
    There would be, but a lot of docker containers are misconfigured or unnecessarily privileged, allowing for escape. Also, if you've been compromised, you may have a rootkit that hides itself from the filesystem, so you can't be sure of a file's existence through a simple `ls` or `stat`.
    • miladyincontrol 10mo ago
      > but a lot of docker containers are misconfigured or unnecessarily privileged, allowing for escape Honestly, citation needed. Very rare unless you're literally giving the container access to write to /usr/bin or other binaries the host is running, to reconfigure your entire /etc, access to sockets like docker's, or some other insane level of over reach I doubt even the least educated docker user would do. While of course they should be scoped properly, people act like some elusive 0-day container escape will get used on their minecraft server or personal blog that has otherwise sane mounts, non-admin capabilities, etc. You arent that special.
      • fomine3 10mo ago
        I've seen many articles with `-v /var/run/docker.sock:/var/run/docker.sock` without scary warning
        • boomlinde 10mo ago
          What would the intended use case for that be?
          • jp191919 10mo ago
            Diun
      • cyphar 10mo ago
        As a maintainer of runc (the runtime Docker uses), if you aren't using user namespaces (which is the case for the vast majority of users) I would consider your setup insecure. And a shocking number of tutorials recommend bind-mounting docker.sock into the container without any warning (some even tell you to mount it "ro" -- which is even funnier since that does nothing). I have a HN comment from ~8 years ago complaining about this.
      • vultour 10mo ago
        Half the vendor software I come across asks you to mount devices from the host, add capabilities or run the container in privileged mode because their outsourced lowest bidder developers barely even know what a container is. I doubt even the smallest minority of their customers protest against this because apparently the place I work at is always the first one to have a problem with it.
  • TheRealPomax 10mo ago
    Docker containers with root have rootish rights on the host machine too because the userid will just be 0 for both. So if you have, say, a bind mount that you play fast and loose with, the docker user can create 0777 files outside the docker container, and now we're almost done. Even worse if "just to make it work" someone runs the container with --privileged and then makes the terminal mistake of exposing that container to the internet.
    • V__ 10mo ago
      Can you explain this a bit further? Wouldn't that 0777 file outside docker be still executed inside the container and not on the host?
      • necovek 10mo ago
        I believe they meant you could create an executable that is accessible outside the container (maybe even as setuid root one), and depending on the path settings, it might be possible to get the user to run it on the host. Imagine naming this executable "ls" or "echo" and someone having "." in their path (which is why you shouldn't): as long as you do "ls" in this directory, you've ran compromised code. There are obviously other ways to get that executable to be run on the host, this just a simple example.
        • marwamc 10mo ago
          Another example is they would enumerate your directories and find the names of common scripts and then overwrite your script. Or to be even sneakier, they can append their malicious code to an existing script in your filesystem. Now each time you run your script, their code piggybacks. OTH if I had written such a script for linux I'd be looking to grab the contents of $(hist) $(env) $(cat /etc/{group,passwd})... then enumerate /usr/bin/ /usr/local/bin/ and the XDG_{CACHE,CONFIG} dirs - some plaintext credentials are usually here. The $HOME/.{aws,docker,claude,ssh} Basically the attacker just needs to know their way around your OS. The script enumerating these directories is the 0777 script they were able to write from inside the root access container.
  • Havoc 10mo ago
    I think a root container can talk to docker daemon and launch additional containers...with volume mounts of additional parts of file system etc. Not particularly confident about that one though
    • minitech 10mo ago
      Unintentional vulnerabilities in Docker and the kernel aside, it can only do that if it has access to the Docker API (usually through a bind mount of the Unix socket). Having access to the Docker API is equivalent to having root on the host.
      • czbond 10mo ago
        Well $hit. I have been using Docker for installing NPM modules in interactive projects I was testing out. I believed Docker blocked access to the underlying host (my computer). Thanks for mentioning it - but now... how does one deal with this?
        • minitech 10mo ago
          If you didn’t mount docker.sock or any directory above it (i.e. / or /run by default) or run your containers as --privileged, you’re probably fine with respect to this angle. I’d still recommend rootless containers under unprivileged users* or VMs for extra comfort. Qubes (https://www.qubes-os.org/ https://www.qubes-os.org/) is good, even if it’s a little clunkier than it could be. * but if you’re used to bind-mounting, they’ll be a hassle Edit: This is by no means comprehensive, but I feel compelled to point it out specifically for some reason: remember not to mount .git writable, folks! Write access to .git is arbitrary code execution as whoever runs git.
          • deleted 10mo ago
            [deleted]
        • 3np 10mo ago
          As sibling mentioned, unless you or the runtime explicitly mount the docker socket, this particular scenario shouldn't affect you. You might still want to tighten things up. Just adding on the "rootless" part - running the container runtime as an unprivileged user on the host instead of root - you also want to run npm/node as unprivileged user inside the container. I still see many defaulting to running as root inside the container since that's the default of most images. OP touches on this. For rootless podman, this will run as a user with your current uid and map ownership of mounts/volumes: podman run -u$(id -u) --userns=keep-id
  • deleted 10mo ago
    [deleted]
  • d4mi3n 10mo ago
    While this is true, the general security stance on this is: Docker is not a security boundary. You should not treat it like one. It will only give you _process level_ isolation. If you want something with better security guarantees, you can use a full VM (KVM/QEMU), something like gVisor[1] to limit the attack surface of a containerized process, or something like Firecracker[2] which is designed for multi-tenancy. The core of the problem here is that process isolation doesn't save you from whole classes of attack vectors or misconfigurations that open you up to nasty surprises. Docker is great, just don't think of it as a sandbox to run untrusted code. 1. https://gvisor.dev/ https://gvisor.dev/ 2. https://firecracker-microvm.github.io/ https://firecracker-microvm.github.io/
    • socalgal2 10mo ago
      that's a really good point .. but, I think 99% of docker users believe it is a a sandbox and treat it as such.
      • freedomben 10mo ago
        And not without cause. We've been pitching docker as a security improvement for well over a decade now. And it is a security improvement, just not as much as many evangelists implied.
        • fragmede 10mo ago
          Must depend on who you've been talking to. Docker's not been pitched for security in the circles I run in, ever.
      • dist-epoch 10mo ago
        it is a sandbox against unintentional attacks and mistakes (sudo rm -rf /) but will not stop serious malware
      • TacticalCoder 10mo ago
        Not 99%. Many people run an hypervisor and then a VM just for Docker. Attacker now needs a Docker exploit and then a VM exploit before getting to the hypervisor (and, no, pwning the VM ain't the same as pwning the hypervisor).
  • michaelt 10mo ago
    Firstly, the attacker just wants to mine Monero with CPU, they can do that inside the container. Second, even if your Docker container is configured properly, the attacker gets to call themselves root and talk to the kernel. It's a security boundary, sure, but it's not as battle-tested as the isolation of not being root, or the isolation between VMs. Thirdly, in the stock configuration processes inside a docker container can use loads of RAM (causing random things to get swapped to disk or OOM killed), can consume lots of CPU, and can fill your disk up. If you consider denial-of-service an attack, there you are. Fourthly, there are a bunch of settings that disable the security boundary, and a lot of guides online will tell you to use them. Doing something in Docker that needs to access hot-plugged webcams? Hmm, it's not working unless I set --privileged - oops, there goes the security boundary. Trying to attach a debugger while developing and you set CAP_SYS_PTRACE? Bypasses the security boundary. Things like that.
  • Nextgrid 10mo ago
    Container escapes exist. Now the question is whether the attacker has exploited it or not, and what the risk is. Are you holding millions of dollars in crypto/sensitive data? Better assume the machine and data is compromised and plan accordingly. Is this your toy server for some low-value things where nothing bad can happen besides a bit of embarrassment even if you do get hit by a container escape zero-day? You're probably fine. This attack is just a large-scale automated attack designed to mine cryptocurrency; it's unlikely any human ever actually logged into your server. So cleaning up the container is most likely fine.
  • trhway 10mo ago
    >there still would need to be a vulnerability in docker for it to “attack” the host, or am I missing something? non necessary vulnerability per. se. Bridged adapter for example lets you do a lot - few years ago there were a story of something like how a guy got a root in container and because the container used bridged adapter he was able to intercept traffic of an account info updates on GCP
  • easterncalculus 10mo ago
    If the container is running in privileged mode you can just talk to the docker socket to the daemon on the host, spawn a new container with direct access to the root filesystem, and then change anything you want as root.
    • CGamesPlay 10mo ago
      Notably, if you run docker-in-docker, Docker is probably not a security boundary. Try this inside any dind container (especially devcontainers): docker run -it --rm --pid=host --privileged -v /:/mnt alpine sh I disagree with other commenters here that Docker is not a security boundary. It's a fine one, as long as you don't disable the boundary, which is as easy as running a container with `--privileged`. I wrote about secure alternatives for devcontainers here: https://cgamesplay.com/recipes/devcontainers/#docker-in-devcontainer https://cgamesplay.com/recipes/devcontainers/#docker-in-devc...
      • flaminHotSpeedo 10mo ago
        Containers are never a security boundary. If you configure them correctly, avoid all the footguns, and pray that there's no container escape vulnerabilities that affect "correctly" configured containers then they can be a crude approximation of a security boundary that may be enough for your use case, but they aren't a suitable substitute for hardware backed virtualization. The only serious company that I'm aware of which doesn't understand that is Microsoft, and the reason I know that is because they've been embarrassed again and again by vulnerabilities that only exist because they run multitenant systems with only containers for isolation
        • vel0city 10mo ago
          Virtual machines are never a security boundary. If you configure them correctly, avoid all the footguns, and pray that there's no VM escape vulnerabilities that affect "correctly" configured VMs then they can be a crude approximation of a security boundary that may be enough for your use case, but they aren't a suitable substitute for entirely separate hardware. Its all turtles, all the way down.
  • cyphar 10mo ago
    You really need to use user namespaces to get this kind of security protection -- running as root inside a container without user namespaces is not secure. Yes, breakouts often require some other bug or misconfiguration but the margin for error is non-existent (for instance, if you add CAP_SYS_PTRACE to your containers it is trivial to break out of them and container runtimes have no way of protecting against that). Almost all container breakouts in the past decade were blocked by user namespaces. Unfortunately, user namespaces are still not the default configuration with Docker (even though the core issues that made using them painful have long since been resolved).
• deleted 10mo ago
  [deleted]
• minitech 10mo ago
  > Here’s the test. If /tmp/.XIN-unix/javae exists on my host, I’m fucked. If it doesn’t exist, then what I’m seeing is just Docker’s default behavior of showing container processes in the host’s ps output, but they’re actually isolated. /tmp/.XIN-unix/javae & rm /tmp/.XIN-unix/javae This article’s LLM writing style is painful, and it’s full of misinformation (is Puppeteer even involved in the vulnerability?).
  • jakelsaunders94 10mo ago
    Yeah fair, I asked claude to help because honestly this was a little beyond my writing skills. I'm real though. Sorry. Will change
    • seafoamteal 10mo ago
      Hi Jake! Cool article, and it's something I'll keep in mind when I start giving my self-hosted setup a remodel soon. That said, I have to agree with the parent comment and say that the LLM writing style dulled what would otherwise have been a lovely sysadmin detective work article and didn't make me want to explore your site further. I'm glad you're up to writing more of your own posts, though! I'm right there with you that writing is difficult, and I've definitely got some posts on similar topics up on my site that are overly long and meandering and not quite good, but that's fine because eventually once I write enough they'll hopefully get better. Here's hoping I'll read more from you soon!
      • jakelsaunders94 10mo ago
        Thanks for the encouragement! I find it difficult to write articles beyond simply stating a series of facts. I tried handwriting https://blog.jakesaunders.dev/schemaless-search-in-postgres/ https://blog.jakesaunders.dev/schemaless-search-in-postgres/ bit I thought it came off as rambling. Maybe I'll have a go at redrafting this tomorrow in non LLM-ese.
        • lmf4lol 10mo ago
• _fqqr 10mo ago
  This article is very interesting at first but I once again get disappointed after reading clear signs of AI like "Why this matters" and "The moment of truth", and then the whole thing gets tainted with signs all over the place.
  • dinkleberg 10mo ago
    Yeah personally I’d much rather read a poorly constructed article with actually interesting content than the same content put into the formulaic AI voice.
    • venturecruelty 10mo ago
      Article's been edited: >Edit: A few people on HN have pointed out that this article sounds a little LLM generated. That’s because it’s largely a transcript of me panicking and talking to Claude. Sorry if it reads poorly, the incident really happened though! For what it's worth, this is not an excuse, and I still don't appreciate being fed undisclosed slop. I'm not even reading it.
• zamadatix 10mo ago
  I don't use Docker for my containers at home, but I take it by the concern that user namespacing is not the employed by them or something?
  • heavyset_go 10mo ago
    If you're root in a namespace and manage to escape, you can have root privileges outside of it.
    • zamadatix 10mo ago
      Are you referring to user namespaces and, if so, how does that kind of break out to host root work? I thought the whole point of user namespaces was your UID 0 inside the container is UID 100000 or whatever from the perspective of outside the container. Escaping the container shouldn't inherently grant you ability to change your actual UID in the host's main namespace in that kind of setup, but I'm not sure Docker actually leverages user namespaces or not. E.g. on my systemd-nspawn setup with --private-users=pick (enables user namespacing) I created a container and gave it a bind mount. From the container it appears like files in the bind mount created by the container namespace's UID 0 are owned by UID 0 but from outside the container the same file looks owned by UID 100000. Inverted, files owned by the "real" UID 0 on the host look owned by 0 to the host but as owned by 65534 (i.e. "nobody") from the container's perspective. Breaking out of the container shouldn't inherently change the "actual" user of the process from 100000 to 0 any more than breaking out of the container as a non-0 UID in the first place - same as breaking out of any of the other namespaces doesn't make the "UID 0" user in the container turn into "UID 0" on the host.
      • heavyset_go 10mo ago
        Users in user namespaces are granted capabilities that root has, user namespaces themselves need to be locked down to prevent that, but if a user with root capabilities escapes the namespace, they have the capabilities on the host. They also expose kernel interfaces that, if exploited, can lead to the same. In the end, namespaces are just for partitioning resources, using them for sandboxes can work, but they aren't really sandboxes.
• j45 10mo ago
  Never expose your server IP directly to the internet, vps or baremetal.
  • miramba 10mo ago
    Is there a way to do that and still be able to access the server?
    • deleted 10mo ago
      [deleted]
    • Carrok 10mo ago
      Many ways. Using a "bastion host" is one option, with something like wireguard or tinc. Tailscale and similar services are another option. Tor is yet another option.
      • venturecruelty 10mo ago
        >Never expose your server IP directly to the internet, vps or baremetal.
      • cortesoft 10mo ago
        The bastion host is a server, though, and would be exposed to the internet.
        • j45 10mo ago
          It can run a firewall and forward to internal traffic as well.
    • sh3rl0ck 10mo ago
      Either via a VPN or a tunnel.
    • iLoveOncall 10mo ago
      Yes, CloudFlare ZeroTrust. It's entirely free, I use it for loads of containers on multiple hosts and it works perfectly.
• iLoveOncall 10mo ago
  > ls -la /tmp/.XIN-unix/javae Unless ran as root this could return file not found because of missing permissions, and not just because the file doesn't actually exist, right? > “I don’t use X” doesn’t mean your dependencies don’t use X That is beyond obvious, and I don't understand how anyone would feel safe from reading about a CVE on a widely used technology when they run dozens of containers on their server. I have docker containers and as soon as I read the article I went and checked because I have no idea what technology most are built with. > No more Umami. I’m salty. The CVE was disclosed, they patched it, but I’m not running Next.js-based analytics anymore. Nonsensical reaction.
  • qingcharles 10mo ago
    Yeah, my Umami box was hit, but the time between the CVE disclosure and my box getting smacked was incredibly low. Umami patched it very quickly. And then patched it again a second time when the second CVE dropped right after. Nothing is immune. What analytics are you going to run? If you roll your own you'll probably leave a hole somewhere.
  • Hackbraten 10mo ago
    > No more Umami. I’m salty. But kudos for the word play!
• grekowalski 10mo ago
  Recently, those Monero miners were installing themselves everywhere that had a vulnerable React 19. I had exactly the same problem.
  • qingcharles 10mo ago
    I had to nuke my Oracle Cloud box that runs my Umami server. It got hit. Was a good excuse to upgrade version and upgrade all my backup systems etc. Lost a few hours of data while it was returning 500 errors.
  • tgsovlerkhgsel 10mo ago
    I love mining malware - it's reasonably visible and causes almost no damage. Essentially, it's like a bug bounty program that you don't have to manage, doesn't generate costly bullshit reports, and only costs you a few bucks of electricity when a vulnerability is found. If you have decent network or process level monitoring, you're likely to find it, while you might not realize the vulnerable software itself or some stealthier, more dangerous malware that might exploit it.
• codegeek 10mo ago
  tl:dr: He got hacked but the damage was only restricted to one docker container runn ing Umami (that is built on top of NextJS). Thankfully, he was running the docker container as a non privileged non-root user which saved him big time considering the fact that the attack surface was limited only within the container and could not access the entire host/filesystem. Is there ever a reason someone should run a docker container as root ?
  • d4mi3n 10mo ago
    If you're using the container to manage stuff on the host, it'll likely need to be a process running as root. I think the most common form of this is Docker-in-Docker style setups where a container is orchestrating other containers directly through the Docker socket.
• wnevets 10mo ago
  Sure does seem like the primary outcome of cryptocurrencies being released onto the world has been criminals making money.
  • dylan604 10mo ago
    Is that really a surprise though?
    • venturecruelty 10mo ago
      Not for anyone who doesn't have a financial stake in said fraud, no.
  • nrhrjrjrjtntbt 10mo ago
    And fast malware detection.
  • BLKNSLVR 10mo ago
    Criminals and the porn industry are almost invariably early adopters of new technologies. For better or worse their use-cases are proof-of-concepts that get expanded and built on, if successful, by more legitimate industries. Re: the Internet. Re: Peer-to-peer. Re: Video streaming. Re: AI.
    • lapetitejort 10mo ago
      What is the average length of time for new tech to escape porn and crime and integrate into real applications? Longer than 15 years?
      • BLKNSLVR 10mo ago
        Some kind of function of how quickly regulation comes to the technology.
    • denismenace 10mo ago
      How were criminals the early adopters of Internet, Video streaming and AI?
• meisel 10mo ago
  Is mining via CPU even worthwhile for the hackers? I thought ASICs dominated mining
  • jsheard 10mo ago
    ASICs do dominate Bitcoin mining but Monero's POW algorithm is supposed to be ASIC resistant. Besides, who cares if it's efficient when it's someone else's server?
    • deleted 10mo ago
      [deleted]
  • minitech 10mo ago
    Hard for it not to be worthwhile, since it’s free for them. Same automated exploit run across the entire internet.
  • Bender 10mo ago
    Optimal hardware costs money. Easy to hack machines are free and in nearly unlimited numbers.
  • justinsaccount 10mo ago
    If the effectiveness of mining is represented as profit divided by the cost of running the infrastructure, then a CPU that someone else is paying for is worth it as long as the profit is greater than zero.
  • heavyset_go 10mo ago
    Yes, for Monero it is the only real viable option. I'd also assume that the OP's instance is one of many other victims whose total mining might add up to a significant amount of crypto.
  • edm0nd 10mo ago
    Its easily worth it as they are not spending any money on compute or power. If they can enslave 100s or even 1000s of machine mining XMR for them, easy money if you set aside the legality of it.
• tolerance 10mo ago
  Was dad notified of the security breach? If not he may want to consider switching hosting providers. Dad deserves a proper LLM-free post mortem.
  • jakelsaunders94 10mo ago
    Hahaha, I did tell him this afternoon. This is the bloke who has the same password for all his banking apps despite me buying him 1password though. The imminent threat from RCE's just didn't land.
    • dylan604 10mo ago
      Buying someone 1Pass, or the like, and calling it good is not enough. People using password managers forget how long it takes to visit all of the sites you use to create that site's record, then update the password to a secure one, and then log out and log back in with the new password to test it is good. For a lot of people having a password manager bought for them is going to be over it after the second site. Just think about how many videos on TikTok they could have been watching instead
      • venturecruelty 10mo ago
        Yeah, mom and I sat down one afternoon and we changed all of her passwords to long, secure ones, generated by 1Password. It was a nice time! It also helped her remember all of the different services she needs to access, and now they're all safely stored with strong passwords. And it was a nice way to connect and spend some time together. :)
    • suspended_state 10mo ago
      Careful, HN isn't your average IRC channel.
• heavyset_go 10mo ago
  I wouldn't trust that boot image or storage again, I'd nuke it for peace of mind. That said, do you have an image of the box or a container image? I'm curious about it.
  • jakelsaunders94 10mo ago
    Yeah I did consider just killing it, I'm going to keep an eye on it for a few days with a gun to it just in case. I was lucky in that my DB backups were working so all my persistence wax backed up to S3. I think I could stand up another one in an hour. Unfortunately I didn't keep an image no. I almost didn't have the foresight to investigate before yeeting the whole box into the sun!
    • muppetman 10mo ago
      Enable connection tracking (if it's not already) and keep looking at the conntrack entires. That's a good way to spot random things doing naughty stuff.
• qingcharles 10mo ago
  As an aside, if you're using a Hetzner VPS for Umami you might be over-specced. I just cut my Hetzner bill by $4/mo by moving my Umami box to one of the free Oracle Cloud VPS after someone on here pointed out the option to me. Depends whether this is a hobby thing or something more serious, but that option is there.
  • tgtweak 10mo ago
    The manageability of having everything on one host is kind of nice at that scale, but yeah you can stack free tiers on various providers for less.
  • ianschmitz 10mo ago
    I would pay $4/mo to stay as far away from Oracle as possible
  • spiderfarmer 10mo ago
    I pay for Hetzner because it’s an EU based, sane company without a power hungry CEO.
    • mos87 10mo ago
      Orange Man Bad?!?
      • eb0la 10mo ago
        Yes: Orange clown very bad.
        • mos87 10mo ago
          >ebola lol.
  • angulardragon03 10mo ago
    All fine and well, but oracle will threaten to turn off your instance if you don’t maintain a reasonable average CPU usage on the free hosts, and will eventually do so abruptly. This became enough of a hassle that I stopped using them.
• tgtweak 10mo ago
  Just a note - you can very much limit cpu usage on the docker containers by setting --cpus="0.5" (or cpus:0.5 in docker compose) if you expect it to be a very lightweight container, this isolation can help prevent one roudy container from hitting the rest of the system regardless of whether it's crypto-mining malware, a ddos attempt or a misbehaving service/software.
  • freedomben 10mo ago
    This is true, but it's also easy to set at one point and then later introduce a bursty endpoint that ends up throttled unnecessarily. Always a good idea to be familiar with your app's performance profile but it can be easy to let that get away from you.
  • jakelsaunders94 10mo ago
    This is a great shout actually. Thanks for pointing it out!
  • fragmede 10mo ago
    The other thing to note is that docker is for the most part, stateless. So if you're running something that has to deal with questionable user input (images and video or more importantly PDFs), is to stick it on its own VM and then cycle the docker container every hour and the VM every 12, and then still be worried about it getting hacked and leaking secrets.
    • tgtweak 10mo ago
      Most of this is mitigated by running docker in an LXC containers (like proxmox does) which grants a lot more isolation than docker on it's own - closer in nature to running separate VMs.
      • butvacuum 10mo ago
        Too bad it straight doesn't work without heavy mods in pve9
• venturecruelty 10mo ago
  I still can't believe that there are so many people out here popping boxen and all they do is solve drug sudokus with the hardware. Hacks are so lame now.
• ryanto 10mo ago
  Sorry to hear you got hacked. I know we aren't supposed to rely on containers as a security boundary, but it sure is great hearing stories like this where the hack doesn't escape the container. The more obstacles the better I guess.
  • DANmode 10mo ago
    Hacks are humans. For like, ten more minutes anyway. If the human involved can’t escalate, the hack can’t.
• pigbearpig 10mo ago
  You might want to harden that those outbound firewall rules as another step. Did the Umami container need the ability to initiate connections? If not, that would eliminate the ability to do the outbound scans. Also could prevent something to exfiltrate sensitive data.
• danparsonson 10mo ago
  No firewall! Wow that's brave. Hetzner will let you configure one that runs outside of the box so you might want to add that too, as part of your defense in depth - that will cover you if you make a mistake with ufw. Personally I keep SSH firewalled only to my home address in this way; if I'm out and about and need access, I can just log into Hetzner's website and change it temporarily.
  • Nextgrid 10mo ago
    But the firewall wouldn't have saved them if they're running a public web service or need to interact with external services. I guess you can have the appserver fully firewalled and have another bastion host acting as an HTTP proxy, both for inbound as well as outbound connections. But it's not trivial to set up especially for the outbound scenario.
    • danparsonson 10mo ago
      No you're right, I didn't mean the firewall would have saved them, but just as a general point of advice. And yes a second VPS running opnSense or similar makes a nice cheap proxy and then you can firewall off the main server completely. Although that wouldn't have saved them either - they'd still need to forward HTTP/S to the main box.
      • Nextgrid 10mo ago
        A firewall blocking outgoing connections (except those whitelisted through the proxy) would’ve likely prevented the download of the malware (as it’s usually done by using the RCE to call a curl/wget command rather than uploading the binary through the RCE) and/or its connection to the mining server.
        • denkmoon 10mo ago
          How many people do proper egress filtering though, even when running a firewall
        • 10mo ago
• OutOfHere 10mo ago
  You're lucky that Hetzner didn't delete your server and terminate your account.
  • croemer 10mo ago
    With which justification?
    • OutOfHere 10mo ago
      Cryptocurrency software usage. It is strictly against their policy. Afaik, their policy does not differentiate with voluntary and involuntary use. They have done it to others.
  • mythrwy 10mo ago
    If they do that they don't get paid for the service the subsequent month.
• nodesocket 10mo ago
  I also run Umami, but patched once the CVE patch was released. Also, I only expose the tracking js endpoint and /api/send via Caddy publically (though, /api/send might be enough to exploit the vul). To actually interact with Umami UI I use Twingate (similar to Tailscale) to tunnel into the VPC locally.
• Computer0 10mo ago
  Still confused what I am supposed to do to avoid all this.
  • movedx 10mo ago
    Learning to manage an operating system in full, and having a healthy amount of paranoia, is a good first step.
    • doublerabbit 10mo ago
      Then, write all your own software to please the paranoia for the next 15 years. Next year is the 5th year of my current personal project. Ten to go.
  • nikanj 10mo ago
    Host your personal blog on wordpress.com or similar
• seymon 10mo ago
  What's considered nowadays the best practice (in terms of security) for running selfhosted workloads with containers? Daemon less, unprivileged podman containers? And maybe updating container images with a mechanism similar to renovate with "minimumReleaseTime=7days" or something similar!?
  • movedx 10mo ago
    You’ll set yourself up for success if you check the dependencies of anything you run, regardless of it being containerised. Use something like Snyk to scan containers and repositories for known exploits and see if anything stands out. Then you need to run things with as least privilege as possible. Sadly, Docker and containers in general are an anti-pattern here because they’re about convenience first, security second. So the OP should have run the contains as read-only with tight resource limits and ideally IP restrictions on access if it’s not a public service. Another thing you can do is use Tailscale, or something like it, to keep things being a zero trust, encrypted, access model. Not suitable for public services of course. And a whole host of other things.
  • elric 10mo ago
    As always: never run containers as root. Never expose ports to the internet unless needed. Never give containers outbound internet access. Run containers that you trust and understand, and not random garbage you find on the internet that ships with ancient vulnerabilities and a full suite of tools. Audit your containers, scan them for vulnerabilities, and nuke them from orbit on the regular. Easier said than done, I know. Podman makes it easier to be more secure by default than Docker. OpenShift does too, but that's probably taking things too far for a simple self hosted app.
• 3np 10mo ago
  > I also enabled UFW (which I should have done ages ago) I disrecommend UFW. firewalld is a much better pick in current year and will not grow unmaintainable the way UFW rules can. firewall-cmd --persistent --set-default-zone=block firewall-cmd --persistent --zone=block --add-service=ssh firewall-cmd --persistent --zone=block --add-service=https firewall-cmd --persistent --zone=block --add-port=80/tcp firewall-cmd --reload Configuration is backed by xml files in /etc/firewalld and /usr/lib/firewalld instead of the brittle pile of sticks that is the ufw rules files. Use the nftables backend unless you have your own reasons for needing legacy iptables. Specifically for docker it is a very common gotcha that the container runtime can and will bypass firewall rules and open ports anyway. Depending on your configuration, those firewall rules in OP may not actually do anything to prevent docker from opening incoming ports. Newer versions of firewalld gives an easy way to configure this via StrictForwardPorts=yes in /etc/firewalld/firewalld.conf.
  • arein3 10mo ago
    Does it have a gui?
  • exceptione 10mo ago
    > Specifically for docker it is a very common gotcha that the container runtime can and will bypass firewall rules and open ports anyway. Like I said in another comment, drop Docker, install podman.
    • 3np 10mo ago
      This affects podman too.
      • jsheard 10mo ago
        Not if you run it in rootless mode, which is more of a first class citizen in Podman compared to Docker.
        • 3np 10mo ago
• marwamc 10mo ago
  Hahaha OP could be in deep trouble depending on what types of creds/data they had in that container. I had replied to a child comment but I figure best to reply to OP. From the root container, depending on volume mounts and capabilities granted to the container, they would enumerate the host directories and find the names of common scripts and then overwrite one such script. Or to be even sneakier, they can append their malicious code to an existing script in the host filesystem. Now each time you run your script, their code piggybacks. OTOH if I had written such a script for linux I'd be looking to grab the contents of $(hist) $(env) $(cat /etc/{group,passwd})... then enumerate /usr/bin/ /usr/local/bin/ and the XDG_{CACHE,CONFIG} dirs - some plaintext credentials are usually here. The $HOME/.{aws,docker,claude,ssh} Basically the attacker just needs to know their way around your OS. The script enumerating these directories is the 0777 script they were able to write from inside the root access container.
  • jakelsaunders94 10mo ago
    Nothing in that container luckily, just what Umami needed to run, so no creds at all. Thanks for the info though!
  • cobertos 10mo ago
    Luckily umami in docker is pretty compartimentalized. All data is in the and the DB runs in another container. The biggest thing is the DB credentials. The default config requires no volume mounts so no worries there. It runs unprivileged with no extra capabilities. IIRC don't think the container even has bash, a few of the exploits that tried to run weren't able to due to lack of bash in the scripts they ran. Deleting and remaking the container will blow away all state associated with it. So there isn't a whole lot to worry about after you do that.
    • simulator5g 10mo ago
      You could just chain this with another exploit, just because it doesn’t run as root by default doesn’t mean it’s not a big deal.
• hoppp 10mo ago
  This nextjs vulnerability is gonna be exploited everywhere because its so easy. This is just the start
  • christophilus 10mo ago
    I didn’t think it was possible for me to dislike nextjs any more, but here we are. It’s the Sharepoint of the JS ecosystem.
• exceptione 10mo ago
  The first step I would take is running podman instead of Docker to prevent container escapes. Podman can be run truly rootless and doesn't mess with your firewall. Next I would drop all caps if possible.
  • doodlesdev 10mo ago
    What's the difference between running Podman and running Docker in rootless mode? (Other than Docker messing with the firewall, which apparently OP doesn't know about… yet). I understand Podman doesn't require a daemon, but is that all there is to it, or is there something I'm missing?
    • exceptione 10mo ago
      The runtime has been designed from the ground up to be run daemonless and rootless. They also have a K8s runtime, that has an extremely small surface, just enough to be K8s compliant. But podman has also great integration with systemd. With that you could use a socket activated systemd unit, and stick the socket inside the container, instead of giving the container any network at all. And even if you want networking in the container, the podman folks developed slirp4netns, which is user space networking, and now something even better: passt/pasta.
    • crimsonnoodle58 10mo ago
      Rootless docker is more compatible than podman I found. I experienced crash dumps in say mssql with podman, but not with rootless docker. Also rootless docker does not bypass ufw like rootful docker does.
• hughw 10mo ago
  You can run Docker Scout on one repo for free, and that would alert you that something was using Next.js and had that CVE. AWS ECR has pretty affordable scanning too: 9 cents/image and 1 cent/rescan. Continuous scanning even for these home projects might be worth it. [*] https://aws.amazon.com/inspector/pricing/ https://aws.amazon.com/inspector/pricing/
• xp84 10mo ago
  I wonder in a case like this how hard it would be to "steal" the crypto that you've paid to mine. But I assume these people are probably smart enough to where everything is instantly forwarded to their C&C server to prevent that.
  • akimbostrawman 10mo ago
    Unless you know the wallets seed phrase you can not access the mined funds. At best you could replace there wallet with your own to mine it yourself.
  • tgsovlerkhgsel 10mo ago
    There is no need for the node doing the mining calculations to have access to the private key of the payout wallet.
• croemer 10mo ago
  Not proof read by a human. It claims more than once the vulnerability was related to Puppeteer. Hallucination! "CVE-2025-66478 - Next.js/Puppeteer RCE)"
  • loloquwowndueo 10mo ago
    TFA mentions it’s mostly a transcript of a Claude session literally in the first paragraph.
    • croemer 10mo ago
      That doesn't excuse publishing errors.
    • themafia 10mo ago
      That was added as an edit. It does not cover the inaccuracies contained within. It should more realistically say "this article was generated by an LLM and may contain several errors which I didn't bother to find or correct."
      • loloquwowndueo 10mo ago
        That’s fair!
• egberts1 10mo ago
  This Monero mining also happened with one of my VPS over at interserv.net, when I forgot to log out of the root console in web-based terminal console to one of my VPS and closed its browser tab instead. It has since been fixed: Lesson learned.
  • tgsovlerkhgsel 10mo ago
    > when I forgot to log out of the root console in web-based terminal console to one of my VPS and closed its browser tab instead. This should not enable the compromise of your session, i.e. either there was some additional vulnerability (and not logging out widened the window during which it could be exploited), or the attack was totally unrelated. Either way, you should find out the real cause.
    • egberts1 10mo ago
      And they did.
• zrn900 10mo ago
  Just use Hetzner managed servers? Very high specs, they manage everything, and you can install a lot of languages, apps etc.
• kopirgan 10mo ago
  Only lesson seems to be use ufw! (or equivalent)
  • scottyeager 10mo ago
    As others have mentioned, a firewall might have been useful in restricting outbound connections to limit the usefulness of the machine to the hacker after the breach. An inbound firewall can only help protect services that aren't meant to be reachable on the public internet. This service was exposed to the internet intentionally so a firewall wouldn't have helped avoid the breach. The lesson to me is that keeping up with security updates helps prevent publicly exposed services from getting hacked.
    • kopirgan 10mo ago
      Yes thanks for the clarification.
• eyberg 10mo ago
  a) containers don't contain b) if you want to limit your hosting environment to only the language/program you expect to run you should provision with unikernels which enforce it
  • tgsovlerkhgsel 10mo ago
    > a) containers don't contain Except it seems to have done so in this case?
• elif 10mo ago
  This is a perfect example of how honeypots, anti-malware organizations, and blacklists are so important to security. Even if you are an owasp member who reads daily vulnerability reports, it's so easy to think you are unaffected.
• LelouBil 10mo ago
  Something similar happened to me last year, it was with an unsecured user account accessible over ssh with password authentication, something like admin:admin that I forgot about. At least that's what I think happened because I never found out exactly how it was compromised. The miner was running as root and it's file was even hidden when I was running ls ! So I didn't understand what was happening, it was only after restarting my VPS from with a rescue image, and after mounting the root filesystem, that I found out the file I was seeing in the processes list did indeed exist.
• CGamesPlay 10mo ago
  I took issue with this paragraph of the article, on account of several pieces of misinformation, presumably courtesy of Claude hallucinations: > Here’s the test. If /tmp/.XIN-unix/javae exists on my host, I’m fucked. If it doesn’t exist, then what I’m seeing is just Docker’s default behavior of showing container processes in the host’s ps output, but they’re actually isolated. 1. Files of running programs can be deleted while the program is running. If the program were trying to hide itself, it would have deleted /tmp/.XIN-unix/javae after it started. The nonexistence of the file is not a reliable source of information for confirming that the container was not escaped. 2. ps shows program-controlled command lines. Any program can change what gets displayed here, including the program name and arguments. If the program were trying to hide itself, it would change this to display `login -fp ubuntu` instead. This is not a reliable source of information for diagnosing problems. It is good to verify the systemd units and crontab, and since this malware is so obvious, it probably isn't doing these two hiding methods, but information-stealing malware might not be detected by these methods alone. Later, the article says "Write your own Dockerfiles" and gives one piece of useless advice (using USER root does not affect your container's security posture) and two pieces of good advice that don't have anything to do with writing your own Dockerfiles. "Write your own Dockerfiles" is not useful security advice.
  • 3np 10mo ago
    > "Write your own Dockerfiles" is not useful security advice. I actually think it is. It makes you more intimate with the application and how it runs, and can mitigate one particular supply-chain security vector. Agreeing that the reasoning is confused but that particular advice is still good I think.
• esaym 10mo ago
  So this is part of the "React2Shell" CVE-2025-55182 issue? I find it interesting that this seems to get so little publicity. Almost like the issue is normal or expected. And it looks like the affected versions go back a little over a year. So if you've deployed anything with Next.js over the last 12 months your web app is now probably part of a million node bot net. And everyone's advice is just "use docker" or "install a firewall". I'm not even sure what to say, or think, or even how to feel about the frontend ecosystem at this point. I've been debating on leaving the whole "web app" ecosystem as my main employment ventures and applying to some places requiring C++. C++ seems much easier to understand than what ever the latest frontend fad is. /rant
  • h33t-l4x0r 10mo ago
    I'm hearing about it like crazy because I deployed around 100 Next frontends in that time period. I didn't use server components though so I'm not affected.
    • mnahkies 10mo ago
      My understanding of the issue is that even if you don't use server components, you're still vulnerable. Unless you're running a static html export - eg: not running the nextjs server, but serving through nginx or similar
      • abustamam 10mo ago
        Yeah, crucially it says > If your app’s React code does not use a server, your app is not affected by this vulnerability. If your app does not use a framework, bundler, or bundler plugin that supports React Server Components, your app is not affected by this vulnerability. https://react.dev/blog/2025/12/03/critical-security-vulnerability-in-react-server-components#immediate-action-required https://react.dev/blog/2025/12/03/critical-security-vulnerab... So if you have a backend that supports RSC, even if you don't use it, you can still be vulnerable. GP said they only shipped front ends but that can mean a lot. Edit:link
• aborsy 10mo ago
  If I’m not wrong, a hetzner VM by default has no firewall enabled. If you are coming from providers with different default settings, that might bite you. Containers that you thought were not open to internet have been open all this time. Two firewalls failed: They bypassed ufw and there was no external firewall either. You have to define a firewall policy and attach it to the VM.
• p0w3n3d 10mo ago
  $ sudo ufw default deny incoming $ sudo ufw default allow outgoing $ sudo ufw allow ssh $ sudo ufw allow 80/tcp $ sudo ufw allow 443/tcp $ sudo ufw enable As a user of iptables this order makes me anxious. I used to cut myself out from the server many times because first blocking then adding exceptions. I can see that this is different here as the last command commits the rules...
  • kgeist 10mo ago
    I had this one too: I first denied all incoming requests and was about to allow SSH, but my SSH connection dropped :) Fortunately, I was able to restore the VM with the provider's VM console.
    • p0w3n3d 10mo ago
      I have a funny story, when I did it in the startup script, then I ran it. I lost my ssh, and moreover this server was in another country, France. And moreover I set up the internal keyboard layout there to be US as I am writing without looking at the keyboard. In the result, the Polish-French guy there, who was at site, was unable to enter the password correctly unless I translated it to the french keyboard for him.
• kachapopopow 10mo ago
  I find it interesting that the recent trend of moving to self-hosted solutions is sparking this rediscovery of security issues that come with self-hosting. One more time and it will be a cycle!
  • Aachen 10mo ago
    What trend? All I'm seeing here is further centralisation: Search engines try to fight slop results with collateral damage mostly in small or even personal websites. Restaurants are happy to be on one platform only: Google Maps. Who needs an expensive website if you're on there and someone posts your menu as one of the pictures? (Ideally an old version so the prices seem cheaper and you can't be pinned down for false advertising.) Open source communities use Github, sometimes Gitlab or Codeberg, instead of setting up a Forgejo (I host a ton of things myself but notice that the community effect is real and also moved away from self hosting a forge). The cherry in top is when projects use Discord chats as documentation and bug reporting "form". Privacy people use Signal en masse, where Matrix is still as niche as it was when I first heard of it. The binaries referred to as open source just because they're downloadable can be found on huggingface, even the big players use that exclusively afaik. Some smaller projects may be hosted on Github but I have yet to see a self-hosted one. Static websites go on (e.g. Github) Pages and back-ends are put on Firebase. Instead of a NAS, individuals as well as small businesses use a storage service like Onedrive or Icloud. Some more advanced users may put their files on Backblaze B2. Those who newly dip their toes in self-hosting increasingly use a relay server to reach their own network, not because they need it but to avoid dealing with port forwarding or setting up a way to privately reach internal services. Security cameras are another good example of this: you used to install it, set a password, and forward the port so you can watch it outside the home. Nowadays people expect that "it just works" on their phone when they plug it in, no matter where they are. That this relies on Google/Amazon and that they can watch all the feeds is acceptable for the convenience. And that's all not even mentioning the death of the web: people who don't use websites anymore the way they were meant (as hyperlinked pages) but work with an LLM as their one-stop shop Not that the increased convenience, usability, and thus universal accessibility of e.g. storage and private chats is necessarily bad, but the trend doesn't seem to me as you seem to think it is I can't think of any example of something that became increasingly often self-hosted instead of less across the last 1, 5, or 10 years If you see a glimmer of hope for the distributed internet, do share because I feel increasingly as the last person among my friends who hosts their own stuff
• spoaceman7777 10mo ago
  > "No more exposed PostgreSQL ports, no more RabbitMQ ports open to the internet." Yikes. I would still recommend a server rebuild. That is _not_ a safe configuration in 2025, whatsoever. You are very likely to have a much better engineered persistent infection on that system.
  • microtonal 10mo ago
    Also, apparently they run an IoT platform for other users on the same host that cannot only visualize sensors, but also trigger (mains-powered) devices. The right thing to do is to roll out a new server (you have a declarative configuration right?), migrate pure data (or better, get it from the latest backup), remove the attacked machine off the internet to do a full audit. Both to learn about what compromises there are for the future and to inform users of the IoT platform if their data has been breached. In some countries, you are even required by law to report breaches. IANAL of course.
• tgsovlerkhgsel 10mo ago
  Would "user root" without --privileged and excessive mounts have enabled a container escape, or just exposed additional attack surface that potentially could have allowed the attacker to escape if they had another exploit?
  • PlqnK 10mo ago
    They would need a vulnerability in containerd or the kernel to escape the sandbox and being root in the sandbox would give them more leeway to exploit that vulnerability. But if they do have a vulnerability and manage to escape the sandbox then they will be root on your host. Running your processes as an unprivileged user inside your containers reduces the possibility of escaping the sandbox, running your containers themselves as un unprivileged user (rootless podman or docker for example) reduces the attack surface when they manage to escape the sandbox.
• gppmad 10mo ago
  Well written blog post. Well done, I've learned something new.
• mxxc 10mo ago
  regardless of firewalls and best practices and all, i'd just put all sorts of admin-related stuff behind tailscale (or similar), including ssh. hetzner allows you to have ssh closed in your public ip and still open a terminal via the console if ssh-on-tailscale fails. for the web stuff you should do a similar trick. blog and public websites on the public address, while admin stuff goes on tailscale. and if you do it nicely with letsencrypt you can even have nice hostnames pointing to your private stuff.
• cachius 10mo ago
  Does anybody know how to just list the processes running inside a single container from within that container? And isn’t it a design flaw if you can see all processes from inside a container? This could provide useful information for escaping it.
• dilippkumar 10mo ago
  I’m sorry you went through this. But I am interested in the monero aspect here. Should I treat this as some datapoint on monero’s security having held up well so far?
  • Hendrikto 10mo ago
    The main reason to use Monero for stuff like this is their mining algo. They made big efforts and changed algorithms several times to make and keep it GPU and ASIC resistant. If you used the server to mine Bitcoin, you would make approximately zero (0) profit, even if somebody else pays for the server. But also yes, Monero has technically held up very well.
    • cylemons 10mo ago
      Didn't Qubic manage to attack Monero?
      • akimbostrawman 10mo ago
        They tried to do a 51% attack which at worst could result in double spends. They have never reached more than 35%. The attack did not and could not compromise or weaken moneros privacy and anonymity features.
  • tgsovlerkhgsel 10mo ago
    > Should I treat this as some datapoint on monero’s security having held up well so far? No. The reason attackers mine Monero and not some other cryptocurrency isn't the anonymity. Most cryptocurrencies aren't (meaningfully) mineable with CPUs, Monero (apparently) is. There may be others, but I suspect that they either deliver less $ per CPU-second (due to not being valuable), or are sufficiently unknown and/or painful to set up that the attackers just go with the known default. Trying to mine Bitcoin directly would be pointless, you'd get no money because you're competing with ASIC miners. Some coins were designed to be ASIC resistant, these are mostly mined on GPUs. Monero (and some other coins) were designed to also be GPU resistant (I think). You could see it as a sign that that property has held up (well enough), but nothing else.
• rendaw 10mo ago
  I didn't see it mentioned, but wouldn't having a RO root filesystem with writable directories mounted noexec also have been sufficient?
• bradley13 10mo ago
  I had something similar, but I was lucky enough to catch it myself. I've used SSH for years, and never knew that it - by default - also accepts password logins. Maybe dumb on my part, but there you go...
• broken_broken_ 10mo ago
  I am not an expert in incident reaction, but I thought the safe way was to image the affected machine, turn it off, take a clean machine, boot a clean OS image with the affected image mounted read only in a VM, and do the investigation like that ? Assume that the malware has replaced system commands, possibly used a kernel vulnerability to lie to you to hide its presence, so do not do anything in the infected system directly ?
  • dewey 10mo ago
    Maybe if your company infrastructure is affected but not the server you use to host your side projects on with “coolify” unless IT security is your hobby.
• jlengrand 10mo ago
  TY for the article. You made me look into my server, and also found some strange activity on my docker containers. Good call!
• mos87 10mo ago
  so what's the point of containers here? seems only to make things less transparent and more complex to manage. js scripts running on frameworks running inside containers PS so I see the host ended up staying uncompromised
• ChrisMarshallNY 10mo ago
  Good job. Examples like this, are why I don’t run a VPS. I could definitely do it (I have, in the past), but I’m a mediocre admin, at best. I prefer paying folks that know their shit to run my hosting.
• andix 10mo ago
  I was surprised by the same thing, a tool I run that uses Next.js without me knowing before. I noticed this, because Hetzner forwarded me an email from the German government agency for IT security (BSI) that must have scanned all German based IP addresses for this Next.js vulnerability. It was a table of IP addresses and associated CVEs. Great service from their side, and a lot of thanks to the German tax payers wo fund my free vulnerability scans ;)
• kalaksi 10mo ago
  After reading some comments: this probably goes without saying, but one should be very careful what to expose to the internet. Sounds like the analytics-service maybe could have been available only over VPN (or similar, like mTLS etc.) And for basic web sites, it's much better if it requires no back-end. Every service exposed increases risk and requires additional vigilance to maintain. Which means more effort.
• hnarn 10mo ago
  Interesting that this got posted today, I also have a server on Hetzner (although I don't think it's relevant) and noticed yesterday that a Monero miner had been installed. Luckily for me, the software I had installed[1] was in an LXC container running under Incus, so the intrusion never escaped the application environment, and the container itself was configured with low CPU priority so I didn't even notice it until I tried to visit the page and it didn't load. I looked around a bit and it seemed like an SSH key had been added under the root user, and there were some kind of remote management agents installed. This container was running Alpine so it was pretty easy to identify what processes didn't belong from a simple ps output of the remaining processes after shutting down the actual web application. In the end, I just scrapped the container, but I did save it in case I ever feel like digging around (probably not). In the end I did learn some useful things: - It's a good idea to assume your system will get taken over, so ensure it's isolated and suitably resource constrained (looking at you, pay-as-you-go cloud users). - Make sure you have snapshots and backups, in my case I do daily ZFS snapshots in Incus which makes rolling back to before the intrusion a breeze. - While ideally anything compromised should be scrapped, rolling back, locking it down and upgrading might be OK depending on the threat. Regarding the miner itself: - from what I could see in its configuration it hadn't actually been correctly configured, so it's possible they do some kind of benchmark and just leave the system silently compromised if it's not "worth it", they still have a way in to use it for other purposes. - no attempt had been made at file system obfuscation, which is probably the only reason I really discovered it. There were literally folders in /root lying around with the word "monero" in them, this could have been easily hidden. - if they hadn't installed a miner and just silently compromised the system, leaving whatever running on it alone (or even doing a better job at CPU priority), I probably never would have noticed this. [1]: https://github.com/umami-software/umami https://github.com/umami-software/umami
• nikanj 10mo ago
  It makes me irrationally angry that cryptos have given a clear monetary value to raw CPU time, and a very strong incentive to create botnets.
• majorbugger 10mo ago
  OK, so am I right that this guy had a completely unsecured metrics endpoint running on his server? Why would you do that in the first place?
• nunodonato 10mo ago
  The world will be a better place when all crypto just disappears
  • tgsovlerkhgsel 10mo ago
    Would it be better for the victim if that was ransomware (asking for Apple gift cards) or some malware that stealthily siphons off data until it finds something valuable?
• prmoustache 10mo ago
  Author forgot one important take: to limit the attack surface what doesn't need to be public facing should not be public facing. Things such as analytics software can easily be accessed through wireguard or even simpler using ssh as socks proxy.
• grugdev42 10mo ago
  Roughly 20 years ago, PHP sites were getting hacked everywhere. Now, JS sites everywhere are getting hacked. JS has turned into the very thing it strived to replace. It's good to see developers haven't changed. ;)
  • urban_alien 10mo ago
    It's almost like... the problem wasn't the tool at all :P
  • TwoNineFive 10mo ago
    Webdevs still security ignorant. Actually just ignorant. Not sorry to say that because it's beyond true.
• IlikeMadison 10mo ago
  I don't think using key-based authentication for SSH and enabling Fail2ban is necessary. Fail2ban is only useful if you keep password authentication. But I might be wrong.
  • Sohcahtoa82 10mo ago
    I should check my SSH logs. My intuition is that since the SSH server reports what auth methods are available, once a bot sees that password auth is disabled, they will disconnect and not try again. But I also know that bots can be dumb.
• hank2000 10mo ago
  I've bene working with a GPU security company for the last few months... I can tell you that neo clouds (generally) do not see security as a high priority—or often, even their responsibility. Many do not have hte ability to even know if your GPUs have been compromised and they expect you'll take responsibility. Meanwhile companies think the clouds are looking at it.... anyhow. it is a real problem.
  • Sohcahtoa82 10mo ago
    > I can tell you that neo clouds (generally) do not see security as a high priority—or often, even their responsibility. AWS explicitly spells this out in their Shared Responsibility Model page [0] It is not your cloud provider's responsibility to protect you if you run outdated and vulnerable software. It's not their responsibility to prevent crypto-miners from running on your instances. It's not even their responsibility to run a firewall, though the major players at least offer it in some form (ie, AWS Security Groups and ACL). All of that is on the customer. The provider should guarantee the security of the cloud. The customer is responsible for security in the cloud. [0] https://aws.amazon.com/compliance/shared-responsibility-model/ https://aws.amazon.com/compliance/shared-responsibility-mode...
• scottydelta 10mo ago
  I have a similar setup but with following additional security configurations: - Hetzner firewall (because ufw doesn't work well with docker) to only allow public access to 443. - Self-hosted OpenVPN to access all private ports. I also self-host an additional Wireguard instance as a backup VPN. - Cloudflare Access to protect `*.coolifydomain.com` by default. This would have helped protect the OP's Umami setup since only the OP can access the Umami dashboard. Bypass rules can be created in Cloudflare Access to allow access to other systems that need access using IP or domain. - Cloudflare Access rules to only allow access to internal admin path such as /wp-admin/ through my VPN IP (or via email OTP to specified email ids). - Traefik labels on docker-compose files in Coolify to add basic auth to internal services which can't be behind Cloudflare Access such as self-hosted Prefect. This would have also added a login screen before an attacker would see Umami's app. - I host frontends only on Vercel or Cloudflare workers and host the backend API on the Coolify server. Both of these are confirmed to never have been affected, due to decoupling of application routing. - Finally a bash cron script running on server every 5 minutes that monitors the resources and sends me an alert via Pushover when the usages are above the defined thresholds. We need monitoring and alert as well, security measures alone are not enough. Even with all these steps, there will always be edge cases. That's the caveat of self-hosting but at the same time it's very liberating.
• throwawayffffas 10mo ago
  > I also enabled UFW (which I should have done ages ago): Docker will overwrite your rules when you publish ports. Do not publish ports with docker. Do not run internal services on the publicly accessible system.
• racl101 10mo ago
  This is weird. I viewed this blog post on Chrome and it loaded fine. But I sent the link to my fellow dev and he tried viewing it on Microsoft Edge on MacOS but the browser showed a red page with the "This site has been reported as unsafe" message by the Microsoft Defender SmartScreen. It highlighted the domain: 'jakesaunders.dev' in the address bar in red text.
• eikowagenknecht 10mo ago
  I got another mail from Hetzner on Dec 10, telling me that the BSI told them my web site was having serious security problems with Next.JS. Not having high opinions on the BSI, I first thought this was some very elaborate scam attack but no, it turns out it was legit and also Umamis inbuilt Next.JS for me. But apparently there was no crypto miner or other active abuse yet.
• rishikeshs 10mo ago
  What a coincidence. I also received a similar email and upon investigating, found the same Monero mining on my server. Tried rebuilding the image, but same thing was poping up. Then I did a npm update and that fixed everything!