21 ms·
A Docker footgun led to a vandal deleting NewsBlur's MongoDB database
- deleted 5y ago[deleted]
- killingtime74 5y agoWere you using Docker raw? No orchestration? If so what's the advantage to dockerizing your data stores
- spatley 5y agoHaving all of the context from operating system version to all the dependent software packages has terrific advantages in deployment and provisioning. One single command for the predictable and repeatable launch of the DB and all the services is super useful even without Kubernetes or other orchestration.
- cratermoon 5y agoNewsBlur was using Ansible. https://www.ansible.com/use-cases/orchestration https://www.ansible.com/use-cases/orchestration
- sneak 5y agoThe mistake here is a MongoDB that didn't require authentication, not that docker's clunky iptables setup exposed it to the internet. Relying solely on a host-based firewall for access control is, for reasons which must now be obvious, admin incompetence. They are responsible for securing the containers. They didn't. Your services should be using authentication even if they are only bound to localhost.
- chipotle_coyote 5y ago> The mistake here is a MongoDB that didn't require authentication, not that docker's clunky iptables setup exposed it to the internet. If you're backing up diligently, have a problem, go to restore backup, and discover that the backups are unusable due to an issue caused by the backup software's default configuration, you could say "the mistake here is you relying on a single backup system, pardner, not the backup software's bad default configuration," and in the sense that it's good advice to have multiple backup strategies, sure, you're technically correct. But that doesn't excuse the backup software from having a dangerously bad default configuration. That would be relatively trivial to fix. That they've apparently known about for years.
- edoceo 5y agoYea. Why blame softwares (with multiple user filed bugs) when you can blame the user (again)?
- qbasic_forever 5y agoThe docker issue and original HN thread have countless instances of folks who have hit and been hacked by the exact same issue. There is clearly a major problem with the docker documentation, usage, etc. that is causing people to continually be taken by surprise with its iptables behavior. The blame game doesn't help solve the real problem that people are unknowingly putting services directly on the public internet when they think they are secured by a firewall. With how many millions of people use docker every day IMHO the severity of this issue is far too great to ignore. A simple mitigation to fail and warn a user when it appears they might be hitting this issue (i.e. before docker run changes iptables config, give a quick look to see if it looks like it's already configured for the most common ufw or other setup) would stop this from happening for 99% of cases.
- kortex 5y agoI consider myself pretty docker savvy. I had no idea about this footgun, and my brain is full of all kinds of docker minutiae. Everyone knows docker runs as root, and therefore be careful using it, just as you would sudo. It seems this is a major fail in docs. I kinda get why it's the default, but it needs to be made way better known that by default -p punches a hole through ufw.
- sneak 5y agoIf anything here has a bug it's the mongo container image that launches without requiring authentication, not docker. You're confusing the proximate cause for the root cause.
- mdellavo 5y agoI don't know why this is being downvoted. Multiple overlapping layers of security would have given newsblur a backup in case of accidental "footguns". Unauthenticated mongodb instances are a pretty common problem - it's why a "script kiddie" was so successful.
- qbasic_forever 5y agoA simple runtime check by docker to detect the most common configuration issue (ufw enabled for a service port that's about to be run by a container) is just another layer of protection to add to the stack IMHO. How can it hurt?
- eganist 5y ago> I don't know why this is being downvoted. Multiple overlapping layers of security would have given newsblur a backup in case of accidental "footguns". Security specialist here. Startups are built by generalists. Good decisions were made here which made other defense-in-depth considerations not as critical when reconciled with go-to-market needs. If every generalist focused on every security risk in their product, they'd expose themselves to the business risk of not moving quickly enough. It's pretty clear NewsBlur did what they could and relied on the expertise of others to not fail them in exceptionally basic, entirely avoidable ways. They were betrayed by that reliance.
- arp242 5y ago> I don't know why this is being downvoted. The core message ("using auth on MongoDB would have prevented this, it's always a good idea to add password auth just in case") is perfectly reasonable; we can all learn from this, and it's perfectly fine to point out such things. But the way it was phrased was absolutely not okay. People make mistakes all the time and they are not "incompetent". This is the classic "I am very smart, I never make mistakes, if you made a mistake then you're an idiot. You probably eat poop. I am smart btw"-attitude that's just ... ugh... People rely on firewalls to prevent mistakes from becoming disastrous. Defeating that silently is super surprising. People don't know everything about every piece of tech they use either; very few people do: it's just too much information. And it's not like auth alone is perfect. Remember when a bug in MySQL allowed people to bypass the auth? Good thing I put a firewall in front of my Doc... oh, no, wait... So this is why I downvoted it.
- yjftsjthsd-h 5y ago> The mistake here is a MongoDB that didn't require authentication, not that docker's clunky iptables setup exposed it to the internet. It can be both.
- jbverschoor 5y agoIt’s both indeed.. start to blame redis too. It’s a docker problem, which should listen to localhost by default
- mnisjk2 5y agoAgree that authentication should be table stakes, though I would argue that the actual mistake is that the MongoDB application/docker container was on a host/VM with a network interface on the public internet.
- eganist 5y agoIt's not all that great blaming the victim. They clearly made the right moves with at least some of their configuration decisions and leaned on the underlying platform not being bonkers (but alas, it was: https://github.com/moby/moby/issues/4737 https://github.com/moby/moby/issues/4737 and https://github.com/moby/moby/issues/22054 https://github.com/moby/moby/issues/22054). Should they have hardened in all the other ways for defense in depth, e.g requiring authentication from localhost? Sure. Should the attacker have not done the attack? Without a doubt. But should docker have not completely undone a security control? Yes, and the fact that docker seems to have persisted with the current state is the topic of discussion. Previously: https://news.ycombinator.com/item?id=27613217 https://news.ycombinator.com/item?id=27613217
- Firerouge 5y agoThis same one got me after years of using Docker, I only discovered it after using the combination of Ubuntu Server (and it's ufw) on a DMZed device. I was running what I thought was an internal FTP instance for almost a week. Luckily it was about as hardened as regular ftp can be, but I noticed the problem when my service wasn't able to log in as the (very low) connection limit was filled by someone attempting passwords. I've been using https://github.com/shinebayar-g/ufw-docker-automated https://github.com/shinebayar-g/ufw-docker-automated to make docker compliant with ufw, and defining firewall rules as labels for the containers. It needs some work still, namely the service should be hosted in it's own container for easier updating, but it works reasonably well.
- cbo100 5y agoYep. This exact interaction between UFW and Docker had me stumped for a while too. Totally unexpected outcome. Luckily I discovered it in testing, so didn't make it to production. But annoying that those issues still remain for so long.
- 1337shadow 5y ago> Published ports > This creates a firewall rule which maps a container port to a port on the Docker host to the outside world. Source : https://docs.docker.com/config/containers/container-networking/ https://docs.docker.com/config/containers/container-networki... Requiring authentication from localhost does not seem relevant to me, given that the creds would be stored somewhere, either in memory either in a file anyway, but exposing a port is not "binding on localhost". However testing your firewall after publishing a docker port seems common sense. Indeed, that's how I found out Docker was using the DOCKER-USER iptables chain that you can customize: https://docs.docker.com/network/iptables/ https://docs.docker.com/network/iptables/ And that's how I made a simple firewall that works: https://yourlabs.io/oss/yourlabs.docker/-/blob/master/tasks/main.yml#L77-115 https://yourlabs.io/oss/yourlabs.docker/-/blob/master/tasks/... Another thing, instead of using exposing ports like that, the easiest is to use Docker-Compose, so that your containers of a stack have their own private shared network, then you won't have to publish ports to make your services communicate. Otherwise, just create a private network yourself and containerize your stuff in it. So for me that's two newbie mistakes which conducted to falling for this non targeted, script kiddie attack which has been going on since 2017. But yeah, go ahead publish your ports instead of using docker networks how you should, "believe" in your firewall while you're at it, and then blame "docker footgun".
- foobarbazetc 5y agoWait... people run their databases on public IPs?
- edoceo 5y agoYep. It happens. Welcome to earth.
- jbverschoor 5y agoNo, they run databases that listen to localhost, and then use docker to forward that private binding to the public ip
- drdaeman 5y agoNo, they run database that listens to 0.0.0.0/0 and/or ::0/0 on a private isolated network interface (eth0 inside Docker namespace), then let Docker create a NAT to forward packets from a public network interface on the host to this database. The latter is a mistake. It doesn't ever make sense to expose hosted container ports to host, with the exception of public-facing stuff (like a webserver). The whole backoffice should be on a separate network. And this is how Docker works by default, since forever, unless one very explicitly requests that it exposes ("publishes") the ports. Just don't expose stuff. If the host needs to talk to a database or something, it can always talk to it via that `docker0` (or however it's called, I don't really remember the details) interface. (I believe it doesn't always add a route, because on one of my machines I have to manually add a route so Traefik on a host is able to talk to the containers on a isolated virtual network.)
- jbverschoor 5y agoThat depends in if you use docker networks. By default, -p maps a port, afaik to 0.0.0.0:port. So this is more an issue with the default docker settings than anything else. Yes they should’ve configured, checked, etc. But defaults sound be secure by default
- deleted 5y ago[deleted]
- mdoms 5y agoAlternative headline: how my unauthenticated database ended up on the public web.
- ljm 5y agoThe only question I can ask is, why Docker? Newsblur presumably existed before Docker.
- Marsymars 5y agoIt did, but by interesting coincidence, only really got traction when Google Reader shut down, which happened to be the same month of the initial release of Docker.
- cratermoon 5y agoThe attack reads a lot like Meow https://searchsecurity.techtarget.com/news/252486611/Meow-attacks-wipe-more-than-1000-exposed-databases https://searchsecurity.techtarget.com/news/252486611/Meow-at...
- mattacular 5y agoOff-topic but I had never seen the Tor exit node's default text before. Pretty hilarious that it states Tor provides privacy to people need it most: "average computer users." Yep, that's definitely who is using Tor...
- cratermoon 5y agoI can't help but be reminded that zero trust architecture for security has been a thing for at least a decade, and that the 2004 Jericho Forum concluded that perimeter security was illusory, more akin to a picket fence than a wall.
- qbasic_forever 5y agoEveryone embracing immutable server configuration would help too. We give so many tools like docker complete trust to touch configuration and do the right thing. Sometimes it bites us though. In a perfect world you'd see docker try to change iptables and it fail, then investigate what's up and understand that a specific change has to be allowed and all the implications of that change.
- closeparen 5y agoPerimeter security would have been just fine here. The breach occurred because the host was exposed directly to the internet, rather than e.g. sharing a private network with a load balancer.
- cratermoon 5y agoClearly it was not fine, but in fact leaky, and depending only on perimeter security was (and is) flawed. See https://collaboration.opengroup.org/jericho/commandments_v1.2.pdf https://collaboration.opengroup.org/jericho/commandments_v1....
- closeparen 5y agoThere was no perimeter security here. The attacker did not first enter a private network and then pivot to MongoDB; he dialed MongoDB right from the internet. Had Mongo been un-authenticated on a private network it still might have been owned, but the bar would have been a lot higher. Side note: everything that’s ever existed is “flawed,” it’s a weird word to use in the context of something you want to discredit, because then your alternative had better be “flawless” and it obviously isn’t.
- rubyist5eva 5y agoCrap like this is why I dont run docker. I'm glad RedHat took a principled stance on it and dropped it. Podman doesn't punch holes you didn't ask for in your firewall. What a ludicrous anti-feature.
- thatwasunusual 5y agoThat might be true, but regardless of docker (or other, similar solutions), shouldn't MongoDB have had auth protection?
- haram_masala 5y agoTotally agreed with this. MongoDB’s default security posture is ridiculous, and amazingly it used to be even worse.
- jbverschoor 5y agoIt’s exactly the same for redis, which means if used for jobs, you can insert arbitrary code.
- berkes 5y agoAll my async job code lives in a codebase and redis merely contains the payload and some ids. What system, framework or stack are you using that stores the code of the job in the database? I'm curious, because I can imagine it solves some issues, e.g. deploying new code while allowing old code to finish running and scheduled jobs.
- arp242 5y agoI think what they mean is Redis' Lua scripting stuff with EVAL and such. Also, if you have stuff like this in Redis, which will get picked up by a job runner: {"runjob": "foojob", "params": ["one", "two"]} Depending on how your job runner interprets this, it can be a big problem if people can write to this. At the very least it'll be a DoS in most cases, at worst (SQL) database, shell access, or leaking of passwords.
- tedk-42 5y agoI'm a bit confused here about how this was a docker issue. Initially I thought there was some stuff happening with uPnP with a router/gateway of some sort (https://en.wikipedia.org/wiki/Universal_Plug_and_Play https://en.wikipedia.org/wiki/Universal_Plug_and_Play) If I run a docker container on my macbook and expose a port over 0.0.0.0 (docker run -d -p 8080:8080 nginx) and it's available on my LAN network, how does it get exposed over the internet? Unless there's a rule in my router/gateway which does the port forwarding, it's not possible. I wish the author goes into a bit more detail onto how his setup was vulnerable rather than bringing up an old Github Issue.
- qbasic_forever 5y agoIt's not a problem on docker desktop for mac or windows. It's only an issue on linux, and only for systems that use iptables as a firewall (typically ubuntu/debian & ufw). And even more specifically it really only affects servers that are in a hosting environment with no other firewall or protection in front--Digital Ocean's default droplet config is a prime example, on AWS by default you have a cloud-specific firewall provided by Amazon's networking. On your system docker is running in a little virtual machine that has its own linux kernel, virtual peripherals, etc. This gives you an extra layer of security (by pure happenstance) such that the VM has to allow traffic in too. And this is kind of why this issue is so nasty. You as a mac user might never know that this is a problem or potential footgun. You could happily develop entire production services on your mac, then move them to a shared linux host like Digital Ocean and... uh oh, now your services are open and you had no idea it was even possible.
- tedk-42 5y agoBut this isn't a linux issue - this is by design. If you're deploying code without knowing how networking works, you're always going to have problems with this like this. Perhaps my example was a little too simplified as AWS is my day-to-day cloud host but I find it hard to believe opening up ports on your VM and exposing it to the world and then placing the blame on docker is fair.
- boardwaalk 5y ago
- whybotherwith 5y ago>In NewsBlur’s case, because NewsBlur is a home of free speech, allowing users in countries with censored news outlets to bypass restrictions and get access to the world at large, the continuing risk of supporting anonymous Internet traffic is worth the cost. This, the backups, the write-up, all make it really hard for me to want to victim-blame the dev for not catching a very silly Docker default. That having been said - public ip? - no fw? - no password on the mongod instance? Idk, couldn't be me, not even in dev, just take the 20 seconds to plumb the pw to both sides. Modding me down won't change these facts and won't keep you from being compromised if you take the same lazy steps
- ViViDboarder 5y agoThe problem with the Docker default is that they did have a firewall configured but Docker will reconfigure it. That said, some external firewall would prevent this. My VPS provider allows me to configure which ports I want to expose to the outside world which would mitigate this kind of issue.
- dheera 5y ago> write to me in the mail with your DB IP: FTHISGUY@recoverme.one why the redaction? name and shame.
- Rapzid 5y agoBuck stops at NewsBlur on this one. Let's call a spade a spade; unqualified. Newb. It happens. It happened. Port scan next time? Metasploit maybe? "Script Kiddie".. It's in the name.
- deleted 5y ago[deleted]
- joshgree88 5y agoCommmmme on a DB without authentication or user permissions on a public network...
- whalesalad 5y agoWhat surprises me most is the machine running Docker was 100% connected directly to a public network interface. That’s the real root of the problem here.
- tedk-42 5y agoExactly - and there's no mention either of how the fix was implemented nor what remediations were done to prevent it happening again in the future...
- alexchamberlain 5y agoThere's a section st the end detailing 3 fixes, including using a private network.
- fulafel 5y agoAmbiguous addresses are generally a pretty big headache and have a lot of security downsides, network level firewalls are better and would have worked here too. It's literally the old NAT vs firewall case.
- onebike 5y agoThis (Docker opening a hole in my firewall) is why I moved my dev server from Linode to Digital Ocean. DO provides a “cloud firewall” that provides something akin to AWS security groups and therefore can’t be messed by Docker. Linode doesn’t have anything like that (last time I checked at least).
- LinuxBender 5y agoThey recently added one. In fact I had to move many of my VM's to new hypervisors because the ones that didn't support the cloud firewall were deprecated. I don't even use their cloud firewall.
- bombcar 5y agoThis is perhaps the best arguments I’ve seen for a separate firewall device even if it’s in the cloud (and just software) - something on your box running as root may bypass your rules just to help you.
- eru 5y agoAlternatively, running all your services as VMs also helps. Having root in a VM doesn't typically give you any rights on the hypervisor (at least not on eg Xen).
- deleted 5y ago[deleted]
- ahepp 5y agoWell, if they get root on your mongo vm they can still drop all your tables (or ransomware you) right? So would it make a difference in this particular case? Outside the VM tooling probably not being so insane as to bypass the firewall?
- eru 5y agoWell, in this case docker was trying to be helpful. On a hypervisor, it's much harder for VMs to influence each other. Linux containers (and docker amongst them) started out as convenient and reasonably performant, and added security later. One patch at a time. Historically, hypervisors typically started secure and added performance and convenience over time. (Very simplified. But I used to work for XenSource back in the day.)
- digianarchist 5y agoSeems like poor network architecture to me. The fact that you can misconfigure a firewall on a VM or server which in turn allows public traffic is pretty dangerous.
- _tom_ 5y agoTo be fair, he was in the process of fixing the poor network architecture when this happened.
- sarak12070 5y agoFree Download Sketchup 2015 Full Crack https://download2021software.blogspot.com/2021/06/free-download-sketchup-2015-full-crack.html https://download2021software.blogspot.com/2021/06/free-downl... Review mamonde lip tint with Discount offer https://topsaleproductsonline.blogspot.com/2021/06/review-mamonde-lip-tint-with-discount.html https://topsaleproductsonline.blogspot.com/2021/06/review-ma... The aerodynamics of Argentavis, the world's largest flying bird https://www.interestingnews.club/2021/06/the-aerodynamics-of-argentavis-worlds.html https://www.interestingnews.club/2021/06/the-aerodynamics-of... https://techtips2021.blogspot.com/2021/06/best-way-to-get-quora-followers.html https://techtips2021.blogspot.com/2021/06/best-way-to-get-qu...
- caymanjim 5y agoDocker has worked like this for a long time. It's a really bad default, and they damn well ought to stop operating this way by default, but it's also like Docker Devops 101. When you install Docker on a Linux system, you should configure the DOCKER-USER chain to drop everything originating on your public network interface. You should also stop running services bound to localhost and instead run them on a private network. You can proxy any traffic that really ought to be allowed through from the outside, or if you can't do that for some reason, then make a single exception to the iptables and host binding rules for that one container.
- xeromal 5y agoThis feels like something you might forget if it's just been running for a while.
- spockz 5y agoWhen I moved my docket setup to a new fedora server a few months ago I was surprised I had to add firewall rules to allow traffic to the ports exposed in the containers. I’m not sure whether this is for any port or specially for things running under the docker user but RHEL should be safe by default.
- j1elo 5y agoWould you know of some blog post or guide in the vein of "essential sane(r) defaults to apply after installing Docker" kind of tutorial? Which included, for example, the chain config you mentioned
- wyldfire 5y agoDoes mongodb still ship in a no-auth-required-by-default mode? I could've sworn that was the case years ago at the early NoSQL days. But if they're still around, surely they must have fixed that by now?
- haimez 5y agoBased on my comments and the reply from a mongo employee in the previous thread about this: yes, they changed the defaults in 2017: https://news.ycombinator.com/item?id=27613761 https://news.ycombinator.com/item?id=27613761 That said, it also seems like the docker images are a different beast, so be careful what containers you ship and with what configuration…
- ramraj07 5y agoThe biggest issue with docker is the false sense of security it seems to give a lot of engineers into thinking they know infra when they really don't. I stay away from this because I don't understand it fundamentally, and now this proves it's better to not think these new technologies are your friend unless you actually know what you're doing (which apparently most don't).
- judge2020 5y agoStaying away from it is still not the best strategy - at least learn and play with it to understand its strengths and weaknesses. > it seems to give a lot of engineers into thinking they know infra when they really don't Maybe, but technology changes over time - I don’t see many new projects choosing VMware over docker/OCI for new infrastructure deployment since you usually don’t need a full VM for applocations that just need isolation and easy static deployments.
- ramraj07 5y agoI suppose my premise is that engineers often delude themselves into thinking that they know it well enough to implement docker when they dont. Id much rather prefer that the regular developers focus on the coding and leave those decisions to actual devops folks who can focus on getting this stuff right. But the simplicity of dockerfile lulls many into thinking if they run something with it it's production ready.
- lvncelot 5y agoDoesn't that run counter to what devops actually should be? I know that in practice, this split is what happens at companies, but we should work to close that rift, not widen it.
- kijin 5y agoWhatever devops is supposed to be, it shouldn't involve relying on a tool whose security characteristics the team does not fully understand. If requiring such knowledge results in widening a pre-existing rift, so be it.
- buzzert 5y ago> In case of refusal to pay, we will contact the General Data Protection Regulation, GDPR and notify them that you store user data in an open form and is not safe. Under the rules of the law, you face a heavy fine or arrest and your base dump will be dropped from our server! Does anybody know if this threat is at all credible?
- DharmaPolice 5y agoThe way it's phrased doesn't make much sense but they could contact whoever is responsible for GDPR enforcement in a particular country (if in the EU) and make a complaint against you. But it's unlikely to come to anything (especially if all the data was deleted) and you won't be arrested.
- rndgermandude 5y agoYou could end up paying a fine if the regulatory authority investigates and finds that personal data was exposed (or deleted) due to lack of "appropriate technical and organisational measures to ensure a level of security appropriate to the risk"[1]. Relatedly, you can be fined for not telling the regulatory authority about breaches yourself: "In the case of a personal data breach, the controller shall without undue delay and, where feasible, not later than 72 hours after having become aware of it, notify the personal data breach to the supervisory authority competent in accordance with Article 55, unless the personal data breach is unlikely to result in a risk to the rights and freedoms of natural persons."[2] British Airways[3] for example was fined for inadequate security measures. I also remember a case where some company (cannot quite remember which) itself told the regulator about some security snafu as they should, a snafu that most likely was not exploited. They were still issued a fine but a reduced one because they were very cooperative and most likely no actual damage happened. There is an enforcement tracker website[4] where you can filter for e.g. "security" in the type column. But nobody will go to prison for being the victim of a hack, that's just fearmongering by that attacker. [1] https://gdpr-info.eu/art-32-gdpr/ https://gdpr-info.eu/art-32-gdpr/ [2] https://gdpr-info.eu/art-33-gdpr/ https://gdpr-info.eu/art-33-gdpr/ [3] https://ico.org.uk/about-the-ico/news-and-events/news-and-blogs/2020/10/ico-fines-british-airways-20m-for-data-breach-affecting-more-than-400-000-customers/ https://ico.org.uk/about-the-ico/news-and-events/news-and-bl... [4] https://www.enforcementtracker.com/ https://www.enforcementtracker.com/
- overgard 5y agoI mean, that's a bad docker feature, but so many other things went wrong to get there in the first place. No db password?! I'm guessing that if it was talking over localhost the db was on the same system as the web server? Agh.
- grayhatter 5y agoThis isn't a footgun. A footgun is when something happens that should be expected, but isn't for reasons of negligence or ignorance to the thing that should be expected. The example that everyone seems to love is pointer arithmetic. If you make a basic error in your math, invalid memory access may occur and then likely more bad stuff. Docker altering firewall rules without explicit instructions to do so is either a feature, if you're looking to defend docker, Or a security defect if you're feeling rational. There's an argument to be made for the root cause of the issue, The docker container manifest, or the application itself. But to pretend like this is what docker should be doing, users should know better just seems wrong.
- superasn 5y agoAnd now Google is picking up the definition of footgun(1) from Hacker news! (1) https://i.imgur.com/pHlLFJA.png https://i.imgur.com/pHlLFJA.png
- thih9 5y agoYour screenshot shows an earlier date (25-Jun-2018) next to the definition.
- superasn 5y agoYes it links to this comment: https://news.ycombinator.com/item?id=17393292#:~:text=Generally%20a%20footgun%20is%20something,shoot%20yourself%20in%20the%20foot https://news.ycombinator.com/item?id=17393292#:~:text=Genera.... The funny thing is there are definitions for it in wikionary and urbandictionary yet it is picking the HN one.
- tremon 5y agoThe other funny thing is that that definition is also from a thread involving MongoDB.
- 5y ago
- plasma 5y agoI feel there are two issues to fix: 1. Docker doing this iptables change out of the box, and 2. MongoDB not having a password set out of the box The life of a developer (and solo dev) means you often have limited time you need to navigate a project and try and do your best to understand, deploy and use it -- this is just one of many tasks on your TODO today to get you closer to operating your product. I really wish these kinds of things were more secure in a few ways: 1. Defensive defaults (passwords, not opening holes in firewalls), and 2. Not making the security hard to use If its painful to work with the security feature and "get it working", someone with limited time may just undo the defensive defaults to get things working again (doh). But I get it, sometimes a security piece on by default is so cryptically painful to understand that you get so frustrated with it you just turn it off.
- laurent123456 5y agoOne of the main issue with security is that there's no visible difference between a well secured system and one that's open to any script kiddie. And you might install something or make a mistake in a config file, and suddenly your system is completely unsecure, and again there be will nothing obvious about it. I think what's missing is an easy to use tool, installed by default - you run it and see a clear overview of your system security - what ports are opened, how is SSH accessible, how secure are the running services, etc. with plain ticks/crosses to show what's good or not. The kind of tool that a developer can install on their server and then check that at least the basics are right.
- macintux 5y agoThe enterprise space is rife with such tools, but unfortunately they tend to tell you everything, which makes it hard to identify what actually matters.
- Saint_Genet 5y agoUnderstanding security is no longer optional if you try to run a business in the internet. People like to talk about how they want to focus on the product and building features, but if you don’t have security you don’t have a functional product. No, security is not easy, but it’s not an optional skill set anymore. Learn it of fail.
- aborsy 5y agoAnother issue that is not discussed, when I type “sudo ufw status verbose” I expect to see open ports in my system. It’s an issue that ufw doesn’t show some open ports (all relevant iptable rules).
- Tepix 5y agoufw assumes you use only ufw to configure the firewall. If something else messes with iptables, all bets are off.
- fulafel 5y agoA big thing is that this also happens on your dev machine with "docker run". (Unless you know to use -p 127.0.0.1:1234:1234 instead of just -p 1234:1234 like all the examples on the web tell you.)
- ryukafalz 5y ago> If a rogue database user starts deleting stories, it would get noticed a whole lot faster than a database being dropped all at once. This feels like an odd statement. Surely a database being dropped all at once is about the loudest possible thing that could happen to a database-reliant application?
- fulafel 5y agoThere's not much distance between current Docker behaviour and it calling a AWS/GCP/whatever API to route traffic to it. I predict some piece of sw, if not Docker, will be caught doing that in the near future.
- nneonneo 5y agoNetwork services really should have auth by default even if they only bind to localhost. There’s so many ways that a localhost service can become accessible to attackers. Any process with network access, no matter what user it’s running as, can access a localhost service - UNIX sockets on the other hand can be restricted by the usual user/group permissions. A localhost service can be exposed by e.g. an SSRF bug from a web server on the same machine. Or, accessed by a browser on the same machine browsing untrusted sites (an attack model for desktop services which open local ports), and so on. (In case you believe that the HTTP protocol restriction in the latter two cases protects you - protocol smuggling over HTTP is frighteningly effective!) Of course, as seen here, a network service binding to more than localhost is strictly more exposed - and auth should be required in those cases (even when the service should ostensibly be LAN-only). Secure-by-default is the best kind of security.
- stouset 5y agoIt’s an unpopular opinion but it’s 100% a good idea. If at all possible, bind local services to UNIX sockets instead of localhost ports. At least that way you’ll get some measure of access control effectively for free. If you’re writing software that deals with network connections (as a client or server), write the code for UNIX sockets first. It’s usually trivial to bolt network connection code on top, and being able to deal with sockets directly can make automated testing significantly easier.
- alephu5 5y agoA similar but more flexible solution I like is to create a wireguard network between all nodes and developer workstations, which creates a separate interface called wg0. Bind every private service to this and publish what you want through a reverse proxy. It has an added benefit of not needing vendor-specific TLS configuration and firewall rules easy to configure since everything goes through the single VPN port.
- z3t4 5y agoSecure by default is best for those people who do not know every little config detail. Secure by default helps those that you try to make life easier for, eg. you open a port because you assume your user don't know how to... having security by default is the best option for those people. And by "those people" I mean everyone, including myself.
- scottydelta 5y agoI have been using ufw-docker script[1] to patch the ufw and docker issue on my servers. The docker containers are publically accessible even if you have ufw enabled and rules applied. These days its better to use the hosting provider firewall on top of ufw. Leading providers like AWS, Hetzner etc all provide the feature to add firewall rules via their UI. [1] https://github.com/chaifeng/ufw-docker https://github.com/chaifeng/ufw-docker
- fest 5y agoI was bitten by this as well, luckily I was setting up ufw after docker (and was really surprised that dockerized services showed up in port scan). To be fair, it did not look to me that docker intentionally punched holes in firewall- it purposefully places it's rules in separate table/queue, which due to way iptables work, hampers the firewall rules. However, once it became apparent to docker developers, it should have been made a priority to be fixed, as it is counterintuitive default.
- naranha 5y agoI think docker/podman with firewalld behaves the same way. I was astonished when I found it out. I think you can however bind to localhost exclusively by using the ip in the argument: docker run -p 127.0.0.1:5555:5555 ....
- oynqr 5y agoI am running a docker-compose setup with various services, only ever exposing ports so my reverse proxy on the same machine, which is not in a container, can hit them. The config explicitly specifies "127.0.0.1:external:internal". To my dismay, I can actually reach those external ports from the internet, even though I have some simple iptables rules to drop anything but 22, 80 and 443. I never really liked the whole container idea, but this is ridiculous.
- ripperdoc 5y agoAre you sure? I also have "127.0.0.1:external:internal" on some services and I cannot reach them with nc, but I can access those that are just configured as "externa:internal"
- yrro 5y agoSounds like your config is doing the right thing. It would be interesting to see what 'iptables -S -t nat' and 'iptables -S -t filter' say, as well as 'docker port CONTAINER_ID'.
- meow112012 5y agoI don't know how docker is to blame here; to me it's a bit unfair. It's more about the setting of the ufw. Docker relies on their own NAT table, and setting ufw accordingly requires a bit of setup. That said, having a good setting from hosting provider is quite easier ans safer.
- Tepix 5y agoDocker's default goes against best practices. Just look at all those opened issues on Github. Clearly people get caught by surprise and may suffer a security compromise as a result. Docker is 100% to blame here and they need to change the default! Just put it in the release notes and get it over with.
- hughrr 5y agoThe real issue here is that the “backend” database engine was exposed on the public internet. It should have been a layer behind that with suitable protection, as mentioned a VPC with security group rules or equivalent firewall. Regardless of the docker foot gun this could have been a human fudge as well. Having done exactly this myself before with SQL server and Slammer I didn’t blame the default configuration, I blamed myself.
- metafunctor 5y agoThe issue is that docker circumvents a platform security control without warning. Docker on Ubuntu has a critical security vulnerability, with a track record of being exploited. That, clearly, is something that should be fixed ASAP. If you have just a single layer, sure, blame yourself for poor security practices. Services should listen to unix domain sockets or internal networks only. They should have authentication in place. Machines should not have routable IP addresses. Machines should have externally applied firewalls. And so on. That doesn't mean the security layer that did fail shouldn't be fixed.
- hughrr 5y agoCorrect. my point is that security is no good full stop if there is only one layer. Even temporary rules and default states aren’t protected against until your automation is complete if your box is on the public internet. Or if you screw your automation up, the same happens.
- darkwater 5y agoSorry if it sounds like a blame the victim (I'm probably doing it, yeah) but if you know all that, and even if you don't, having a DB server meant to be accessed only internally with a public IP is basically wrong under all circumstances. I mean, the docker "footgun" can be valid if you are running a single host with multiple services, some of them public and some of them private, and dockerizing the private ones gets them exposed to the Internet, even if you had a firewall rule to manage that. That's fine, let's blame Docker. But in this case I'm sorry but the Docker behavior just exposed a broken design. I really hope they learn from all this the right lesson, which is not to just blame docker, but to carefully think whether you really need to use the Internet as a mean of internal communication for your services.
- KronisLV 5y agoProps to the developer for being this open and letting others learn and discuss the implications of this situation! I rather enjoyed the original discussion as well, even though the situation itself was unfortunate: https://news.ycombinator.com/item?id=27613217 https://news.ycombinator.com/item?id=27613217 I agree with the other posters, that Docker can cause problems with firewalls, as pointed out both in the article, in the GitHub issue, and in the other thread as well: https://github.com/moby/moby/issues/4737 https://github.com/moby/moby/issues/4737 Furthermore, it also seems to me, that not only Docker should respect the firewall rules (even though it'd confuse another group of people about why their services aren't accessible externally and would necessitate manual firewall rule management, short of some explicit way to do that, such as docker run ... -p 80:3000 --expose-in-firewall 80 ...) and that MongoDB also should have secure defaults, as any other piece of software! That said, at least to me it appears that Docker Compose, Docker Swarm and other technologies have attempted to introduce a networking abstraction to allow running services (more) securely and privately, by not exposing their ports on the host directly. For example, see the following example of a Compose file, which would only expose a web server to the world: version: '3.4' services: # you would probably want to run MongoDB not exposed to the outside world at all # notice that there are no ports exposed here, given that we're using Docker networking mongodb_container: image: mongo:bionic restart: "unless-stopped" environment: - MONGO_INITDB_ROOT_USERNAME=lazy_example_of_a_single_file_config - MONGO_INITDB_ROOT_PASSWORD=but_you_should_be_able_to_store_this_in_secrets # see docs for MONGO_INITDB_ROOT_PASSWORD_FILE at https://hub.docker.com/_/mongo deploy: replicas: 1 placement: constraints: - node.hostname == your-mongo-server.com # just an example, probably use labels instead resources: limits: cpus: "4" memory: "8G" # your web application wouldn't need to be exposed to the outside world either # it could just receive the requests from a web server that ensures SSL/TLS web_application: image: hello-world # which you would replace with your actual app restart: "unless-stopped" environment: - APP_CONFIG=variables_go_here_or_in_a_config_map - MONGO_HOST=mongodb_container - MONGO_PORT=27017 - MONGO_USERNAME=either_the_username_above_or_a_non_root_one - MONGO_PASSWORD=though_the_non_root_needs_separate_creation # maybe with a helper container, like some people use dbmate for RDBMS deploy: replicas: 4 placement: constraints: - node.hostname == your-application-server.com # just an example, probably use labels instead resources: limits: cpus: "2" memory: "2G" # finally, the actual web server is the ingress point, which you can self host # in the case of Kubernetes and other fancy tech you'd be able to use external solutions, probably web_server: image: caddy:alpine # or another web server, that would actually be exposed restart: "unless-stopped" ports: - 80:80 - 443:443 # though sometimes you need the long format for the above so client IPs resolve correctly volumes: - some_config_directory:/etc/caddy/Caddyfile - some_other_config_directory:/data - yet_another_config_directory:/config deploy: replicas: 4 placement: constraints: - node.hostname == your-ingress-server.com # just an example, probably use labels instead resources: limits: cpus: "2" memory: "2G" That's not to say that everyone should use Swarm or other orchestrators, but personally i find it a good approach, which allows defining some of the network topology in the same file as the application deployment, thus making the risk of human error slightly smaller. Docker even has a page on networking, that provides information about the additional functionality that's available as well: https://docs.docker.com/network/ https://docs.docker.com/network/ Of course, that's not to say that defense in depth and external solutions wouldn't perhaps be a better option.
- tigrezno 5y agoHoly shit, I've been running open ports for a year with docker, completely ignoring my firewall! This has to be fixed ASAP.
- alias_neo 5y agoOne of the best things one can do with Docker, for security, but at the expense of convenience is to configure it to disable the docker-proxy and IPTables manipulation and manage IPTables yourself. It'll be slow at first, and you'll need new rules each time you bring a container up but you'll have far more control. We switched to this approach when we had an IP dual-stack issue with the proxy and not only do we know it's secure because we wrote the rules, we get better performance without the userland proxy and more control over routing. It's important to understand that because your containers sit on a bridged network, INPUT rules do nothing to stop incoming connections, it's all about the FORWARD table.
- mercora 5y agonote that if the containers are bridged the iptables FORWARD table wont affect packet forwarding between containers either. you would need to use ebtables for layer 2 firewalling i am almost certain. Edit: apparently there is a sysctl setting that makes the bridge netfilter code call the iptables filter routines (bridge-nf-call-iptables) so i am kinda wrong...
- heurisko 5y agoDocker should at least add a warning message that they are editing IP table rules. I remember experimenting with docker running on a VM, and being confused for a long time why my ufw rules didn't seem to be working, until I found out this was a known issue.
- dmix 5y agoThank you for continuing to support censored users by allowing Tor. Cloudflare, for ex, has wrecked havoc on these vulnerable people.
- abiro 5y agoI think the real lesson here should be to use managed services whenever possible. There is just no excuse to manage your own infra on the VPS level for a typical web app in 2021. There are a myriad of PaaS providers where legacy VPS setups can be migrated to without too much pain.
- tigerBL00D 5y agoI feel sorry for the victim, but it sounds like this whole setup is more or less careless. If your Docker configuration is the only thing that's stopping your database from being exposed - you should reconsider your approach to network and database security. At the very least there should be firewall protection at host level. Further is a standard procedure to put unsecure endpoints in a private VPC that's not accessible from the internet (and a bastion for administration). Relying solely on docker for something as important is bad even if you configure it correctly - it's still a single layer of software protecting you and it may have vulnerabilities of its own.
- berkes 5y ago> At the very least there should be firewall protection at host level If you read the Footgun at https://github.com/moby/moby/issues/4737 https://github.com/moby/moby/issues/4737, this is exactly what happens: someone sets up a conservative firewall, then Docker drills holes in it and opens itself up to the world, regardless of your firewall.
- tigerBL00D 5y agoOkay, good point, and that's exactly why you isolate your networks. You don't want to be one configuration option from being wide open.
- berkes 5y agoTrue. In this case having a separate (hardware) firewall, a vpn setup or some other networking, protects you against this. That is poorly documented on Docker too. Yet, what Docker does is still wrong: it should not disable Ubuntu's default security. Ever. Even if that security is inadequate. An analogy would be to say "While watering your plants last night, I left all your windows open. Now your stuff is stolen. I did this because you should have installed window grills: without window grills your security sucks anyway."
- simiones 5y agoIsn't this to a much greater extent UFW's fault? If it claims to manage the firewall for your system, I would expect it to manage all of iptables, not just one particular chain that it thinks is important. At the very least, I would expect it to signal when there are other chains configured in iptables that it can't/won't manager for you.
- fletchowns 5y agoAlong the same lines, I remember coming across a lot of bad information when learning about Docker & iptables. For example, this article about using fail2ban & docker will leave your system unprotected: https://chlee.co/how-to-secure-and-protect-nginx-on-linux-with-fail2ban/ https://chlee.co/how-to-secure-and-protect-nginx-on-linux-wi... You have to tell fail2ban to use the DOCKER-USER chain when adding rules, otherwise none of the rules it adds do anything for docker.
- Saint_Genet 5y agoIt's amazing to see everyone pointing out the poor infrastructure security being downvoted. I guess engineering doesn't matter anymore, just blame your tools.
- yrro 5y agoKinda surprised so few people have any idea how docker networking works... The OP didn't (I think) say but... were they running their containers with the --expose or --publish options? Without those, there shouldn't have been anything listening. And with them, well, the clue's in the option name... That said, I've never liked the way docker implements its networking. Not only does it monkey with iptables (not too bad once you expect it) but, worse, it silently enables IP forwarding on all interfaces! Any system where it is installed is now a router; this should make anyone who uses a VPN to connect to work rather nervous... I really wish docker used a different solution for exposing container ports by default, such as having the docker daemon proxy traffic. Yes, it wouldn't be as fast. But this wouldn't matter for the vast majority of users, who would benefit from having behaviour that matches other listening ports, doesn't bypass the firewall, etc. Have the iptables implementation available as an option that is there when you really need the performance--and know what you're doing!
- dirkc 5y agoJust want to confirm I understand the issue here. 1. `docker run -p 0.0.0.0:6666:6666 ...` - docker will update the firewall to allow traffic to port 6666 2. `docker run -p 127.0.0.1:6666:6666 ...` - docker won't update the firewall. 3. `docker run -p 6666` - the port is published on 0.0.0.0 and a random port is opened on the host I see 3 being a little surprising, but 1 and 2 does what I would expect of it.
- aorth 5y ago> 1. `docker run -p 0.0.0.0:6666:6666 ...` - docker will update the firewall to allow traffic to port 6666 I don't think Docker should touch the firewall whatsoever. Apache doesn't, nginx doesn't, PostgreSQL doesn't, etc. Why is Docker different? The sysadmin should decide that!
- maple3142 5y agoMaybe it is more convenient to developer so that they don't have to touch the firewall? Docker is not only used by sysadmin.
- Liquid_Fire 5y agoWhy not add a --open-firewall-ports option and let the user decide? Maybe print a warning without that option if the port is not open if you really want to make it easy.
- aorth 5y agoHaha yes of course it's easier for developers, but they write the software that I have to deploy and manage on my server!
- rmetzler 5y agoI think the one case you missed is `docker run -p 6666:6666 ...` which defaults to `-p 0.0.0.0:6666:6666` I can't find it in the article, but I suspect this what actually happened. It's a bad default and unfortunately most examples build on it. Similar, it's easy to deploy MongoDB without a password. But it's also easy to set one, the docs even mention it https://hub.docker.com/_/mongo https://hub.docker.com/_/mongo And everyone can run a portscan with nmap against their servers. Unfortunately the MongoDB default ports 27017-27019 are not part of the top 1000 ports which nmap scans by default.
- lenlorijn 5y agoThis is why security in depth is important. A simple DB user could have prevented this, and many other consequences from misconfiguration. This time it's docker, next time it's a broken private cloud setting or a compromised node elsewhere on your network.
- simiones 5y agoI see everyone discussing how much Docker is at fault, how much Mongo DB is at fault, how much NewsBlur should have had better settings, and I do agree to some extent. However, the much worse problem seems to be the fact that NewsBlur didn't test their network connectivity with something as basic as a port scan. This wasn't some complex attack based on some complex code injection jumping through legitimate ports or anything: their machine had the Mongo port open when they thought it should be closed. This is the kind of thing that absolutely shouldn't make it past basic testing.
- gerdesj 5y agoThank goodness that someone noticed the real problem: You can't see network traffic. You don't write a program and deploy it without testing, so don't do the same with your firewalls. You need to actually test your firewall policy and not blindly assume it does what you think it should.
- papito 5y agoAs the after-action report concludes, which is painfully obvious as you read this - should have been in a VPC.
- simiones 5y agoBut even before talking about the fix: the setup should have been tested, ideally before it was deployed (if possible).
- tbarbugli 5y agoyes, the author made a junior dev mistake and he is trying to blame [tool name here]. Making mistakes like this sucks but passing the blame makes it even worse.
- simiones 5y agoMaking mistakes is not the problem - everyone makes mistakes. But you need testing to catch your mistakes - this is a process failure.
- one2three4 5y agoIs there any easy way to guard against this without putting everything in a VPC?
- nickjj 5y ago> When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world Can Docker really be blamed here? It sounds like when they ran MongoDB they explicitly published Mongo's port to the internet by either adding the ports property to docker-compose.yml or the -p flag with a Docker command to open Mongo's port to the outside world. Not to pour salt on an open wound but why are they blaming Docker for this? Surely an ops person should have thought to check if any internal services are exposed to the outside world? This goes with or without using Docker. This would be like explicitly opening a port with iptables by copy / pasting something off the internet and then blaming iptables because it opened the port. If anyone is curious, years ago I've blogged about the difference between expose and publish for Docker at https://nickjanetakis.com/blog/docker-tip-59-difference-between-exposing-and-publishing-ports https://nickjanetakis.com/blog/docker-tip-59-difference-betw....
- XorNot 5y agoOr there just should've been a password. This whole incident would've been prevented and hopefully eventually rectified if they'd put auth on the localhost interface as well. Let your tools save passwords, but it's insanity to have anything just blindly accept requests - you're one mis-directed script away from people wiping out your prod DB, let alone dedicated hackers.
- nickjj 5y agoYeah, passwords are good. I remember back in 2016 I had a similar issue with Redis. I ran it with -p 6379:6379 with no password and someone got into my Redis container back when "crackit"[0] was a thing. But I never blamed Docker or Redis. I blamed myself because I was careless for not fully understanding the implications of what -p did at the time. [0]: http://antirez.com/news/96 http://antirez.com/news/96
- masa331 5y agoWhat about not running production in docker? One less layer to worry about
- _def 5y agoCan anybode recommend some usfeul resources regarding currently popular firewalls? Never touched that topic personally so far, but I'm interested.
- figassis 5y agoI have talked about this before. This is completely non standard behavior, but the way the docker team simply washes their hands is incredible. https://github.com/chaifeng/ufw-docker/issues/31 https://github.com/chaifeng/ufw-docker/issues/31 https://news.ycombinator.com/item?id=22299693 https://news.ycombinator.com/item?id=22299693
- coding123 5y agoI learned my lesson of not trusting a machine's own ability to secure itself in 1998 or 99 when an IIS machine was installed on Thursday and was completely taken over by viruses on Friday. Block the ports using private networks or use the IP security provided by the cloud provider (which does not use your machine's security to accomplish this). Defense in depth and all that. But sure, docker has a footgun, everything does, otherwise linux would not ship with sudo. (someone's going to reply with a linux that does not come with sudo).
- amphorasource 5y agoWhile I'm glad no data was leaked and that Samuel is taking the layered defense approach seriously as it was wildly debated on the previous HN thread, I'm still a bit miffed that we only got confirmation that no leak actually occurred now. As a Newsblur customer, would have really preferred the following course of action: lock down infrastructure, confirmation that no leak occurred, communication of that fact to customers and THEN service recovery. While this was all happening, the only indication that most likely no data was leaked was a single HN commenter that had a similar experience: https://news.ycombinator.com/item?id=27615708 https://news.ycombinator.com/item?id=27615708
- Annatar 5y agoSo instead of just going through http://ftp.rpm.org/max-rpm/ http://ftp.rpm.org/max-rpm/ and learning how to make RPM's, the author continues employing all this complex technology, because why keep it simple when it can be ultra-complex, hack-prone and Dockerized with a gratis footgun to boot? Makes perfect sense!
- tbarbugli 5y agoSo basically he messed up something basic and he is passing the blame to the tool he used? I would fell much more sympathetic if he acknowledged his mistakes instead of flexing muscles and throwing Docker/MongoDB under the bus. The clear mistake here is allowing incoming traffic on 0.0.0.0/32 Disclaimer: I am not into Docker or Mongodb
- throw7 5y agoA security device (firewall) should not be running all types of services and it certainly shouldn't be allowing firewall rules to be inserted automagically.
- AtNightWeCode 5y agoWould be interesting to get some stats on how common it is to put Docker on a public facing server. I never met anybody in real life that trusts Docker that much.
- rhabarba 5y ago"I put my database on the internet without a password. People used it. HACKERS!!!!!"
- Saris 5y agoI'm a bit curious as to why they had the MongoDB docker container running with the 27017 port mapped in the first place, maybe for testing? You don't need to map ports for inter-container communications, and maybe that's where some of the confusion comes in for people using docker. I have noticed a few friends who have been playing with docker for home server stuff thinking that they need to map ports for everything, just to get containers in a stack talking to each other. At the same time docker really shouldn't be messing with the hosts incoming firewall rules. And MongoDB really should force auth by default.
- Cpoll 5y ago> ufw firewall ... kept on a strict allowlist with only my internal servers > I switched the MongoDB cluster over to the new servers. It sounds like they had multiple Mongo nodes on separate machines, communicating over the internet. Presumably the application(s) are also on separate machines and would need to communicate with Mongo. I think `docker network` has options to create a network across Docker daemon hosts? But that has some tradeoffs, I'm sure.
- rasguanabana 5y agoThat’s why you need more than one layer of security over your critical assets. Network based access control to me always feels easy to misconfigure. Basic auth would delay attacker. Encryption of data at rest would render stolen data unusable to attacker. General advice - perform threat modeling of your services to uncover weak links.