14 ms·

NewsBlur's founder here. I'll attempt to explain what's happening. This situation is more of a script kiddie than a hacker. I'm in the process of moving everyt

NewsBlur's founder here. I'll attempt to explain what's happening. This situation is more of a script kiddie than a hacker. I'm in the process of moving everything on NewsBlur over to Docker containers in prep for the big redesign launching next week. It's been a great year of maintenance and I've enjoyed the fruits of Ansible + Docker for NewsBlur's 5 database servers (PostgreSQL, MongoDB, Redis, Elasticsearch, and soon ML models). About two hours before this happened, I switched the MongoDB cluster over to the new servers. When I did that, I shut down the original primary in order to delete it in a few days when all was well. (Thank goodness I did that! It'll come in handy a few hours from now). Turns out the ufw firewall I enabled and diligently kept on a strict allowlist with only my internal servers didn't work on a new server because of Docker. When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world. So while my firewall was "active", doing a `sudo iptables -L | grep 27017` showed that MongoDB was open the world. More info on SO[1]. To be honest, I'm a bit surprised it took over 3 hours from when I flipped the switch to when a script kiddie dropped NewsBlur's MongoDB collections, and ransomed about 250GB of data. I am now running a snapshot on that old primary, just in case it reconnects to a network and deletes everything. Once done, I'll boot it up, secondary it out, and be back in business. Let's hope my assumptions hold. [1]: https://stackoverflow.com/questions/30383845/what-is-the-best-practice-of-docker-ufw-under-ubuntu https://stackoverflow.com/questions/30383845/what-is-the-bes...

dawnerd 5y ago
This actually got me a while ago but with redis and some script kiddy turning my dev server into a bitcoin miner. Anyone else running docker and using iptables really needs to read this https://docs.docker.com/network/iptables/ https://docs.docker.com/network/iptables/
- squeaky-clean 5y ago
  Almost exactly the same thing has happened to me except Selenium and they were trying to log into Playstation Network accounts.
  - deleted 5y ago
    [deleted]
- qbasic_forever 5y ago
  The insidious thing is that there's no indication, failure, log or anything to tell you something is out of the ordinary either. It would be one thing if it just exploded and failed to run, but it's even worse that it silently interacts with iptables & ufw to allow all the traffic through. The exact opposite of what you want or intended.
  - dawnerd 5y ago
    Only reason I noticed was a command was taking slightly longer than normal and I checked htop and saw redis using 100% cpu. Absolutely insane this is the Docker default.
- frakkingcylons 5y ago
  Same exact thing happened to my test redis instance two years ago.
- bushbaba 5y ago
  I’m kind of shocked this is even deemed acceptable architecture. You’d think docker wouldn’t even touch iptables unless explicitly told to.
  - qbasic_forever 5y ago
    It's how they can make containers feel like isolated little subnets without resorting to vxlan or other kernel-level stuff. It's a great development experience and I'd be sad to see it go. But.... it really needs to proactively detect and warn users. The issue has been known for many years. A quick little check and error out on startup if you're running on Ubuntu or have ufw enabled would probably save 99% of the pain people have had with it over the years.
    - 1337shadow 5y ago
      It works fine when users create docker networks for their containers to communicate, which docker-compose does by default, instead of publishing ports on 0.0.0.0 like yolo
  - AtNightWeCode 5y ago
    I would say that this is the expected behavior. Also, I don't think one should rely on firewalls in this way.
    - Shank 5y ago
      Relying on firewalls to do what firewalls do and have done and continue to do seems perfectly acceptable. Yes, your database should have authentication enabled too, but expecting ports to not be unexpectedly open is the entire point of firewalls.
      - AtNightWeCode 5y ago
        Kind of what I said. Firewalls are for blocking unwanted traffic. It should not be used as a replacement for other security measures. "unexpectedly open", well, there I simply disagree.
- js4ever 5y ago
  Redis can be exploited to run executables ???
  - qbasic_forever 5y ago
    It can execute Lua, I'm sure there's plenty of fun for hackers to have with this: https://redis.io/commands/eval https://redis.io/commands/eval
  - dawnerd 5y ago
    Yeah - kinda crazy theres no auth by default AND eval is allowed. Pretty trivial for someone to have it download a script and run it pretty much with free reign.
    - slavak 5y ago
      Redis doesn't accept unauthenticated external connections by default for a while now, specifically to try and eliminate this footgun. https://github.com/redis/redis/commit/edd4d555df57dc84265fdfb4ef59a4678832f6da https://github.com/redis/redis/commit/edd4d555df57dc84265fdf...
      - dawnerd 5y ago
        Interesting, this definitely happened with a more recent version... Wonder if theres some other exploit at play too (could also be the containerized version?)
      - SilverRed 5y ago
        I had an issue where I used the redis docker image and didn't understand docker networking properly so I set the network mode as host so my other container could connect. Not knowing this had exposed redis to the world unauthenticated (in about 2018). Eventually a kind script set a password on redis which caused me to notice and fix this issue.
  - bigiain 5y ago
    From antirez the guy who wrote redis's blog: 'The Redis security model is: “it’s totally insecure to let untrusted clients access the system, please protect it from the outside world yourself”.' -- http://antirez.com/news/96 http://antirez.com/news/96 That blog post also helpfully shows how to write you own key into .ssh/authorized_keys to you can log in as the redis user over ssh. From there use your favourite lunar priv escalation bug to p0wn the box completely. (Or just run your cryptominer as the redis user...) Note: that's about 5 years old now...
- tpetry 5y ago
  I guess most people have faced this. I got years ago too but after a few seconds some internal monitoring system alerted me that new ports have been exposed. I am not sure anymore how i solved it but after some time (and none of the documented solutions working) i decided to let docker do to iptables whatever it wants to and used another firewall which filtered the traffic again after iptables was done "filtering" it.
- Aeolun 5y ago
  What is this black magic? Why is docker concerned with iptables at all? And these docs read a lot like: “We are going to totally ignore any of your firewall rules unless you follow all these steps exactly and do a lot of manual work.”
29athrowaway 5y ago
There are search engines for services exposed to the internet, like https://www.shodan.io/ https://www.shodan.io/ If your mongoDB server is exposed to the Internet it will show up there. When that happens, it's only a matter of time until someone targets you. You can write an alert that probes for sensitive services exposed to the Internet. In that way, if this happens again, you get an alert that you can use to detect the problem early. Also... use authentication for your database, it doesn't take much effort to do. With a secure password.
- mr_o47 5y ago
  How does shodan works like how do they know if something is exposed to the internet. Are they scanning networks 24/7 I’m just a noob in security so therefore learning
  - 29athrowaway 5y ago
    They go through the entire ip address range scanning specific ports.
  - zootboy 5y ago
    They do a monthly scan, with additional spot checks available on-demand: https://help.shodan.io/the-basics/on-demand-scanning https://help.shodan.io/the-basics/on-demand-scanning
    - achillean 5y ago
      We actually scan on average once a week. I used that language to be ultra conservative but I'll need to change it. For the past 8+ years we've been doing weekly scans.
      - exikyut 5y ago
        In case you see this, the most interesting question I think I could possibly ask is, what does the current real-world impact of IPv6 appear to be practically speaking? Abstractly and intuitively, IPv6's massiveness would seem to put an end to the interesting closed loop of address space vs backhaul capacity that has developed around v4. I can't help but wonder though - with for example some providers leasing out ginormous blocks of address space according to fairly predictable patterns (and customers just using the first v6 address that pops out - if at all), this makes me wonder if it'll be possible to steer v6 scans using a mix of statistics, machine learning, and Perl if statements :). The other thing I'm idly curious about is how you actually scan on a regular basis. Broadly speaking about long-term viability, I guess the TL;DR probably boils down to coordination and careful nurturing of reputation similar to what the large-scale email providers maintain. But from a technical perspective, I do wonder if/how much things like peering, and BGP, and noise-cancelling routing (if you will), etc, come into the picture - and how big the links are :D I would be very happy to coincidentally discover writeups touching on these questions anytime. Thanks for reading :)
shitloadofbooks 5y ago
I don't know what your monitoring setup is like, but I'd recommend something like Prometheus' blackbox-exporter to help alarm when your network "changes". I setup alerting rules so that certain subnets or servers suddenly being exposed to certain instances of blackbox will wake people up!
fiddlerwoaroof 5y ago
docker has so many of these footguns.
qbasic_forever 5y ago
Oooph, good luck. And when you have time please make Docker aware that this well known foot-gun has finally done serious harm. They have known and ignored for years that iptables/ufw on Linux is totally broken and wide open when using Docker: https://github.com/moby/moby/issues/4737 https://github.com/moby/moby/issues/4737
- gnyman 5y ago
  Glad someone else highlighted this old ticket. I bet this, in combination with the extremely irresponsible solution to ship mongodb without auth as default has caused countless of data leaks and destruction events. We just haven't heard about most of them. Elastic provides the same foot-gun. Last year someone deleted almost 4000 open mongodb and elastic databases in what was called the Meow attack [1]. In my opinion it's as irresponsible as HW manufacturers shipping with default passwords, something which finally got the attention of regulators[2]. So I wouldn't be surprised if we at some point see some attempts to keep sw-developers accountable for what they give out. I have for some time been using the data-oil analogy to describe where we are at. If data is the new oil, and a database is a tanker, then we are at the single hull tanker stage. We need double hulls, but just like the oil industry, the sw industry has little incentive to fix it themselves. I am hoping we get some regulation which improves the situation, because otherwise this will keep happening. 1: https://www.bleepingcomputer.com/news/security/new-meow-attack-has-deleted-almost-4-000-unsecured-databases/ https://www.bleepingcomputer.com/news/security/new-meow-atta... 2: https://techcrunch.com/2018/10/05/california-passes-law-that-bans-default-passwords-in-connected-devices/ https://techcrunch.com/2018/10/05/california-passes-law-that...
  - qbasic_forever 5y ago
    Yeah, it seems like there's a weird inbetween phase when projects go from "awesome tool used and loved by some core people" to "this is the new normal, it's everywhere" where these issues get lost. I could see back in 2014 moby not really feeling like the quirks of ufw & iptables were its problem. But now in 2021 with how many millions of times docker run is used per day on machines all across the internet... it's just irresponsible security. It sucks, it isn't really docker's problem in the first place... but it's reality and someone needs to grab the hot potato and keep it from burning people for the good of us all.
conesus 5y ago
In case anybody's interested, here's what the "hack" looks like: nbset:PRIMARY> show dbs READ__ME_TO_RECOVER_YOUR_DATA 0.000GB admin 0.000GB local 16.471GB newsblur 0.718GB nbset:PRIMARY> use READ__ME_TO_RECOVER_YOUR_DATA switched to db READ__ME_TO_RECOVER_YOUR_DATA nbset:PRIMARY> show collections README system.profile nbset:PRIMARY> db.README.find() { "_id" : ObjectId("60d3e112ac48d82047aab95d"), "content" : "All your data is a backed up. You must pay 0.03 BTC to XXXXXXFTHISGUYXXXXXXX 48 hours for recover it. After 48 hours expiration we will leaked and exposed all your data. In case of refusal to pay, we will contact the General Data Protection Regulation, GDPR and notify them that you store user data in an open form and is not safe. Under the rules of the law, you face a heavy fine or arrest and your base dump will be dropped from our server! You can buy bitcoin here, does not take much time to buy https://localbitcoins.com or https://buy.moonpay.io/ After paying write to me in the mail with your DB IP: FTHISGUY@recoverme.one and you will receive a link to download your database dump." }
- deleted 5y ago
  [deleted]
- 29athrowaway 5y ago
  What happened to your service is the security equivalent of the scholar's mate. It's important to be able to lose with dignity and move on. Your adversary was unsophisticated but this incident was your fault.
- pmontra 5y ago
  > you face a heavy fine or arrest Heavy fine yes but not arrest AFAIK. Anyway this is a script programed to scary the target. Do you even store personal data inside that database?
  - xref 5y ago
    From their Twitter feed: mongodb is just RSS feed data, personal data is in postgres and wasn’t accessible to the script kiddy
DevX101 5y ago
Put passwords on your production databases. Even if it's behind a firewall.
- fastball 5y ago
  Yeah, my setup is: private network + whitelist + password.
steventhedev 5y ago
I'm sorry to hear that you got bit by the docker networking thing. It bit me twice in the past. Once with a new server and once when they changed the config format from envvars to json (we were disabling dockers iptables nonsense). Do you know if they simply encrypted the data in place or if they succeeded in exfiltrating a full copy?
ralphington 5y ago
What kind of database auth did you have? Wouldn't they have had to access config files or related in order to obtain your passwords, usernames, etc?
- beermonster 5y ago
  I think by default mongodb has no enabled access control, so there is no default user or password.
  - pm90 5y ago
    How is this acceptable… requiring a password, even a weak one might have at least bought some time in this situation.
    - beermonster 5y ago
      See https://acowebs.com/mongodb-security/ https://acowebs.com/mongodb-security/
  - vultour 5y ago
    Am I misunderstanding or do people launch their Mongo container without even MONGO_INITDB_ROOT_{USERNAME,PASSWORD}? It's clearly mentioned in the image README. Takes 15 seconds to set. I'd be incredibly concerned if anybody with more than a day of infrastructure experience did this, even worse on a production database.
- dheera 5y ago
  Mongo is so insecure that it's commonplace to not bother with usernames and passwords and just firewall the hell out of it instead. Plus that's one more plaintext password you'll end up storing all over the place. Its default configuration requires no authentication. Not saying it's a good practice but it's a common pattern I've seen.
serial_dev 5y ago
Why do you call them "script kiddie" and not a hacker? IMO it's still a hacker even if the attack is not very sophisticated or even if you made a big security mistake.
- bitexploder 5y ago
  I think it’s fine. Script kiddie is a strict subset of “hacker” in the negative sense of the word “hacker”. It’s a way to convey to the reader the level of sophistication used in the attack by describing the hacker in this way.
  - haimez 5y ago
    Right, and there should be a sense of shame associated with being pwned by, for example, not setting a password on your public internet accessible (redis || postgres || mongo) instances. You didn’t get hacked, you let a child have their way with your application. Hence: script kiddie
    - kbatten 5y ago
      script kiddie is simply a hacker term, it has nothing to do with age or even skill.
      - haimez 5y ago
        > it has nothing to do with age… It’s in the name, so it doesn’t seem worth disputing that at least at the time when the term was coined- it did have to do with age > or even skill Skill is literally the defining feature. What exactly do you mean by “it’s simply a hacker term” anyway? Are words just sounds we make with our mouths?
- gumby 5y ago
  No hacking skill or knowledge required to simply download and run someone else’s script. That’s why they are called “script kiddies”.
tkojames 5y ago
Did you have default or something setup for mongo db users? Because even if it open to world a strong password and fail2ban would stop this. Still dumb of docker to not be more clear. Sometimes I wish BSD jails become more popular.
rubatuga 5y ago
I ran into this iptables issue in 2017 while setting up Monica in a Docker container, decided to never use docker since.
vngzs 5y ago
I think there are some good lessons here: 1. Even if you have one way to protect your database (e.g., firewall rules), you should have another. In this case, use a database password or (better) client TLS certificate to authenticate traffic. We're all human and we mess up. You should be designing systems that are graceful in response to your inevitable mistakes. 2. If you can afford another server/a hosting provider with VPC-like features, don't put your databases on the open internet. Run them in a private network (RFC 1918), behind a NAT and a load balancer/entrypoint that only routes to your internet-serving applications. Allow only those application servers to hit your databases. If you had done this, the attacker wouldn't have noticed your mistake, because all they could hit was your public server. 3. Keep regular backups. Oh, and test them! If you don't test your backups, you don't have backups - you have archives. In the GitLab data loss incident [0], they had 3 methods of data backup, but all of them failed. A regular test would have discovered this. Don't make that mistake. Good on you for sharing the events as they happen. I think people tend to be much more forgiving in response to openness. Don't freak out, and write a public postmortem when you're done. [0]: https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com-database-incident/ https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com-data...
- lmm 5y ago
  I agree with only the third of those. The other lessons I'd take would be: 1. Be cautious about trendy technologies that promise to make life easy - often they cut corners to do so, and often security is one of those corners 2. Use real authentication rather than network firewalling. Make your datastore TLS-only and require a valid client certificate to connect; that way it doesn't matter if it's exposed to the internet (indeed I'd argue you probably should expose every server to the internet - much like Chaos Monkey, it's counterintuitive but it forces you to build your systems with the right kind of resilience from day one)
  - vngzs 5y ago
    Yeah, there's the whole "zero trust" possibility which I didn't mention. If you do authentication/authorization really well, you can stop doing (2). For things like databases, I think it's better to treat them as if they _could_ be exposed to the open internet, without actually doing so. It's generally not the case that anyone on the internet needs to query your DBs. As described, you're _only_ relying on client TLS to protect your database. What if the TLS key leaks? So you're down to one layer again - one mistake and it's game over. So maybe you need Hashicorp Vault so you can have client certificates with very short validity periods. So do you expose Vault to the Internet, so you can fetch your client certificate to query the DB? What if Vault has a bug? And it's turtles all the way down ... I love zero trust designs. But I think saying you can just slap client TLS on the problem and be done is laying the foundation for a repeat of this event - however you look at it, you want redundancy in your security.
rank0 5y ago
Good luck! I appreciate your transparent account of the situation but what does it say about your company if your database got popped by a “script kiddie”?
- jacquesm 5y ago
  It says more about Docker than anything else. This is an insane default setting, it's something that should have been fixed when it was first brought to their attention. Computer security is hard enough without loaded footguns like these lying around.
  - beermonster 5y ago
    Debian has no firewall rules by default. Up until recently also home directory permissions that were not good for multi user systems. Both by design. Defaults are often insecure but maximise interoperability or general usability. Look at Windows !
    - jacquesm 5y ago
      Yes, Debian - and Ubuntu, for that matter - have some pretty bad defaults in some places. Having users' homedirs UGO rwxr-xr-x is pretty bad. The defaults should be secure with explicit unlock steps for those that know their environment well enough that they can explicitly relax some restrictions.
  - rank0 5y ago
    I understand your point but how do you explain the complete absence of database security controls? That part is on Newsblur. Defense in depth is important!
    - jacquesm 5y ago
      Yes, they absolutely have some culpability, but making a change like this without alerting the administrator of the system is the rough equivalent of any process with 'root' privileges on any one of your servers suddenly executing an iptables command to allow all access. You'd only know about it because you got hacked. Such drastic changes to the security model should only be one after explicit instruction. The number of companies that operate without access controls between servers on the same segment is unfortunately quite large, database security controls are - again - more often than not left at their default setting and those too are quite often insecure. Defense in depth always has limited depth, though I totally agree that running a database without access controls is not the way to go.
rbut 5y ago
Use the new rootless mode and you won’t have issues with it inserting it’s rules above UFW. You can then expose ports to a specific IP and use UFW to allow it. Much cleaner than any UFW-docker hacks out there, and more secure. https://docs.docker.com/engine/security/rootless/ https://docs.docker.com/engine/security/rootless/
- sascha_sl 5y ago
  The real workaround has always been to disable iptables and masq in the docker daemon and set up those things yourself, with your existing firewall. Binding your port to loopback works too, but that's more prone to accidents.
metafunctor 5y ago
Yeah, configure `ufw default deny incoming` and Docker will sneakily bypass that and gives the internet unfettered access to your ElasticSearch and MongoDB and whatever. And somehow, that's apparently "not a bug". It's one of the biggest footguns I know of, and could not believe it was expected behavior when I first encountered it. We have a strict policy that servers must not have a routable IP address at all, and must have an external firewall applied. It's a good idea in any case, but turns out it's absolutely necessary with Docker.
pm90 5y ago
Is 3 hours a large time for an open server to be discovered? Do the attackers just have a giant list of ips that they constantly scan and can instantly know if it’s suddenly open to traffic?
- Shank 5y ago
  Check out masscan [0]. It’s extremely easy to scan IPv4 very rapidly and find targets in an automated fashion. It advertises scanning the internet in 5 minutes. [0]: https://github.com/robertdavidgraham/masscan https://github.com/robertdavidgraham/masscan
- shakna 5y ago
  ZMap claims to be able to scan the entire IPv4 space of the Internet in about 45mins. [0] There's no reason not to believe that claim, either. With many people doing this, it is kind of surprising that it took so long for a vulnerable server to be discovered. [0] https://zmap.io/ https://zmap.io/
- gnyman 5y ago
  Yes if you publish a known insecure service like mongodb, on the standard port, on a well known VPS provider you can expect it to be automatically compromised within hours if not minutes. As others commented, scanning the whole Internet is even not a problem so scanning a "limited" part where you are likely to see these services pop up is even less of a problem. I think the takeaway is that you cannot hide in the masses on the Internet anymore, 10-20 years ago you could throw up a insecure server and it could be fine for a long time. Nowadays you must assume someone will find and try to login to your service, even if you put it on a non-standard port. Also, if it's a HTTPS service take note they when you get a certificate you will be announcing that domain to the whole world and publish it to a searchable database (for example https://crt.sh/ https://crt.sh/ ).
  - jd_mongodb 5y ago
    If you want to secure your on-premise MongoDB we publish a checklist here https://docs.mongodb.com/manual/administration/security-checklist/ https://docs.mongodb.com/manual/administration/security-chec... better still use MongoDB Atlas and get our best security practices baked in.
TheChaplain 5y ago
The docker part bit me in the behind as well, had absolutely no idea it would circumvent ufw by design. Angry at myself for not reading the docs carefully but who has time for that? :/
vinayan3 5y ago
Thank you for letting us know and being clear about it. Really like NewsBlur. Hope it comes back soon!
watermelon0 5y ago
It seems that you are using DigitalOcean. They offer a cloud firewall [1], which sits in front of your droplets, and you can limit inbound ports to only the necessary ones (e.g. 80, 443, SSH). I'm always using this with providers that support it, since I've managed to mistakenly open ports that should be private (either by misconfigured firewall, or due to the Docker "issue"). [1] https://docs.digitalocean.com/products/networking/firewalls/ https://docs.digitalocean.com/products/networking/firewalls/
EnigmaCurry 5y ago
Here's the fix to prevent docker messing with ufw rules: https://github.com/chaifeng/ufw-docker https://github.com/chaifeng/ufw-docker
herpderperator 5y ago
I'm sorry that you have to go through this, it seems inevitable these days. However, while it's nice that you're sharing your analysis of the situation, you start off by downplaying the attack and calling them a script kiddie. If for example someone finds out they can brute-force Facebook's 6-digit password reset token because they didn't put any rate-limiting in[0], are they a hacker? Is there major skill involved in doing so or just a million-iteration loop to go through all the combinations? They received a $15,000 payout indicating that Facebook values their mistake seriously (although I'd say it's worth much more given it's a guaranteed full account takeover.) So regardless of what you think of the attack, easy or not, you still made a security mistake and you should admit that first and foremost rather than brushing it off as a "script kiddie situation". Other than that, it's commendable that you have working backups and are responding calmly and with a plan. I hope you get everything back in working order smoothly :) [0] https://www.theverge.com/2016/3/8/11179926/facebook-account-security-flaw-bug-bounty-payout https://www.theverge.com/2016/3/8/11179926/facebook-account-...
- tommica 5y ago
  I think the "script kiddie situation" comes from the part of trying to ransom the data
- deleted 5y ago
  [deleted]
- black3r 5y ago
  hackers start with a target and try to find a vulnerability. script kiddies start with a vulnerability and try to find sites vulnerable to it. it's not about the skill involved in making the exploit, it's about the effort around that. the case you mention is more of a hacker feat because that exploit had to be crafted specifically for facebook. meanwhile in this case it was most probably someone who just continuously scans the IPv4 address space for open mongo instances and applies the same generic "exploit" against them if it finds some in a fully automatized process
  - hutzlibu 5y ago
geenat 5y ago
Absolutely horrid that Docker still bypasses iptables / ufw by default.
1337shadow 5y ago
Don't pay, they have deleted your data and you're not getting it back. This non targeted attack has been running since 2017 and is pretty well documented now, nobody ever reported getting their data back after paying. ALWAYS read for existing documentation about an attack you have been victim of, should be one of the first thing you do, especially prior to pay. https://www.imperva.com/blog/ransomware-attacks-on-mysql-and-mongodb/ https://www.imperva.com/blog/ransomware-attacks-on-mysql-and... https://www.itproportal.com/news/ransomware-attacks-on-mongodb-servers-use-gdpr-to-blackmail-victims/ https://www.itproportal.com/news/ransomware-attacks-on-mongo... https://security.stackexchange.com/questions/237048/mongo-db-hacked-read-me-to-recover-without-the-port-exposed-in-the-firewall https://security.stackexchange.com/questions/237048/mongo-db... Everybody falls for that, I mean, look at the BTC these guys made, it's crazy! Anyway, Docker uses the DOCKER-USER firewall chain: https://docs.docker.com/network/iptables/ https://docs.docker.com/network/iptables/ Example: https://yourlabs.io/oss/yourlabs.docker/-/blob/master/tasks/main.yml#L77-115 https://yourlabs.io/oss/yourlabs.docker/-/blob/master/tasks/... People should really test their firewalls after setting it up. Another thing, instead of using Ansible+Docker and exposing ports like that, use Ansible+Docker-Compose, so that your containers of a stack have their own private shared network, then you won't have to publish ports to make your services communicate. https://docs.ansible.com/ansible/latest/collections/community/general/docker_compose_module.html https://docs.ansible.com/ansible/latest/collections/communit...
python273 5y ago
The same thing happened to me a few years ago. I used DigitalOcean's Docker image and it had some message about UFW in motd, so I assumed it works with Docker. So I created a container with passwordless mongodb and it got wiped in a few hours. And DO still have this in motd for newly created droplets: Welcome to DigitalOcean's 1-Click Docker Droplet. To keep this Droplet secure, the UFW firewall is enabled. All ports are BLOCKED except 22 (SSH), 2375 (Docker) and 2376 (Docker). Full motd: https://pastebin.com/cdaecHU8 https://pastebin.com/cdaecHU8 Though it links to https://do.co/3j6j3po https://do.co/3j6j3po and it mentions ufw problem: > Note: The default firewall for the Docker One-Click is UFW, which is a front end to iptables. However, Docker modifies iptables directly to set up communication to and from containers. This means that UFW won’t give you a full picture of the firewall settings. You can override this behavior in Docker by adding --iptables=false to the Docker daemon.
- jbuhbjlnjbn 5y ago
  So the makers know of the security issue, but still leave it in by default? That's bad. Either fix the issue, or put warnings all over the place that cannot be missed to inform the user. This is just what another poster commented on, sacrificing security for ease of use.
  - python273 5y ago
    Yeah, I reported it to DO support and suggested to add a warning, but seems like it was never added. DigitalOcean Support Thursday, March 15, 2018 9:53 PM Hello, Thank you very much for bringing this to our attention. I will create an internal escalation to our images team to review this. :) [...]
z3t4 5y ago
My theory is that its MongoDB that is behind the ransomware, why else would they 1. Not have auth protection 2. Open up the firewall !?
- alpacaillama 5y ago
  If you read what the author said and understood it, you would see that mongodb didn’t open up the firewall and it does have auth protection but the author chose not to enable it.
deleted 5y ago
[deleted]
xtat 5y ago
Docker is such a massively leaky abstraction :( Thanks for the detail. Newsblur is great.
andrewstuart 5y ago
You have backups, right?
shocks 5y ago
> Turns out the ufw firewall I enabled and diligently kept on a strict allowlist with only my internal servers didn't work on a new server because of Docker. I’ve been got by this too.
UrielFanelli 5y ago
I can infer so many errors in the architecture, I wonder how this may have survived so far. 1. you put your DB in a server which is exposed to the internet. 2. you have no VIP/NAT in front of your systems. 3. you rely in iptables , while knowing some automatic system is manipulating it. 3 hours? I wonder it took so long. I expect this infrastructure will be a script kiddies party room within a few minutes.
- deleted 5y ago
  [deleted]
- Kiro 5y ago
  As someone who has been running multiple services with millions of users for decades: 1. I need to be able to connect to my DB from anywhere. 2. No idea what that even means. 3. Don't know. Never even touched the firewall. I have a PW on my DB and that's it. Why do I need more than that?
  - unionpivo 5y ago
    > 1. I need to be able to connect to my DB from anywhere. Understandable and reasonable. there are ways to achieve this without exposing the DB but it take some more effort. > 3. Don't know. Never even touched the firewall. Firewalls are good, because they add an extra layer on security. Personally I believe, there should be firewall on most servers (firewall should be the default). But firewalls aren't magic and they are just one layer. > I have a PW on my DB and that's it. Why do I need more than that? Because this means you have 0 margins for error. Either you PW auth works flawlessly and without bugs, or you are screwed. Also you expose one more endpoint that can potentially be a ddos target.
  - LunaSea 5y ago
    You need to: - put the database in a virtual private cloud (VPC), an internal network - setup a Virtual Private Network (VPN) also placed in the same VPC from which developers can connect to to access the internal network - setup at least two MongoDB users, one `readWrite` user that can connect from the internal network and one administrative user that can only connect from localhost - setup a key based SSH connection only accessible from the VPN to the MongoDB instance - setup Security Groups (firewall) to lock all the unused ports and IP origins out That way you'll need a VPN key, an SSH key and the MongoDB admin user's access to fully compromise the database.
slver 5y ago
> This situation is more of a script kiddie than a hacker. Is this a useful distinction? Define the sharp line between a "script kiddie" and a "hacker". You're describing a person who detected your vulnerability and acted within a 3 hour window of it appearing and closing. If I were you I'd be more focused on reflection and not on putting labels on whoever hacked you.
ghgr 5y ago
> When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world. Yeah I had the same problem. If you are not using orchestration engines like Swarm of Kubernetes you can just avoid it by explicitly binding the port to your local interface (i.e. -p 127.0.0.1:27017:27017 instead of just -p 27017:27017)
roboben 5y ago
This is just another reason to laugh more about dockers "enterprise production ready" offering. So many things I encountered in Docker are so _not_ enterprise at all.
sneak 5y ago
Is there a colo or host that will run honeypots within your /24 and detect and automatically block malicious traffic sources like this? Seems like it would be a dead simple detection system, and would be a huge value-add. You could of course order an additional IP and do it yourself, too, but it seems like the colo doing it would be more efficient in terms of scale. (And they'd likely be more diligent in doing it, avoiding obvious DoS vectors like triggering the honeypot from AWS netblocks.) Hetzner (my host) sends me nastygrams from their IDS when my box tries to connect to RFC1819 space (which isn't even routable via them!) when running p2p software like ipfs. You'd think if they are willing to complain to customers about zero-impact stuff like that, they'd be willing to blackhole non-customers for nonzero impact malicious traffic.
qxmat 5y ago
Once this is over consider looking at network topology as a security mechnism in it's own right. Professionally I try operate a subnet hierarchy: public, intermediate, private where there's no routing information between public and private and private has no internet connectivity.
tluyben2 5y ago
I was caught out by this too[0]. I now have a fw script which runs automatically for demos etc. [0] https://github.com/docker-library/redis/issues/259#issuecomment-755928566 https://github.com/docker-library/redis/issues/259#issuecomm...
goatinaboat 5y ago
When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world. This is crazy. Your network should have been on a private IP address space behind a firewall running static NAT exposing only ports 80 and 443 on a routable IP address. This is network architecture 101.
k_ 5y ago
Similar thing happened to me on side projects a couple years ago. A docker update (or something like that, I don't remember the details) rewrote iptables config and opened my mongodb to the world. I didn't notice and the whole thing got ransomed over and over... Had some fun using mongodb but I don't think I'll ever use it again =/
uniqueuid 5y ago
Completely unrelated: NewsBlur was the first rss service I paid for after google reader closed down. I used it intensely for a long time and have very fond memories. I especially liked that you had open-sourced the code and spent some time looking at the architecture. I'm now self-hosting a rss reader, but NewsBlur will always remain dear to my heart.
carlsborg 5y ago
Been bitten by servers listening on 0:0:0:0 before. Nice thing about deploying on AWS is this class of problems is avoided by security groups defined in code or yaml and managed in github.
corobo 5y ago
That's a lot of blame being placed outwards there. It doesn't matter how script kiddie a person is if they got past your security. Disappointing response, this. What data got leaked? Please let haveibeenpwned.com know if your system leaked emails or worse.
- marcos100 5y ago
  Yeah, downplaying the guy who hacked you doesn't make it any better. Actually it makes it worse, since your security is so bad that any "script kiddie" can hack into your system.
- samhw 5y ago
  I think his point was _precisely_ that it was a stupid mistake on his part, and not a 'master hacker' subverting his defences. But I agree it's odd that he doesn't seem to have heard of defence in depth, and relied entirely on one UFW rule as opposed to also setting a password on Mongo and/or (preferably) configuring firewalls on the cloud provider level.
throwawinsider 5y ago
I also found frustrating that docker ignored ufw. Both ufw and iptables are very impractical apps. Add to it that docker integrates those rules in a obscure way. The result is a time sink and a huge security risk. Thanks god the server provider supports firewall rules in my case.
habosa 5y ago
Just want to say that I love Newsblur have I have been using it for free for many years. Reading this today made me stop and think about how hard you must work on it, so today I will donate/subscribe!
throwaway9870 5y ago
In addition to the comments I see here, one more note: seems like a lot of change at one time. * Ansible * Docker * Big redesign * New database cluster * New firewall config I have found great benefit to breaking problems down into smaller parts even though some times it causes some extra work.
coolsunglasses 5y ago
I think I can imagine why you would need a combination of PostgreSQL, Elasticsearch, and Redis, but what problem does MongoDB solve for you?
nirui 5y ago
> Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world Oh boi. I just knew(0) that default value is problematic. Too bad this time it caused somebody real money. (0): https://news.ycombinator.com/item?id=26678025 https://news.ycombinator.com/item?id=26678025

dawnerd 5y ago

This actually got me a while ago but with redis and some script kiddy turning my dev server into a bitcoin miner. Anyone else running docker and using iptables really needs to read this https://docs.docker.com/network/iptables/ https://docs.docker.com/network/iptables/

squeaky-clean 5y ago
Almost exactly the same thing has happened to me except Selenium and they were trying to log into Playstation Network accounts.
- deleted 5y ago
  [deleted]
qbasic_forever 5y ago
The insidious thing is that there's no indication, failure, log or anything to tell you something is out of the ordinary either. It would be one thing if it just exploded and failed to run, but it's even worse that it silently interacts with iptables & ufw to allow all the traffic through. The exact opposite of what you want or intended.
- dawnerd 5y ago
  Only reason I noticed was a command was taking slightly longer than normal and I checked htop and saw redis using 100% cpu. Absolutely insane this is the Docker default.
frakkingcylons 5y ago
Same exact thing happened to my test redis instance two years ago.
bushbaba 5y ago
I’m kind of shocked this is even deemed acceptable architecture. You’d think docker wouldn’t even touch iptables unless explicitly told to.
- qbasic_forever 5y ago
  It's how they can make containers feel like isolated little subnets without resorting to vxlan or other kernel-level stuff. It's a great development experience and I'd be sad to see it go. But.... it really needs to proactively detect and warn users. The issue has been known for many years. A quick little check and error out on startup if you're running on Ubuntu or have ufw enabled would probably save 99% of the pain people have had with it over the years.
  - 1337shadow 5y ago
    It works fine when users create docker networks for their containers to communicate, which docker-compose does by default, instead of publishing ports on 0.0.0.0 like yolo
- AtNightWeCode 5y ago
  I would say that this is the expected behavior. Also, I don't think one should rely on firewalls in this way.
  - Shank 5y ago
    Relying on firewalls to do what firewalls do and have done and continue to do seems perfectly acceptable. Yes, your database should have authentication enabled too, but expecting ports to not be unexpectedly open is the entire point of firewalls.
    - AtNightWeCode 5y ago
      Kind of what I said. Firewalls are for blocking unwanted traffic. It should not be used as a replacement for other security measures. "unexpectedly open", well, there I simply disagree.
js4ever 5y ago
Redis can be exploited to run executables ???
- qbasic_forever 5y ago
  It can execute Lua, I'm sure there's plenty of fun for hackers to have with this: https://redis.io/commands/eval https://redis.io/commands/eval
- dawnerd 5y ago
  Yeah - kinda crazy theres no auth by default AND eval is allowed. Pretty trivial for someone to have it download a script and run it pretty much with free reign.
  - slavak 5y ago
    Redis doesn't accept unauthenticated external connections by default for a while now, specifically to try and eliminate this footgun. https://github.com/redis/redis/commit/edd4d555df57dc84265fdfb4ef59a4678832f6da https://github.com/redis/redis/commit/edd4d555df57dc84265fdf...
    - dawnerd 5y ago
      Interesting, this definitely happened with a more recent version... Wonder if theres some other exploit at play too (could also be the containerized version?)
    - SilverRed 5y ago
      I had an issue where I used the redis docker image and didn't understand docker networking properly so I set the network mode as host so my other container could connect. Not knowing this had exposed redis to the world unauthenticated (in about 2018). Eventually a kind script set a password on redis which caused me to notice and fix this issue.
- bigiain 5y ago
  From antirez the guy who wrote redis's blog: 'The Redis security model is: “it’s totally insecure to let untrusted clients access the system, please protect it from the outside world yourself”.' -- http://antirez.com/news/96 http://antirez.com/news/96 That blog post also helpfully shows how to write you own key into .ssh/authorized_keys to you can log in as the redis user over ssh. From there use your favourite lunar priv escalation bug to p0wn the box completely. (Or just run your cryptominer as the redis user...) Note: that's about 5 years old now...
tpetry 5y ago
I guess most people have faced this. I got years ago too but after a few seconds some internal monitoring system alerted me that new ports have been exposed. I am not sure anymore how i solved it but after some time (and none of the documented solutions working) i decided to let docker do to iptables whatever it wants to and used another firewall which filtered the traffic again after iptables was done "filtering" it.
Aeolun 5y ago
What is this black magic? Why is docker concerned with iptables at all? And these docs read a lot like: “We are going to totally ignore any of your firewall rules unless you follow all these steps exactly and do a lot of manual work.”

29athrowaway 5y ago

There are search engines for services exposed to the internet, like https://www.shodan.io/ https://www.shodan.io/ If your mongoDB server is exposed to the Internet it will show up there. When that happens, it's only a matter of time until someone targets you. You can write an alert that probes for sensitive services exposed to the Internet. In that way, if this happens again, you get an alert that you can use to detect the problem early. Also... use authentication for your database, it doesn't take much effort to do. With a secure password.

mr_o47 5y ago
How does shodan works like how do they know if something is exposed to the internet. Are they scanning networks 24/7 I’m just a noob in security so therefore learning
- 29athrowaway 5y ago
  They go through the entire ip address range scanning specific ports.
- zootboy 5y ago
  They do a monthly scan, with additional spot checks available on-demand: https://help.shodan.io/the-basics/on-demand-scanning https://help.shodan.io/the-basics/on-demand-scanning
  - achillean 5y ago
    We actually scan on average once a week. I used that language to be ultra conservative but I'll need to change it. For the past 8+ years we've been doing weekly scans.
    - exikyut 5y ago
      In case you see this, the most interesting question I think I could possibly ask is, what does the current real-world impact of IPv6 appear to be practically speaking? Abstractly and intuitively, IPv6's massiveness would seem to put an end to the interesting closed loop of address space vs backhaul capacity that has developed around v4. I can't help but wonder though - with for example some providers leasing out ginormous blocks of address space according to fairly predictable patterns (and customers just using the first v6 address that pops out - if at all), this makes me wonder if it'll be possible to steer v6 scans using a mix of statistics, machine learning, and Perl if statements :). The other thing I'm idly curious about is how you actually scan on a regular basis. Broadly speaking about long-term viability, I guess the TL;DR probably boils down to coordination and careful nurturing of reputation similar to what the large-scale email providers maintain. But from a technical perspective, I do wonder if/how much things like peering, and BGP, and noise-cancelling routing (if you will), etc, come into the picture - and how big the links are :D I would be very happy to coincidentally discover writeups touching on these questions anytime. Thanks for reading :)

shitloadofbooks 5y ago

I don't know what your monitoring setup is like, but I'd recommend something like Prometheus' blackbox-exporter to help alarm when your network "changes". I setup alerting rules so that certain subnets or servers suddenly being exposed to certain instances of blackbox will wake people up!

fiddlerwoaroof 5y ago

docker has so many of these footguns.

qbasic_forever 5y ago

Oooph, good luck. And when you have time please make Docker aware that this well known foot-gun has finally done serious harm. They have known and ignored for years that iptables/ufw on Linux is totally broken and wide open when using Docker: https://github.com/moby/moby/issues/4737 https://github.com/moby/moby/issues/4737

gnyman 5y ago
Glad someone else highlighted this old ticket. I bet this, in combination with the extremely irresponsible solution to ship mongodb without auth as default has caused countless of data leaks and destruction events. We just haven't heard about most of them. Elastic provides the same foot-gun. Last year someone deleted almost 4000 open mongodb and elastic databases in what was called the Meow attack [1]. In my opinion it's as irresponsible as HW manufacturers shipping with default passwords, something which finally got the attention of regulators[2]. So I wouldn't be surprised if we at some point see some attempts to keep sw-developers accountable for what they give out. I have for some time been using the data-oil analogy to describe where we are at. If data is the new oil, and a database is a tanker, then we are at the single hull tanker stage. We need double hulls, but just like the oil industry, the sw industry has little incentive to fix it themselves. I am hoping we get some regulation which improves the situation, because otherwise this will keep happening. 1: https://www.bleepingcomputer.com/news/security/new-meow-attack-has-deleted-almost-4-000-unsecured-databases/ https://www.bleepingcomputer.com/news/security/new-meow-atta... 2: https://techcrunch.com/2018/10/05/california-passes-law-that-bans-default-passwords-in-connected-devices/ https://techcrunch.com/2018/10/05/california-passes-law-that...
- qbasic_forever 5y ago
  Yeah, it seems like there's a weird inbetween phase when projects go from "awesome tool used and loved by some core people" to "this is the new normal, it's everywhere" where these issues get lost. I could see back in 2014 moby not really feeling like the quirks of ufw & iptables were its problem. But now in 2021 with how many millions of times docker run is used per day on machines all across the internet... it's just irresponsible security. It sucks, it isn't really docker's problem in the first place... but it's reality and someone needs to grab the hot potato and keep it from burning people for the good of us all.

conesus 5y ago

In case anybody's interested, here's what the "hack" looks like: nbset:PRIMARY> show dbs READ__ME_TO_RECOVER_YOUR_DATA 0.000GB admin 0.000GB local 16.471GB newsblur 0.718GB nbset:PRIMARY> use READ__ME_TO_RECOVER_YOUR_DATA switched to db READ__ME_TO_RECOVER_YOUR_DATA nbset:PRIMARY> show collections README system.profile nbset:PRIMARY> db.README.find() { "_id" : ObjectId("60d3e112ac48d82047aab95d"), "content" : "All your data is a backed up. You must pay 0.03 BTC to XXXXXXFTHISGUYXXXXXXX 48 hours for recover it. After 48 hours expiration we will leaked and exposed all your data. In case of refusal to pay, we will contact the General Data Protection Regulation, GDPR and notify them that you store user data in an open form and is not safe. Under the rules of the law, you face a heavy fine or arrest and your base dump will be dropped from our server! You can buy bitcoin here, does not take much time to buy https://localbitcoins.com or https://buy.moonpay.io/ After paying write to me in the mail with your DB IP: FTHISGUY@recoverme.one and you will receive a link to download your database dump." }

deleted 5y ago
[deleted]
29athrowaway 5y ago
What happened to your service is the security equivalent of the scholar's mate. It's important to be able to lose with dignity and move on. Your adversary was unsophisticated but this incident was your fault.
pmontra 5y ago
> you face a heavy fine or arrest Heavy fine yes but not arrest AFAIK. Anyway this is a script programed to scary the target. Do you even store personal data inside that database?
- xref 5y ago
  From their Twitter feed: mongodb is just RSS feed data, personal data is in postgres and wasn’t accessible to the script kiddy

DevX101 5y ago

Put passwords on your production databases. Even if it's behind a firewall.

fastball 5y ago
Yeah, my setup is: private network + whitelist + password.

steventhedev 5y ago

I'm sorry to hear that you got bit by the docker networking thing. It bit me twice in the past. Once with a new server and once when they changed the config format from envvars to json (we were disabling dockers iptables nonsense). Do you know if they simply encrypted the data in place or if they succeeded in exfiltrating a full copy?

ralphington 5y ago

What kind of database auth did you have? Wouldn't they have had to access config files or related in order to obtain your passwords, usernames, etc?

beermonster 5y ago
I think by default mongodb has no enabled access control, so there is no default user or password.
- pm90 5y ago
  How is this acceptable… requiring a password, even a weak one might have at least bought some time in this situation.
  - beermonster 5y ago
    See https://acowebs.com/mongodb-security/ https://acowebs.com/mongodb-security/
- vultour 5y ago
  Am I misunderstanding or do people launch their Mongo container without even MONGO_INITDB_ROOT_{USERNAME,PASSWORD}? It's clearly mentioned in the image README. Takes 15 seconds to set. I'd be incredibly concerned if anybody with more than a day of infrastructure experience did this, even worse on a production database.
dheera 5y ago
Mongo is so insecure that it's commonplace to not bother with usernames and passwords and just firewall the hell out of it instead. Plus that's one more plaintext password you'll end up storing all over the place. Its default configuration requires no authentication. Not saying it's a good practice but it's a common pattern I've seen.

serial_dev 5y ago

Why do you call them "script kiddie" and not a hacker? IMO it's still a hacker even if the attack is not very sophisticated or even if you made a big security mistake.

bitexploder 5y ago
I think it’s fine. Script kiddie is a strict subset of “hacker” in the negative sense of the word “hacker”. It’s a way to convey to the reader the level of sophistication used in the attack by describing the hacker in this way.
- haimez 5y ago
  Right, and there should be a sense of shame associated with being pwned by, for example, not setting a password on your public internet accessible (redis || postgres || mongo) instances. You didn’t get hacked, you let a child have their way with your application. Hence: script kiddie
  - kbatten 5y ago
    script kiddie is simply a hacker term, it has nothing to do with age or even skill.
    - haimez 5y ago
      > it has nothing to do with age… It’s in the name, so it doesn’t seem worth disputing that at least at the time when the term was coined- it did have to do with age > or even skill Skill is literally the defining feature. What exactly do you mean by “it’s simply a hacker term” anyway? Are words just sounds we make with our mouths?
gumby 5y ago
No hacking skill or knowledge required to simply download and run someone else’s script. That’s why they are called “script kiddies”.

tkojames 5y ago

Did you have default or something setup for mongo db users? Because even if it open to world a strong password and fail2ban would stop this. Still dumb of docker to not be more clear. Sometimes I wish BSD jails become more popular.

rubatuga 5y ago

I ran into this iptables issue in 2017 while setting up Monica in a Docker container, decided to never use docker since.

vngzs 5y ago

I think there are some good lessons here: 1. Even if you have one way to protect your database (e.g., firewall rules), you should have another. In this case, use a database password or (better) client TLS certificate to authenticate traffic. We're all human and we mess up. You should be designing systems that are graceful in response to your inevitable mistakes. 2. If you can afford another server/a hosting provider with VPC-like features, don't put your databases on the open internet. Run them in a private network (RFC 1918), behind a NAT and a load balancer/entrypoint that only routes to your internet-serving applications. Allow only those application servers to hit your databases. If you had done this, the attacker wouldn't have noticed your mistake, because all they could hit was your public server. 3. Keep regular backups. Oh, and test them! If you don't test your backups, you don't have backups - you have archives. In the GitLab data loss incident [0], they had 3 methods of data backup, but all of them failed. A regular test would have discovered this. Don't make that mistake. Good on you for sharing the events as they happen. I think people tend to be much more forgiving in response to openness. Don't freak out, and write a public postmortem when you're done. [0]: https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com-database-incident/ https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com-data...

lmm 5y ago
I agree with only the third of those. The other lessons I'd take would be: 1. Be cautious about trendy technologies that promise to make life easy - often they cut corners to do so, and often security is one of those corners 2. Use real authentication rather than network firewalling. Make your datastore TLS-only and require a valid client certificate to connect; that way it doesn't matter if it's exposed to the internet (indeed I'd argue you probably should expose every server to the internet - much like Chaos Monkey, it's counterintuitive but it forces you to build your systems with the right kind of resilience from day one)
- vngzs 5y ago
  Yeah, there's the whole "zero trust" possibility which I didn't mention. If you do authentication/authorization really well, you can stop doing (2). For things like databases, I think it's better to treat them as if they _could_ be exposed to the open internet, without actually doing so. It's generally not the case that anyone on the internet needs to query your DBs. As described, you're _only_ relying on client TLS to protect your database. What if the TLS key leaks? So you're down to one layer again - one mistake and it's game over. So maybe you need Hashicorp Vault so you can have client certificates with very short validity periods. So do you expose Vault to the Internet, so you can fetch your client certificate to query the DB? What if Vault has a bug? And it's turtles all the way down ... I love zero trust designs. But I think saying you can just slap client TLS on the problem and be done is laying the foundation for a repeat of this event - however you look at it, you want redundancy in your security.

rank0 5y ago

Good luck! I appreciate your transparent account of the situation but what does it say about your company if your database got popped by a “script kiddie”?

jacquesm 5y ago
It says more about Docker than anything else. This is an insane default setting, it's something that should have been fixed when it was first brought to their attention. Computer security is hard enough without loaded footguns like these lying around.
- beermonster 5y ago
  Debian has no firewall rules by default. Up until recently also home directory permissions that were not good for multi user systems. Both by design. Defaults are often insecure but maximise interoperability or general usability. Look at Windows !
  - jacquesm 5y ago
    Yes, Debian - and Ubuntu, for that matter - have some pretty bad defaults in some places. Having users' homedirs UGO rwxr-xr-x is pretty bad. The defaults should be secure with explicit unlock steps for those that know their environment well enough that they can explicitly relax some restrictions.
- rank0 5y ago
  I understand your point but how do you explain the complete absence of database security controls? That part is on Newsblur. Defense in depth is important!
  - jacquesm 5y ago
    Yes, they absolutely have some culpability, but making a change like this without alerting the administrator of the system is the rough equivalent of any process with 'root' privileges on any one of your servers suddenly executing an iptables command to allow all access. You'd only know about it because you got hacked. Such drastic changes to the security model should only be one after explicit instruction. The number of companies that operate without access controls between servers on the same segment is unfortunately quite large, database security controls are - again - more often than not left at their default setting and those too are quite often insecure. Defense in depth always has limited depth, though I totally agree that running a database without access controls is not the way to go.

rbut 5y ago

Use the new rootless mode and you won’t have issues with it inserting it’s rules above UFW. You can then expose ports to a specific IP and use UFW to allow it. Much cleaner than any UFW-docker hacks out there, and more secure. https://docs.docker.com/engine/security/rootless/ https://docs.docker.com/engine/security/rootless/

sascha_sl 5y ago
The real workaround has always been to disable iptables and masq in the docker daemon and set up those things yourself, with your existing firewall. Binding your port to loopback works too, but that's more prone to accidents.

metafunctor 5y ago

Yeah, configure `ufw default deny incoming` and Docker will sneakily bypass that and gives the internet unfettered access to your ElasticSearch and MongoDB and whatever. And somehow, that's apparently "not a bug". It's one of the biggest footguns I know of, and could not believe it was expected behavior when I first encountered it. We have a strict policy that servers must not have a routable IP address at all, and must have an external firewall applied. It's a good idea in any case, but turns out it's absolutely necessary with Docker.

pm90 5y ago

Is 3 hours a large time for an open server to be discovered? Do the attackers just have a giant list of ips that they constantly scan and can instantly know if it’s suddenly open to traffic?

Shank 5y ago
Check out masscan [0]. It’s extremely easy to scan IPv4 very rapidly and find targets in an automated fashion. It advertises scanning the internet in 5 minutes. [0]: https://github.com/robertdavidgraham/masscan https://github.com/robertdavidgraham/masscan
shakna 5y ago
ZMap claims to be able to scan the entire IPv4 space of the Internet in about 45mins. [0] There's no reason not to believe that claim, either. With many people doing this, it is kind of surprising that it took so long for a vulnerable server to be discovered. [0] https://zmap.io/ https://zmap.io/
gnyman 5y ago
Yes if you publish a known insecure service like mongodb, on the standard port, on a well known VPS provider you can expect it to be automatically compromised within hours if not minutes. As others commented, scanning the whole Internet is even not a problem so scanning a "limited" part where you are likely to see these services pop up is even less of a problem. I think the takeaway is that you cannot hide in the masses on the Internet anymore, 10-20 years ago you could throw up a insecure server and it could be fine for a long time. Nowadays you must assume someone will find and try to login to your service, even if you put it on a non-standard port. Also, if it's a HTTPS service take note they when you get a certificate you will be announcing that domain to the whole world and publish it to a searchable database (for example https://crt.sh/ https://crt.sh/ ).
- jd_mongodb 5y ago
  If you want to secure your on-premise MongoDB we publish a checklist here https://docs.mongodb.com/manual/administration/security-checklist/ https://docs.mongodb.com/manual/administration/security-chec... better still use MongoDB Atlas and get our best security practices baked in.

TheChaplain 5y ago

The docker part bit me in the behind as well, had absolutely no idea it would circumvent ufw by design. Angry at myself for not reading the docs carefully but who has time for that? :/

vinayan3 5y ago

Thank you for letting us know and being clear about it. Really like NewsBlur. Hope it comes back soon!

watermelon0 5y ago

It seems that you are using DigitalOcean. They offer a cloud firewall [1], which sits in front of your droplets, and you can limit inbound ports to only the necessary ones (e.g. 80, 443, SSH). I'm always using this with providers that support it, since I've managed to mistakenly open ports that should be private (either by misconfigured firewall, or due to the Docker "issue"). [1] https://docs.digitalocean.com/products/networking/firewalls/ https://docs.digitalocean.com/products/networking/firewalls/

EnigmaCurry 5y ago

Here's the fix to prevent docker messing with ufw rules: https://github.com/chaifeng/ufw-docker https://github.com/chaifeng/ufw-docker

herpderperator 5y ago

I'm sorry that you have to go through this, it seems inevitable these days. However, while it's nice that you're sharing your analysis of the situation, you start off by downplaying the attack and calling them a script kiddie. If for example someone finds out they can brute-force Facebook's 6-digit password reset token because they didn't put any rate-limiting in[0], are they a hacker? Is there major skill involved in doing so or just a million-iteration loop to go through all the combinations? They received a $15,000 payout indicating that Facebook values their mistake seriously (although I'd say it's worth much more given it's a guaranteed full account takeover.) So regardless of what you think of the attack, easy or not, you still made a security mistake and you should admit that first and foremost rather than brushing it off as a "script kiddie situation". Other than that, it's commendable that you have working backups and are responding calmly and with a plan. I hope you get everything back in working order smoothly :) [0] https://www.theverge.com/2016/3/8/11179926/facebook-account-security-flaw-bug-bounty-payout https://www.theverge.com/2016/3/8/11179926/facebook-account-...

tommica 5y ago
I think the "script kiddie situation" comes from the part of trying to ransom the data
deleted 5y ago
[deleted]
black3r 5y ago
hackers start with a target and try to find a vulnerability. script kiddies start with a vulnerability and try to find sites vulnerable to it. it's not about the skill involved in making the exploit, it's about the effort around that. the case you mention is more of a hacker feat because that exploit had to be crafted specifically for facebook. meanwhile in this case it was most probably someone who just continuously scans the IPv4 address space for open mongo instances and applies the same generic "exploit" against them if it finds some in a fully automatized process
- hutzlibu 5y ago

geenat 5y ago

Absolutely horrid that Docker still bypasses iptables / ufw by default.

1337shadow 5y ago

Don't pay, they have deleted your data and you're not getting it back. This non targeted attack has been running since 2017 and is pretty well documented now, nobody ever reported getting their data back after paying. ALWAYS read for existing documentation about an attack you have been victim of, should be one of the first thing you do, especially prior to pay. https://www.imperva.com/blog/ransomware-attacks-on-mysql-and-mongodb/ https://www.imperva.com/blog/ransomware-attacks-on-mysql-and... https://www.itproportal.com/news/ransomware-attacks-on-mongodb-servers-use-gdpr-to-blackmail-victims/ https://www.itproportal.com/news/ransomware-attacks-on-mongo... https://security.stackexchange.com/questions/237048/mongo-db-hacked-read-me-to-recover-without-the-port-exposed-in-the-firewall https://security.stackexchange.com/questions/237048/mongo-db... Everybody falls for that, I mean, look at the BTC these guys made, it's crazy! Anyway, Docker uses the DOCKER-USER firewall chain: https://docs.docker.com/network/iptables/ https://docs.docker.com/network/iptables/ Example: https://yourlabs.io/oss/yourlabs.docker/-/blob/master/tasks/main.yml#L77-115 https://yourlabs.io/oss/yourlabs.docker/-/blob/master/tasks/... People should really test their firewalls after setting it up. Another thing, instead of using Ansible+Docker and exposing ports like that, use Ansible+Docker-Compose, so that your containers of a stack have their own private shared network, then you won't have to publish ports to make your services communicate. https://docs.ansible.com/ansible/latest/collections/community/general/docker_compose_module.html https://docs.ansible.com/ansible/latest/collections/communit...

python273 5y ago

The same thing happened to me a few years ago. I used DigitalOcean's Docker image and it had some message about UFW in motd, so I assumed it works with Docker. So I created a container with passwordless mongodb and it got wiped in a few hours. And DO still have this in motd for newly created droplets: Welcome to DigitalOcean's 1-Click Docker Droplet. To keep this Droplet secure, the UFW firewall is enabled. All ports are BLOCKED except 22 (SSH), 2375 (Docker) and 2376 (Docker). Full motd: https://pastebin.com/cdaecHU8 https://pastebin.com/cdaecHU8 Though it links to https://do.co/3j6j3po https://do.co/3j6j3po and it mentions ufw problem: > Note: The default firewall for the Docker One-Click is UFW, which is a front end to iptables. However, Docker modifies iptables directly to set up communication to and from containers. This means that UFW won’t give you a full picture of the firewall settings. You can override this behavior in Docker by adding --iptables=false to the Docker daemon.

jbuhbjlnjbn 5y ago
So the makers know of the security issue, but still leave it in by default? That's bad. Either fix the issue, or put warnings all over the place that cannot be missed to inform the user. This is just what another poster commented on, sacrificing security for ease of use.
- python273 5y ago
  Yeah, I reported it to DO support and suggested to add a warning, but seems like it was never added. DigitalOcean Support Thursday, March 15, 2018 9:53 PM Hello, Thank you very much for bringing this to our attention. I will create an internal escalation to our images team to review this. :) [...]

z3t4 5y ago

My theory is that its MongoDB that is behind the ransomware, why else would they 1. Not have auth protection 2. Open up the firewall !?

alpacaillama 5y ago
If you read what the author said and understood it, you would see that mongodb didn’t open up the firewall and it does have auth protection but the author chose not to enable it.

deleted 5y ago

[deleted]

xtat 5y ago

Docker is such a massively leaky abstraction :( Thanks for the detail. Newsblur is great.

andrewstuart 5y ago

You have backups, right?

shocks 5y ago

> Turns out the ufw firewall I enabled and diligently kept on a strict allowlist with only my internal servers didn't work on a new server because of Docker. I’ve been got by this too.

UrielFanelli 5y ago

I can infer so many errors in the architecture, I wonder how this may have survived so far. 1. you put your DB in a server which is exposed to the internet. 2. you have no VIP/NAT in front of your systems. 3. you rely in iptables , while knowing some automatic system is manipulating it. 3 hours? I wonder it took so long. I expect this infrastructure will be a script kiddies party room within a few minutes.

deleted 5y ago
[deleted]
Kiro 5y ago
As someone who has been running multiple services with millions of users for decades: 1. I need to be able to connect to my DB from anywhere. 2. No idea what that even means. 3. Don't know. Never even touched the firewall. I have a PW on my DB and that's it. Why do I need more than that?
- unionpivo 5y ago
  > 1. I need to be able to connect to my DB from anywhere. Understandable and reasonable. there are ways to achieve this without exposing the DB but it take some more effort. > 3. Don't know. Never even touched the firewall. Firewalls are good, because they add an extra layer on security. Personally I believe, there should be firewall on most servers (firewall should be the default). But firewalls aren't magic and they are just one layer. > I have a PW on my DB and that's it. Why do I need more than that? Because this means you have 0 margins for error. Either you PW auth works flawlessly and without bugs, or you are screwed. Also you expose one more endpoint that can potentially be a ddos target.
- LunaSea 5y ago
  You need to: - put the database in a virtual private cloud (VPC), an internal network - setup a Virtual Private Network (VPN) also placed in the same VPC from which developers can connect to to access the internal network - setup at least two MongoDB users, one `readWrite` user that can connect from the internal network and one administrative user that can only connect from localhost - setup a key based SSH connection only accessible from the VPN to the MongoDB instance - setup Security Groups (firewall) to lock all the unused ports and IP origins out That way you'll need a VPN key, an SSH key and the MongoDB admin user's access to fully compromise the database.

slver 5y ago

> This situation is more of a script kiddie than a hacker. Is this a useful distinction? Define the sharp line between a "script kiddie" and a "hacker". You're describing a person who detected your vulnerability and acted within a 3 hour window of it appearing and closing. If I were you I'd be more focused on reflection and not on putting labels on whoever hacked you.

ghgr 5y ago

> When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world. Yeah I had the same problem. If you are not using orchestration engines like Swarm of Kubernetes you can just avoid it by explicitly binding the port to your local interface (i.e. -p 127.0.0.1:27017:27017 instead of just -p 27017:27017)

roboben 5y ago

This is just another reason to laugh more about dockers "enterprise production ready" offering. So many things I encountered in Docker are so _not_ enterprise at all.

sneak 5y ago

Is there a colo or host that will run honeypots within your /24 and detect and automatically block malicious traffic sources like this? Seems like it would be a dead simple detection system, and would be a huge value-add. You could of course order an additional IP and do it yourself, too, but it seems like the colo doing it would be more efficient in terms of scale. (And they'd likely be more diligent in doing it, avoiding obvious DoS vectors like triggering the honeypot from AWS netblocks.) Hetzner (my host) sends me nastygrams from their IDS when my box tries to connect to RFC1819 space (which isn't even routable via them!) when running p2p software like ipfs. You'd think if they are willing to complain to customers about zero-impact stuff like that, they'd be willing to blackhole non-customers for nonzero impact malicious traffic.

qxmat 5y ago

Once this is over consider looking at network topology as a security mechnism in it's own right. Professionally I try operate a subnet hierarchy: public, intermediate, private where there's no routing information between public and private and private has no internet connectivity.

tluyben2 5y ago

I was caught out by this too[0]. I now have a fw script which runs automatically for demos etc. [0] https://github.com/docker-library/redis/issues/259#issuecomment-755928566 https://github.com/docker-library/redis/issues/259#issuecomm...

goatinaboat 5y ago

When I containerized MongoDB, Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world. This is crazy. Your network should have been on a private IP address space behind a firewall running static NAT exposing only ports 80 and 443 on a routable IP address. This is network architecture 101.

k_ 5y ago

Similar thing happened to me on side projects a couple years ago. A docker update (or something like that, I don't remember the details) rewrote iptables config and opened my mongodb to the world. I didn't notice and the whole thing got ransomed over and over... Had some fun using mongodb but I don't think I'll ever use it again =/

uniqueuid 5y ago

Completely unrelated: NewsBlur was the first rss service I paid for after google reader closed down. I used it intensely for a long time and have very fond memories. I especially liked that you had open-sourced the code and spent some time looking at the architecture. I'm now self-hosting a rss reader, but NewsBlur will always remain dear to my heart.

carlsborg 5y ago

Been bitten by servers listening on 0:0:0:0 before. Nice thing about deploying on AWS is this class of problems is avoided by security groups defined in code or yaml and managed in github.

corobo 5y ago

That's a lot of blame being placed outwards there. It doesn't matter how script kiddie a person is if they got past your security. Disappointing response, this. What data got leaked? Please let haveibeenpwned.com know if your system leaked emails or worse.

marcos100 5y ago
Yeah, downplaying the guy who hacked you doesn't make it any better. Actually it makes it worse, since your security is so bad that any "script kiddie" can hack into your system.
samhw 5y ago
I think his point was _precisely_ that it was a stupid mistake on his part, and not a 'master hacker' subverting his defences. But I agree it's odd that he doesn't seem to have heard of defence in depth, and relied entirely on one UFW rule as opposed to also setting a password on Mongo and/or (preferably) configuring firewalls on the cloud provider level.

throwawinsider 5y ago

I also found frustrating that docker ignored ufw. Both ufw and iptables are very impractical apps. Add to it that docker integrates those rules in a obscure way. The result is a time sink and a huge security risk. Thanks god the server provider supports firewall rules in my case.

habosa 5y ago

Just want to say that I love Newsblur have I have been using it for free for many years. Reading this today made me stop and think about how hard you must work on it, so today I will donate/subscribe!

throwaway9870 5y ago

In addition to the comments I see here, one more note: seems like a lot of change at one time. * Ansible * Docker * Big redesign * New database cluster * New firewall config I have found great benefit to breaking problems down into smaller parts even though some times it causes some extra work.

coolsunglasses 5y ago

I think I can imagine why you would need a combination of PostgreSQL, Elasticsearch, and Redis, but what problem does MongoDB solve for you?

nirui 5y ago

> Docker helpfully inserted an allow rule into iptables, opening up MongoDB to the world Oh boi. I just knew(0) that default value is problematic. Too bad this time it caused somebody real money. (0): https://news.ycombinator.com/item?id=26678025 https://news.ycombinator.com/item?id=26678025