16 ms·
The MongoDB hack and the importance of secure defaults
- UlrichC 10y agoI cannot believe, that so many users install MongoDB, only using default settings and installed on servers, which are accessible via the internet. That is fundamentally wrong application design. The application connects to the database, so the database must not be public accessible via internet. I mean, if you push something into a production environment, you must think a bit about the tools you are using and how they are configured for security. On a development system, I don't want to setup users or auth-mechanisms. There are so many resources from MongoDB, how you can secure your MongoDB database: - Security Architecture White Paper: https://www.mongodb.com/collateral/mongodb-security-architecture https://www.mongodb.com/collateral/mongodb-security-architec... - Security Best Practices blog post: https://www.mongodb.com/blog/post/how-to-avoid-a-malicious-attack-that-ransoms-your-data https://www.mongodb.com/blog/post/how-to-avoid-a-malicious-a... There is even a checklist which you can work through: https://docs.mongodb.com/manual/administration/security-checklist https://docs.mongodb.com/manual/administration/security-chec... And for those, who don't want to read anything ;-), there is a comprehensive Video-Course on the MongoDB-university: https://university.mongodb.com/courses/M310/about https://university.mongodb.com/courses/M310/about If I enter a car and don't use the seat-belt or switch on the light at night (defaults are off), then I cannot blame the car-manufacturer. If I use something important (here the database in a production environment), then I have to learn the ropes and read the material specifically on security topics.
- 1_2__3 10y agoI'll say this is even worse because whether or not to bind to localhost, all interfaces, or something in between is often highly dependent on how the software works - which you're less likely to know when just trying it for the first time. Even someone security conscious would have to pause and do some investigations to find out what mongo expected in terms of port and interface bindings. If the default was to bind to 0.0.0.0 my first assumption would be they require that.
- jbverschoor 10y agoMongoDB has a history of using the insane defaults - not secure, and willing to loose your data.
- 51Cards 10y agoI have never used MongoDB so I admit I'm talking blind here, but can someone explain how/why a piece of highly popular software gets to version 2.6 allowing unsecured remote connections by default? Further to that is that type of thinking you want in the development process of something as critical as a database engine? It just seems amazing to me that it got so far before the community in general pushed back that this was really bad design? Hopefully someone with MongoDB history can explain if this was a long term sticking point, etc. Thanks.
- andersonmvd 10y agoWe usually can't see the same problems happening in front of our noses. Why shouldn't we discuss why docker images run as root by default then?
- jcrawfordor 10y agoI think we absolutely should discuss this - the number of images I see on Docker Hub that run complex network services as the root user really concerns me. The security measures implemented by Docker reduce the risk but absolutely do not eliminate it, you're taking a big risk there. This issue, along with its related issues with permissions on persistent storage, is one of the major reasons I'm hoping to move away from Docker as soon as possible.
- tannhaeuser 10y agoWhat Docker security measures do you mean? Actually, I was wondering recently what isolation Docker provides that a basic chroot-jail doesn't. Re root permisions I guess if you're running a typical network daemon (Apache, say), then requiring Docker to run as root is no worse than running the service on the host; root would be a requisite anyway for the daemon to open privileged ports/gain logging permissions etc. before dropping to "nobody" or what have you.
- kccqzy 10y agoDocker drops capabilities of root by default. It doesn't drop everything but better than not dropping at all. If you are not using docker, you can manually add the specific privilege to open low-numbered ports without running the program in question as root.
- grobbles 10y agoMongoDB was hardly alone -- almost all of these projects favored getting people started quickly, and avoiding anything that slowed them for even a moment. And we've seen it in project after project that it absolutely helps uptake, and if your project requires an hour of analysis and configuration before being usable, even if this saves tens of thousands of hours in the future, it puts it at a dramatic disadvantage. https://dennisforbes.ca/index.php/2012/10/02/trouble-with-securing-riak/ https://dennisforbes.ca/index.php/2012/10/02/trouble-with-se... I realize this post is dead (I disagreed with dang once), and that's okay. This is for the edification for those who can see ghosts.
- fdim 10y agoWhy would someone still use v2.6? There isn't much trouble to upgrade.
- discreditable 10y agoI find it interesting that there's no firewall with a default deny rule between these exposed mongodb installs and the Internet. All I can think is that most of them are on cloud services which are directly exposed. It reminds me of the fiasco with all of the directly-connected vulnerable network cameras on the Internet.
- 00deadbeef 10y agoI think the problem is developers running apt-get install mongodb and assuming all other considerations, like a firewall, are somehow magically taken care of, then patting themselves on the back for not needing a sysadmin.
- pavel_lishin 10y agoI think it's more of a case of assuming that the database wouldn't expose itself to the internet at large, because why would it? It's like buying a new car, and checking for holes in the gas tank before rolling off the lot.
- rblatz 10y agoTo make this analogy a bit more accurate. With this car instead of filling the gas tank by opening the gas tank door and removing a gas cap you just put the pump into a hole in the side of the car/tank. Then act like you had no idea gas was sloshing out the side hole of this tank. You didn't do anything special to connect to MongoDB, and are able to connect to it from your other servers. Why would you expect the same wouldn't apply to everyone? I will say that people who ran MongoDB on their app server get somewhat of a pass on this.I could imagine that they would find the idea of MongoDB running open to the world with no auth to be so nuts that they assumed it was only allowing connections from the local machine.
- 00deadbeef 10y agoYour car analogy fails because safety is highly regulated by the government, and the manufacturer can be sued for their failings. When it comes to server security, you are responsible, and should never make assumptions. If you don't know how, or simply can't be bothered to check, then you shouldn't be administering a server. Also a firewall is a most basic requirement, even something simple like UFW would do.
- mikestew 10y agoMicrosoft took a rash of shit some time ago (15 years?) for shipping MS Proxy Server with every port open by default. From the POV of employee-at-the-time, it took them a disappointingly long time for them to not do that anymore. Since then, I've learned to not assume that products are secure-by-default. At the same time, I kind of thought we learned our lesson and cut that shit out low these many years later. Add a line to a text config file that's probably buried eight directories down in a hierarchy that's owned by root? (I'm just hyperbolically guessing for effect; I generally avoid Mongo.) Do it, or you're hacked? And it's been this way for years? Come on.
- nailer 10y agoExchange 5.0 was an open mail relay and not possible to lock down - you actually had to pay for Exchange 5.5 in order to get relaying control. But that's bad old NT4-era Microsoft. Not 201x MongoDB.
- onion2k 10y agoDo it, or you're hacked? And it's been this way for years? Come on. It's not though. And it never really has been. A simple approach to security is to only expose the absolute minimum to the internet - you close everything and then open one thing at a time until your service works. Had those 30,000 MongoDB instances been sitting on IP addresses behind routers that only allowed local traffic, or specific IP addresses, then they would have been a lot harder to steal data from. The fact that they were willing to give up data to an unauthorised user would still be a problem, but it'd be a local network problem rather than something anyone with a port scanner could do. This is not a case of discovering some arcane setting you have to change to make sure your data is secure. This hack was a case of people putting services on open connections without bothering to follow even basic practises. It's bad that MongoDB gave up data without auth details, but your infrastructure can be designed to block an attacker even when they have security details so the fact that MongoDB had this problem is never going to be the whole story.
- debacle 10y agoMicrosoft is a different beast. They ship things based on the principal of least surprise. There are incredible features in newer versions of SQL Server that aren't turned on by default when upgrading, even though no one would want to not have them on.
- deleted 10y ago[deleted]
- jstoja 10y agoI stumbled upon this (awesome) article that made me realize tons of things: http://www.ranum.com/security/computer_security/editorials/dumb/index.html http://www.ranum.com/security/computer_security/editorials/d... I'm not a security expert (far from it) but I hope that I understand enough the importance of security to learn a bit about it and implement it as much as I can. Secure defaults is now maybe the first concept I'm trying to explain to people in my company.
- deleted 10y ago[deleted]
- nailer 10y agoRelevant laptop sticker: https://twitter.com/ag_dubs/status/603273801783234560 https://twitter.com/ag_dubs/status/603273801783234560 Source code to print your own: https://github.com/mikemaccana/stickers https://github.com/mikemaccana/stickers
- mark_l_watson 10y agoNot defaulting to 'localhost' (or 127.0.0.1) is pretty bad in general.
- orblivion 10y agoI don't mean to defend Mongo here, but as a counter argument to the broad principle: Ubuntu doesn't ship with iptables blocking all incoming connections by default. Should it?
- user5994461 10y agoWell, if you're on a laptop, Linux may ship with no Wifi card support and thus no internet whatsoever. Guess that's even more secure!
- rodgerd 10y agoYes.
- peterwwillis 10y agoThe modern consensus is that it should ask you what class of network you're connected to and configure it accordingly, as more permissive rules are better for home networks and the opposite for public networks. But also, if Ubuntu didn't run network services by default, there would be no need to secure them with iptables. So defaults affect defaults.
- kalleboo 10y agoHonestly, in this day and age, it probably should. As well as whitelisting software that is allowed to make outgoing connections. At least for server installations. Network security is still a complete joke.
- hashkb 10y agoPostgres' defaults barely let you connect to the DB. You don't hear stories like this about Postgres.
- cpuguy83 10y agoWell, sure but then people just open the flood gates because it's such a pain in the ass to change postgres access policies. I guess secure by default means there aren't likely to be massive hacks like this, but it does not mean people are running securely.
- Kudos 10y agoIs it just that no one's running exploit scanners for Postgres then?
- dspillett 10y agoBut back when I was starting out you did get a hell of a lot of people choosing mysql over postgres partly based on the fact that the latter was easier to connect to out of the box, and/or filling the support channels with connection questions that are already answered in an installation guide in the documentation. Convenience "sells" even at the expense of security (and in this case, actually working properly at all!) and if you don't make it as easy as possible be prepared to support people who refuse to read the instructions.
- kosma 10y agopg_hba.conf is one of the reasons it took me so long to learn Postgres. Of course, now that I know it, I don't look back - but making the first step in learning a database "understand this weird model of AF_UNIX username == dbname trust" delayed the learning process by years.
- anarazel 10y agoFWIW, I'm a postgresql committer these days, and pg_hba.conf was probably the bit I was uncomfortable with for the longest (user level, not code level).
- tormeh 10y agoWho, when wanting to write a secure piece of software, thinks: "I know! I'll use JavaScript!"? Security was always going to be an afterthought at best. There are legitimate reasons for writing things in JavaScript, and none of them apply to a database.
- golergka 10y agoWhat does this have to do with Javascript?
- tormeh 10y agoMongoDB is written in Javascript.
- grzm 10y agoIn part. According to the Github repo, it's 75% C++, 18% JS. https://github.com/mongodb/mongo https://github.com/mongodb/mongo
- aioprisan 10y agoThe JS is from docs and examples, not from core MongoDB code.
- grzm 10y agoThanks for the further clarification!
- tormeh 10y agoSo https://en.wikipedia.org/wiki/MongoDB https://en.wikipedia.org/wiki/MongoDB is wrong? Well, guess I trusted Wikipedia a bit too much.
- aioprisan 10y agoI'm confused, what about that page are you claiming to be right or wrong? The code is here for all to see: https://github.com/mongodb/mongo/tree/master/src/mongo https://github.com/mongodb/mongo/tree/master/src/mongo
- bjt2n3904 10y agoIf I were a mongo dev, I wouldn't want to write authentication and have to manage crypto. Default bind to localhost. If you want to handle external connections, the network layer provides your security.
- dhatch387 10y agoThe v2.6 open access issue in MongoDB is no new revelation, and has been rehashed over and over again. It's worth noting that this post comes from a company that profits from convincing its customers that the products they are using are insecure.
- koolba 10y agoWhile we should hope that server software is secure by default, you shouldn't rely on it either. A good rule of thumb is: Distrust until verified If you're running any servers yourself, there's no reason to open up network access to the world. At most the specific ports you want to expose, to the specific IP ranges you want to allow, should be white listed. A default of everything to everywhere is insane. Even with SSH key based authentication (v.s. say passwords) I'd consider it inept to expose all your infrastructure publicly. Set up a VPC or whatever the local equivalent is for your hosting environment and proxy SSH access via a bastion host. If you whitelist the inbound addresses to the bastion host (let's say to restrict it to just your office) then you've also eliminated the vast majority of auth log spam too. Unfortunately the people that would understand and implement things like this are the same set of people that wouldn't have publicly exposed MongoDB instances in the first place.
- mkagenius 10y agoWhile we are at this, this checklist is worth looking at: https://github.com/FallibleInc/security-guide-for-developers/blob/master/security-checklist.md https://github.com/FallibleInc/security-guide-for-developers...
- andreyf 10y agoNo matter what your defaults are, if a sysadmin doesn't understand that some software is OK to connect to a public-internet addressable interface while other software is not, their shit is going to get hacked (or, as in this case, accessed). This is the equivalent of leaving your data in a file cabinet in your lobby and being surprised someone took it. The answer isn't to add self-locking little office locks to the file cabinets by default, it's to educate folks that this is a file cabinet that's designed to be inside the office, not in a publicly accessible place.
- jldugger 10y agoNetwork oriented software created after 2000 has zero excuse for allowing unauthenticated writes from the general internet by default.
- stouset 10y agoTell that to redis. No, seriously. Please. I've been banging that drum for years.
- andreyf 10y agoAlso, MySQL [1]. Are you sure that you're right and all these major db projects are wrong? 1. note the default value of "bind to *": http://dev.mysql.com/doc/refman/5.7/en/server-options.html#option_mysqld_bind-address http://dev.mysql.com/doc/refman/5.7/en/server-options.html#o...
- 10y ago
- guypod 10y agoIt's worth noting this isn't unique to MongoDB. The "Marked" npm package, with it's 2 million downloads, doesn't sanitize input by default. "st", another popular package, allows directory listing by default. Quite a few of those...
- mike-cardwell 10y agoWhen our sysadmin set up our Mongo cluster, he firewalled out all IPs except our production systems, turned on authentication, set things up to ensure we used SSL, and configured backups. He didn't do this because he's an amazing sysadmin. He did it because he's competent, knows how to put a service on the Internet, and RTFM. I mean. If you can connect to something and use it without having to authenticate yourself, wouldn't it naturally cross your mind to check that others can't do the same? It's just common sense.
- nevi-me 10y agoI first started using Mongo in 2013/4, when I deployed my app, first thing I did was to change the default port and add authentication, as the manual recommended. I'm an accountant who's a hobbyist developer. I knew very little about security then, but I read the manual. The insecure defaults were an issue sure, but anyone installing a piece of software in production without at least reading up on config options needs to find another job.
- danielweber 10y agoMeanwhile, the FTC sued D-Link over insecure defaults in their cameras. We need good and consistent rules about this, and "well I was giving it away for free" isn't as clear a boundary as people will think it is.
- tedunangst 10y agoThe FTC sued over insecure defaults because it said "secure" on the box.
- hackits 10y agoIP Cameras from China have a number of issues of `calling home` adhoc. Granted to say even looking at their kernel's I tend to keep them completely on their own sub-net away from the net.
- 10y ago
- Pxtl 10y agoRealistically, insecure defaults are part of the reason Mongo was adopted in the first place. Other databases are a hellscape of configuration, meanwhile Mongo is starting to get work done from the moment it's installed. Security and usability are always at odds. If you make security a pain, people will give up on security. Mongo is the ultimate manifestation of that principle. Developers were driven to Mongo by opaque and painful admin tasks.
- Florin_Andrei 10y agoYes, but there is such a thing as minimum viable security. You know, that magical place where you're one step above absolute zero.
- rosser 10y agoToo bad. If you put something on the internet, it's on you to keep it secure.
- camus2 10y ago> Security and usability are always at odds. No they are not. Start the server for the first time? ask users to create an admin account, then display a notice saying they didn't configure SSL or something else properly. Problem solved. Each time the server is started and the setup isn't secure enough, display a message. That's how you do it.
- fuzzy2 10y ago> Security and usability are always at odds. Absolutely not. How does having to log in affect usability of a system itself in any way? Security and convenience are often (but not always!) at odds.
- scott_karana 10y agoA "hellscape of configuration"? `apt-get install mysql-server` makes you enter a "root" password out of the box. `apt-get install postgresql` create a local user `postgres` with access the local database. Both bind to only localhost by default. Those sound like easy, reasonable defaults to me. (And this isn't exclusive to Debian-likes: RHEL almost certainly shares the exact same benefits)
- achillean 10y agoThe most common MongoDB versions that are being exploited with the ransomware come with secure defaults! https://www.shodan.io/report/kjVzjr3O https://www.shodan.io/report/kjVzjr3O Originally, I thought the problem would disappear once MongoDB changed the default from "0.0.0.0" to "localhost". However, that has had no impact on the number of exposed MongoDB instances on the Internet. Bad defaults are definitely a problem but in this case that isn't what's happened. Also note that the most popular location where these instances are hosted is on AWS where you explicitly have to open up the firewall.
- exogen 10y agoJust a guess, but those people probably updated their installation with an existing (insecure-by-default) configuration in place. Very little software like this will mess with an existing configuration, so that upgrading doesn't appear to break anything. That's how they'd be on a secure-by-default version but still be insecure.
- achillean 10y agoYeah, I could definitely see that happening though the overall number of exposed MongoDBs has also increased. And there are Docker images that come w/ insecure defaults ala: https://hub.docker.com/r/swcc/docker-mongodb/~/dockerfile/ https://hub.docker.com/r/swcc/docker-mongodb/~/dockerfile/ One of my concerns is that people are piling onto MongoDB because "it's web scale" and ignoring potentially systemic security issues that aren't specific to MongoDB. The same issue exists for Riak, Cassandra, Memcache etc.: https://blog.shodan.io/memory-as-a-service/ https://blog.shodan.io/memory-as-a-service/
- gitignore0 10y agoI have recently been working with mongodb wanted to start the service with auth enabled with defined users and security. Here is the way I did. https://medium.com/gitignore/automatically-start-mongodb-in-authentication-mode-using-predefined-admin-users-48dca5d1f075 https://medium.com/gitignore/automatically-start-mongodb-in-...
- peterwwillis 10y agoDefaults are stupid and dangerous. First off, there's the constant problem of default passwords. The big IoT compromises lately were due to default passwords, and ever since they became popular they've been a bane to security everywhere. Wifi APs used to use defaults, but luckily most come with a randomly assigned password now (except those provided by an ISP, occasionally). All software that matters should not use default passwords. Second, a "default" is not intended to be an operational setting. When developing software, a default is literally a placeholder setting so that your software won't immediately crash if a configuration value is missing. Some software is effectively meaningless without a configuration, but if they are obscenely permissive they can allow a user to still take advantage of it, which is how MongoDB is set up. Finally, there is effectively no such thing as a secure default, because something isn't really secure until it's been vetted through a process. Pretending your software is secure when you have literally not given it a second thought is just setting you up for eventual disaster. People should be actively thinking about their security if they care about it; defaults just leave them blissfully ignorant. In my ideal world we would require software to no longer have defaults. Demand that users be aware of the configuration and running of their software so they understand what's involved with how it runs. This would teach them more about the software they're using, which would enable them to take better advantage of it, as well as make it more secure. If you don't have time to do that you probably shouldn't be using it.
- unquietcode 10y agoThe MongoDB hack and the importance of hiring competent people.
- sfilargi 10y agoI wouldn't blame MongoDB. Any machine connected to the Internet should start with a configuration equivalent to: ufw default deny incoming ufw allow ssh ufw enable And then start punching very specific holes as needed.
- ggregoire 10y agoA bit off-topic but related: how secure are the defaults of an AWS RDS MySQL (created by a BeanStalk environment, if it matters)?
- BillFinchDba 10y agoIt's pretty simple folks, RTFM. If you are running MonogoDB on your laptop to build an MVP, sure you can run it unsecured no worries. When you go to production, you go secure. My hope is that no DBA worth his under-appreciated skills would drop a totally open DB on the public internet. That said, it obviously happens... Read the directions, have a plan, and follow the checklist: https://docs.mongodb.com/manual/administration/security-checklist/?jmp=community-hub https://docs.mongodb.com/manual/administration/security-chec... Everyone has a different security model. In my case all my DB servers live behind and API layer on the internal network, and the DMZ web layer talks to the API. That makes keeping things secure MUCH easier...
- Pxtl 10y agoTo play devil's advocate for a second, isn't Mongo following the unix philosophy here? Small, composable tools instead of bolting on redundant features to everything? Opening/closing ports is the firewall's job. Mongo's job is storing and retrieving data (now, how poorly it does at that is another conversation). Having both Mongo and the Firewall handle ports means you do everything twice, which violates OnceAndOnlyOnce.
- StavrosK 10y agoYes, it is. Apparently there's a time and a place to be following the Unix philosophy, and this is not that.
- scott_karana 10y agoIf that's the case, it should be behind something like inetd, instead of being able to daemonize itself ;)
- gator-io 10y agoI had a situation where their cloud backup product was not backing up data and not issuing alerts when it wasn't working. I gave them full access to logs, etc. and the support response was basically that they couldn't determine why it wasn't working or alerting, but it it works for them, so I should migrate to their hosted solution. I sometimes think of the potential company-ending cluster-fudge that could have caused, while I stare wide-eyed at the ceiling at night.
- virtuabhi 10y agoI just had to deal with an insecure default in our Hadoop cluster. Hadoop, by default, enables WebHDFS, which provides read+write access to HDFS over HTTP. The service runs on name node using the same port which displays the Hadoop Jobs Web UI. Now, it is very easy to not know anything about WebHDFS and every day authenticate/login to server, submit jobs, and track the status on WebUI (which is public). But then anyone can send a POST request pretending to be any user and Hadoop will happily execute the command (including deleting all files). Maybe Hadoop developers believe that all Hadoop installations are behind a firewall (ours was, but an exception was made for Jobs Web UI as it seemed to be a read-only status page for running jobs) and/or have Kerberos integrated with Hadoop for authentication. But that being said, enabling HDFS access over HTTP by default seems to a debatable choice. Also, if you have a small Hadoop cluster running, do set "dfs.webhdfs.enabled" property to False in hdfs-site.xml.
- ns8sl 10y agoSpeaking of defaults, if you install Mongo on EC2 using a default instance of Ubuntu using the default engine, Wired Tiger, you will get ceaseless server crashes. Wired Tiger is NOT STABLE on the default volume - ext4. You have to use xfs. This was only recently added as a startup check.
- balgan 10y agoThis isnt a mongodb problem. redis has already been targeted and memcached and elastic will. e next. for more info: ise.binaryedge.io or http://blog.binaryedge.io/2015/08/10/data-technologies-and-security-part-1/ http://blog.binaryedge.io/2015/08/10/data-technologies-and-s...
- satokano 10y agoI found another piece of useful information from mlab, so I will share it. http://blog.mlab.com/2016/07/mongodb-tips-tricks-collection-level-access-control/ http://blog.mlab.com/2016/07/mongodb-tips-tricks-collection-...
- nirajadhikary 10y agoThe news is too late for me. Two of our development servers are already compromised. Anyway I added those servers under NAT so those servers are inaccessible from internet. Added admin access credentials also. The news should have been spread without any delay.
- nuha 10y agoEducation, experience, diligence. I'd expect that of anyone basing their livelihood on delivering production systems. Secure out of the box is nice, except that when the boxing is done by others, and delivered to under-informed, unexperienced people. It's too easy to npm yourself a "full stack" or just pick a Docker container with full stack on it, then place it on public inter-web... there are dangers to be had, and others can save you. It's up to you. And yeah, the click-bait titles including the word "hack" are laughable.
- jlaustill 10y agoThis was 100% preventable, MongoDB comes with great resources for securing your data. I pity whomever puts ANY database on the internet without taking proper security measures. Here's some documentation to get you started. https://www.mongodb.com/blog/post/how-to-avoid-a-malicious-attack-that-ransoms-your-data?jmp=community-hub https://www.mongodb.com/blog/post/how-to-avoid-a-malicious-a... https://docs.mongodb.com/manual/administration/security-checklist/?jmp=community-hub https://docs.mongodb.com/manual/administration/security-chec... https://www.mongodb.com/collateral/mongodb-security-architecture?jmp=community-hub https://www.mongodb.com/collateral/mongodb-security-architec... This really should have been titled "the importance of good DBA's..."
- beebeebush 10y agoits amazing how most people mock mongodb in forums. I would point fingers to sysadmins and developers. Is anybody taking really care how your business services are really running ? This tell a lot of current ways of companies, where nobody care/or have time/don't bother to check whole process through.