14 ms·

When our sysadmin set up our Mongo cluster, he firewalled out all IPs except our production systems, turned on authentication, set things up to ensure we used S

When our sysadmin set up our Mongo cluster, he firewalled out all IPs except our production systems, turned on authentication, set things up to ensure we used SSL, and configured backups. He didn't do this because he's an amazing sysadmin. He did it because he's competent, knows how to put a service on the Internet, and RTFM. I mean. If you can connect to something and use it without having to authenticate yourself, wouldn't it naturally cross your mind to check that others can't do the same? It's just common sense.

nevi-me 10y ago
I first started using Mongo in 2013/4, when I deployed my app, first thing I did was to change the default port and add authentication, as the manual recommended. I'm an accountant who's a hobbyist developer. I knew very little about security then, but I read the manual. The insecure defaults were an issue sure, but anyone installing a piece of software in production without at least reading up on config options needs to find another job.
- danielweber 10y ago
  Meanwhile, the FTC sued D-Link over insecure defaults in their cameras. We need good and consistent rules about this, and "well I was giving it away for free" isn't as clear a boundary as people will think it is.
  - tedunangst 10y ago
    The FTC sued over insecure defaults because it said "secure" on the box.
    - hackits 10y ago
      IP Cameras from China have a number of issues of `calling home` adhoc. Granted to say even looking at their kernel's I tend to keep them completely on their own sub-net away from the net.
    - lenkite 10y ago
      So MongoDB should state 'insecure out of the box' on their home page ? Just kidding sorry.
      - chopin 10y ago
        I don't think it's far fetched. I'd prefer a disclaimer "For the convenience of easy testing the defaults are NOT secure. If you intend to use it productively read the manual about secure configuration".
  - nevi-me 10y ago
    I'll comment without knowledge about the D-Link issue that you mention. I would imagine that if someone were to sue MongoDB Inc. today about the issue in the article, their first defence would be clear documentation that explained/recommended production guidelines. I don't know if D-Link had similar, but then again legal systems sometimes produce weird results that don't seem rational. I agree that we need good and consistent rules, in addition to that, we need DevOps/SysAdmins/SRE's that are responsible enough to know what they are doing. Carefully read documentation instead of "quickly deploy" only to come back a year later writing soppy "don't use MongoDB or XYZ because we didn't read the manual". :) MongoDB is on 3.4 now, so I would rhetorically wonder why some people/companies are still on <=2.6. If the data that is being ransomed is that important, it'll be a good lesson to those DB maintainers to upgrade and secure their stack.
  - benchaney 10y ago
    There is a huge difference between consumer facing and developer facing products when it comes to needing security out of the box. This isn't even close to a real issue.
- deleted 10y ago
  [deleted]
peterwwillis 10y ago
The article's argument is that users have no common sense and don't think about what they're doing, so we should reinforce this same behavior.
- thfuran 10y ago
  Never trust the users.
- ajross 10y ago
  Lots of these companies were storing data on behalf of other users who weren't necessarily aware of the level of "common sense" practiced by their suppliers. Whose behavior, exactly, are you trying to avoid "reinforcing" here? Is a world where no one trusts startups because they don't "think about what they're doing" one you really want to live in? Can I just say I'm horrified at the amount of glibertarian nutjobery in this thread? People are seriously arguing against the most obvious of security practices (seriously: don't open unauthenticated ports by default) because they appear to want the dummies (and their customers) to be punished. Welcome to Trump's world, I guess.
- acobster 10y ago
  > you may decide that not binding a database to local connections would be a reasonable default...But assumptions like that are dangerous because when not if someone does something that breaks that assumption, they’re now vulnerable to attacks—and they may not even know it. Understanding of the potential security risks presented by these defaults is not exactly mainstream knowledge. I wouldn't call this reinforcing behavior...I'd call it protecting your users. Common sense is something that comes with experience, and MongoDB is popular enough that it's likely to be in many web devs' first projects in production. Is it their fault they haven't developed this "common sense"? Yes and no. On the other hand, is vulnerable-by-default ever a good idea?
camus2 10y ago
Well, you have a sysadmin, that's the point. Developers aren't sysadmin. Should they have a deep knowledge of network security? you bet they should. Now I'm sorry, but MongoDB inc IS at fault as well, for not forcing developers to create credentials upon installation from the beginning. Any vendor that doesn't do that with its product isn't serious about security. Let's take Wordpress, imagine it didn't force its users to create an admin account upon deployment, everybody would be mad about it. But somehow MongoDB got a pass all this years? bullshit. I hope this hack will permanently damage MongoDB brand.
tetha 10y ago
Thoughts like this are, and will forever be, the one dividing thing between a more ops-y person versus a more dev-y person. But that's fine. At our place, our mysql cookbook is maturing. This mysql cookbook makes it really easy to say "hey, we need a mysql master with these 2 databases". The only security overhead consist of generating 4 passwords in that case (and we're planning to automate that). Once you've done that, you get a mysql master with passwords, restricted root access, a local firewall, backups and all kinds of good practices. It's secure because our cookbooks are good, and people use our cookbooks because they are easier than handling mysql directly. And that's a kind of cooperation I'm really happy about. Devs want services, and we provide easy ways to setup services in decently secure ways.
- LinuxBender 10y ago
  ops-y + dev-y, isn't that supposed to evolve into DevOps? I know the term is a bit overloaded. Do the community chef, ansible and puppet cookbooks use secure defaults for MongoDB? Asking because I have never used MongoDB.
  - tetha 10y ago
    > ops-y + dev-y, isn't that supposed to evolve into DevOps? I know the term is a bit overloaded. Yes, but I've grown to dislike that term, though it's my current job title. There's a number of people who're titled with DevOps and they're yelling about Docker this and CD that, and AWS/RDS those. That crowd is way to excited about some tools and some solutions in specific use cases, and they sometimes tend to drown out the real value of config management and close cooperation of devs, ops and other involved people with their own specific focus - for a very large amount of use cases. > Do the community chef, ansible and puppet cookbooks use secure defaults for MongoDB? Asking because I have never used MongoDB. I'm a chef guy. First google result for "mongodb cookbook" doesn't set authentication, but makes it easy to enable required authentication. Second cookbook result doesn't enable authorization by default. This makes sense though. If a community cookbook manages mongodb, that cookbook is supposed to support all use cases of mongodb and it usually tries to mimic the default use case of the application in question. To maintain that, the chef cookbooks for mongodb don't enforce authorization. However, if I was supposed to implement a mongodb cookbook to use in my place, I'd intentionally fail the chef run if authentication is disabled and stop mongodb in that case. This would be trivial in both cookbooks I looked at.
ajross 10y ago
That's the same logic that says people who get fired from their jobs should be homeless and starve, or that those who get sick without insurance should be denied care. We don't do that in our society. There's a moral distinction between culpability and impact. There are profoundly stupid things that people do, yet still need protection from. Those DB admins were incompetent by lots of measures, but their data still has value and its seizure is a public harm. It's the job of the rest of us (in this case, MongoDB's developers) to take reasonable steps to minimize the chance of that happening. Secure defaults are a very reasonable precaution. MongoDB fucked up.
- deleted 10y ago
  [deleted]
- theGimp 10y ago
  I'm no Mongo fan, but that's a false equivalence. You don't need to use Mongo in order to survive.
  - ajross 10y ago
    And having your data stolen is preferable to starving. The point is that, morally: X is bad X being all Y's fault doesn't imply that Y should be unprotected from the consequences of X. We help each other out in this society. So in this case if you're a database developer with a good handle on deployment security, you don't put a insecure-by-default product in the hands of people who aren't. I genuinely can't understand why people are arguing to the contrary.
    - woliveirajr 10y ago
      Cars can be dangerous and everybody should read the manual before using one, but it doesn't mean they are sold in an unsafe state where the user has to configure something first, otherwise it'll kill everybody. Even knifes are sold with some package that prevents them from cutting before the package is removed. I agree, I don't think your job is done just because you wrote somewhere "pay attention to this".
pmlnr 10y ago
sysadmin - that title is becoming rare. Would you expect the same competancy from a devops, or a site reliability engineer? ( Not trolling, real question; as a sysadmin myself I do see that priorities differ in many cases when a dev or a manager wants to see a service running and you have until yesterday. )
- Rapzid 10y ago
  I'm not sure the way titles are being thrown around in this thread is entirely helpful. opsy vs devsy, sysadmin vs devops? What exactly is a "devops"? I would expect anyone responsible for something to have competency in it or find someone who is to help. Titles are irrelevant.
- edgan 10y ago
  It is becoming rare, because management who ultimately hires and fires people, wants things done faster. They think that making devs responsible for ops stuff, combined with automation and new technology will magically make it all good. I am a former Sr. Systems Administrator, and Sr. Systems Engineer. My current title is Sr. Devops Engineer.
  - jsmthrowaway 10y ago
    Sr. Site Reliability Engineer here. (Itself a complex title, varying by employer.) I have taken to identifying myself as an Operations Engineer for this very reason. I'm an engineer focused on operations. Tada. You want some operating and I do some engineering to get there, and all of that is clearly printed on my tin. Some might disagree with my reduction, but I feel that it fits well what I do. Pretty much everybody who knows their way around a compiler and administers systems could probably take this title and be fine, and then you're not suffering under the yoke of devops or whatever. Because when I hear "developer," I can't help but think a feature engineer on the product side for some reason. Same with that word creeping into "devops," and that leads to things like "full stack" engineering -- creating the impression that only a select few work with or consider "the whole stack," which is a scary subliminal message. Even if you're a developer simply changing the colors on the Facebook feed or something, you should always be thinking about the full scope of things and how your changes will be operated. (Your operations nerds will love you if you think about us ahead of time.) As an aside, systems administrator with no software skills is a totally and completely fine career, with a number of very smart people (and friends, still proudly announcing every year of uptime in texts), and in a lot of these discussions I see them denigrated. Just remember that, not saying anyone is doing it.
rodgerd 10y ago
> When our sysadmin Yeah, well, we're in the middle of a trend to act as though dedicated systems people are bad, pains in the arse, you shouldn't need or want them.
- lacampbell 10y ago
  Wait, then who is supposed to administer our systems? Do people think developers are up to the task!?
  - jlgaddis 10y ago
    What administration? If something happens to the system, just toss it out and spin up a new one. Come on, man, this is 2017. Get with the program. /s
    - lacampbell 10y ago
      Oh god. That's a crazy thought process. People use VPSs more than physical servers these days. But someone still has to spin them up and configure them and deploy stuff to them. Seems insane for that to be a job for devs.
  - SoftwareCSpace 10y ago
    It depends on the developers mate, properly experienced ones are often up to the task, while the latest fad chasing buzzword inducing cool kids probably aren't .. it's no coincidence they are the demographic that put Mongo on the map :)
    - lacampbell 10y ago
      I'm a developer. I can do some sys admin stuff at a pinch. But it's not my area of expertise at all. I mean you might as well say since experienced sysadmins do a lot of scripting they might as well develop as well.
z3t4 10y ago
Next time you cross a bridge, make sure it will handle your weight ... And think about everything that can possible go wrong, what materials are used, and how old it is, etc.
- cookiecaper 10y ago
  Different. This isn't some deep, underlying fault that requires a lot of specialized knowledge to recognize, like a bridge that has the wrong type of struts or whatever. If the program doesn't make you put in a password ... it probably doesn't require a password. Refusing to deploy a process into production without setting a password is more like refusing to drive over a bridge that's obviously missing chunks and crumbling -- that is, it should be obvious, even to the untrained eye, that there is a massive structural issue that makes it unsafe. I will grant that some junior people may assume that local connections are coming in over a trusted socket and that TCP connections are required to hit an authentication process, but even this is kind of a stretch, and no one should be leaving production deployments for real companies in the hands of someone that inexperienced. I will also grant that guilt-by-association is probably not appropriate here. There are a lot of good people at companies that have grossly incompetent and/or disorganized managerial superstructures that may have caused something like this to end up exposed to the internet. I have personally worked at multiple places where the daily comedy of errors we called "work" could've resulted in an unpassworded Mongo ending up on the public network. Any way you slice it, driving over what is a clearly destroyed, ruined bridge is someone's fault, somewhere (and per regular management practice, the person least responsible for the fault will probably take the fall :) ). Let's just not pretend that this is an ordinary complication or oversight in the course of sysadmin.
  - z3t4 10y ago
    The problem is that once you work with something every day for years it becomes common sense. You should always assume your users have zero experience and don't RTFM. Messing with configurations does require a lot of specialized knowledge. Not everyone has the knowledge of a Sysadmin.
    - 10y ago
twelvedogs 10y ago
this is why we have botnets running on security cameras. you can't just demand the user be smarter, you put sane defaults on there so they actually have to go to at least a small amount of effort to shoot themselves in the foot.
gregmac 10y ago
> configured backups. This part seems to be glossed over but is a HUGE issue. It sounds like several companies have tried to pay the ransom with varying levels of success [0] ... why are they not just restoring from backup? I can only assume they don't have backups. (!) What is their DR plan if the server dies? Or someone accidentally pushes code that messes up the contents of the DB? Or someone tries to drop the development database but oops: they didn't notice they were connected to the production server? Even if you're using a hosted service, what if they go down? Get hacked? Lock you out because of a billing dispute/TOS violation/DMCA takedown/accident? Hired bad sysadmins that didn't do backups (correctly)? Not having backups of your data is inexcusable and just reeks of utter incompetence, and has nothing to do with configuration defaults or documentation. [0] https://krebsonsecurity.com/2017/01/extortionists-wipe-thousands-of-databases-victims-who-pay-up-get-stiffed/ https://krebsonsecurity.com/2017/01/extortionists-wipe-thous...
- btown 10y ago
  For small data volumes, MongoDB Inc. will actually back things up for you for $2.50/GB/mo storage (free bandwidth), and it's a live backup with multiple historical versions (an agent on your database machine tails the oplog, so you're at most a couple seconds out of date). And their free monitoring tier is great alongside it. I'm truly surprised that they don't sell it harder; it took me years of using Mongo before I knew this service existed. https://www.mongodb.com/cloud/cloud-manager https://www.mongodb.com/cloud/cloud-manager
  - gregmac 10y ago
    If you're also hosting your data with them though, it's not a complete backup solution. You're still subject to essentially a single point of failure: if they are hacked, or suspend your account, get a DMCA notice against you, or are shut down by the SEC[1], both your live database and backups are subject to loss. You still need to do external backups. You may have a lot of trust in the provider and these less frequently, but you should still do them. [1] Had this happen to me once in the early 2000's: company I worked for had a dedicated server at a colo facility. After several days of them not responding to phone/email/etc, their answering machine was changed to a message saying the SEC had seized all assets and had all the owners 'under investigation' or something like that. We had external backups, but immediately took the latest stuff and got everything migrated to a new system in a new facility. Server stayed up for a few weeks after that, but then suddenly their whole IP space went offline. We never did get our server back.
jlgaddis 10y ago
In the new, DevOps-centered world, doing all that is unnecessary and only serves to slow down the launching of $product. Besides, you should be using containers for everything anyways. If something happens to it, just throw it out and spin up a new one. /s
developer2 10y ago
These kinds of articles about "attacks" are annoying. Let's be absolutely clear: there is no "attack" or "hack" involved. No real company was affected by this. There is no operating system bug involved. No software vulnerability. No zero-day exploit. No social engineering angle. There's no real problem. This is simply first-time developers with no server experience, deploying their first project, who don't understand the first thing about having a server on the internet. Publicly exposing an open port for software that is innately designed to run on a private network has nothing to do with... whatever you want to call all these "exposé" pieces about "attacks". You could write an article to explain the introductory concepts of system administration to newcomers to the industry. But to even mention the word "attack" when the only thing involved is an open port is... sigh. I feel for newcomers who need to learn. I really do; the amount of information one needs to absorb to be even remotely competent is vast, and takes at least a few years to pick up (and then another decade or two to fine tune that knowledge). But this is not a situation where the maintainers of software packages are to blame for not educating their users, or insinuating that their products are not "secure out of the box". It has nothing to do with individual software packages, and everything to do with the very core aspects of having a computer connected to a public network.
- raesene6 10y ago
  I'm afraid your suggestion that "No real company was affected by this" doesn't bear out with the evidence that I've seen or the experiences I've had in security. Real companies with real data make some pretty elementary mistakes with regards to security. I'm a security tester and the number of times I've got access to systems deployed by real companies who've really paid money for an external security reviewer using things like default creds is quite high. It's tempting to think that this is just an education issue and that once people know how to do security well things will get better but personally, my opinion after 16 years in security is that this isn't the case. Effort spent on security is a trade-off with other things and in many cases people make the choice (either unconsciously or deliberately) not to prioritise it.

nevi-me 10y ago

I first started using Mongo in 2013/4, when I deployed my app, first thing I did was to change the default port and add authentication, as the manual recommended. I'm an accountant who's a hobbyist developer. I knew very little about security then, but I read the manual. The insecure defaults were an issue sure, but anyone installing a piece of software in production without at least reading up on config options needs to find another job.

danielweber 10y ago
Meanwhile, the FTC sued D-Link over insecure defaults in their cameras. We need good and consistent rules about this, and "well I was giving it away for free" isn't as clear a boundary as people will think it is.
- tedunangst 10y ago
  The FTC sued over insecure defaults because it said "secure" on the box.
  - hackits 10y ago
    IP Cameras from China have a number of issues of `calling home` adhoc. Granted to say even looking at their kernel's I tend to keep them completely on their own sub-net away from the net.
  - lenkite 10y ago
    So MongoDB should state 'insecure out of the box' on their home page ? Just kidding sorry.
    - chopin 10y ago
      I don't think it's far fetched. I'd prefer a disclaimer "For the convenience of easy testing the defaults are NOT secure. If you intend to use it productively read the manual about secure configuration".
- nevi-me 10y ago
  I'll comment without knowledge about the D-Link issue that you mention. I would imagine that if someone were to sue MongoDB Inc. today about the issue in the article, their first defence would be clear documentation that explained/recommended production guidelines. I don't know if D-Link had similar, but then again legal systems sometimes produce weird results that don't seem rational. I agree that we need good and consistent rules, in addition to that, we need DevOps/SysAdmins/SRE's that are responsible enough to know what they are doing. Carefully read documentation instead of "quickly deploy" only to come back a year later writing soppy "don't use MongoDB or XYZ because we didn't read the manual". :) MongoDB is on 3.4 now, so I would rhetorically wonder why some people/companies are still on <=2.6. If the data that is being ransomed is that important, it'll be a good lesson to those DB maintainers to upgrade and secure their stack.
- benchaney 10y ago
  There is a huge difference between consumer facing and developer facing products when it comes to needing security out of the box. This isn't even close to a real issue.
deleted 10y ago
[deleted]

peterwwillis 10y ago

The article's argument is that users have no common sense and don't think about what they're doing, so we should reinforce this same behavior.

thfuran 10y ago
Never trust the users.
ajross 10y ago
Lots of these companies were storing data on behalf of other users who weren't necessarily aware of the level of "common sense" practiced by their suppliers. Whose behavior, exactly, are you trying to avoid "reinforcing" here? Is a world where no one trusts startups because they don't "think about what they're doing" one you really want to live in? Can I just say I'm horrified at the amount of glibertarian nutjobery in this thread? People are seriously arguing against the most obvious of security practices (seriously: don't open unauthenticated ports by default) because they appear to want the dummies (and their customers) to be punished. Welcome to Trump's world, I guess.
acobster 10y ago
> you may decide that not binding a database to local connections would be a reasonable default...But assumptions like that are dangerous because when not if someone does something that breaks that assumption, they’re now vulnerable to attacks—and they may not even know it. Understanding of the potential security risks presented by these defaults is not exactly mainstream knowledge. I wouldn't call this reinforcing behavior...I'd call it protecting your users. Common sense is something that comes with experience, and MongoDB is popular enough that it's likely to be in many web devs' first projects in production. Is it their fault they haven't developed this "common sense"? Yes and no. On the other hand, is vulnerable-by-default ever a good idea?

camus2 10y ago

Well, you have a sysadmin, that's the point. Developers aren't sysadmin. Should they have a deep knowledge of network security? you bet they should. Now I'm sorry, but MongoDB inc IS at fault as well, for not forcing developers to create credentials upon installation from the beginning. Any vendor that doesn't do that with its product isn't serious about security. Let's take Wordpress, imagine it didn't force its users to create an admin account upon deployment, everybody would be mad about it. But somehow MongoDB got a pass all this years? bullshit. I hope this hack will permanently damage MongoDB brand.

tetha 10y ago

Thoughts like this are, and will forever be, the one dividing thing between a more ops-y person versus a more dev-y person. But that's fine. At our place, our mysql cookbook is maturing. This mysql cookbook makes it really easy to say "hey, we need a mysql master with these 2 databases". The only security overhead consist of generating 4 passwords in that case (and we're planning to automate that). Once you've done that, you get a mysql master with passwords, restricted root access, a local firewall, backups and all kinds of good practices. It's secure because our cookbooks are good, and people use our cookbooks because they are easier than handling mysql directly. And that's a kind of cooperation I'm really happy about. Devs want services, and we provide easy ways to setup services in decently secure ways.

LinuxBender 10y ago
ops-y + dev-y, isn't that supposed to evolve into DevOps? I know the term is a bit overloaded. Do the community chef, ansible and puppet cookbooks use secure defaults for MongoDB? Asking because I have never used MongoDB.
- tetha 10y ago
  > ops-y + dev-y, isn't that supposed to evolve into DevOps? I know the term is a bit overloaded. Yes, but I've grown to dislike that term, though it's my current job title. There's a number of people who're titled with DevOps and they're yelling about Docker this and CD that, and AWS/RDS those. That crowd is way to excited about some tools and some solutions in specific use cases, and they sometimes tend to drown out the real value of config management and close cooperation of devs, ops and other involved people with their own specific focus - for a very large amount of use cases. > Do the community chef, ansible and puppet cookbooks use secure defaults for MongoDB? Asking because I have never used MongoDB. I'm a chef guy. First google result for "mongodb cookbook" doesn't set authentication, but makes it easy to enable required authentication. Second cookbook result doesn't enable authorization by default. This makes sense though. If a community cookbook manages mongodb, that cookbook is supposed to support all use cases of mongodb and it usually tries to mimic the default use case of the application in question. To maintain that, the chef cookbooks for mongodb don't enforce authorization. However, if I was supposed to implement a mongodb cookbook to use in my place, I'd intentionally fail the chef run if authentication is disabled and stop mongodb in that case. This would be trivial in both cookbooks I looked at.

ajross 10y ago

That's the same logic that says people who get fired from their jobs should be homeless and starve, or that those who get sick without insurance should be denied care. We don't do that in our society. There's a moral distinction between culpability and impact. There are profoundly stupid things that people do, yet still need protection from. Those DB admins were incompetent by lots of measures, but their data still has value and its seizure is a public harm. It's the job of the rest of us (in this case, MongoDB's developers) to take reasonable steps to minimize the chance of that happening. Secure defaults are a very reasonable precaution. MongoDB fucked up.

deleted 10y ago
[deleted]
theGimp 10y ago
I'm no Mongo fan, but that's a false equivalence. You don't need to use Mongo in order to survive.
- ajross 10y ago
  And having your data stolen is preferable to starving. The point is that, morally: X is bad X being all Y's fault doesn't imply that Y should be unprotected from the consequences of X. We help each other out in this society. So in this case if you're a database developer with a good handle on deployment security, you don't put a insecure-by-default product in the hands of people who aren't. I genuinely can't understand why people are arguing to the contrary.
  - woliveirajr 10y ago
    Cars can be dangerous and everybody should read the manual before using one, but it doesn't mean they are sold in an unsafe state where the user has to configure something first, otherwise it'll kill everybody. Even knifes are sold with some package that prevents them from cutting before the package is removed. I agree, I don't think your job is done just because you wrote somewhere "pay attention to this".

pmlnr 10y ago

sysadmin - that title is becoming rare. Would you expect the same competancy from a devops, or a site reliability engineer? ( Not trolling, real question; as a sysadmin myself I do see that priorities differ in many cases when a dev or a manager wants to see a service running and you have until yesterday. )

Rapzid 10y ago
I'm not sure the way titles are being thrown around in this thread is entirely helpful. opsy vs devsy, sysadmin vs devops? What exactly is a "devops"? I would expect anyone responsible for something to have competency in it or find someone who is to help. Titles are irrelevant.
edgan 10y ago
It is becoming rare, because management who ultimately hires and fires people, wants things done faster. They think that making devs responsible for ops stuff, combined with automation and new technology will magically make it all good. I am a former Sr. Systems Administrator, and Sr. Systems Engineer. My current title is Sr. Devops Engineer.
- jsmthrowaway 10y ago
  Sr. Site Reliability Engineer here. (Itself a complex title, varying by employer.) I have taken to identifying myself as an Operations Engineer for this very reason. I'm an engineer focused on operations. Tada. You want some operating and I do some engineering to get there, and all of that is clearly printed on my tin. Some might disagree with my reduction, but I feel that it fits well what I do. Pretty much everybody who knows their way around a compiler and administers systems could probably take this title and be fine, and then you're not suffering under the yoke of devops or whatever. Because when I hear "developer," I can't help but think a feature engineer on the product side for some reason. Same with that word creeping into "devops," and that leads to things like "full stack" engineering -- creating the impression that only a select few work with or consider "the whole stack," which is a scary subliminal message. Even if you're a developer simply changing the colors on the Facebook feed or something, you should always be thinking about the full scope of things and how your changes will be operated. (Your operations nerds will love you if you think about us ahead of time.) As an aside, systems administrator with no software skills is a totally and completely fine career, with a number of very smart people (and friends, still proudly announcing every year of uptime in texts), and in a lot of these discussions I see them denigrated. Just remember that, not saying anyone is doing it.

rodgerd 10y ago

> When our sysadmin Yeah, well, we're in the middle of a trend to act as though dedicated systems people are bad, pains in the arse, you shouldn't need or want them.

lacampbell 10y ago
Wait, then who is supposed to administer our systems? Do people think developers are up to the task!?
- jlgaddis 10y ago
  What administration? If something happens to the system, just toss it out and spin up a new one. Come on, man, this is 2017. Get with the program. /s
  - lacampbell 10y ago
    Oh god. That's a crazy thought process. People use VPSs more than physical servers these days. But someone still has to spin them up and configure them and deploy stuff to them. Seems insane for that to be a job for devs.
- SoftwareCSpace 10y ago
  It depends on the developers mate, properly experienced ones are often up to the task, while the latest fad chasing buzzword inducing cool kids probably aren't .. it's no coincidence they are the demographic that put Mongo on the map :)
  - lacampbell 10y ago
    I'm a developer. I can do some sys admin stuff at a pinch. But it's not my area of expertise at all. I mean you might as well say since experienced sysadmins do a lot of scripting they might as well develop as well.

z3t4 10y ago

Next time you cross a bridge, make sure it will handle your weight ... And think about everything that can possible go wrong, what materials are used, and how old it is, etc.

cookiecaper 10y ago
Different. This isn't some deep, underlying fault that requires a lot of specialized knowledge to recognize, like a bridge that has the wrong type of struts or whatever. If the program doesn't make you put in a password ... it probably doesn't require a password. Refusing to deploy a process into production without setting a password is more like refusing to drive over a bridge that's obviously missing chunks and crumbling -- that is, it should be obvious, even to the untrained eye, that there is a massive structural issue that makes it unsafe. I will grant that some junior people may assume that local connections are coming in over a trusted socket and that TCP connections are required to hit an authentication process, but even this is kind of a stretch, and no one should be leaving production deployments for real companies in the hands of someone that inexperienced. I will also grant that guilt-by-association is probably not appropriate here. There are a lot of good people at companies that have grossly incompetent and/or disorganized managerial superstructures that may have caused something like this to end up exposed to the internet. I have personally worked at multiple places where the daily comedy of errors we called "work" could've resulted in an unpassworded Mongo ending up on the public network. Any way you slice it, driving over what is a clearly destroyed, ruined bridge is someone's fault, somewhere (and per regular management practice, the person least responsible for the fault will probably take the fall :) ). Let's just not pretend that this is an ordinary complication or oversight in the course of sysadmin.
- z3t4 10y ago
  The problem is that once you work with something every day for years it becomes common sense. You should always assume your users have zero experience and don't RTFM. Messing with configurations does require a lot of specialized knowledge. Not everyone has the knowledge of a Sysadmin.
  - 10y ago

twelvedogs 10y ago

this is why we have botnets running on security cameras. you can't just demand the user be smarter, you put sane defaults on there so they actually have to go to at least a small amount of effort to shoot themselves in the foot.

gregmac 10y ago

> configured backups. This part seems to be glossed over but is a HUGE issue. It sounds like several companies have tried to pay the ransom with varying levels of success [0] ... why are they not just restoring from backup? I can only assume they don't have backups. (!) What is their DR plan if the server dies? Or someone accidentally pushes code that messes up the contents of the DB? Or someone tries to drop the development database but oops: they didn't notice they were connected to the production server? Even if you're using a hosted service, what if they go down? Get hacked? Lock you out because of a billing dispute/TOS violation/DMCA takedown/accident? Hired bad sysadmins that didn't do backups (correctly)? Not having backups of your data is inexcusable and just reeks of utter incompetence, and has nothing to do with configuration defaults or documentation. [0] https://krebsonsecurity.com/2017/01/extortionists-wipe-thousands-of-databases-victims-who-pay-up-get-stiffed/ https://krebsonsecurity.com/2017/01/extortionists-wipe-thous...

btown 10y ago
For small data volumes, MongoDB Inc. will actually back things up for you for $2.50/GB/mo storage (free bandwidth), and it's a live backup with multiple historical versions (an agent on your database machine tails the oplog, so you're at most a couple seconds out of date). And their free monitoring tier is great alongside it. I'm truly surprised that they don't sell it harder; it took me years of using Mongo before I knew this service existed. https://www.mongodb.com/cloud/cloud-manager https://www.mongodb.com/cloud/cloud-manager
- gregmac 10y ago
  If you're also hosting your data with them though, it's not a complete backup solution. You're still subject to essentially a single point of failure: if they are hacked, or suspend your account, get a DMCA notice against you, or are shut down by the SEC[1], both your live database and backups are subject to loss. You still need to do external backups. You may have a lot of trust in the provider and these less frequently, but you should still do them. [1] Had this happen to me once in the early 2000's: company I worked for had a dedicated server at a colo facility. After several days of them not responding to phone/email/etc, their answering machine was changed to a message saying the SEC had seized all assets and had all the owners 'under investigation' or something like that. We had external backups, but immediately took the latest stuff and got everything migrated to a new system in a new facility. Server stayed up for a few weeks after that, but then suddenly their whole IP space went offline. We never did get our server back.

jlgaddis 10y ago

In the new, DevOps-centered world, doing all that is unnecessary and only serves to slow down the launching of $product. Besides, you should be using containers for everything anyways. If something happens to it, just throw it out and spin up a new one. /s

developer2 10y ago

These kinds of articles about "attacks" are annoying. Let's be absolutely clear: there is no "attack" or "hack" involved. No real company was affected by this. There is no operating system bug involved. No software vulnerability. No zero-day exploit. No social engineering angle. There's no real problem. This is simply first-time developers with no server experience, deploying their first project, who don't understand the first thing about having a server on the internet. Publicly exposing an open port for software that is innately designed to run on a private network has nothing to do with... whatever you want to call all these "exposé" pieces about "attacks". You could write an article to explain the introductory concepts of system administration to newcomers to the industry. But to even mention the word "attack" when the only thing involved is an open port is... sigh. I feel for newcomers who need to learn. I really do; the amount of information one needs to absorb to be even remotely competent is vast, and takes at least a few years to pick up (and then another decade or two to fine tune that knowledge). But this is not a situation where the maintainers of software packages are to blame for not educating their users, or insinuating that their products are not "secure out of the box". It has nothing to do with individual software packages, and everything to do with the very core aspects of having a computer connected to a public network.

raesene6 10y ago
I'm afraid your suggestion that "No real company was affected by this" doesn't bear out with the evidence that I've seen or the experiences I've had in security. Real companies with real data make some pretty elementary mistakes with regards to security. I'm a security tester and the number of times I've got access to systems deployed by real companies who've really paid money for an external security reviewer using things like default creds is quite high. It's tempting to think that this is just an education issue and that once people know how to do security well things will get better but personally, my opinion after 16 years in security is that this isn't the case. Effort spent on security is a trade-off with other things and in many cases people make the choice (either unconsciously or deliberately) not to prioritise it.