9 ms·

If the databases in question (Elastic, MongoDB, others) make it too easy to set up unsecured access, possibly because they default to an unsecured state on inst

If the databases in question (Elastic, MongoDB, others) make it too easy to set up unsecured access, possibly because they default to an unsecured state on installation, then some good may come of this: The reputation hit to the database vendors should encourage them to mend their ways. If that happens, then the attack can arguably be justified despite the damage — consider all the future database installations which wouldn't otherwise have been secured which are now spared from not only destruction but theft.

lol768 6y ago
I've said it on here before, but the way in which Elasticsearch used to lock away critical security functionality (like TLS support and RBAC) behind a paid subscription whilst making just enough functionality available for free such that users could shoot their foot off is disgusting. This only ever changed after Open Distro for Elasticsearch came onto the scene and forced Elastic's hand. I entirely agree the vendors are (partially) to blame here.
- bmcahren 6y ago
  This is why we are refactoring our database to be able to migrate to Amazon documentdb from MongoDB. Encryption at rest.... Pay up!
  - tyre 6y ago
    Curious, why do you use Mongo? Does it give you something that a JSONB column in Postgres wouldn’t?
    - scandox 6y ago
      Maybe they have 10 years of code sitting on it? It's what a colleague calls sound historical reasons.
      - deleted 6y ago
        [deleted]
    - BossingAround 6y ago
      Surprisingly, I feel Mongo has been kind of surging back into developers' minds. I thought Mongo was utterly dead, and I also don't know why would one use Mongo instead of JSONB. The last I heard (3-4 years ago?), there were some fundamental problems with Mongo. That said, I've incidentally heard a lot about Mongo in the last half a year. Might be my bubble. Might be MongoDB actually maturing and getting really good. I hope it's the latter.
      - lallysingh 6y ago
        I mostly hear complaints about email spam from their salespeople.
    - StriverGuy 6y ago
      If you have ever used Mongo Atlas, you would understand why people use mongo. The administration ease that the platform provides is unmatched by any other DBaaS I am aware of.
    - cpursley 6y ago
      Yes, hipster cred. For everyone else, JUP ("Just Use Postgres") Snark aside, Postgres should hold you over for quite some time. With Postgres you probably don't need Redis, Elastic Search, Mongo or Kafka.
    - hamandcheese 6y ago
      I use jsonb heavily. While it is amazing, I definitely wouldn’t rely on it as a general purpose replacement for NoSQL/schemaless data storage. An example of an issue I am dealing with currently: while you can create a gin index to speed up containment queries, Postgres doesn’t keep any statistics about jsonb columns. This means the query planner will sometimes do stupid things, like using the index even for very non-selective overlap conditions, which is a lot slower than just doing a sequential scan. Less of an issue for me but worth considering: the size of the gin index in my use case seems to be about 5x bigger than the size of the unindexed data. I was surprised by the size increase. I only use the containment operator so I could make a smaller/faster index using the jsonb_path_ops operator class. This is on my todo list :) Like all non-btree indexes in Postgres, the index is unordered. That means sorting by values in the jsonb column will always be slow. This doesn’t matter for selective queries, but exacerbates my already slow non-selective queries that return large result sets. That said, if your queries are selective, jsonb + gin indexes are surprisingly performant (in the 0.5-10ms range for small result sets). My use case is a mix of structured relational data with jsonb for user-defined values (which of course they want to use for querying/sorting and I was dumb enough to say “sure, why not?”) In terms of the magnitude of data, there’s roughly 10 million rows. Each team using this service has the query scoped to about 500k-1 million records, and then additional filters (on the jsonb column) will scope that down to anywhere between 60k-0 results.
      - acidbaseextract 6y ago
        Thanks for the detailed response. I'm curious, is the NoSQL store a canonical store of its own data? If not, how do you replicate from postgres to the secondary store? I ask because where I work we sync postgres to a secondary store for search, but the way it's done in a piecemeal, application-specific way gives me the heebie jeebies. It almost certainly will result in that secondary store drifting. Unfortunately we can't use something like zombodb [1] as we're on amazon RDS. It seems like you know your stuff, and seeing non-deterministic consistency irritates the heck out of me! [1] https://github.com/zombodb/zombodb https://github.com/zombodb/zombodb
    - pavel_odintsov 6y ago
      Thank you for raising this question. Well, for my things I use MongoDB because of very convenient integration with programming languages (Go, C++ via Mongo C++) and zero hassle with schema But I'll be happy to replace it by something else, my load is extremely small and only single requirement is to have DB as network daemon, not as embedded storage as it will be used by 2 applications (main daemon an API). RethinkDB was really nice candidate for it but it's not alive anymore: https://rethinkdb.com/blog/rethinkdb-shutdown/ https://rethinkdb.com/blog/rethinkdb-shutdown/
    - lmm 6y ago
      A JSONB column in Postgres doesn't have: - Proper replication in its first-class, default configuration - Automatically managed cluster membership - Seamless automated failover - First-class async client libraries in many languages - A non-awful query language Focusing on the JSON is beside the point (though it is a convenience). Wake me up when you've got a properly distributed database.
      - andoriyu 6y ago
        Are you trolling? Also, wtf is properly distributed database?
  - mushi 6y ago
    You’re switching from a free database to a paid one because what?
- BossingAround 6y ago
  Well, it's just another attempt at monetizing the product. Nowadays, companies and developers expect everything to be OSS (and I love it) yet it's incredibly expensive to develop SW (and very few people do OSS just because of passion--I tried and failed miserably). Locking RBAC and TLS behind a paid subscription is a sure way to force companies with security teams to pay for it (or not to use it). This particular lesson comes relatively cheap--dropping your data is not the worst an attacker could do with it. Hopefully, more people will research what I'd call "security 101"...
  - pavel_lishin 6y ago
    > Well, it's just another attempt at monetizing the product. Don't make excuses for their shitty business practices.
    - gameswithgo 6y ago
      Don't tell people what to do.
    - cookiecaper 6y ago
      The thing is that no one cares. It's a cultural issue. If Elastic makes it "harder" to get off the starting line by setting secure defaults, they'll get their lunch eaten by a fork called EasyElastic that people perceive as "easier". This probably isn't an excuse for Elastic anymore, but it's how Elastic was born ("easier than Solr!" -- they didn't invent full-text search, after all) and it's how whatever supplants Elastic will be born. Why is MySQL dominant over Postgres even though "referential integrity" didn't come to the game until v5? Why Docker over jails or OpenVZ? etc. People adopt technology because it's fashionable and it becomes fashionable in part because it's perceived as easy to use. Security and ease-of-use are not quite true opposites, but there's definitely some intrinsic tension. We have a lot of hapless practitioners in the space and the root problems here won't go away until we get some standards. Better solutions usually lose because crappy stuff focuses more effort on marketing and an "easy" onboarding process, where better stuff focuses on the operational complications of the real world.
  - 6y ago
nisa 6y ago
It's also easy to get bitten by Docker. You can secure your server with iptables/ufw only to discover that docker happily punches through your firewall and you need to filter on the DOCKER-USER chain - and even that was broken: https://unrouted.io/2017/08/15/docker-firewall/ https://unrouted.io/2017/08/15/docker-firewall/ https://github.com/docker/for-linux/issues/690 https://github.com/docker/for-linux/issues/690
- dawnerd 6y ago
  Seriously this is the most annoying thing ever, especially if someone on your team things you need to expose the ports to redis in a docker compose. I’ve come back from a weekend where my redis instance was being used for crypto mining. Anything that is insecure by default in 2020 should be killed off IMO.
  - usmannk 6y ago
    Isn’t exposing ports in your docker-compose services:redis:ports, how you’d do that? (Docker hobbyist here)
    - jcrawfordor 6y ago
      My guess would be that they went with "ports" instead of "expose" which makes it public. But even with careful composition of your docker-compose you might want to be careful, Docker can interact with iptables in surprising ways and long iptables rulesets are almost comically difficult to validate sometimes. The result is that if you're using both Docker (and even more if you use Compose, Kubernetes, some other orchestrator) and doing anything else with iptables (including using it as a host firewall) you need to take a lot of caution to make sure you don't mess anything up. These tools usually confine their changes to custom chains to make this stuff easier to sort out but at the same time it can make it more difficult to understand what's happening, particularly with the use of tagging and etc. In general I would recommend putting some kind of protection between machines running Docker (and especially orchestrators) and the internet. This could be cloud provider mechanisms (security groups, ACLs, etc), a firewall appliance, NAT gateway configuration, etc. depending on the situation. It's not necessary, but it makes the situation easier to audit/validate, and more layers of protection seldom hurt. If nothing else it means that much of the time you'll need to make a mistake in two places instead of one, in order to have an unintended exposure.
dimitar 6y ago
Apparently these guys don't have a firewall setup and haven't heard about private networks and VPNs. reads TFA again Wait, one of the victims is a VPN provider???
- TwoBit 6y ago
  I've read that there are now VPN providers that just use someone else's VPN engine. So they don't have to know wtf they are doing.
  - dehrmann 6y ago
    For a product that's 50% snakeoil and marketing, that approach seems reasonable? It's easier than actually doing the leg work.
    - _-___________-_ 6y ago
      Only 50%? I'd put it much closer to 100% tbh. Why people are so eager to funnel all their traffic through some of the shadiest businesses on the Internet is beyond me...
tdeck 6y ago
Why not just rename all the tables or something? That's enough to get the developer's attention without being so destructive.
- nkrisc 6y ago
  Because if it's not destructive they have no reason to pay attention. Change names back and it's business as usual.
  - tdeck 6y ago
    IDK, if someone kept changing the table names in my DB every week I'd probably throw a password on it, even if I were really lazy. Most of these people probably didn't realize their DBs were unsecured, and that gets the point across quickly (particularly if the new table names are chosen instructively).
    - nkrisc 6y ago
      That sounds reasonable, but you'd think most people would also be concerned about their databases being publicly accessible in the first place, yet here we are.
      - ivanhoe 6y ago
        I don't think this is the case of people not being concerned, but simply the ignorance on their part about the setup. People just presume that the defaults are safe, and never bother getting into the details.
        nkrisc 6y ago
        At the risk of arguing "no true Scotsman," someone who is concerned about security likely wouldn't make assumptions about defaults. Or rather, someone appropriately paranoid about security concerns would not trust defaults without at least reviewing them.
dilandau 6y ago
What about developers just being competent before deploying a database on the public internet? At least google "securing X" before just pumping data in.
- umvi 6y ago
  How about all professionals in every industry being competent before doing anything? Then we wouldn't have any issues in society.
  - dilandau 6y ago
    Strawman. And a weak one at that. Fuck off with conflating the minimum amount of competency needed to secure the data you needlessly collect with that kind of naivete.
    - umvi 6y ago
      Who says every unsecured db on the internet consists of "needlessly collected" data?
IAmEveryone 6y ago
That's just victim blaming. The same logic applies to every crime: "lock your doors if you don't want your TV to be stolen!". It also works at any level of security: "Lock your doors and hire guards if you don't want clever thieves breaking a window..." But if you require everyone to take adequate measures to physically secure their houses, you don't even need laws and morality! And while this may provide the sort of negative reinforcement you are hoping for, actual damages in each case are going to be all over the place. That makes this one of the worst possible punishments. Imagine the penalty for speeding is a random outcome somewhere between a stern reminder and the death penality. There would be nothing fair and little useful about that, even though it does follow the same basic principle of do bad -> be harmed.
- theevilsharpie 6y ago
  > But if you require everyone to take adequate measures to physically secure their houses, you don't even need laws and morality! If laws and morality were sufficient protection against malicious actors, I might agree with you. However, in a world where cyber vandals are often beyond any accessible jurisdiction (and may even be supported by their local authorities), laws and morality are clearly not effective at keeping unauthorized users out of private systems. As such, the responsibility for keeping private information secure naturally falls on the people running the systems. Putting up a network firewall (or at least requiring authentication) would have prevented the damage described in the OP, and is a rudimentary security measure that has been common practice for decades. The people who suffered significant damage from this attack should strongly consider outsourcing system administration to someone who knows what they're doing.
- jessaustin 6y ago
  You're not thinking of the right victim. You're considering the organizations who compiled the databases in question. If the databases are composed entirely of their own information that they created themselves, then sure they've suffered a loss. In most cases, however, these databases include privacy-sensitive personal information about the customers of the organization. Those customers have been victims of bad security practices ever since MongoDb was first installed, and this "meow" hack has ended their victimization.
exadeci 6y ago
Wasn't there a guy that thoroughly tested databases found that MongoDB was awful? I'm pretty sure many of his articles were posted here.

lol768 6y ago

I've said it on here before, but the way in which Elasticsearch used to lock away critical security functionality (like TLS support and RBAC) behind a paid subscription whilst making just enough functionality available for free such that users could shoot their foot off is disgusting. This only ever changed after Open Distro for Elasticsearch came onto the scene and forced Elastic's hand. I entirely agree the vendors are (partially) to blame here.

bmcahren 6y ago
This is why we are refactoring our database to be able to migrate to Amazon documentdb from MongoDB. Encryption at rest.... Pay up!
- tyre 6y ago
  Curious, why do you use Mongo? Does it give you something that a JSONB column in Postgres wouldn’t?
  - scandox 6y ago
    Maybe they have 10 years of code sitting on it? It's what a colleague calls sound historical reasons.
    - deleted 6y ago
      [deleted]
  - BossingAround 6y ago
    Surprisingly, I feel Mongo has been kind of surging back into developers' minds. I thought Mongo was utterly dead, and I also don't know why would one use Mongo instead of JSONB. The last I heard (3-4 years ago?), there were some fundamental problems with Mongo. That said, I've incidentally heard a lot about Mongo in the last half a year. Might be my bubble. Might be MongoDB actually maturing and getting really good. I hope it's the latter.
    - lallysingh 6y ago
      I mostly hear complaints about email spam from their salespeople.
  - StriverGuy 6y ago
    If you have ever used Mongo Atlas, you would understand why people use mongo. The administration ease that the platform provides is unmatched by any other DBaaS I am aware of.
  - cpursley 6y ago
    Yes, hipster cred. For everyone else, JUP ("Just Use Postgres") Snark aside, Postgres should hold you over for quite some time. With Postgres you probably don't need Redis, Elastic Search, Mongo or Kafka.
  - hamandcheese 6y ago
    I use jsonb heavily. While it is amazing, I definitely wouldn’t rely on it as a general purpose replacement for NoSQL/schemaless data storage. An example of an issue I am dealing with currently: while you can create a gin index to speed up containment queries, Postgres doesn’t keep any statistics about jsonb columns. This means the query planner will sometimes do stupid things, like using the index even for very non-selective overlap conditions, which is a lot slower than just doing a sequential scan. Less of an issue for me but worth considering: the size of the gin index in my use case seems to be about 5x bigger than the size of the unindexed data. I was surprised by the size increase. I only use the containment operator so I could make a smaller/faster index using the jsonb_path_ops operator class. This is on my todo list :) Like all non-btree indexes in Postgres, the index is unordered. That means sorting by values in the jsonb column will always be slow. This doesn’t matter for selective queries, but exacerbates my already slow non-selective queries that return large result sets. That said, if your queries are selective, jsonb + gin indexes are surprisingly performant (in the 0.5-10ms range for small result sets). My use case is a mix of structured relational data with jsonb for user-defined values (which of course they want to use for querying/sorting and I was dumb enough to say “sure, why not?”) In terms of the magnitude of data, there’s roughly 10 million rows. Each team using this service has the query scoped to about 500k-1 million records, and then additional filters (on the jsonb column) will scope that down to anywhere between 60k-0 results.
    - acidbaseextract 6y ago
      Thanks for the detailed response. I'm curious, is the NoSQL store a canonical store of its own data? If not, how do you replicate from postgres to the secondary store? I ask because where I work we sync postgres to a secondary store for search, but the way it's done in a piecemeal, application-specific way gives me the heebie jeebies. It almost certainly will result in that secondary store drifting. Unfortunately we can't use something like zombodb [1] as we're on amazon RDS. It seems like you know your stuff, and seeing non-deterministic consistency irritates the heck out of me! [1] https://github.com/zombodb/zombodb https://github.com/zombodb/zombodb
  - pavel_odintsov 6y ago
    Thank you for raising this question. Well, for my things I use MongoDB because of very convenient integration with programming languages (Go, C++ via Mongo C++) and zero hassle with schema But I'll be happy to replace it by something else, my load is extremely small and only single requirement is to have DB as network daemon, not as embedded storage as it will be used by 2 applications (main daemon an API). RethinkDB was really nice candidate for it but it's not alive anymore: https://rethinkdb.com/blog/rethinkdb-shutdown/ https://rethinkdb.com/blog/rethinkdb-shutdown/
  - lmm 6y ago
    A JSONB column in Postgres doesn't have: - Proper replication in its first-class, default configuration - Automatically managed cluster membership - Seamless automated failover - First-class async client libraries in many languages - A non-awful query language Focusing on the JSON is beside the point (though it is a convenience). Wake me up when you've got a properly distributed database.
    - andoriyu 6y ago
      Are you trolling? Also, wtf is properly distributed database?
- mushi 6y ago
  You’re switching from a free database to a paid one because what?
BossingAround 6y ago
Well, it's just another attempt at monetizing the product. Nowadays, companies and developers expect everything to be OSS (and I love it) yet it's incredibly expensive to develop SW (and very few people do OSS just because of passion--I tried and failed miserably). Locking RBAC and TLS behind a paid subscription is a sure way to force companies with security teams to pay for it (or not to use it). This particular lesson comes relatively cheap--dropping your data is not the worst an attacker could do with it. Hopefully, more people will research what I'd call "security 101"...
- pavel_lishin 6y ago
  > Well, it's just another attempt at monetizing the product. Don't make excuses for their shitty business practices.
  - gameswithgo 6y ago
    Don't tell people what to do.
  - cookiecaper 6y ago
    The thing is that no one cares. It's a cultural issue. If Elastic makes it "harder" to get off the starting line by setting secure defaults, they'll get their lunch eaten by a fork called EasyElastic that people perceive as "easier". This probably isn't an excuse for Elastic anymore, but it's how Elastic was born ("easier than Solr!" -- they didn't invent full-text search, after all) and it's how whatever supplants Elastic will be born. Why is MySQL dominant over Postgres even though "referential integrity" didn't come to the game until v5? Why Docker over jails or OpenVZ? etc. People adopt technology because it's fashionable and it becomes fashionable in part because it's perceived as easy to use. Security and ease-of-use are not quite true opposites, but there's definitely some intrinsic tension. We have a lot of hapless practitioners in the space and the root problems here won't go away until we get some standards. Better solutions usually lose because crappy stuff focuses more effort on marketing and an "easy" onboarding process, where better stuff focuses on the operational complications of the real world.
- 6y ago

nisa 6y ago

It's also easy to get bitten by Docker. You can secure your server with iptables/ufw only to discover that docker happily punches through your firewall and you need to filter on the DOCKER-USER chain - and even that was broken: https://unrouted.io/2017/08/15/docker-firewall/ https://unrouted.io/2017/08/15/docker-firewall/ https://github.com/docker/for-linux/issues/690 https://github.com/docker/for-linux/issues/690

dawnerd 6y ago
Seriously this is the most annoying thing ever, especially if someone on your team things you need to expose the ports to redis in a docker compose. I’ve come back from a weekend where my redis instance was being used for crypto mining. Anything that is insecure by default in 2020 should be killed off IMO.
- usmannk 6y ago
  Isn’t exposing ports in your docker-compose services:redis:ports, how you’d do that? (Docker hobbyist here)
  - jcrawfordor 6y ago
    My guess would be that they went with "ports" instead of "expose" which makes it public. But even with careful composition of your docker-compose you might want to be careful, Docker can interact with iptables in surprising ways and long iptables rulesets are almost comically difficult to validate sometimes. The result is that if you're using both Docker (and even more if you use Compose, Kubernetes, some other orchestrator) and doing anything else with iptables (including using it as a host firewall) you need to take a lot of caution to make sure you don't mess anything up. These tools usually confine their changes to custom chains to make this stuff easier to sort out but at the same time it can make it more difficult to understand what's happening, particularly with the use of tagging and etc. In general I would recommend putting some kind of protection between machines running Docker (and especially orchestrators) and the internet. This could be cloud provider mechanisms (security groups, ACLs, etc), a firewall appliance, NAT gateway configuration, etc. depending on the situation. It's not necessary, but it makes the situation easier to audit/validate, and more layers of protection seldom hurt. If nothing else it means that much of the time you'll need to make a mistake in two places instead of one, in order to have an unintended exposure.

dimitar 6y ago

Apparently these guys don't have a firewall setup and haven't heard about private networks and VPNs. reads TFA again Wait, one of the victims is a VPN provider???

TwoBit 6y ago
I've read that there are now VPN providers that just use someone else's VPN engine. So they don't have to know wtf they are doing.
- dehrmann 6y ago
  For a product that's 50% snakeoil and marketing, that approach seems reasonable? It's easier than actually doing the leg work.
  - _-___________-_ 6y ago
    Only 50%? I'd put it much closer to 100% tbh. Why people are so eager to funnel all their traffic through some of the shadiest businesses on the Internet is beyond me...

tdeck 6y ago

Why not just rename all the tables or something? That's enough to get the developer's attention without being so destructive.

nkrisc 6y ago
Because if it's not destructive they have no reason to pay attention. Change names back and it's business as usual.
- tdeck 6y ago
  IDK, if someone kept changing the table names in my DB every week I'd probably throw a password on it, even if I were really lazy. Most of these people probably didn't realize their DBs were unsecured, and that gets the point across quickly (particularly if the new table names are chosen instructively).
  - nkrisc 6y ago
    That sounds reasonable, but you'd think most people would also be concerned about their databases being publicly accessible in the first place, yet here we are.
    - ivanhoe 6y ago
      I don't think this is the case of people not being concerned, but simply the ignorance on their part about the setup. People just presume that the defaults are safe, and never bother getting into the details.
      - nkrisc 6y ago
        At the risk of arguing "no true Scotsman," someone who is concerned about security likely wouldn't make assumptions about defaults. Or rather, someone appropriately paranoid about security concerns would not trust defaults without at least reviewing them.

dilandau 6y ago

What about developers just being competent before deploying a database on the public internet? At least google "securing X" before just pumping data in.

umvi 6y ago
How about all professionals in every industry being competent before doing anything? Then we wouldn't have any issues in society.
- dilandau 6y ago
  Strawman. And a weak one at that. Fuck off with conflating the minimum amount of competency needed to secure the data you needlessly collect with that kind of naivete.
  - umvi 6y ago
    Who says every unsecured db on the internet consists of "needlessly collected" data?

IAmEveryone 6y ago

That's just victim blaming. The same logic applies to every crime: "lock your doors if you don't want your TV to be stolen!". It also works at any level of security: "Lock your doors and hire guards if you don't want clever thieves breaking a window..." But if you require everyone to take adequate measures to physically secure their houses, you don't even need laws and morality! And while this may provide the sort of negative reinforcement you are hoping for, actual damages in each case are going to be all over the place. That makes this one of the worst possible punishments. Imagine the penalty for speeding is a random outcome somewhere between a stern reminder and the death penality. There would be nothing fair and little useful about that, even though it does follow the same basic principle of do bad -> be harmed.

theevilsharpie 6y ago
> But if you require everyone to take adequate measures to physically secure their houses, you don't even need laws and morality! If laws and morality were sufficient protection against malicious actors, I might agree with you. However, in a world where cyber vandals are often beyond any accessible jurisdiction (and may even be supported by their local authorities), laws and morality are clearly not effective at keeping unauthorized users out of private systems. As such, the responsibility for keeping private information secure naturally falls on the people running the systems. Putting up a network firewall (or at least requiring authentication) would have prevented the damage described in the OP, and is a rudimentary security measure that has been common practice for decades. The people who suffered significant damage from this attack should strongly consider outsourcing system administration to someone who knows what they're doing.
jessaustin 6y ago
You're not thinking of the right victim. You're considering the organizations who compiled the databases in question. If the databases are composed entirely of their own information that they created themselves, then sure they've suffered a loss. In most cases, however, these databases include privacy-sensitive personal information about the customers of the organization. Those customers have been victims of bad security practices ever since MongoDb was first installed, and this "meow" hack has ended their victimization.

exadeci 6y ago

Wasn't there a guy that thoroughly tested databases found that MongoDB was awful? I'm pretty sure many of his articles were posted here.