6 ms·

I think there are some good lessons here: 1. Even if you have one way to protect your database (e.g., firewall rules), you should have another. In this case, u

by vngzs 5y ago

I think there are some good lessons here: 1. Even if you have one way to protect your database (e.g., firewall rules), you should have another. In this case, use a database password or (better) client TLS certificate to authenticate traffic. We're all human and we mess up. You should be designing systems that are graceful in response to your inevitable mistakes. 2. If you can afford another server/a hosting provider with VPC-like features, don't put your databases on the open internet. Run them in a private network (RFC 1918), behind a NAT and a load balancer/entrypoint that only routes to your internet-serving applications. Allow only those application servers to hit your databases. If you had done this, the attacker wouldn't have noticed your mistake, because all they could hit was your public server. 3. Keep regular backups. Oh, and test them! If you don't test your backups, you don't have backups - you have archives. In the GitLab data loss incident [0], they had 3 methods of data backup, but all of them failed. A regular test would have discovered this. Don't make that mistake. Good on you for sharing the events as they happen. I think people tend to be much more forgiving in response to openness. Don't freak out, and write a public postmortem when you're done. [0]: https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com-database-incident/ https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com-data...

lmm 5y ago
I agree with only the third of those. The other lessons I'd take would be: 1. Be cautious about trendy technologies that promise to make life easy - often they cut corners to do so, and often security is one of those corners 2. Use real authentication rather than network firewalling. Make your datastore TLS-only and require a valid client certificate to connect; that way it doesn't matter if it's exposed to the internet (indeed I'd argue you probably should expose every server to the internet - much like Chaos Monkey, it's counterintuitive but it forces you to build your systems with the right kind of resilience from day one)
- vngzs 5y ago
  Yeah, there's the whole "zero trust" possibility which I didn't mention. If you do authentication/authorization really well, you can stop doing (2). For things like databases, I think it's better to treat them as if they _could_ be exposed to the open internet, without actually doing so. It's generally not the case that anyone on the internet needs to query your DBs. As described, you're _only_ relying on client TLS to protect your database. What if the TLS key leaks? So you're down to one layer again - one mistake and it's game over. So maybe you need Hashicorp Vault so you can have client certificates with very short validity periods. So do you expose Vault to the Internet, so you can fetch your client certificate to query the DB? What if Vault has a bug? And it's turtles all the way down ... I love zero trust designs. But I think saying you can just slap client TLS on the problem and be done is laying the foundation for a repeat of this event - however you look at it, you want redundancy in your security.
  - thrashh 5y ago
    All internal services should be protected, even on your home network. If you have 100 devices on a network where everything is unprotected, that’s 100 different ways someone can try to get full access to 99 other devices.
    - deleted 5y ago
      [deleted]
  - pm90 5y ago
    At every layer of “breakage”, you are cutting down the chance for an attacker to access your system. While it’s possible the failures might cascade it seems likely that these failure modes are pretty independent of each other. You’re simply pointing out that no amount of layers are foolproof. The goal is not to reduce failure to 0% but to 0.001%.
    - vngzs 5y ago
      I think you misunderstood me - I mean that you want multiple layers to your security, and you need to be careful to select layers that fail independently. If Vault has a sufficiently bad unauthenticated bug, then attackers can simply use it to request database credentials and query the DB, which is now on the open internet. An easy way to get independent failures is to layer a private network with strong auth and firewall rules. While it's certainly possible to expose your DB to the internet safely - given sufficient protections - you won't get that with just a TLS key. And even if you try to implement the "obvious" additional layers here (Vault, right?), it's easy to inadvertently include design problems that reduce to "only one failure and the system is exposed."
  - lmm 5y ago
    My view is that for most cases the cost/benefit of multiple layers doesn't stack up. Given a fixed amount of available time and effort, you'll generally get better results by focusing that effort on making one really good layer - e.g. putting active monitoring in place so that you detect when your single layer breaks (whether that's an attacker from outside the network being able to connect to inside the network or an attacker with an expired certificate being able to connect to a live system) - than by spending half as much time each on two layers, IME.
- wjossey 5y ago
  Respectfully, all of the suggestions are good, as are your two. But, segregating your public traffic and private traffic is an excellent way to prevent these sorts of issues from happening. While it doesn't prevent someone who's compromised your network from accessing the DB, it means someone can't fat finger a port and open a DB to the public internet.
- lowbloodsugar 5y ago
  Well, your second point is his first point. And his second point is table-stakes as far as I'm concerned. As others have said, do all of them. They are not hard to do.
- kortilla 5y ago
  > 2. Use real authentication rather than network firewalling. This is in vogue but it’s wrong because it presents this as an either/or. In reality, people are going to goof up and auth flag at one point, accidentally bind a service, or just run a service with a 0-day (a.k.a everyone). There is no reason to run a server accepting traffic from every IP if your clients are coming from known ranges. People see the zero trust model and mistakenly think it means no network-level filtering. This is completely wrong and all of the big players still protect backend services with network ACLs on top of required auth.
  - lmm 5y ago
    In reality time and attention are limited resources. Does network-level filtering have a good RoI relative to putting the same effort into improving security at the service level? IMO no (at least not until you've reached a very high level of service security where you're hitting severely diminishing returns) - network-level security is necessarily at least one of a) crude b) complex enough that it becomes an attack surface itself - though reasonable people can disagree.
    - unionpivo 5y ago
      You are looking it as a network level filtering. I am looking at, make private network, and then you have to explicitly add an gateway (usually a load balancer) that can access it. In short don't design system where you need to filter, design the system, where you need to take explicite action, to make something public. This is easily, done on something like AWS.
      - lima 5y ago
        An independent firewall accomplishes the same even if every server has a public IP.
      - lmm 5y ago
        The very concept of a "private network" leads you down the path of making a security boundary that has far too large a surface area, IME. Either it becomes a big bag with all your hosts in it, and you write systems that trust all the requests they get (even if you know you "shouldn't") because you know only your systems are on the network, and then an attacker figures out how to make one of your systems make an arbitrary request and you get owned. Or you put each component on its own private network but then you have to open up every port you're actually listening on so that your components can talk to each other (and you probably automate that in your kubernetes/puppet/whatever setup, like what happened here with docker) and the private network does nothing.
wjossey 5y ago
Just want to yes and you. In general, put everything in private subnets, and make the only way any traffic can get to a server is through a load balancer. There are very few reasons to have a server itself have its own public IP address, and using your load balancer as a chokepoint, means you can set up layers and layers of redundancy to prevent traffic from ever being able to reach a database under your control. This holds true whether we're talking RDS DBs, something you've spun in a K8 cluster, or something you're running on a vanilla compute box. Assume you will screw up at some point and open up a port that shouldn't be. Ask yourself, "what's the impact here?" If you're OK, even with a fat fingered port, great. Assume you will have a dev deploy a DB without a password. "What's the impact here?" If you're ok, even without a password, great! That isn't to say that the above two scenarios are something you should tolerate, but automation can help detect these sorts of issues and make it easy for you to resolve them. While that automation is running, you want to make sure you're not going to get owned. Kudos to the OP for sharing their story. Always lots to learn on this front, and we can always get better at our cloud operations.
- mtippett 5y ago
  +1 on the private subnet. You want to design your system so that if a network a critical misconfiguration occurs you don't open yourself up - you simply stop working. (Too many years chasing EMR clusters getting dropped onto the internet by users with AWS console access).
- discordance 5y ago
  Defense in depth... VNets, TLS certs, RBAC etc.
- AtNightWeCode 5y ago
  Layer 4 and/or 7 proxies -> private networks -> services. Auth and encryption all the way on everything. IP white-listening were suitable. Pretty much the standard way of building web were I live.
- lima 5y ago
  "Private subnet" is just one way to accomplish it - what you want is a firewall that drops inbound traffic by default, and private subnet/NAT is one common way to do it that does have some downsides (extra complexity - you need a NAT gateway now).
nodesocket 5y ago
Can I also add don't run MongoDB into Docker containers. Use MongoDB Atlas or switch to AWS and DocumentDB with proper VPC and a network level firewall (security groups). @conesus It's just not worth using discount hosting providers for this exact reason. Use a M(icrosoft) A(mazon) G(oogle) cloud, yes it costs more.... But your time is worth it!
- EdwardDiego 5y ago
  Why not run it in containers? What's the downsides? I mean, if you can reliably run Kafka or Postgres in containers, what's sufficiently different about MongoDB? Or are you talking about using Docker itself as the container host as opposed to K8s or ECS etc.?
  - higeorge13 5y ago
    I don’t get why the original response is downvoted. Of course you can run those in docker containers, but it is generally suggested that you don’t. From my experience i would run stateless services in containers and persistent storages in vms, dedicated servers, or cloud services.
    - the_duke 5y ago
      People have been running database instances just fine on their own for decades, without the help of "big brother". There are reasons why running a high performance database instance in containers is problematic, but security is not one of them - not any more so than application containers. You just need to know what you are doing, it's not a black art.
      - SOLAR_FIELDS 5y ago
        Truth - I personally avoid doing so, but that’s only because I prefer not to introduce that extra layer of complexity into my stack due to lack of full understanding of the technology. There’s nothing stopping me from bothering to learn all the trade offs and pitfalls and doing so, but so far there hasn’t been enough of a compelling reason for me to go through that effort. I just go with the general advice “you usually don’t want to dockerize stateful applications” and leave it at that.
radicalbyte 5y ago
What strikes me about your reply is that 1, 2, 3 are all absolutely basic table-stakes things. SQL injection mitigation level things. Have we suddenly stopped teaching the basics?
- windowsworkstoo 5y ago
  Youre downvoted but you’re right - this is entry level stuff and it seems to get routinely not taught or simply forgotten and ignored. Everyone seems to want to worry about “nation state actors” and getting with novel 0 days when in fact missing the basic low hanging fruit is likely to result in far more damage
panta 5y ago
Another important missing point is not having docker publicly exposed. Just reach docker through a ssh tunnel (eg via rdocker) or a vpn.
gregoriol 5y ago
I'd also suggest to use another server/vps to monitor your setup (availability, performance, ...), and include a ports open check in the monitoring tools. Having an external check of your usual ports (thoose that should be closed) is a good way to find when something is off.
danielpops 5y ago
Another takeaway, IMO, use infrastructure-as-code to define the components in your infrastructure, so something like this critical firewall configuration won't be missed
- zerd 5y ago
  Docker will happily override your firewall rules even with infrastructure as code. We block the AWS/GCP metadata IP to avoid potential exploits using that to pivot, but Docker also manages to override that sometimes, so we have a cron job that re-applies it.

by vngzs 5y ago

lmm 5y ago

I agree with only the third of those. The other lessons I'd take would be: 1. Be cautious about trendy technologies that promise to make life easy - often they cut corners to do so, and often security is one of those corners 2. Use real authentication rather than network firewalling. Make your datastore TLS-only and require a valid client certificate to connect; that way it doesn't matter if it's exposed to the internet (indeed I'd argue you probably should expose every server to the internet - much like Chaos Monkey, it's counterintuitive but it forces you to build your systems with the right kind of resilience from day one)

vngzs 5y ago
Yeah, there's the whole "zero trust" possibility which I didn't mention. If you do authentication/authorization really well, you can stop doing (2). For things like databases, I think it's better to treat them as if they _could_ be exposed to the open internet, without actually doing so. It's generally not the case that anyone on the internet needs to query your DBs. As described, you're _only_ relying on client TLS to protect your database. What if the TLS key leaks? So you're down to one layer again - one mistake and it's game over. So maybe you need Hashicorp Vault so you can have client certificates with very short validity periods. So do you expose Vault to the Internet, so you can fetch your client certificate to query the DB? What if Vault has a bug? And it's turtles all the way down ... I love zero trust designs. But I think saying you can just slap client TLS on the problem and be done is laying the foundation for a repeat of this event - however you look at it, you want redundancy in your security.
- thrashh 5y ago
  All internal services should be protected, even on your home network. If you have 100 devices on a network where everything is unprotected, that’s 100 different ways someone can try to get full access to 99 other devices.
  - deleted 5y ago
    [deleted]
- pm90 5y ago
  At every layer of “breakage”, you are cutting down the chance for an attacker to access your system. While it’s possible the failures might cascade it seems likely that these failure modes are pretty independent of each other. You’re simply pointing out that no amount of layers are foolproof. The goal is not to reduce failure to 0% but to 0.001%.
  - vngzs 5y ago
    I think you misunderstood me - I mean that you want multiple layers to your security, and you need to be careful to select layers that fail independently. If Vault has a sufficiently bad unauthenticated bug, then attackers can simply use it to request database credentials and query the DB, which is now on the open internet. An easy way to get independent failures is to layer a private network with strong auth and firewall rules. While it's certainly possible to expose your DB to the internet safely - given sufficient protections - you won't get that with just a TLS key. And even if you try to implement the "obvious" additional layers here (Vault, right?), it's easy to inadvertently include design problems that reduce to "only one failure and the system is exposed."
- lmm 5y ago
  My view is that for most cases the cost/benefit of multiple layers doesn't stack up. Given a fixed amount of available time and effort, you'll generally get better results by focusing that effort on making one really good layer - e.g. putting active monitoring in place so that you detect when your single layer breaks (whether that's an attacker from outside the network being able to connect to inside the network or an attacker with an expired certificate being able to connect to a live system) - than by spending half as much time each on two layers, IME.
wjossey 5y ago
Respectfully, all of the suggestions are good, as are your two. But, segregating your public traffic and private traffic is an excellent way to prevent these sorts of issues from happening. While it doesn't prevent someone who's compromised your network from accessing the DB, it means someone can't fat finger a port and open a DB to the public internet.
lowbloodsugar 5y ago
Well, your second point is his first point. And his second point is table-stakes as far as I'm concerned. As others have said, do all of them. They are not hard to do.
kortilla 5y ago
> 2. Use real authentication rather than network firewalling. This is in vogue but it’s wrong because it presents this as an either/or. In reality, people are going to goof up and auth flag at one point, accidentally bind a service, or just run a service with a 0-day (a.k.a everyone). There is no reason to run a server accepting traffic from every IP if your clients are coming from known ranges. People see the zero trust model and mistakenly think it means no network-level filtering. This is completely wrong and all of the big players still protect backend services with network ACLs on top of required auth.
- lmm 5y ago
  In reality time and attention are limited resources. Does network-level filtering have a good RoI relative to putting the same effort into improving security at the service level? IMO no (at least not until you've reached a very high level of service security where you're hitting severely diminishing returns) - network-level security is necessarily at least one of a) crude b) complex enough that it becomes an attack surface itself - though reasonable people can disagree.
  - unionpivo 5y ago
    You are looking it as a network level filtering. I am looking at, make private network, and then you have to explicitly add an gateway (usually a load balancer) that can access it. In short don't design system where you need to filter, design the system, where you need to take explicite action, to make something public. This is easily, done on something like AWS.
    - lima 5y ago
      An independent firewall accomplishes the same even if every server has a public IP.
    - lmm 5y ago
      The very concept of a "private network" leads you down the path of making a security boundary that has far too large a surface area, IME. Either it becomes a big bag with all your hosts in it, and you write systems that trust all the requests they get (even if you know you "shouldn't") because you know only your systems are on the network, and then an attacker figures out how to make one of your systems make an arbitrary request and you get owned. Or you put each component on its own private network but then you have to open up every port you're actually listening on so that your components can talk to each other (and you probably automate that in your kubernetes/puppet/whatever setup, like what happened here with docker) and the private network does nothing.

wjossey 5y ago

Just want to yes and you. In general, put everything in private subnets, and make the only way any traffic can get to a server is through a load balancer. There are very few reasons to have a server itself have its own public IP address, and using your load balancer as a chokepoint, means you can set up layers and layers of redundancy to prevent traffic from ever being able to reach a database under your control. This holds true whether we're talking RDS DBs, something you've spun in a K8 cluster, or something you're running on a vanilla compute box. Assume you will screw up at some point and open up a port that shouldn't be. Ask yourself, "what's the impact here?" If you're OK, even with a fat fingered port, great. Assume you will have a dev deploy a DB without a password. "What's the impact here?" If you're ok, even without a password, great! That isn't to say that the above two scenarios are something you should tolerate, but automation can help detect these sorts of issues and make it easy for you to resolve them. While that automation is running, you want to make sure you're not going to get owned. Kudos to the OP for sharing their story. Always lots to learn on this front, and we can always get better at our cloud operations.

mtippett 5y ago
+1 on the private subnet. You want to design your system so that if a network a critical misconfiguration occurs you don't open yourself up - you simply stop working. (Too many years chasing EMR clusters getting dropped onto the internet by users with AWS console access).
discordance 5y ago
Defense in depth... VNets, TLS certs, RBAC etc.
AtNightWeCode 5y ago
Layer 4 and/or 7 proxies -> private networks -> services. Auth and encryption all the way on everything. IP white-listening were suitable. Pretty much the standard way of building web were I live.
lima 5y ago
"Private subnet" is just one way to accomplish it - what you want is a firewall that drops inbound traffic by default, and private subnet/NAT is one common way to do it that does have some downsides (extra complexity - you need a NAT gateway now).

nodesocket 5y ago

Can I also add don't run MongoDB into Docker containers. Use MongoDB Atlas or switch to AWS and DocumentDB with proper VPC and a network level firewall (security groups). @conesus It's just not worth using discount hosting providers for this exact reason. Use a M(icrosoft) A(mazon) G(oogle) cloud, yes it costs more.... But your time is worth it!

EdwardDiego 5y ago
Why not run it in containers? What's the downsides? I mean, if you can reliably run Kafka or Postgres in containers, what's sufficiently different about MongoDB? Or are you talking about using Docker itself as the container host as opposed to K8s or ECS etc.?
- higeorge13 5y ago
  I don’t get why the original response is downvoted. Of course you can run those in docker containers, but it is generally suggested that you don’t. From my experience i would run stateless services in containers and persistent storages in vms, dedicated servers, or cloud services.
  - the_duke 5y ago
    People have been running database instances just fine on their own for decades, without the help of "big brother". There are reasons why running a high performance database instance in containers is problematic, but security is not one of them - not any more so than application containers. You just need to know what you are doing, it's not a black art.
    - SOLAR_FIELDS 5y ago
      Truth - I personally avoid doing so, but that’s only because I prefer not to introduce that extra layer of complexity into my stack due to lack of full understanding of the technology. There’s nothing stopping me from bothering to learn all the trade offs and pitfalls and doing so, but so far there hasn’t been enough of a compelling reason for me to go through that effort. I just go with the general advice “you usually don’t want to dockerize stateful applications” and leave it at that.

radicalbyte 5y ago

What strikes me about your reply is that 1, 2, 3 are all absolutely basic table-stakes things. SQL injection mitigation level things. Have we suddenly stopped teaching the basics?

windowsworkstoo 5y ago
Youre downvoted but you’re right - this is entry level stuff and it seems to get routinely not taught or simply forgotten and ignored. Everyone seems to want to worry about “nation state actors” and getting with novel 0 days when in fact missing the basic low hanging fruit is likely to result in far more damage

panta 5y ago

Another important missing point is not having docker publicly exposed. Just reach docker through a ssh tunnel (eg via rdocker) or a vpn.

gregoriol 5y ago

I'd also suggest to use another server/vps to monitor your setup (availability, performance, ...), and include a ports open check in the monitoring tools. Having an external check of your usual ports (thoose that should be closed) is a good way to find when something is off.

danielpops 5y ago

Another takeaway, IMO, use infrastructure-as-code to define the components in your infrastructure, so something like this critical firewall configuration won't be missed

zerd 5y ago
Docker will happily override your firewall rules even with infrastructure as code. We block the AWS/GCP metadata IP to avoid potential exploits using that to pivot, but Docker also manages to override that sometimes, so we have a cron job that re-applies it.