3 ms·
I agree with only the third of those. The other lessons I'd take would be: 1. Be cautious about trendy technologies that promise to make life easy - often they
by lmm 5y ago
I agree with only the third of those. The other lessons I'd take would be:
1. Be cautious about trendy technologies that promise to make life easy - often they cut corners to do so, and often security is one of those corners
2. Use real authentication rather than network firewalling. Make your datastore TLS-only and require a valid client certificate to connect; that way it doesn't matter if it's exposed to the internet (indeed I'd argue you probably should expose every server to the internet - much like Chaos Monkey, it's counterintuitive but it forces you to build your systems with the right kind of resilience from day one)
- vngzs 5y agoYeah, there's the whole "zero trust" possibility which I didn't mention. If you do authentication/authorization really well, you can stop doing (2). For things like databases, I think it's better to treat them as if they _could_ be exposed to the open internet, without actually doing so. It's generally not the case that anyone on the internet needs to query your DBs. As described, you're _only_ relying on client TLS to protect your database. What if the TLS key leaks? So you're down to one layer again - one mistake and it's game over. So maybe you need Hashicorp Vault so you can have client certificates with very short validity periods. So do you expose Vault to the Internet, so you can fetch your client certificate to query the DB? What if Vault has a bug? And it's turtles all the way down ... I love zero trust designs. But I think saying you can just slap client TLS on the problem and be done is laying the foundation for a repeat of this event - however you look at it, you want redundancy in your security.
- thrashh 5y agoAll internal services should be protected, even on your home network. If you have 100 devices on a network where everything is unprotected, that’s 100 different ways someone can try to get full access to 99 other devices.
- deleted 5y ago[deleted]
- pm90 5y agoAt every layer of “breakage”, you are cutting down the chance for an attacker to access your system. While it’s possible the failures might cascade it seems likely that these failure modes are pretty independent of each other. You’re simply pointing out that no amount of layers are foolproof. The goal is not to reduce failure to 0% but to 0.001%.
- vngzs 5y agoI think you misunderstood me - I mean that you want multiple layers to your security, and you need to be careful to select layers that fail independently. If Vault has a sufficiently bad unauthenticated bug, then attackers can simply use it to request database credentials and query the DB, which is now on the open internet. An easy way to get independent failures is to layer a private network with strong auth and firewall rules. While it's certainly possible to expose your DB to the internet safely - given sufficient protections - you won't get that with just a TLS key. And even if you try to implement the "obvious" additional layers here (Vault, right?), it's easy to inadvertently include design problems that reduce to "only one failure and the system is exposed."
- lmm 5y agoMy view is that for most cases the cost/benefit of multiple layers doesn't stack up. Given a fixed amount of available time and effort, you'll generally get better results by focusing that effort on making one really good layer - e.g. putting active monitoring in place so that you detect when your single layer breaks (whether that's an attacker from outside the network being able to connect to inside the network or an attacker with an expired certificate being able to connect to a live system) - than by spending half as much time each on two layers, IME.
- lowbloodsugar 5y agoUnless this is literally running on a single 1U host in some colo, there is no excuse for not having defense in depth. An old school DMZ if you are in a datacenter. A VPC if you are on the cloud. Then client certs for everything. Two factor for ssh. Auditing. These are straightforward to set up, with different options if you have time-but-not-money or money-but-not-time. If you have not-money-and-not-time then this is probably going to end badly no matter what.
- lmm 5y ago> there is no excuse for not having defense in depth. An old school DMZ if you are in a datacenter. A VPC if you are on the cloud. Then client certs for everything. Two factor for ssh. Auditing. Maybe there is no excuse, but literally every company I've worked for (including Fortune 500s) has been missing at least one item from your list. So "industry best practice" means committing less time and money than it would take to implement all those things properly (rightly or wrongly) and you need to triage and prioritise.
- lowbloodsugar 5y agoSo the excuse is "my company doesn't take infosec seriously". Like every similar issue of "should do but don't" (testing, formal promotion processes, diversity, harassment response), you get to decide to tolerate it or get a job elsewhere. My experience is that the companies that take this stuff seriously also do a better job of converting my skills into cash, and as a result, pay me better.
- wjossey 5y agoRespectfully, all of the suggestions are good, as are your two. But, segregating your public traffic and private traffic is an excellent way to prevent these sorts of issues from happening. While it doesn't prevent someone who's compromised your network from accessing the DB, it means someone can't fat finger a port and open a DB to the public internet.
- lowbloodsugar 5y agoWell, your second point is his first point. And his second point is table-stakes as far as I'm concerned. As others have said, do all of them. They are not hard to do.
- kortilla 5y ago> 2. Use real authentication rather than network firewalling. This is in vogue but it’s wrong because it presents this as an either/or. In reality, people are going to goof up and auth flag at one point, accidentally bind a service, or just run a service with a 0-day (a.k.a everyone). There is no reason to run a server accepting traffic from every IP if your clients are coming from known ranges. People see the zero trust model and mistakenly think it means no network-level filtering. This is completely wrong and all of the big players still protect backend services with network ACLs on top of required auth.
- lmm 5y agoIn reality time and attention are limited resources. Does network-level filtering have a good RoI relative to putting the same effort into improving security at the service level? IMO no (at least not until you've reached a very high level of service security where you're hitting severely diminishing returns) - network-level security is necessarily at least one of a) crude b) complex enough that it becomes an attack surface itself - though reasonable people can disagree.
- unionpivo 5y agoYou are looking it as a network level filtering. I am looking at, make private network, and then you have to explicitly add an gateway (usually a load balancer) that can access it. In short don't design system where you need to filter, design the system, where you need to take explicite action, to make something public. This is easily, done on something like AWS.
- lima 5y agoAn independent firewall accomplishes the same even if every server has a public IP.
- lmm 5y agoThe very concept of a "private network" leads you down the path of making a security boundary that has far too large a surface area, IME. Either it becomes a big bag with all your hosts in it, and you write systems that trust all the requests they get (even if you know you "shouldn't") because you know only your systems are on the network, and then an attacker figures out how to make one of your systems make an arbitrary request and you get owned. Or you put each component on its own private network but then you have to open up every port you're actually listening on so that your components can talk to each other (and you probably automate that in your kubernetes/puppet/whatever setup, like what happened here with docker) and the private network does nothing.