3 ms·
The part that irks me is that if you’re doing any VPC design that’s going to even potentially include peering you need to carefully understand the limitations f
by devonkim 8y ago
The part that irks me is that if you’re doing any VPC design that’s going to even potentially include peering you need to carefully understand the limitations first. This means that within the same region you can refer to security groups in rules as if they’re in the same VPC, but you can’t do that if you’re peering across regions. Then add in some DNS restrictions (like not being able to directly resolve a peer VPC’s entries, somewhat solvable by use of VPC private zones to serve as DNS across regions) and it can be real awkward. Then there’s overlapping VPC CIDR issues (VPC Transit Gateways can only sorta help this)
The primary caveats beyond basic networks that impact designs is that multicasting is not enabled by the network layers but at the network interface (ENI) layer and you need to carefully look at how security groups really work (they’re attached to an ENI fundamentally, which is how you can route between networks with a single instance as long as it’s within the same AZ)
All of this I’ve found was completely disregarded / unknown by almost every company outside the F500 or high end tech start-ups when they first started with AWS and I’ve spent a lot of my career having to migrate production environments between VPCs so that we can get enough room to grow adequately. Making subnets as small as possible is not what you should be doing in AWS, folks. In fact, making them real small means you spent a fair bit of effort which means you decided to put in a lot of effort without stopping to read the documentation in earnest for a couple hours. And using a default VPC CIDR repeatedly from the console is a pretty grand way to make sure you can never let two VPCs communicate with each other via anything other than a third intermediate VPC that you’ll have to migrate to eventually.
Some of the overly-cautious networking approaches I’ve seen include making a VPC for every single application / service, using a NACL for every application (multiplied by every AZ used to isolate each subnet and cutting off cross-AZ routing thereby, of course), creating your own NAT instance that doesn’t do anything better than a NAT gateway, NAT gateways in every AZ (for a whole $1 of traffic / mo each). The story of problems in AWS infrastructure is the same - trying to plan too far ahead for the wrong things and not realizing the limitations of the right things that are not flexible anymore. This is much more common when companies hire traditionally experienced network engineers that have just a little too much confidence.
- jugg1es 8y agoPlanning your network is something everyone has to do whether or not they are using AWS. I think its unreasonable to expect to just throw stuff in AWS without thinking about it and then complain when something comes up unexpectedly.
- devonkim 8y agoThe issue is that lots of folks get started on AWS (or any cloud provider) and because everything’s built around getting developers in a hurry to push out some code this step is increasingly taking less and less time. Sure, it’s not a big deal for the usual start-up that fizzles in a couple years but once it’s going and there’s actual users the system is setup that there will be a forced outage of some sort when a couple hours investment would have prevented a lot of headaches. The common pivot for a company is to go from a b2c company to b2b and that means regulations. That means you can’t do cowboy infrastructure setup like people setup their home network (in fact, most home networks are more secure than what most SaaS devs do as a rule sans router firmware exploits).