12 ms·
Networking on AWS (2018)
- mooreds 8y agoI taught some courses on AWS for a year and a half. The networking piece is something that is trivial for any network engineer, but for any developer (which is my background) working through the network piece is crucial. It takes a while and this looks like a good reference. However, it's best to also check out the AWS docs https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html https://docs.aws.amazon.com/vpc/latest/userguide/what-is-ama... . They are not always the easiest read, but I find them to be pretty authoritative. I also like this video https://www.oreilly.com/library/view/amazon-web-services/9781771373944/video217078.html https://www.oreilly.com/library/view/amazon-web-services/978... (part of http://shop.oreilly.com/product/0636920040415.do http://shop.oreilly.com/product/0636920040415.do ). Full disclaimer, I used to work with Jon.
- orangejewce 8y agoThis piece excludes some incredibly common networking complications: VPC Peering Connections, VPC Endpoint Services, VPN connections.
- vvanders 8y agoBeen recently putting together a homelab and it's done wonders to help make some of these more abstract things like routing tables and CIDR mentioned a lot more concrete.
- voltagex_ 8y agoAre you able to share any details? I started putting together a more complex setup and ended up flattening things out because I couldn't get routing between e.g. 10.0.0.1 and 10.1.0.1 working.
- vvanders 8y agoYeah, totally depends on what you're using for a router/gateway/firewall. I've got a mix of Ubiquiti gear and pfSense. Most of it was a matter of just setting up a static route(like in the article) where when I'm on the 192.x.x.x(192.0.0.0/24) network and want to talk to 10.0.0.0/24 I'd put in the gateway(10.0.0.1) as the next hop. Without knowing more about your setup it's hard to say. r/homelab is also a pretty decent place with a lot of helpful people interested in networking and homelabs.
- voltagex_ 8y agoI've got an all-Ubiquiti setup (and once you start digging into the forums / need IPv6, boy is it unimpressive) It seems like going with the Ubiquiti USG instead of the EdgeRouter or a pfsense box was a big mistake. I'm heavily space/heat/power constrained though.
- vvanders 8y agoSame USG here. If you're on two different networks in the Unifi setup you'll also probably need a firewall rule between them. In my case the 10.0.0.0/24 was on pfSense so the Unifi router just passed it along and didn't do anything else.
- mvanbaak 8y agoFor the public subnets where the NAT gateways are, you can use 1 route table for all public subnets together. Besides that: nice article
- grahamlyons 8y agoWould the default route table do that job? Subnets would be associated with that if they weren't explicitly assigned to another route table, right?
- mvanbaak 8y agoYes. But i prefer to create my own. Create a riutetable, call it “public” using a tag, attach the internetgateway and attach all public subnets, add a default route that points to the internetgateway and you’re done
- becauseiam 8y agoThere's quite a few things you've missed that are significant and should have been included, maybe one for part two: * Network ACLs, which describe the ruleset (consider it like a stateless firewall) for subnets and their respective routes. Whilst they are optional, having a default set it straightens out a lot of duplication that may end up in Security Groups (which are more stateful in nature). * Elastic (public) IPs. NAT instances/gateways require their use, and there is dance to be done around their allocation in account, and attaching to instance interfaces. * IPv6 components. Egress-only Internet Gateways operate differently to IGWs, as there is no NAT they need a route applied across all subnets both public and private. IPv6 CIDR which allocates the VPCs /56 (and thus each subnet gets a /64, and each instance's interface thus gets a /128 which is bananas, but IPv6 is a second class citizen on AWS). Finally updating the subnets so automatic IPv6 address assignment happens. * VPC Gateways - these are broken into two types, the older type that support S3/DynamoDB and effectively allow traffic in a public/private subnet to bypass NAT. These enabled can have significant advantages to access and throughput. The newer "PrivateLink" services are different and having pricing costs associated with them. * DNS and DHCP: It's a rule in the VPC that the delegated resolver lives on ".2" of the VPC's CIDR, and operates in dual-horizon - EC2 hostnames setup accordingly resolved by instances inside the VPC will get the private VPC CIDR address, not any Elastic IP.
- himangshuj 8y agoNetwork ACLS are quite tricky to debug. For one of my connections, network calls were failing because esp was blocked at acl layer, ACL blocks all non tcp traffic by default. Funnily, network calls with same data-center was working but was failing when calling to another data-center. I had to look at VPC flow logs to figure that non tcp protocals were being blocked.
- cure 8y ago+1 for VPC Gateways. If you want to get decent aggregate performance out of S3 when you have many readers/writers in your VPC, you want one of those.
- cxmcc 8y agoA few more here: VPC Peering. VPC Endpoints.
- ninetax 8y agoNice, I found this explanation a bit more in depth and super helpful as well https://start.jcolemorrison.com/aws-vpc-core-concepts-analogy-guide/ https://start.jcolemorrison.com/aws-vpc-core-concepts-analog...
- cygned 8y agoIn my opinion, the most annoying thing about AWS networking - and some other services - is that they often use IDs and do not show labels which forces me to remember them partly, go back and forth or have multiple windows open. The AWS console is not the best UX piece on the web, but this part is especially error prone.
- bmurphy1976 8y agoHaving to reference security groups by id instead of name in cloudformation stacks, terraform, and a variety of other places is one of the most infuriating things. Makes everything much more difficult to configure and maintain because the IDs are so opaque and unique. Somebody has to do the grunt work of looking them and and copy/pasting them or writing scripts to propagate configuration forward. What a waste of time and effort.
- chrisacky 8y agoI agree, it's frustrating. However, considering you can share security groups, and even share VPC as a resource to different AWS accounts, and that "names" are non-unique, how would you solve the problem of allowing people to select a non-unique resource and know which one they actually mean. It's a pain, sure, but there's no better solution... explicitness creates certainty in this case..
- bmurphy1976 8y agoYou would use the arns. Those are unique but predictable.
- scrollaway 8y agoIn terraform you should never be using IDs directly unless it's in a variable or, preferably, a data source. The reference should look like aws_something.mything.id.
- mvanbaak 8y agoSince you mention cloudformation here my tips to make it easier: - tag all groups so its easier to remember - use Export in outputs so you can use ImportValue in other templates - use only cloudformation, no edit/create/add actions in the console Now the ID thing is 0 problems as you never have to worry about them anymore Same goes for terraform or any other InfrastructureAsCode
- p4lindromica 8y agoI created a collection of terraform modules that gets a minimal AWS network set up for a single-region webapp: https://github.com/lopopolo/hyperbola/tree/master/terraform/modules/aws/network https://github.com/lopopolo/hyperbola/tree/master/terraform/...
- dvtrn 8y agoKudos for this, sincerely!
- Gelob 8y agoAWS security groups and ACLs are the most worthless things. you cant treat them like a real firewall. you end up just allowing anything outbound or inbound. they dont let you be detailed enough
- AzMoo_ 8y agoWhy are you just allowing anything outbound or inbound? You can specify Allow/Deny on any combination of source subnet, dest subnet, source port, dest port for starters. That gets you a pretty comprehensive ability to lock down a VPC on its own.
- Gelob 8y agoSay I want to allow outbound http/https to 10 different IPs. I can't do that in 1 rule like a traditional firewall.
- justnoise 8y agoJust in case those IPs are within your AWS account: you can apply a single security group to those machines and then use that security group as the destination in the outbound rule. If they're outside your account then, you're right, that's a shortcoming in AWS (Azure and GCP both allow multiple destinations in a single rule).
- Gelob 8y agoYes coming from outside aws, you're fucked
- AzMoo_ 8y agoIt's about 2 seconds work in CloudFormation though.
- iheartpotatoes 8y agoDumb question: if the IPs are coming from Route53 for web addresses, why don't you just point them as aliases to the same load balancer? Done and done, right?
- benmanns 8y agoNAT gateways are one of the things that blindsided me on the whole "serverless" idea for hobby projects. To have a Lambda function with access to the outside world and your private network resources your $0.01/month function becomes a $35/month+ expense if you don't want to manage your own t2 NAT instance (and required patches, upgrades, scaling, monitoring, etc). See https://forums.aws.amazon.com/thread.jspa?threadID=234959 https://forums.aws.amazon.com/thread.jspa?threadID=234959
- ceejayoz 8y agoA VPC with a public subnet that's locked down largely via security groups is probably fine for a project that can't justify a $35/month spend. edit: Apparently not. See below, my mistake.
- davidjnelson 8y agoHave you tried cloudflare workers? The networking gets taken care of for you, plus they are obscenely fast as they run on the edge closest to the client and use v8 isolates to drop 95th percentile latency from cold starts from ~1.5 seconds to about 300ms.
- kentonv 8y agoHmm, cold starts for Workers depends on your script size, but should be around 10ms, never 300ms. Are you seeing 300ms? Is that actually Workers cold start time, or cold start for a larger application (e.g. that might include things like HTTP cache warm-up)? (I'm the tech lead for Workers.)
- gravypod 8y agoWould be really good to go into managed VPNs and VPC peering. These are some of the amazing things that the VPCs provide you that took me a while to figure out.
- abalone 8y agoTangential question: This guy's blog has fantastic content but I don't see an RSS feed or any other way of subscribing (apart from a much broader Twitter feed). What's the best way to keep up?
- make3 8y agofollow him on twitter : https://twitter.com/grahamlyons https://twitter.com/grahamlyons
- grahamlyons 8y agoFlattery will get you everywhere. It's a Jekyll blog so I'm sure RSS is just a plugin away. And then I'd better write some more content...
- abalone 8y agoThank you! But seriously, do you rely exclusively on Twitter to get the word out that you've posted something? I'm just so curious how people do this in the Modern Age. I am totally dependent on RSS readers to subscribe to things but I feel I am living in the past. I get that Medium tried to address the distribution problem, but not everyone posts on Medium. And I get that everyone posts things on Twitter, but I want to scream when people say just follow someone on Twitter because obviously there is a very, very high chance that I will miss if someone posted something new because Twitter is not meant to be read comprehensively.
- llama052 8y agoOff topic, but as a network guy by heart I've always been fairly happy with how AWS implements the network side of things, especially in comparison to something like Azure. AWS you have the same basic concepts of a network, and the terminology aligns enough that you can make sense of it fairly quick if you're in the network realm. Azure however takes all of that 'network' stuff and turns it into this abstraction where you have to carefully follow one of their guides to realize it's out of date, or the UI doesn't show the appropriate information etc. Also you have Azure network portions that block ICMP because of 'security'. This is all anecdotal from my experience of course, but it's why I keep referring to Azure as the "Excel spreadsheet of the cloud" because the entire design of it is in your face and non intuitive. For instance if I wanted to make a direct connection like DirectConnect to multiple VPC's in AWS, I'd use the Transit Gateway, connect to it from on-prem, add the VPC and the route, and be done. In Azure, I'd use expressroute, add the Expressroute circuit to a Subscription, add a gateway for that, and then an additional gateway for each VPC equivalent, create an authorization key for each 'VPC' equivalent and sync them, and then define routing per gateway. Then when you go in to trace the network path ICMP is blocked. I know AWS is more mature than Azure, so it's not entirely fair to criticize them, but every time I touch Azure I miss AWS, or even GCP. Perhaps it's just me not being familiar enough with Azure. ¯\_(ツ)_/¯
- filmgirlcw 8y agoMicrosoft employee here - I don’t work specifically on the networking side of Azure but this is really good feedback that I’ll share with the product teams.
- ggm 8y agoAs long as IPv6 is a second-class citizen, things are going to continue to be painful in AWS. I did the whole "here is your /56, now segment it yourself" thing. Its crude. It should not be neccessary, if V6 was central to the model, you'd be assigned /64 from your covering prefix automatically, as you deploy regional nodes.
- scubbo 8y agoSomething that's missing from this (otherwise great!) guide, that has puzzled me for a while - what's the point? What does this configuration actually gain you/AWS? My best guess is that private subnets are for DDOS protection, but that seems like something that would be better handled by throttling. Given the amount of complaints I've heard about how difficult VPC/Subnet setup is, why bother with it at all? Staving off IP address exhaustion? Or, to ask it another way - what would be the downside of all your resources being in 1 single-Subnet VPC, spread evenly across AZs?
- llama052 8y agoNot sure I'm qualified to answer all of your questions on this but, from a networking perspective.. Private subnets will allow you to reduce your exposure to the Internet, also can reduce costs with something like a NAT gateway. It's useful for things that don't need to be public facing. Generally things on the private subnet can go outbound directly but not have anything come direct into that subnet, you'd need a solution that interfaces with the public side to facilitate that, or manually create a public IP association per instance. You generally don't want one big subnet in general, it's a broadcast domain and it can be quite chatty when you get a lot of devices on it. Alongside that if you're doing multi-AZ and spanning layer-2 you end up with a lot of additional complexity to get that network to span and be highly available over multiple AZ's, while another subnet can be mostly independent. I know of some weird edge cases where you'd have to span layer-2, but if you're doing anything cloud-native you should be able to build around it.
- motive 8y agoJust as an fyi, inside Amazon's virtual network topology, there is no such thing as layer 2, and thus, no broadcast topology. Normally you'd be 100% correct in seeking to limit that bandwidth, but in Amazon everything works just a little differently.
- llama052 8y agoAh yes are right. I was thinking specifically networking, not AWS.
- simonebrunozzi 8y agoThe title says "everything you need to know about networking on AWS". I wish it were this simple. The article is well written, but it simply represents maybe 1% of what you need to know here. I would have called it "A simple introduction to networking on AWS".
- grahamlyons 8y agoI'd contest that this is everything that you need to know. Or perhaps more accurately, everything I need to know. It's certainly not intended to be exhaustive and I am definitely not a network engineer but I think you could operate at a reasonable scale within a single well laid out VPC. Of course there'll be a point for peering etc. but you might not need that.
- tambourine_man 8y agoI’ve been banging my head against the wall for a week trying to set up a site to site VPN in AWS with a Cisco ASA. The auto generated config file have a lot of missing info. If anyone knows of a good resource on the subject it would be greatly appreciated.
- mattbillenstein 8y agoIMO closed-source stuff is just impossible to get working unless you're certified with that equipment. Try OpenVPN: https://openvpn.net/vpn-server-resources/site-to-site-routing-explained-in-detail/ https://openvpn.net/vpn-server-resources/site-to-site-routin...
- iheartpotatoes 8y agoUgh! That's the most complex explanation of AWS I've ever seen. He just described a NETWORK, not AWS. AWS has renamed lots of things, but all the scary text configs that used to be the domain of wizened sysadmins have been replaced with very simple single-page-app GUI controls. LIke routers and gateways: those terms are largely gone from the AWS vocabulary. No need to get into subnets and route tables I think. The majority of clients I've worked with use AWS for web hosting with an ELB load balancer (the most important part), an EC2 instance policy & image (for handling traffic fluctuations), an RDS (database), an S3 and Route53 (external DNS entries) Point the load balancer to the outside world and then let it spin up instances. That's the most common model I've encountered. IT's almost cartoonishly simple compared to what the OP wrote here. Almost. Having an understanding of network architecture helps, but not THAT much.
- baseballMan 8y agoIt depends on how complex the environment you're working in is. If you're at a large enterprise that wants to build a platform capable of scaling to thousands of apps, you most definitely do need to care about everything written here, plus a lot more networking specific things not mentioned.
- sl1ck731 8y agoAs someone who has worked full time putting companies into AWS for almost 4 years as a cloud consultant, I've experienced only a couple occasions where your description is accurate and only for a small portion of the customer's portfolio. Even when beanstalk is used, all of the stuff mentioned in the OP and more are usually required. Simple weekend projects maybe, but only small enterprise services work in your model.
- jrockway 8y agoHaving just gone through the process of using EKS, which requires a VPC, I think the article is quite applicable to anyone doing anything of average complexity. I found myself quite often wondering things like "do I have to run an Internet Gateway in every availability zone?" (no) and "do I attach my NAT Gateway to the public subnet or the private subnet in each AZ?" (public, and then add an entry for the gateway in the private subnet). Amazon does not clearly document any of this. (I think if you read enough you'll eventually figure it out, but experimentation was the most straightforward procedure here). As for just using a single EC2 instance and RDS... that is something you can do, but not everyone's workload is so simple that they can run it on one machine. And not everyone can afford do be down simply because one AZ is down. Hence, multi-AZ VPC setups.
- rkangel 8y agoDoes anyone know of any similar resources for Google Cloud?
- wslh 8y agoI want to add a few notes useful for packet crafting. AWS, Google Cloud, and Azure don't work at layer 2 (Ethernet) as expected since they provide services at layer 3 and up. For example, if you modify the MAC destination address it will not work in AWS. To be able to do that you should disable source/destination checks as is specified in [1]. The last time I checked you cannot do that in Google Cloud or Microsoft Azure. When we experienced this issue, Reddit was the best resource for answers. I put the Reddit threads as they can help others working in projects requiring packet crafting: https://www.reddit.com/r/sysadmin/comments/51xypj/vpc_amazon_virtual_private_cloud_issues_with/ https://www.reddit.com/r/sysadmin/comments/51xypj/vpc_amazon... https://www.reddit.com/r/networking/comments/51y52n/aws_vpc_issues_with_crafted_network_packets/ https://www.reddit.com/r/networking/comments/51y52n/aws_vpc_... https://www.reddit.com/r/sysadmin/comments/533e14/google_compute_engine_issues_with_crafted_network/ https://www.reddit.com/r/sysadmin/comments/533e14/google_com... [1] https://docs.aws.amazon.com/vpc/latest/userguide/VPC_NAT_Instance.html#EIP_Disable_SrcDestCheck https://docs.aws.amazon.com/vpc/latest/userguide/VPC_NAT_Ins...
- CloudNetworking 8y agoAnd for those curious as to how/why this works this way, Azure has published papers on their virtual switch: https://www.microsoft.com/en-us/research/project/azure-virtual-filtering-platform/#!publications https://www.microsoft.com/en-us/research/project/azure-virtu...
- devonkim 8y agoThe part that irks me is that if you’re doing any VPC design that’s going to even potentially include peering you need to carefully understand the limitations first. This means that within the same region you can refer to security groups in rules as if they’re in the same VPC, but you can’t do that if you’re peering across regions. Then add in some DNS restrictions (like not being able to directly resolve a peer VPC’s entries, somewhat solvable by use of VPC private zones to serve as DNS across regions) and it can be real awkward. Then there’s overlapping VPC CIDR issues (VPC Transit Gateways can only sorta help this) The primary caveats beyond basic networks that impact designs is that multicasting is not enabled by the network layers but at the network interface (ENI) layer and you need to carefully look at how security groups really work (they’re attached to an ENI fundamentally, which is how you can route between networks with a single instance as long as it’s within the same AZ) All of this I’ve found was completely disregarded / unknown by almost every company outside the F500 or high end tech start-ups when they first started with AWS and I’ve spent a lot of my career having to migrate production environments between VPCs so that we can get enough room to grow adequately. Making subnets as small as possible is not what you should be doing in AWS, folks. In fact, making them real small means you spent a fair bit of effort which means you decided to put in a lot of effort without stopping to read the documentation in earnest for a couple hours. And using a default VPC CIDR repeatedly from the console is a pretty grand way to make sure you can never let two VPCs communicate with each other via anything other than a third intermediate VPC that you’ll have to migrate to eventually. Some of the overly-cautious networking approaches I’ve seen include making a VPC for every single application / service, using a NACL for every application (multiplied by every AZ used to isolate each subnet and cutting off cross-AZ routing thereby, of course), creating your own NAT instance that doesn’t do anything better than a NAT gateway, NAT gateways in every AZ (for a whole $1 of traffic / mo each). The story of problems in AWS infrastructure is the same - trying to plan too far ahead for the wrong things and not realizing the limitations of the right things that are not flexible anymore. This is much more common when companies hire traditionally experienced network engineers that have just a little too much confidence.
- jugg1es 8y agoPlanning your network is something everyone has to do whether or not they are using AWS. I think its unreasonable to expect to just throw stuff in AWS without thinking about it and then complain when something comes up unexpectedly.