Years ago, I worked in support at AWS. Because of the rather peculiar way AWS accounts are linked, any company with access to AWS GovCloud had all of their acco

Years ago, I worked in support at AWS. Because of the rather peculiar way AWS accounts are linked, any company with access to AWS GovCloud had all of their accounts flagged as GovCloud accounts -- and all of their support requests were routed to my team, a team full of US citizens (most of whom either had security clearance or were pursuing it - but that's another story). Anyway, this meant we got a really, uh, interesting collection of customer support requests -- and, more important, architectures. My favorite is what I decided to dub "the InterNAT." The InterNAT involved a VPC subnet which had its routing rules configured to route all traffic via an EC2 NAT instance (these were the days before NAT gateway). All well and good... except that this VPC also had a VPN configured, which routed their traffic to an EC2 instance in a VPC in another of their accounts. That EC2 instance then routed the connections out of a NAT instance in its own VPC, before eventually connections reached the Internet. What's especially interesting about this is that EC2's network stack is designed to detect and mitigate tromboning[0], but by creating this particular architecture, they had unwittingly managed to bypass those protections. So, not only were they routing through multiple NAT instances across multiple VPCs just to reach the Internet, but they were managing to route outside of the EC2 network fabric in the process, substantially increasing network latency. Coda: Our bathrooms in the support office in Seattle had stark white walls in the stalls, which actually worked okay as whiteboards. People would doodle while taking a shit. So, of course, I diagrammed the InterNAT in a stall a few days after discovering it. Probably two years later, I saw a picture of it pop up in my Facebook feed, and had to cop to having drawn it. Bonus: My second-favorite story (albeit much shorter) was the curious case of SSL onloading. You may have heard of SSL offloading -- using a reverse proxy or load balancer to handle SSL termination, rather than doing it on-host, in order to save resources on your actual webservers. Well, I had a customer -- also in the (in)famous GovCloud queue -- who had decided to implement the reverse. Plaintext HTTP into the load balancer, but HTTPS into the webserver. I guess this is trying to protect against the case where the call is coming from inside the house? 0. https://www.catonetworks.com/blog/the-sound-of-the-trombone/ https://www.catonetworks.com/blog/the-sound-of-the-trombone/