4 ms·

Cool tool and the problem it’s trying to solve is a real one. However on the complex mesh of connections across accounts isn’t the new way to do this now not th

Cool tool and the problem it’s trying to solve is a real one. However on the complex mesh of connections across accounts isn’t the new way to do this now not the Transit Gateway? A tool that could manage complex TGW setups across many accounts would also be cool. Terraform can do that but it’s not the most elegant solution.

jcrites 7y ago
In addition to Transit Gateway, there's also AWS PrivateLink [1], which is my preferred approach for scenarios where it works. As an alternative to peering entire networks together, PrivateLink allows you to establish minimal connectivity between specific clients and services that need to communicate. When a service supports PrivateLink, any authorized client can create a private endpoint for that service in their VPC, and the endpoint receives a local IP address (in that VPC). This has a couple of benefits. The first is that you don't need to coordinate IP address ranges in different VPCs, which is necessary if you're peering them together (unless you're using NAT). IPv4 address management in large networks can be challenging. The second benefit is that although clients can communicate with the service, there isn't an actual route between the two networks. This better achieves the principle of least privilege at the network level; it avoids the security risks of connecting networks together such that any machine can connect to any other. When you design an application network to use PrivateLink from the ground up, you can start with the expectation that the VPC doesn't need any connections to any other networks, not even the Internet; it can potentially be an isolated network bubble, with only PrivateLinks allowing traffic in and out (using e.g. Session Manager for administration). There are also practical limits on how large a federated network of peered VPCs can grow (probably helped by Transit Gateway), but there isn't any limit I'm aware of on the number of client/server connections that can be established with PrivateLink. For these reasons I prefer to use PrivateLink when building systems that communicate across VPCs, falling back to network peering or Internet-facing endpoints in scenarios where it doesn't work. However, one downside compared to network peering is that you need to set up a PrivateLink connection for each service that each client needs to call, though this process can be automated during infrastructure provisioning. Services need to have specifically onboarded with PrivateLink; you can't just deploy a service on the network and have it work. [1] https://aws.amazon.com/privatelink/ https://aws.amazon.com/privatelink/
- bmcalary_atl 7y ago
  We love PrivateLink but there are limits and it needs to be provisioned on a service basis, or fronting a proxy for usage across thousands of services. :) And as you can imagine, it too would need some complex orchestration if used across thousands of VPCs or hundreds/thousands of accounts. But definitely a great tool in the toolbox.
- gregwebs 7y ago
  How do you deal with the connections being to only one availability zone?
  - jcrites 7y ago
    The connection can be present in multiple availability zones, though the service does need to decide which ones it will support at setup time. One simple approach is for the service to accept connections in many or all AZs, allowing the client to decide which ones are relevant to it. From the endpoint service documentation [1]: > For low latency and fault tolerance, we recommend using a Network Load Balancer with targets in every Availability Zone of the AWS Region. To help achieve high availability (...) you can enable cross-zone load balancing. Cross-zone load balancing enables the load balancer to distribute traffic across the registered targets in all enabled Availability Zones. For more information, see Cross-Zone Load Balancing in the User Guide for Network Load Balancers. When establishing a connection, the client can specify multiple subnets in which it would like to create an endpoint [2], and those subnets can be in any AZs supported by the service, if I recall correctly. The client can thus establish a connection that spans multiple AZ by requesting endpoints in subnets in those AZs. In most scenarios it's fine if client/service traffic can cross between AZs; I've only found it necessary to partition traffic by AZ in specialized applications (usually applications that are themselves trying to provide a zonal availability model, or that follow cell-based architecture at the zone level -- most business services don't fall into these categories, though low-level infrastructure services sometimes do). [1] https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-service.html https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-se... [2] https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateVpcEndpoint.html https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_C...
bmcalary_atl 7y ago
Yup, Transit Gateway is definitely a solution!
therockspush 7y ago
Terraform gets unruly at scale, and makes it hard to pass it off to other teams or take a vacation if you're responsible when it breaks. Aviatrix popped up on my radar recently trying to solve the operations side of this. Handles cross account TGW attachments and simplifies the routing.

jcrites 7y ago

In addition to Transit Gateway, there's also AWS PrivateLink [1], which is my preferred approach for scenarios where it works. As an alternative to peering entire networks together, PrivateLink allows you to establish minimal connectivity between specific clients and services that need to communicate. When a service supports PrivateLink, any authorized client can create a private endpoint for that service in their VPC, and the endpoint receives a local IP address (in that VPC). This has a couple of benefits. The first is that you don't need to coordinate IP address ranges in different VPCs, which is necessary if you're peering them together (unless you're using NAT). IPv4 address management in large networks can be challenging. The second benefit is that although clients can communicate with the service, there isn't an actual route between the two networks. This better achieves the principle of least privilege at the network level; it avoids the security risks of connecting networks together such that any machine can connect to any other. When you design an application network to use PrivateLink from the ground up, you can start with the expectation that the VPC doesn't need any connections to any other networks, not even the Internet; it can potentially be an isolated network bubble, with only PrivateLinks allowing traffic in and out (using e.g. Session Manager for administration). There are also practical limits on how large a federated network of peered VPCs can grow (probably helped by Transit Gateway), but there isn't any limit I'm aware of on the number of client/server connections that can be established with PrivateLink. For these reasons I prefer to use PrivateLink when building systems that communicate across VPCs, falling back to network peering or Internet-facing endpoints in scenarios where it doesn't work. However, one downside compared to network peering is that you need to set up a PrivateLink connection for each service that each client needs to call, though this process can be automated during infrastructure provisioning. Services need to have specifically onboarded with PrivateLink; you can't just deploy a service on the network and have it work. [1] https://aws.amazon.com/privatelink/ https://aws.amazon.com/privatelink/

bmcalary_atl 7y ago
We love PrivateLink but there are limits and it needs to be provisioned on a service basis, or fronting a proxy for usage across thousands of services. :) And as you can imagine, it too would need some complex orchestration if used across thousands of VPCs or hundreds/thousands of accounts. But definitely a great tool in the toolbox.
gregwebs 7y ago
How do you deal with the connections being to only one availability zone?
- jcrites 7y ago
  The connection can be present in multiple availability zones, though the service does need to decide which ones it will support at setup time. One simple approach is for the service to accept connections in many or all AZs, allowing the client to decide which ones are relevant to it. From the endpoint service documentation [1]: > For low latency and fault tolerance, we recommend using a Network Load Balancer with targets in every Availability Zone of the AWS Region. To help achieve high availability (...) you can enable cross-zone load balancing. Cross-zone load balancing enables the load balancer to distribute traffic across the registered targets in all enabled Availability Zones. For more information, see Cross-Zone Load Balancing in the User Guide for Network Load Balancers. When establishing a connection, the client can specify multiple subnets in which it would like to create an endpoint [2], and those subnets can be in any AZs supported by the service, if I recall correctly. The client can thus establish a connection that spans multiple AZ by requesting endpoints in subnets in those AZs. In most scenarios it's fine if client/service traffic can cross between AZs; I've only found it necessary to partition traffic by AZ in specialized applications (usually applications that are themselves trying to provide a zonal availability model, or that follow cell-based architecture at the zone level -- most business services don't fall into these categories, though low-level infrastructure services sometimes do). [1] https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-service.html https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-se... [2] https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateVpcEndpoint.html https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_C...

bmcalary_atl 7y ago

Yup, Transit Gateway is definitely a solution!

therockspush 7y ago

Terraform gets unruly at scale, and makes it hard to pass it off to other teams or take a vacation if you're responsible when it breaks. Aviatrix popped up on my radar recently trying to solve the operations side of this. Handles cross account TGW attachments and simplifies the routing.