3 ms·

Correct me if I am wrong: fly's anycast has its limitations compared to Global Accelerator (GA) though: On ocassion, it breaks UDP protocols that are "connecti

Correct me if I am wrong: fly's anycast has its limitations compared to Global Accelerator (GA) though: On ocassion, it breaks UDP protocols that are "connection oriented" (like QUIC and WireGuard, though both have built-in capabilities to recover). There is no way to pin traffic to VMs (route / client affinities) or shape traffic. 100+ locations with GA, and two Anycast IPs (in two distinct "zones"). --- Alternatives to Fly on AWS that I know of: Anycast on AWS without global accelerator: S3 buckets with transfer acceleration); (edge optimized) API Gateway to Lamba / Fargate; S3 + CloudFront. AWS AppRunner + Copilot (which are comparable to Fly + Flyctl) can be geo-routed to nearest instance by DNS-based load balancing with Route53 (not anycast specifically). --- Fly's killer feature (and why we are transitioning to it) is its cost-effectiveness and almost 'zero-devops' setup. - Super cheap bandwidth ($2 per 100GB!) - Free deploys (AppRunner charges for deploys) - Free monitoring (versus expensive but comprehensive CloudWatch) - Free orchestration - Free anycast transit (expensive on aws) - Cheaper, zero-touch, global/cross-region private-network across VMs in the same fly org (zero-muck: transit gateway, nat gateway, internet gateway, vpc private endpoints, iam policies...). - Super cheap and fast disks ($0.15 per GB/disk!) - Easier HA Postgres and HA Redis setups. - GA's TCP proxy does not preserve source/client ip-ports (Fly can).

sudhirj 5y ago
For the most part, yes, everything makes sense. There are some things worth noting though: > On ocassion, it breaks UDP protocols that are "connection oriented" (like QUIC and WireGuard, though both have built-in capabilities to recover). Yes, and no, in that QUIC and WireGuard do work consistently, it's not that they break. But Fly doesn't currently offer UDP flow hashing or sessions or pinning. > There is no way to pin traffic to VMs (route / client affinities) or shape traffic. No, but the system is built to obviate the need for this — you can choose which regions your app runs in and Fly will balance them for you based on the strategy you choose. I'm not sure what benefit is being missed out on by not having it — if there is a clear benefit that's not achievable under the current design we can make a case for building it. > 100+ locations with GA, and two Anycast IPs (in two distinct "zones"). Fly lets you allocate and buy more IPs under Anycast, so more than two should be possible. Regarding the 100+ locations, that's technically true but irrelevant — GA doesn't serve requests, so they still need to hit apps deployed on one of the AWS regions (usually provisioned and managed separately). With Fly your app is running on the edge regions pretty much automatically. The closest alternative to Fly on AWS would be (1) Global Accelerator pointing at (20) Application Load Balancers in every region, each backed with (1+) Fargate containers maybe? Would also need Lambda triggers to turn off traffic to a region if it wasn't worth running containers there, and turn them back on again.
- rstupek 5y ago
  Could one run caddy and serve their own TLS termination?
  - sudhirj 5y ago
    Yes, it’s possible to ask for the raw connection to be passed to the application and self-manage TLS.
NathanKP 5y ago
AWS engineer from the container services team here. One small point of clarification on "AppRunner charges for deploys": We only charge for deploys if you are using App Runner's built-in integration to watch your repo, and automatically build/rebuild your container image from source code. This is not a required feature for App Runner to function though. For example if you are using Copilot with App Runner you can drive the build and release to App Runner from your local dev machine, so there is no extra deployment charge beyond what it costs you for electricity to build your container on your own laptop. You only get charged for App Runner deployment automation when you are using AppRunner as a Github Actions / Jenkins replacement to do the Docker image build on the server side.
trevyn 5y ago
>Super cheap bandwidth ($2 per 100GB!) That’s only “super cheap” if you’re comparing to AWS’s outright highway robbery bandwidth pricing.
- mwcampbell 5y ago
  Sure, there are lots of VPS and dedicated server providers that offer lots of bandwidth, but they're not playing the same game as the big cloud providers, or even fly.io, when it comes to auto-scaling, self-healing, multi-data-center deployments.

sudhirj 5y ago

For the most part, yes, everything makes sense. There are some things worth noting though: > On ocassion, it breaks UDP protocols that are "connection oriented" (like QUIC and WireGuard, though both have built-in capabilities to recover). Yes, and no, in that QUIC and WireGuard do work consistently, it's not that they break. But Fly doesn't currently offer UDP flow hashing or sessions or pinning. > There is no way to pin traffic to VMs (route / client affinities) or shape traffic. No, but the system is built to obviate the need for this — you can choose which regions your app runs in and Fly will balance them for you based on the strategy you choose. I'm not sure what benefit is being missed out on by not having it — if there is a clear benefit that's not achievable under the current design we can make a case for building it. > 100+ locations with GA, and two Anycast IPs (in two distinct "zones"). Fly lets you allocate and buy more IPs under Anycast, so more than two should be possible. Regarding the 100+ locations, that's technically true but irrelevant — GA doesn't serve requests, so they still need to hit apps deployed on one of the AWS regions (usually provisioned and managed separately). With Fly your app is running on the edge regions pretty much automatically. The closest alternative to Fly on AWS would be (1) Global Accelerator pointing at (20) Application Load Balancers in every region, each backed with (1+) Fargate containers maybe? Would also need Lambda triggers to turn off traffic to a region if it wasn't worth running containers there, and turn them back on again.

rstupek 5y ago
Could one run caddy and serve their own TLS termination?
- sudhirj 5y ago
  Yes, it’s possible to ask for the raw connection to be passed to the application and self-manage TLS.

NathanKP 5y ago

AWS engineer from the container services team here. One small point of clarification on "AppRunner charges for deploys": We only charge for deploys if you are using App Runner's built-in integration to watch your repo, and automatically build/rebuild your container image from source code. This is not a required feature for App Runner to function though. For example if you are using Copilot with App Runner you can drive the build and release to App Runner from your local dev machine, so there is no extra deployment charge beyond what it costs you for electricity to build your container on your own laptop. You only get charged for App Runner deployment automation when you are using AppRunner as a Github Actions / Jenkins replacement to do the Docker image build on the server side.

trevyn 5y ago

>Super cheap bandwidth ($2 per 100GB!) That’s only “super cheap” if you’re comparing to AWS’s outright highway robbery bandwidth pricing.

mwcampbell 5y ago
Sure, there are lots of VPS and dedicated server providers that offer lots of bandwidth, but they're not playing the same game as the big cloud providers, or even fly.io, when it comes to auto-scaling, self-healing, multi-data-center deployments.