5 ms·
Cloud bandwidth costs are a rip off. For this particular situation though, make sure to replicate your files across multiple buckets, one in each GCP zone, othe
by KenoFischer 7y ago
Cloud bandwidth costs are a rip off. For this particular situation though, make sure to replicate your files across multiple buckets, one in each GCP zone, otherwise you incur the cross zone transfer costs. We do something similar to serve julia downloads, because it turns out that most downloads are from people running on the cloud (so we basically replicate our binaries to every cloud provider and then to every region for that cloud provider). Everything else then goes through fastly for us (we also use fastly to serve custom redirects if your request comes from one of aforementioned cloud providers). That works pretty well. You do have to monitor it though to make sure that load doesn't suddenly shift to a cloud provider you didn't account for. For example, after GitHub actions became widely used, we suddenly started seeing TB/day traffic from Azure, which we hadn't deployed any caches to, so our bandwidth utilization on Fastly shot through the roof. (Side note: Shout out to fastly for hosting our binaries for free!). Without the cloud provider caching setup, we'd probably be at similar $/day costs, but this way it's basically free (even without the free fastly service, we'd only be at ~$1000/month or so).
- tedunangst 7y agoCan you describe a little more how you direct downloads to an in-zone copy?
- brentonator 7y agoI imagine using known IP ranges (https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html#aws-ip-download https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.... & https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html#subscribe-notifications https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges....) you could redirect to local resources within their region. I'd probably use Lamda/NodeJS with a ~20 minute in-memory range cache that should stay under 128MB of memory per instance. Perhaps I'd store all the range starts and ends each as two 64-bit ints on a database of your choice for persistence, indexing, and comparison. Finally, some code to convert IPv6 and IPv4 into 128-bit (2x64-bit ints) IPv6 integer space and back. A second service could listen for IP range updates avoiding any bandwidth fiasco if a new range opens up with a sudden influx of traffic.
- foota 7y agoI think they were looking for a solution where you replicate the data natively and direct to that copy for a certain region, rather than cache and serve
- KenoFischer 7y agoThe cloud providers all have public lists of their IP ranges, so there's a fastly ACL for each provider/zone pair and for matching IPs, some custom VCL serves a HTTP redirect to the HTTP url for the appropriate cloud provider's cloud storage solution (S3, GCS, etc.). On the VMs, these resolve to appropriate internal IPs to obtain data from the storage service without charge.
- zaroth 7y agoThis is good advice for only a particularly narrow use case. If your clients are likely to be on-prem in the same cloud as you, I think most cloud providers will charge you local data transfer fees (e.g. nothing) rather than egress fees, even if the request is made over the public IP address. If your users aren’t mostly in the cloud with you, than this strategy doesn’t help at all.
- KenoFischer 7y ago> If your users aren’t mostly in the cloud with you, than this strategy doesn’t help at all. Sure, but in the situation in the linked tweet 100% of the users where in the same cloud, just in the wrong region. For julia, the distribution is much more mixed, which is why we have the fastly fallback, which works as a traditional CDN. Still caching locally in each of the clouds is useful as people often download a fresh tarball of nightly julia when they CI their packages, so the load from the clouds is quite high.
- crazypython 7y agoOVH, BuyVM, Scaleway, etc provide cheap bandwith: serverhunter.com Disclaimer: serverhunter.com sponsored my game