5 ms·

Couple things: Cross-region networking. To this day, I'm flabbergasted this isn't something that AWS has. All the other networking bits seem more sensible to

Couple things: Cross-region networking. To this day, I'm flabbergasted this isn't something that AWS has. All the other networking bits seem more sensible to me as well. Another, is 2Gbps per core up to 8 cores which is way more throughput than I've seen on any AWS or Azure instance. On workloads where I care about Network IO:CPU ratio, I'm using 8 core nodes and seeing 16gbps throughput between them consistently.

ti_ranger 8y ago
> Cross-region networking. To this day, I'm flabbergasted this isn't something that AWS has. All the other networking bits seem more sensible to me as well. AWS tries to avoid letting customers create multi-region or global failure modes, like: * https://status.cloud.google.com/incident/compute/18005 https://status.cloud.google.com/incident/compute/18005 22-hour incident with GLOBAL impact resulting in many VMs getting duplicate IPs (this no network connectivity) on 2018-06-15 * https://status.cloud.google.com/incident/cloud-networking/18001 https://status.cloud.google.com/incident/cloud-networking/18... 3-hour (seemingly GLOBAL) impact to load-balancers (no updates or creations) on 2018-01-03 * https://status.cloud.google.com/incident/compute/16007 https://status.cloud.google.com/incident/compute/16007 - 18-minute GLOBAL outage on 2016-04-11 * https://status.cloud.google.com/incident/compute/15055 https://status.cloud.google.com/incident/compute/15055 - 5-minute GLOBAL outage on 2015-08-04 * https://groups.google.com/forum/#!topic/gce-operations/fynnXnb2OFs https://groups.google.com/forum/#!topic/gce-operations/fynnX... 43-minute multi-region packet loss due to multi-region configuration deployment on 2015-03-07 * https://groups.google.com/forum/#!topic/gce-operations/1uw-qEqjBdo https://groups.google.com/forum/#!topic/gce-operations/1uw-q... unscheduled reboot of 28% of instances in multiple regions in ~1.5h on 2014-09-17 There have been recent changes in many AWS services (e.g. DynamoDB, S3, Aurora) to allow cross-region replication without introducing any multi-region dependency, precisely to make it easier to implement multi-region infrastrucure to tolerate a single-region failure (however rare that is on AWS compared to global outages on GCP). > Another, is 2Gbps per core up to 8 cores which is way more throughput than I've seen on any AWS or Azure instance. Which instance types did use on AWS? Many recent instance types (r4, i3, c5, r5, z1) support up to 10Gbps with once core (2 vCPU instances). However, you may need to use placement groups ( https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placemen... ) in large regions in order to get the full throughput on low TCP connection numbers. The only reason I can think of that would explain why GCP doesn't have this problem is they don't have any regions anywhere near as large as large AWS regions ... Edit: white-space only formatting changes
- halbritt 8y ago
  Your point about global failure modes is well-taken. Even still, one could reasonably expect inter-region networking not to be so difficult. As for the instances I tested, they were m4 and r4. To get full 10GE out of either, I needed to use m4.10xl and r4.8xl, both roughly around $900 per month. By comparison an n1-standard-8 is about $60/mo and the n1-highcpu-8 is $50/mo. I've tested the former and got something like 15.8 or 15.9gbps using iperf. IMHO, that's pretty significant.
gazoakley 8y ago
You can do inter region VPC peering: https://aws.amazon.com/about-aws/whats-new/2017/11/announcing-support-for-inter-region-vpc-peering/ https://aws.amazon.com/about-aws/whats-new/2017/11/announcin...

ti_ranger 8y ago

> Cross-region networking. To this day, I'm flabbergasted this isn't something that AWS has. All the other networking bits seem more sensible to me as well. AWS tries to avoid letting customers create multi-region or global failure modes, like: * https://status.cloud.google.com/incident/compute/18005 https://status.cloud.google.com/incident/compute/18005 22-hour incident with GLOBAL impact resulting in many VMs getting duplicate IPs (this no network connectivity) on 2018-06-15 * https://status.cloud.google.com/incident/cloud-networking/18001 https://status.cloud.google.com/incident/cloud-networking/18... 3-hour (seemingly GLOBAL) impact to load-balancers (no updates or creations) on 2018-01-03 * https://status.cloud.google.com/incident/compute/16007 https://status.cloud.google.com/incident/compute/16007 - 18-minute GLOBAL outage on 2016-04-11 * https://status.cloud.google.com/incident/compute/15055 https://status.cloud.google.com/incident/compute/15055 - 5-minute GLOBAL outage on 2015-08-04 * https://groups.google.com/forum/#!topic/gce-operations/fynnXnb2OFs https://groups.google.com/forum/#!topic/gce-operations/fynnX... 43-minute multi-region packet loss due to multi-region configuration deployment on 2015-03-07 * https://groups.google.com/forum/#!topic/gce-operations/1uw-qEqjBdo https://groups.google.com/forum/#!topic/gce-operations/1uw-q... unscheduled reboot of 28% of instances in multiple regions in ~1.5h on 2014-09-17 There have been recent changes in many AWS services (e.g. DynamoDB, S3, Aurora) to allow cross-region replication without introducing any multi-region dependency, precisely to make it easier to implement multi-region infrastrucure to tolerate a single-region failure (however rare that is on AWS compared to global outages on GCP). > Another, is 2Gbps per core up to 8 cores which is way more throughput than I've seen on any AWS or Azure instance. Which instance types did use on AWS? Many recent instance types (r4, i3, c5, r5, z1) support up to 10Gbps with once core (2 vCPU instances). However, you may need to use placement groups ( https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placemen... ) in large regions in order to get the full throughput on low TCP connection numbers. The only reason I can think of that would explain why GCP doesn't have this problem is they don't have any regions anywhere near as large as large AWS regions ... Edit: white-space only formatting changes

halbritt 8y ago
Your point about global failure modes is well-taken. Even still, one could reasonably expect inter-region networking not to be so difficult. As for the instances I tested, they were m4 and r4. To get full 10GE out of either, I needed to use m4.10xl and r4.8xl, both roughly around $900 per month. By comparison an n1-standard-8 is about $60/mo and the n1-highcpu-8 is $50/mo. I've tested the former and got something like 15.8 or 15.9gbps using iperf. IMHO, that's pretty significant.

gazoakley 8y ago

You can do inter region VPC peering: https://aws.amazon.com/about-aws/whats-new/2017/11/announcing-support-for-inter-region-vpc-peering/ https://aws.amazon.com/about-aws/whats-new/2017/11/announcin...