2 ms·

The issue isn't that they needed a backup to cloudflare. The problem was they only have a single internet provider at their datacenter, so they couldn't communi

by tedivm 3y ago

The issue isn't that they needed a backup to cloudflare. The problem was they only have a single internet provider at their datacenter, so they couldn't communicate with Cloudflare. I've honestly never had a service with a single outbound path. Most datacenters where you rent colo have two or three providers as part of their network. In the cases where I've had to manage my own networking inside of a datacenter I always pick two providers in case one fails. > Work is now underway to select a provider for a second transit connection directly into our servers — either via Megaport, or from a service with their own physical presence in 365’s New Jersey datacenter. Once we have this, we will be able to directly control our outbound traffic flow and route around any network with issues. Having multiple transit options is High Availability 101 level stuff.

withinboredom 3y ago
You’d be surprised at how many things break when different routes are chosen. Like etcd, MySQL, and so much more.
- tedivm 3y ago
  Those are generally on internal networks and rarely need to communicate with the internet. They shouldn't be affected by this.
  - withinboredom 3y ago
    Twould be nice…
toast0 3y ago
> The issue isn't that they needed a backup to cloudflare. The problem was they only have a single internet provider at their datacenter, so they couldn't communicate with Cloudflare. That's not the issue. With Cloudflare MagicTransit, packets come in from Cloudflare, and egress normally. They were able to get packets from Cloudflare, but egress wasn't working to all destinations. I wasn't able to communicate with them from my CenturyLink DSL in Seattle, but when I forced a new IP that happened to be in a different /24, because I was seeing some other issues too, the fastmail issues resolved (although timing may be coincidental). Connecting via Verizon and T-Mobile, or a rented server in Seattle also worked. It's kind of a shame they don't provide services with IPv6, because if 5% of IPv4 failed and 5% of IPv6 failed, chances are good that the overall impact to users would be less than 5%, possibly much less, depending on exactly what the underlying issue was (which isn't disclosed); if it was a physical link issue, that's going to affect v4 and v6 traffic that is routed over it, but if it's a BGP announcement issue, those are often separate.

by tedivm 3y ago

withinboredom 3y ago

You’d be surprised at how many things break when different routes are chosen. Like etcd, MySQL, and so much more.

tedivm 3y ago
Those are generally on internal networks and rarely need to communicate with the internet. They shouldn't be affected by this.
- withinboredom 3y ago
  Twould be nice…

toast0 3y ago

> The issue isn't that they needed a backup to cloudflare. The problem was they only have a single internet provider at their datacenter, so they couldn't communicate with Cloudflare. That's not the issue. With Cloudflare MagicTransit, packets come in from Cloudflare, and egress normally. They were able to get packets from Cloudflare, but egress wasn't working to all destinations. I wasn't able to communicate with them from my CenturyLink DSL in Seattle, but when I forced a new IP that happened to be in a different /24, because I was seeing some other issues too, the fastmail issues resolved (although timing may be coincidental). Connecting via Verizon and T-Mobile, or a rented server in Seattle also worked. It's kind of a shame they don't provide services with IPv6, because if 5% of IPv4 failed and 5% of IPv6 failed, chances are good that the overall impact to users would be less than 5%, possibly much less, depending on exactly what the underlying issue was (which isn't disclosed); if it was a physical link issue, that's going to affect v4 and v6 traffic that is routed over it, but if it's a BGP announcement issue, those are often separate.