11 ms·

Why the heck would you send traffic out transit links using GRE tunnels, when it's more cost effective and you have more control over your own private backbone

by hhw 10y ago

Why the heck would you send traffic out transit links using GRE tunnels, when it's more cost effective and you have more control over your own private backbone links? At the scale I speculated 1/10th of what you're suggesting, it would have already been cost effective to operate a backbone. If they're anywhere near the scale you're suggesting, then it should be a no brainer for them to operate their own backbone of 100Gb waves. I'm quite skeptical of the numbers you're citing though. Their PeeringDB profile suggests they only have 10-20Gbps of peering per city. Considering their profile was updated just a few days ago, I would be inclined to consider those listed capacities accurate. Although they mention 'hundreds of Gbps' of capacity per city, they also mention sending up to 50% of traffic through peering in London, where they only have 50Gbps of total peering capacity. Perhaps you're right on that 300Gbps of capacity per location, in which case they would run much lower utilization rates on transits than peers. But that would be even worse allocation of spending than in my initial assessment, considering the cost of a port at an exchange is much cheaper than a transit link with CDR. It also leaves them highly vulnerable to DDoS attacks through exchanges. For a content network, public peering at exchanges in North America just with route servers and networks that have open policies would result in 30-40% of traffic going through the exchange. They would easily do more traffic at the exchange than any one transit in a mix of 3-4, let alone in a mix of 5-7. With any significant private peering, easily 60% of traffic could be settlement free. And guess what, most significant peers require peering at multiple locations with a full set of prefixes, which requires you have a backbone. With the traffic levels you're suggesting they run, they should be able to negotiate settlement free peering with many major regional Tier 2's, making it even less sensible to be purchasing from multiple ones. Considering most Tier 2's within a given region will all peer with each other, there are very few improvements to be had by turning up additional ones. Where there's the most room for improvement is being on the right long haul fiber paths. In which case, given their North American focus, they should be buying from Level3 and they probably could at comparable rates to their current agreements by concentrating more of their commits at fewer providers. If they had their own transport, they could also determine which fiber paths they take across their backbone to ensure optimal latency. Beyond that, given that Tier 1's all peer with each other by definition, it's just a matter of dumping traffic out any one of them for local traffic without traversing a congested peering link. The microseconds it takes to go an extra AS hop within a city has indistinguishable impact on performance. I'm not sure what best practices you're referring to. Who else can you name that utilizes up to 7 transit providers in a given city, without operating their own backbone? The only one that I can personally think of Internap, when they abandoned building their own backbone halfway through turning it up. Ask their former network engineers, from their golden years when they had their highest market share, how that worked for their network and for their business. Are you a current Linode customer in one or more locations? If you were, you'd probably have experienced packet loss issues on a regular basis due to DDoS attacks. There's a reason why they're performing these network upgrades; they've had near daily network interruptions due to DDoS attacks since Christmas of last year, with some outages lasting up to almost a day. Smart network engineering would have never let their network become that unreliable in the first place. And if you were going to blame a lack of budget for that, my suggestions would be even more appropriate for them as they would them to scale their network in a much more cost effective way. An external scrubbing service makes sense, when they've been ineffective at mitigating attacks to date. Your network is only as resilient towards DDoS attacks as your weakest links, and spreading out capacity to a larger number of providers instead of concentrating higher capacities with fewer ones makes it much easier to saturate connectivity to one of them. The only way Linode's current network strategy makes sense, assuming that it's not due to technical oversight, is if it's marketing driven. That's a fair reason, but it should also be fair to call them out on it. I'm not sure why you feel that strategy is in any way optimal, when it's the opposite of the models of hosting companies most renowned for their networks. Take for example SoftLayer, who went to great lengths to build out their own backbone fairly early on. I may halfheartedly agree that Linode's network upgrade strategy might be smart marketing, but I would wholeheartedly disagree that it's smart network engineering. It's not cost effective, is sub-optimal for resiliency against attacks, and fails to leverage peering effectively.

alexforster 10y ago
Welp, there's a lot that needs addressing in this comment, but I'll stick to the heavy-hitters. > they've had near daily network interruptions due to DDoS attacks since Christmas of last year, with some outages lasting up to almost a day Whoa, nope[1]. > It also leaves them highly vulnerable to DDoS attacks through exchanges. We aggressively de-peer with networks that regularly originate attack traffic, allowing us to size our ports according to utilization rather than worst case attack sizes. Multilateral peering is actually a bit more expensive than transit these days - much more expensive if you intend to significantly overprovision capacity. > Who else can you name that utilizes up to 7 transit providers in a given city That's fair, but missing context. We're figuring out who works best for our traffic profile. We will scale back/remove the underperformers and scale up those that prove their worth. > It's not cost effective, is sub-optimal for resiliency against attacks, and fails to leverage peering effectively. All of this is dramatically incorrect. [1] https://cloudharmony.com/status https://cloudharmony.com/status
- hhw 10y ago
  > Whoa, nope[1]. Perhaps fewer issues in the last month, but your own status page shows a litany of issues in September and earlier months. And I am fairly certain your status page has not covered every instance of attacks / packet loss on your network(s). > We aggressively de-peer with networks that regularly originate attack traffic, allowing us to size our ports according to utilization rather than worst case attack sizes. The Internet is far from perfect, and every network of sufficient scale is going to have a number of compromised machines on it, especially eyeball networks, who should be your most desirable peers. Rather than blanket de-peering such networks, you should be looking at PNI's with them. True, there are a number of 'bulletproof' networks, but they are few and far between. > Multilateral peering is actually a bit more expensive than transit these days - much more expensive if you intend to significantly overprovision capacity. That is simply not true either. Even if taking the worst case of leasing waves across 3 major long-haul routes across the continent, you're looking at about $0.20/Mb. 100Gb waves have gotten cheap in the last year, almost cut in half of what they were not too long ago. Unlike transit though, you get full use of both directions, so your effective rate as a content heavy network is going to be closer to $0.10/Mb. And this is just a small minority of your traffic; most peering traffic is local and much of it would travel much shorter distances as the majority of non-local destinations aren't going to be at extreme ends of the continent. And at the end of the day, you can simply limit not accepting peering routes in a given city from other cities that are too far away, and push those out through local transit instead if you prefer. Some exchanges are even one-time fee only, and most others are priced quite reasonably. You peer with networks available on those exchanges first, and only peer with networks at the more expensive exchanges if they're exclusively there. Believe it or not, other networks like to save money too, and those worth peering with are usually at the more cost effective exchanges. > That's fair, but missing context. We're figuring out who works best for our traffic profile. We will scale back/remove the underperformers and scale up those that prove their worth. That's a very expensive way to do things, considering minimum 1 year terms for most providers. Would've been significantly cheaper for you to hire some consultants with extensive experience working with the networks you were considering. Heck, you could have just sent some of the more outgoing members of your network team to a NANOG and gathered feedback for free over a few vendor sponsored beers. It's also not exactly rocket science to figure out which Tier 1's are strongest in which corner of the world. At the end of the day, you're not dependent on their support if you're multi-homing (and shouldn't be because most of them are terrible). > All of this is dramatically incorrect. See points above.

by hhw 10y ago

alexforster 10y ago

Welp, there's a lot that needs addressing in this comment, but I'll stick to the heavy-hitters. > they've had near daily network interruptions due to DDoS attacks since Christmas of last year, with some outages lasting up to almost a day Whoa, nope[1]. > It also leaves them highly vulnerable to DDoS attacks through exchanges. We aggressively de-peer with networks that regularly originate attack traffic, allowing us to size our ports according to utilization rather than worst case attack sizes. Multilateral peering is actually a bit more expensive than transit these days - much more expensive if you intend to significantly overprovision capacity. > Who else can you name that utilizes up to 7 transit providers in a given city That's fair, but missing context. We're figuring out who works best for our traffic profile. We will scale back/remove the underperformers and scale up those that prove their worth. > It's not cost effective, is sub-optimal for resiliency against attacks, and fails to leverage peering effectively. All of this is dramatically incorrect. [1] https://cloudharmony.com/status https://cloudharmony.com/status

hhw 10y ago
> Whoa, nope[1]. Perhaps fewer issues in the last month, but your own status page shows a litany of issues in September and earlier months. And I am fairly certain your status page has not covered every instance of attacks / packet loss on your network(s). > We aggressively de-peer with networks that regularly originate attack traffic, allowing us to size our ports according to utilization rather than worst case attack sizes. The Internet is far from perfect, and every network of sufficient scale is going to have a number of compromised machines on it, especially eyeball networks, who should be your most desirable peers. Rather than blanket de-peering such networks, you should be looking at PNI's with them. True, there are a number of 'bulletproof' networks, but they are few and far between. > Multilateral peering is actually a bit more expensive than transit these days - much more expensive if you intend to significantly overprovision capacity. That is simply not true either. Even if taking the worst case of leasing waves across 3 major long-haul routes across the continent, you're looking at about $0.20/Mb. 100Gb waves have gotten cheap in the last year, almost cut in half of what they were not too long ago. Unlike transit though, you get full use of both directions, so your effective rate as a content heavy network is going to be closer to $0.10/Mb. And this is just a small minority of your traffic; most peering traffic is local and much of it would travel much shorter distances as the majority of non-local destinations aren't going to be at extreme ends of the continent. And at the end of the day, you can simply limit not accepting peering routes in a given city from other cities that are too far away, and push those out through local transit instead if you prefer. Some exchanges are even one-time fee only, and most others are priced quite reasonably. You peer with networks available on those exchanges first, and only peer with networks at the more expensive exchanges if they're exclusively there. Believe it or not, other networks like to save money too, and those worth peering with are usually at the more cost effective exchanges. > That's fair, but missing context. We're figuring out who works best for our traffic profile. We will scale back/remove the underperformers and scale up those that prove their worth. That's a very expensive way to do things, considering minimum 1 year terms for most providers. Would've been significantly cheaper for you to hire some consultants with extensive experience working with the networks you were considering. Heck, you could have just sent some of the more outgoing members of your network team to a NANOG and gathered feedback for free over a few vendor sponsored beers. It's also not exactly rocket science to figure out which Tier 1's are strongest in which corner of the world. At the end of the day, you're not dependent on their support if you're multi-homing (and shouldn't be because most of them are terrible). > All of this is dramatically incorrect. See points above.