6 ms·

Network Update: Multihomed, Increased Transit, Peering

82 points by stenius 10y ago

pjungwir 10y ago
> per-customer VLANs I am looking forward to that! Linode is my go-to hosting service, but it's a little troubling that anyone in the datacenter can hit your private IPs [1]. On the other hand, maybe it shouldn't matter, and you should always act like the network is compromised. Isn't trusting their private network how Google leaked traffic to the NSA? Still, it seems like a nice improvement that would make compromises less likely. [1] https://blog.linode.com/2008/03/14/private-back-end-network-support/ https://blog.linode.com/2008/03/14/private-back-end-network-...
- dimfeld 10y ago
  Agreed. Back when I was using Linode I had a crazy setup that used iptables on each server to accept traffic only from my other servers. With benefit of hindsight I might have maintained it differently, but needless to say, it was a bit fragile.
- misframer 10y ago
  > On the other hand, maybe it shouldn't matter, and you should always act like the network is compromised. What about cases like AWS's VPCs?
  - vgt 10y ago
    To add color to both your comment and parent's. Everything at Google Cloud is encrypted at rest and in transit [0]. Any GCE project is essentially a VPC by default, and a global one at that [1] (aka no need to VPN between regions). Traffic between GCE zones/regions never hits public wire by default ,and Google will carry your packet to the nearest Google POP around the world on its private backbone [2]. (work at Google Cloud, but not on networking/GCE) [0] https://cloud.google.com/security/encryption-at-rest/ https://cloud.google.com/security/encryption-at-rest/ [1] https://cloud.google.com/docs/compare/aws/ https://cloud.google.com/docs/compare/aws/ [2] http://peering.google.com/#/infrastructure http://peering.google.com/#/infrastructure
    - nodesocket 10y ago
      Nice, great information. Google Cloud has the networking model right.
    - bogomipz 10y ago
      Whats the data at rest model? LUKs? Edit. Never mind I see the link.
StanAngeloff 10y ago
This is good news for their users, incl. us given the frequency of DDoS attacks lately. There has been hardly a month go by without their status page flagging an incident report involving increased traffic to one of their datacentres as a result of a DDoS attack.
swalsh 10y ago
I guess it never occurred to me before, but with the increased amounts of attacks lately it's been near the top of my mind. These guys seem to be throwing around the physical addresses of data centers pretty freely. What is the security of these places like? How decentralized are we really? It seems like a few strategic strikes could deal a devastating blow to our edge infrastructure. I know personally, my servers are only hosted in a single datacenter. The company I work for is in 3 datacenters, but I'm not sure the other 2 data centers could handle the full load for an extended period of time if the primary one was completely taken down. Granted not as big of deal as power plants etc, but if you're looking for soft targets, it's a scary thought.
- misframer 10y ago
  > It seems like a few strategic strikes could deal a devastating blow to our edge infrastructure. Imagine a large meteorite impact :).
- jlgaddis 10y ago
  These are all very well-known datacenters with several layers of physical security (there are standards and certifications for datacenters). Their locations aren't exactly secret. Most datacenters have fences/gates, require access cards and/or biometrics to get in and move around inside the building. Once inside, you can only get into your own cages. It's not like you can walk up, knock the door in with a battering ram, and then have access to everything inside.
  - aroch 10y ago
    >It's not like you can walk up, knock the door in with a battering ram, and then have access to everything inside. Well, I mean, yes you can. The actual doors/gates used aren't 'milspec' intrusion rated. They're `better-than-home-depot` doors (all steel, steel door frames, reinforced). Cage door are often hilarious flimsy (thing metal sheet/bars). Certainly breachable by even modestly equipped attackers. The reason you pick real DCs and not the basement of your fortified house is because there's human security in addition to the physical security measures. Which means, some one will notice if you try to bust down the door
neom 10y ago
Reminds me of DigitalOcean in 2015. Curious what a "per-customer VLAN" is in reality.
- VLM 10y ago
  Maybe one way to describe MPLS is its kinda a VPN for VLANs. Or a way to put VLANs in something like a VLAN sorta. I can't speak for them but I worked at what boils down to a semi competitor a decade ago doing network stuff. MPLS is old stuff now and you can google the specific cisco model numbers and MPLS if you'd like to read configuration guides. Superficially only having 4096 VLAN labels on an ethernet connection appears to be a big problem if you have more than 4096 customers. However MPLS label space is 20 bits so you're good to a million customers. Then you have some "fun" mapping games such that your router connects traffic on MPLS label 123456 (which is your customer number) to local ethernet interface port wtf on vlan 100 or whatever you have been given. At least that would have been cutting edge a decade ago and probably still is today. Its unlikely to be any more, or any less, secure than anything else in a virtualized cloudy environment.
  - jlgaddis 10y ago
    > Or a way to put VLANs in something like a VLAN sorta. 802.1ad, a.k.a. "Q-in-Q" [0] > However MPLS label space is 20 bits so you're good to a million customers. VXLAN, cf. RFC 7348 [1], is the latest coolness, allowing for up to 16M "virtual" networks (using 24 bits) and bridging layer 2 over IP (4789/UDP). [0]: https://en.m.wikipedia.org/wiki/IEEE_802.1ad https://en.m.wikipedia.org/wiki/IEEE_802.1ad [1]: https://tools.ietf.org/html/rfc7348 https://tools.ietf.org/html/rfc7348
secure 10y ago
I appreciate how transparent they are about their locations, transit and peering. I’m looking to replace one of my VPS at digitalocean because of stability issues (need to reboot the VM every couple of months, it just entirely drops off the network apparently). Linode seems like a good alternative. My criteria for this application are ≥ 1G of RAM, SSD storage, fast RTT to my other VPS, native IPv6 support.
geuis 10y ago
I fucking love Linode. I've been hosting with them for years and over time I've gotten more performance and more data transfer for the same money. https://jsonip.com https://jsonip.com is hosted with Linode and supports millions of requests a day. It's been a great home for the service.
- grubles 10y ago
  Just don't store cryptocurrency there: https://news.ycombinator.com/item?id=3655137 https://news.ycombinator.com/item?id=3655137
  - tbrownaw 10y ago
    Don't store it on any cloud service.
vetrom 10y ago
Are they still running a hard-to-audit ColdFusion CMS?
- ksec 10y ago
  Wanted to know that as well. They were making a major rewrite. Not sure if it is finished yet.
hhw 10y ago
"we now manage our own true service provider network, allowing us to deliver robust and reliable connectivity." What's needed to combat DDoS attacks is distributed defense. Without their own backbone / private transport links between all of their locations, their network is just a disparate set of data centres and there is no advantage to their having multiple locations, so far as protection from DDoS attacks are concerned. They also fail to mention what capacity each of the links are. They could be anywhere from 1Gbps to 100Gbps, but I presume they'd mention as a selling point anything 40Gbps and up, so let's assume they're using all 10Gbps links and not 1Gbps to give them the benefit of the doubt. So, they range from 50Gbps (Singapore) to 100Gbps (London) per location. It's an impressive list to look at in aggregate, but not really that much for any one location in 2016, especially given a company of their size and visibility, when you can rent shared access to a 200Gbps+ botnet for $19.99. https://www.nanog.org/sites/default/files/20161015_Winward_The_Current_Economics_v1.pdf https://www.nanog.org/sites/default/files/20161015_Winward_T... Instead of buying transit from up to 7 carriers per location, when there are starkly diminishing returns after 3 or 4 so far as routing performance is concerned, they should have instead bought higher capacity to each provider (to ensure at least 10Gbps of unused capacity per provider outside of regular legitimate traffic), external DDoS mitigation, or domestic backbone links and turned up more capacity at the LA Any2 (for Asia) and NYIIX (for Europe) to absorb the majority of DDoS traffic which comes from those regions. With up to 7 carriers, they simply have 7x different points of failure each at only 10Gbps, while getting worse deals on transit pricing due to lower volumes with each provider.
- dsl 10y ago
  You don't need a private backbone to be able to mitigate attacks across multiple locations. I've done it fighting off multi-hundred Gbps attacks and it was never an issue. You can QoS your own intra-site GRE tunnels. Linode is moving 200-300 Gbps globally. That is about 37.5 Gbps per location, and when you figure in a 20% utilization (because you need to be able to burst)... they have about 300 Gbps of transit per location. Spread across 3-5 carriers I would guess they have 40-100 Gbps from each. Way more than your estimated 10 Gbps. As far as "routing performance" they appears to be buying from a few Tier 1 networks per location, and a mix of regional Tier 2s. That is in line with best practices. Sometimes to reach the right networks you do need to spin up circuits with multiple Tier 2s, there is no such thing as "diminishing returns" if you are doing traffic engineering properly. The right way to build networks is to meet your performance needs first and foremost, have enough headroom to grow and serve your customers, and work with your upstreams to manage incoming attacks. An external scrubbing service makes no sense when you can adapt your network as Linode has done so they can easily blackhole targets at their upsteams edge. I applaud their efforts. This is some smart network engineering.

82 points by stenius 10y ago

> per-customer VLANs I am looking forward to that! Linode is my go-to hosting service, but it's a little troubling that anyone in the datacenter can hit your private IPs [1]. On the other hand, maybe it shouldn't matter, and you should always act like the network is compromised. Isn't trusting their private network how Google leaked traffic to the NSA? Still, it seems like a nice improvement that would make compromises less likely. [1] https://blog.linode.com/2008/03/14/private-back-end-network-support/ https://blog.linode.com/2008/03/14/private-back-end-network-...

dimfeld 10y ago
Agreed. Back when I was using Linode I had a crazy setup that used iptables on each server to accept traffic only from my other servers. With benefit of hindsight I might have maintained it differently, but needless to say, it was a bit fragile.
misframer 10y ago
> On the other hand, maybe it shouldn't matter, and you should always act like the network is compromised. What about cases like AWS's VPCs?
- vgt 10y ago
  To add color to both your comment and parent's. Everything at Google Cloud is encrypted at rest and in transit [0]. Any GCE project is essentially a VPC by default, and a global one at that [1] (aka no need to VPN between regions). Traffic between GCE zones/regions never hits public wire by default ,and Google will carry your packet to the nearest Google POP around the world on its private backbone [2]. (work at Google Cloud, but not on networking/GCE) [0] https://cloud.google.com/security/encryption-at-rest/ https://cloud.google.com/security/encryption-at-rest/ [1] https://cloud.google.com/docs/compare/aws/ https://cloud.google.com/docs/compare/aws/ [2] http://peering.google.com/#/infrastructure http://peering.google.com/#/infrastructure
  - nodesocket 10y ago
    Nice, great information. Google Cloud has the networking model right.
  - bogomipz 10y ago
    Whats the data at rest model? LUKs? Edit. Never mind I see the link.

StanAngeloff 10y ago

This is good news for their users, incl. us given the frequency of DDoS attacks lately. There has been hardly a month go by without their status page flagging an incident report involving increased traffic to one of their datacentres as a result of a DDoS attack.

swalsh 10y ago

I guess it never occurred to me before, but with the increased amounts of attacks lately it's been near the top of my mind. These guys seem to be throwing around the physical addresses of data centers pretty freely. What is the security of these places like? How decentralized are we really? It seems like a few strategic strikes could deal a devastating blow to our edge infrastructure. I know personally, my servers are only hosted in a single datacenter. The company I work for is in 3 datacenters, but I'm not sure the other 2 data centers could handle the full load for an extended period of time if the primary one was completely taken down. Granted not as big of deal as power plants etc, but if you're looking for soft targets, it's a scary thought.

misframer 10y ago
> It seems like a few strategic strikes could deal a devastating blow to our edge infrastructure. Imagine a large meteorite impact :).
jlgaddis 10y ago
These are all very well-known datacenters with several layers of physical security (there are standards and certifications for datacenters). Their locations aren't exactly secret. Most datacenters have fences/gates, require access cards and/or biometrics to get in and move around inside the building. Once inside, you can only get into your own cages. It's not like you can walk up, knock the door in with a battering ram, and then have access to everything inside.
- aroch 10y ago
  >It's not like you can walk up, knock the door in with a battering ram, and then have access to everything inside. Well, I mean, yes you can. The actual doors/gates used aren't 'milspec' intrusion rated. They're `better-than-home-depot` doors (all steel, steel door frames, reinforced). Cage door are often hilarious flimsy (thing metal sheet/bars). Certainly breachable by even modestly equipped attackers. The reason you pick real DCs and not the basement of your fortified house is because there's human security in addition to the physical security measures. Which means, some one will notice if you try to bust down the door

neom 10y ago

Reminds me of DigitalOcean in 2015. Curious what a "per-customer VLAN" is in reality.

VLM 10y ago
Maybe one way to describe MPLS is its kinda a VPN for VLANs. Or a way to put VLANs in something like a VLAN sorta. I can't speak for them but I worked at what boils down to a semi competitor a decade ago doing network stuff. MPLS is old stuff now and you can google the specific cisco model numbers and MPLS if you'd like to read configuration guides. Superficially only having 4096 VLAN labels on an ethernet connection appears to be a big problem if you have more than 4096 customers. However MPLS label space is 20 bits so you're good to a million customers. Then you have some "fun" mapping games such that your router connects traffic on MPLS label 123456 (which is your customer number) to local ethernet interface port wtf on vlan 100 or whatever you have been given. At least that would have been cutting edge a decade ago and probably still is today. Its unlikely to be any more, or any less, secure than anything else in a virtualized cloudy environment.
- jlgaddis 10y ago
  > Or a way to put VLANs in something like a VLAN sorta. 802.1ad, a.k.a. "Q-in-Q" [0] > However MPLS label space is 20 bits so you're good to a million customers. VXLAN, cf. RFC 7348 [1], is the latest coolness, allowing for up to 16M "virtual" networks (using 24 bits) and bridging layer 2 over IP (4789/UDP). [0]: https://en.m.wikipedia.org/wiki/IEEE_802.1ad https://en.m.wikipedia.org/wiki/IEEE_802.1ad [1]: https://tools.ietf.org/html/rfc7348 https://tools.ietf.org/html/rfc7348

secure 10y ago

I appreciate how transparent they are about their locations, transit and peering. I’m looking to replace one of my VPS at digitalocean because of stability issues (need to reboot the VM every couple of months, it just entirely drops off the network apparently). Linode seems like a good alternative. My criteria for this application are ≥ 1G of RAM, SSD storage, fast RTT to my other VPS, native IPv6 support.

geuis 10y ago

I fucking love Linode. I've been hosting with them for years and over time I've gotten more performance and more data transfer for the same money. https://jsonip.com https://jsonip.com is hosted with Linode and supports millions of requests a day. It's been a great home for the service.

grubles 10y ago
Just don't store cryptocurrency there: https://news.ycombinator.com/item?id=3655137 https://news.ycombinator.com/item?id=3655137
- tbrownaw 10y ago
  Don't store it on any cloud service.

vetrom 10y ago

Are they still running a hard-to-audit ColdFusion CMS?

ksec 10y ago
Wanted to know that as well. They were making a major rewrite. Not sure if it is finished yet.

hhw 10y ago

"we now manage our own true service provider network, allowing us to deliver robust and reliable connectivity." What's needed to combat DDoS attacks is distributed defense. Without their own backbone / private transport links between all of their locations, their network is just a disparate set of data centres and there is no advantage to their having multiple locations, so far as protection from DDoS attacks are concerned. They also fail to mention what capacity each of the links are. They could be anywhere from 1Gbps to 100Gbps, but I presume they'd mention as a selling point anything 40Gbps and up, so let's assume they're using all 10Gbps links and not 1Gbps to give them the benefit of the doubt. So, they range from 50Gbps (Singapore) to 100Gbps (London) per location. It's an impressive list to look at in aggregate, but not really that much for any one location in 2016, especially given a company of their size and visibility, when you can rent shared access to a 200Gbps+ botnet for $19.99. https://www.nanog.org/sites/default/files/20161015_Winward_The_Current_Economics_v1.pdf https://www.nanog.org/sites/default/files/20161015_Winward_T... Instead of buying transit from up to 7 carriers per location, when there are starkly diminishing returns after 3 or 4 so far as routing performance is concerned, they should have instead bought higher capacity to each provider (to ensure at least 10Gbps of unused capacity per provider outside of regular legitimate traffic), external DDoS mitigation, or domestic backbone links and turned up more capacity at the LA Any2 (for Asia) and NYIIX (for Europe) to absorb the majority of DDoS traffic which comes from those regions. With up to 7 carriers, they simply have 7x different points of failure each at only 10Gbps, while getting worse deals on transit pricing due to lower volumes with each provider.

dsl 10y ago
You don't need a private backbone to be able to mitigate attacks across multiple locations. I've done it fighting off multi-hundred Gbps attacks and it was never an issue. You can QoS your own intra-site GRE tunnels. Linode is moving 200-300 Gbps globally. That is about 37.5 Gbps per location, and when you figure in a 20% utilization (because you need to be able to burst)... they have about 300 Gbps of transit per location. Spread across 3-5 carriers I would guess they have 40-100 Gbps from each. Way more than your estimated 10 Gbps. As far as "routing performance" they appears to be buying from a few Tier 1 networks per location, and a mix of regional Tier 2s. That is in line with best practices. Sometimes to reach the right networks you do need to spin up circuits with multiple Tier 2s, there is no such thing as "diminishing returns" if you are doing traffic engineering properly. The right way to build networks is to meet your performance needs first and foremost, have enough headroom to grow and serve your customers, and work with your upstreams to manage incoming attacks. An external scrubbing service makes no sense when you can adapt your network as Linode has done so they can easily blackhole targets at their upsteams edge. I applaud their efforts. This is some smart network engineering.