4 ms·

Consistently low latency is great. Not just with bufferbloat but with physical latency. A lot of city-based fiber dwellers think that just running fiber out to

by dtaht 3y ago

Consistently low latency is great. Not just with bufferbloat but with physical latency. A lot of city-based fiber dwellers think that just running fiber out to rural areas actually improves performance more than it does. Moving data and interconnectivity out there helps more. They really should get out more. Or, perhaps, learn how to insert artificial, rural like delays into their networks in the city, to grok it. (I am very busy today at the metro connect conference in ft lauderdale today, please expect sparse replies if any. If any of y'all are actually here today, I am outside with Bandwidth is a lie t-shirt on!

vitus 3y ago
Oh, I absolutely believe this. My example of Hawaii being 70ms away from Google was something that I knew about on paper but didn't truly appreciate until I was there on vacation a few years back (where the hotel wifi injected another 30-50ms on unloaded connections). Moving interconnectivity to rural areas is tricky because most peering is where you can convince other network operators to meet you. This is a bit of a catch-22, where if you're not big enough then most network operators won't bother, and you can't get big enough unless you get these network operators on board. There's a secondary issue where just having peering isn't useful without servers (or vice versa), and you often don't have the economies of scale to justify spending millions on routing equipment, servers, etc. if you don't expect to serve many users there. The compromise that some of these big operators (Google, Facebook, Netflix off the top of my head) make is to install cache nodes inside ISP deployments for last-mile delivery. My understanding is that this is generally a win-win for all parties involved: ISPs don't have to build out as much backbone / peering capacity, Google and co can get away with smaller deployments and piggyback off of the ISP's existing network presence, and end users have better quality of experience. That said, for the most part, Google's cache nodes don't serve www.google.com because we generally view them as less secure, and we would have a very bad day if we had a compromise of any high-value TLS certs. (We mostly use them to serve YouTube videos, since that's where we derive the most value from bandwidth offload.)
Dylan16807 3y ago
Please explain how you get a high latency on fiber that doesn't involve bufferbloat or doing something purposefully wrong. Where are these rural delays coming from? Especially because an extra 200 miles of fiber only increases round trip time by 3 milliseconds. How far are signals going before they get routed in the right direction?
- vitus 3y ago
  To give a concrete example: suppose I am in rural West Virginia. I want to access HN. HN's only servers are in San Diego. My RTT to HN is limited by how my ISP can get to transit (most likely AS3356, formerly Level3, now Lumen). Let's suppose that's in Ashburn, Virginia, in Equinix's DC2 datacenter (https://www.equinix.com/data-centers/americas-colocation/united-states-colocation/washington-dc-data-centers/dc2 https://www.equinix.com/data-centers/americas-colocation/uni...). (This is actually where my ISP meets Frontier en route to listserv.wvu.edu, so this isn't a totally contrived example.) So now I have to go east 10ms in order to go 70ms west (80ms). If, say, this ISP could also reach transit in Pittsburgh, that might only be 5ms northwest followed by 60ms west (65ms). Or if HN decided to spin up a second server location in Chicago, then I might only need to go a third of the way across the country instead of all the way (20ms or less). This is what I believe Dave is referring to when he says that moving data and interconnectivity matters more than deploying fiber (which might save 5-10ms over cable).
  - Dylan16807 3y ago
    For your first example, it looks like Ashburn and Pittsburgh should be less than 2 milliseconds apart? I'd expect the difference in routing to be more like 80ms versus 74ms, and that's not very much. Having a second server can definitely have a massive impact on latency, but that doesn't sound like an ISP thing, or a rural thing, or an interconnectivity thing.
    - vitus 3y ago
      Maybe if you had fiber directly connecting Ashburn to Pittsburgh, but it's more likely that you connect Ashburn -> Philadelphia -> Pittsburgh, which is more than double the physical distance. Just looking at distances on a map is insufficient for actually characterizing the network path. Also: bear in mind that you need to double all these numbers when considering round trip time. I recognize I phrased it in such a way that might have been interpreted as one-way latencies, but that wasn't my intent. Even if you had a direct path from Ashburn to Pittsburgh, speed of light through fiber would be about 3.5 ms to travel 450 miles (there and back). And while you might expect that from just plugging numbers into an equation, I have never seen anything resembling 4ms RTT between DC and New York (which are a comparable distance apart from each other) on Google's production network, even though those are definitely directly connected (6-7ms is more realistic).

by dtaht 3y ago

vitus 3y ago

Oh, I absolutely believe this. My example of Hawaii being 70ms away from Google was something that I knew about on paper but didn't truly appreciate until I was there on vacation a few years back (where the hotel wifi injected another 30-50ms on unloaded connections). Moving interconnectivity to rural areas is tricky because most peering is where you can convince other network operators to meet you. This is a bit of a catch-22, where if you're not big enough then most network operators won't bother, and you can't get big enough unless you get these network operators on board. There's a secondary issue where just having peering isn't useful without servers (or vice versa), and you often don't have the economies of scale to justify spending millions on routing equipment, servers, etc. if you don't expect to serve many users there. The compromise that some of these big operators (Google, Facebook, Netflix off the top of my head) make is to install cache nodes inside ISP deployments for last-mile delivery. My understanding is that this is generally a win-win for all parties involved: ISPs don't have to build out as much backbone / peering capacity, Google and co can get away with smaller deployments and piggyback off of the ISP's existing network presence, and end users have better quality of experience. That said, for the most part, Google's cache nodes don't serve www.google.com because we generally view them as less secure, and we would have a very bad day if we had a compromise of any high-value TLS certs. (We mostly use them to serve YouTube videos, since that's where we derive the most value from bandwidth offload.)

Dylan16807 3y ago

Please explain how you get a high latency on fiber that doesn't involve bufferbloat or doing something purposefully wrong. Where are these rural delays coming from? Especially because an extra 200 miles of fiber only increases round trip time by 3 milliseconds. How far are signals going before they get routed in the right direction?

vitus 3y ago
To give a concrete example: suppose I am in rural West Virginia. I want to access HN. HN's only servers are in San Diego. My RTT to HN is limited by how my ISP can get to transit (most likely AS3356, formerly Level3, now Lumen). Let's suppose that's in Ashburn, Virginia, in Equinix's DC2 datacenter (https://www.equinix.com/data-centers/americas-colocation/united-states-colocation/washington-dc-data-centers/dc2 https://www.equinix.com/data-centers/americas-colocation/uni...). (This is actually where my ISP meets Frontier en route to listserv.wvu.edu, so this isn't a totally contrived example.) So now I have to go east 10ms in order to go 70ms west (80ms). If, say, this ISP could also reach transit in Pittsburgh, that might only be 5ms northwest followed by 60ms west (65ms). Or if HN decided to spin up a second server location in Chicago, then I might only need to go a third of the way across the country instead of all the way (20ms or less). This is what I believe Dave is referring to when he says that moving data and interconnectivity matters more than deploying fiber (which might save 5-10ms over cable).
- Dylan16807 3y ago
  For your first example, it looks like Ashburn and Pittsburgh should be less than 2 milliseconds apart? I'd expect the difference in routing to be more like 80ms versus 74ms, and that's not very much. Having a second server can definitely have a massive impact on latency, but that doesn't sound like an ISP thing, or a rural thing, or an interconnectivity thing.
  - vitus 3y ago
    Maybe if you had fiber directly connecting Ashburn to Pittsburgh, but it's more likely that you connect Ashburn -> Philadelphia -> Pittsburgh, which is more than double the physical distance. Just looking at distances on a map is insufficient for actually characterizing the network path. Also: bear in mind that you need to double all these numbers when considering round trip time. I recognize I phrased it in such a way that might have been interpreted as one-way latencies, but that wasn't my intent. Even if you had a direct path from Ashburn to Pittsburgh, speed of light through fiber would be about 3.5 ms to travel 450 miles (there and back). And while you might expect that from just plugging numbers into an equation, I have never seen anything resembling 4ms RTT between DC and New York (which are a comparable distance apart from each other) on Google's production network, even though those are definitely directly connected (6-7ms is more realistic).