8 ms·

there's probably double or triple the switches in between. they're not really switches though. more like super fast packet routers. the latency is much less tha

there's probably double or triple the switches in between. they're not really switches though. more like super fast packet routers. the latency is much less than any normal or carrier grade switch. they don't touch the IP layer at all. (I manage a bunch of them)

walrus01 6y ago
one of the things I keep telling non-network-eng people is that looking at a traceroute gives you absolutely no way of knowing what the OSI layer 2 topology is for transport networks, MPLS, switches, DWDM circuits carried as lit services "glassed through" a location, etc. Just the IP network. Discovering the underlying layer 2 topology of a carrier's network requires inside information and cannot be easily discerned sitting at a computer elsewhere on the internet. You might see two routers that appear to be directly adjacent to each other but it's actually carried as a 10Gbps VLAN across a several-state sized region between two cities many hundreds of km apart, with a lot of intermediate equipment in between.
- gpm 6y ago
  100km = 3 light-ms - which should be noticeable in the relative pings, no?
  - walrus01 6y ago
    Yes, I mean it can be obvious when ISPs have proper consistent reverse DNS with hostnames and POP designations. You might see a router in Sacramento that is a neighbor of something in Portland. Obviously there's a lot of stuff in between there but it's all abstracted away at the purely layer-3 IP network level.
  - chupasaurus 6y ago
    You've missed one zero, 1 light-ms = 300km.
    - tlamponi 6y ago
      In vacuum, yes, in medium it's more like 200 km.
- rmdashrfstar 6y ago
  Do you have any recommendations for where to learn more about network eng for us non-network-engs? My experience is in DevOps (on k8s, mostly) and systems/application programming, if that helps pinpoint any advice you could give. I’m trying to get the bigger picture like network engineers understand, as well as start to understand the layer 2 topology that you described.
  - TheColorYellow 6y ago
    Ditto. Would love to figure out if there's a good way of mapping out layer 1/2 connectivity from a single computer as well. Would be really, really fascinating.
    - walrus01 6y ago
      If you want layer 1 (physical location of dark fiber) you have to get the GIS dataset from the carriers. The data sets exist, but they're often protected as proprietary information or under NDA. I have a QGIS setup with a ton of stuff in it. There are other ways of acquiring layer 1 data which are labor intensive and involve the equivalent of filing FOIAs for construction permits with local city and county agencies, etc. Big facilities based ISPs that have a lot of fiber out there underground and aerial make extensive use of GIS software. Their construction groups will have their own full time GIS staff positions.
    - Hikikomori 6y ago
      You'll have to follow the cable. Since this forwarding is entirely passive there's no way to know.
  - walrus01 6y ago
    The traditional and hard way is to start as a first-tier NOC person for an ISP and work one's way up. Or to start as a field tech for an ISP and show skills worthy of getting promoted (perhaps into a NOC job or a manager-of-field-techs job). That method takes quite a while. Medium to large sized ISPs have a very large amount of BSD/GPL/Apache/misc licensed software running to support back end monitoring and provisioning systems. They do occasionally hire software developers to customize things for their environments, so it certainly wouldn't hurt to reach out to the noteworthy ones in your region and try giving them your CV.
- iso1631 6y ago
  I've seen ISP routers massively increasing latency in conjunction with packet loss -- had a problem with one provider recently where packet loss would increase every evening, and so would the rtt. Suggests large buffers on the congested part. The reverse DNS of the given IP suggested it was a gig-e connection (my side of the hop, which was also into LINX, is 100G) Of course you can hide a router by not decrementing the TTL as it passes through your network at layer 3, you can hide the IP by not responding with ICMP expired messages An ICMP could well return on a different path to the direction it was sent, with a path like this Host -> R1 -> R2 -> R3 // R3 -> R4 -> R5 -> R1 -> Host Traceroute will only show the outbound route, so you should traceroute from both ends The latency will also be affected by ICMP generation on the router, which could be delayed, rate limited, dropping, etc. Doing a quick traceroute to a host of mine in Sydney, from London, shows 5ms to i-91.ulco-core02.telstraglobal.net 202.40.148.33 82ms to i-10104.unse-core01.telstraglobal.net 202.84.141.145 132ms to i-10601.1wlt-core02.telstraglobal.net 202.40.148.106 277ms to i-10406.sydo-core04.telstraglobal.net 202.84.141.226 sydo will be sydney To get the exact map I could talk to Telstra (in this case I peer with telstra directly in London) Or I could look at telstra's map, which ddg helpfully tells me is at https://www.telstraglobal.com/network-infrastructure-map/ https://www.telstraglobal.com/network-infrastructure-map/ Sadly the map doesn't work very well, more form over function ddging 1wlt-core02.telstraglobal.net returns this though https://crowdsupport.telstra.com.au/t5/home-broadband/bad-routing-to-hong-kong-high-ping-latency/td-p/847009 https://crowdsupport.telstra.com.au/t5/home-broadband/bad-ro... Which tells me 1wlt is LA. unse will thus be east coast U.S. I'd have expected routing via Singapore. There's no way to know which way the traffic is actually going without asking Telstra. I have 2 ethernet circuits from the UK to Washington DC, to me it looks like two layer 2 1500MTU circuit. Only by talking to the provier can I work out which circuits it actually travels on trans atlanticly. It's supposed to be separate, but latency changed by 2ms a few days ago. Asked them about it, and there was a failure in their network, they rerouted in a few hundered milliseconds (which isn't good as now both diverse circuits run via the same equipment, thus any issues like another 100ms outage will cause an impact)
  - Hikikomori 6y ago
    >I have 2 ethernet circuits from the UK to Washington DC, to me it looks like two layer 2 1500MTU circuit. Only by talking to the provier can I work out which circuits it actually travels on trans atlanticly. It's supposed to be separate, but latency changed by 2ms a few days ago. Asked them about it, and there was a failure in their network, they rerouted in a few hundered milliseconds (which isn't good as now both diverse circuits run via the same equipment, thus any issues like another 100ms outage will cause an impact) Buying two mpls circuits (assuming they are that) from the same provider is a single point of faliure, your only real redundancy is your handover at each site, if they are separate. You're better of buying an optical link, then you are also not sharing bandwidth with anyone else. The cost isn't that different in my experience, but might be for an trans-Atlantic circuit. IPSEC over normal internet connections with multiple isp's is a better choice.
exikyut 6y ago
Do these have googleable model numbers, if I wanted to see pictures of them ("oh that's what my packets go through")?
- snuxoll 6y ago
  Look at Junipers PTX line, they are fairly common in carrier grade networks AFAICT.
- walrus01 6y ago
  At the small end, a juniper MX204 is a good example of a current-gen router that has enough RAM to take multiple full BGP tables, and has a sufficient number of 10 and 100GbE interfaces.
- kitteh 6y ago
  Juniper MX series. Juniper PTX. Cisco ASR 9000. That's basically a good portion of the internet.
Hikikomori 6y ago
ISP core networks typically have two components, routers and dwdm/opto equipment. The routers are not really special, just your regular ISP core/edge router, adding roughly the same delay for each hop (invisible mpls hops or not) on the order of microseconds. Opto equipment adds practically no delay as they just forward the light without looking at it or processing it. There might be redundant paths that it can switch between if there's a fiber cut.

walrus01 6y ago

one of the things I keep telling non-network-eng people is that looking at a traceroute gives you absolutely no way of knowing what the OSI layer 2 topology is for transport networks, MPLS, switches, DWDM circuits carried as lit services "glassed through" a location, etc. Just the IP network. Discovering the underlying layer 2 topology of a carrier's network requires inside information and cannot be easily discerned sitting at a computer elsewhere on the internet. You might see two routers that appear to be directly adjacent to each other but it's actually carried as a 10Gbps VLAN across a several-state sized region between two cities many hundreds of km apart, with a lot of intermediate equipment in between.

gpm 6y ago
100km = 3 light-ms - which should be noticeable in the relative pings, no?
- walrus01 6y ago
  Yes, I mean it can be obvious when ISPs have proper consistent reverse DNS with hostnames and POP designations. You might see a router in Sacramento that is a neighbor of something in Portland. Obviously there's a lot of stuff in between there but it's all abstracted away at the purely layer-3 IP network level.
- chupasaurus 6y ago
  You've missed one zero, 1 light-ms = 300km.
  - tlamponi 6y ago
    In vacuum, yes, in medium it's more like 200 km.
rmdashrfstar 6y ago
Do you have any recommendations for where to learn more about network eng for us non-network-engs? My experience is in DevOps (on k8s, mostly) and systems/application programming, if that helps pinpoint any advice you could give. I’m trying to get the bigger picture like network engineers understand, as well as start to understand the layer 2 topology that you described.
- TheColorYellow 6y ago
  Ditto. Would love to figure out if there's a good way of mapping out layer 1/2 connectivity from a single computer as well. Would be really, really fascinating.
  - walrus01 6y ago
    If you want layer 1 (physical location of dark fiber) you have to get the GIS dataset from the carriers. The data sets exist, but they're often protected as proprietary information or under NDA. I have a QGIS setup with a ton of stuff in it. There are other ways of acquiring layer 1 data which are labor intensive and involve the equivalent of filing FOIAs for construction permits with local city and county agencies, etc. Big facilities based ISPs that have a lot of fiber out there underground and aerial make extensive use of GIS software. Their construction groups will have their own full time GIS staff positions.
  - Hikikomori 6y ago
    You'll have to follow the cable. Since this forwarding is entirely passive there's no way to know.
- walrus01 6y ago
  The traditional and hard way is to start as a first-tier NOC person for an ISP and work one's way up. Or to start as a field tech for an ISP and show skills worthy of getting promoted (perhaps into a NOC job or a manager-of-field-techs job). That method takes quite a while. Medium to large sized ISPs have a very large amount of BSD/GPL/Apache/misc licensed software running to support back end monitoring and provisioning systems. They do occasionally hire software developers to customize things for their environments, so it certainly wouldn't hurt to reach out to the noteworthy ones in your region and try giving them your CV.
iso1631 6y ago
I've seen ISP routers massively increasing latency in conjunction with packet loss -- had a problem with one provider recently where packet loss would increase every evening, and so would the rtt. Suggests large buffers on the congested part. The reverse DNS of the given IP suggested it was a gig-e connection (my side of the hop, which was also into LINX, is 100G) Of course you can hide a router by not decrementing the TTL as it passes through your network at layer 3, you can hide the IP by not responding with ICMP expired messages An ICMP could well return on a different path to the direction it was sent, with a path like this Host -> R1 -> R2 -> R3 // R3 -> R4 -> R5 -> R1 -> Host Traceroute will only show the outbound route, so you should traceroute from both ends The latency will also be affected by ICMP generation on the router, which could be delayed, rate limited, dropping, etc. Doing a quick traceroute to a host of mine in Sydney, from London, shows 5ms to i-91.ulco-core02.telstraglobal.net 202.40.148.33 82ms to i-10104.unse-core01.telstraglobal.net 202.84.141.145 132ms to i-10601.1wlt-core02.telstraglobal.net 202.40.148.106 277ms to i-10406.sydo-core04.telstraglobal.net 202.84.141.226 sydo will be sydney To get the exact map I could talk to Telstra (in this case I peer with telstra directly in London) Or I could look at telstra's map, which ddg helpfully tells me is at https://www.telstraglobal.com/network-infrastructure-map/ https://www.telstraglobal.com/network-infrastructure-map/ Sadly the map doesn't work very well, more form over function ddging 1wlt-core02.telstraglobal.net returns this though https://crowdsupport.telstra.com.au/t5/home-broadband/bad-routing-to-hong-kong-high-ping-latency/td-p/847009 https://crowdsupport.telstra.com.au/t5/home-broadband/bad-ro... Which tells me 1wlt is LA. unse will thus be east coast U.S. I'd have expected routing via Singapore. There's no way to know which way the traffic is actually going without asking Telstra. I have 2 ethernet circuits from the UK to Washington DC, to me it looks like two layer 2 1500MTU circuit. Only by talking to the provier can I work out which circuits it actually travels on trans atlanticly. It's supposed to be separate, but latency changed by 2ms a few days ago. Asked them about it, and there was a failure in their network, they rerouted in a few hundered milliseconds (which isn't good as now both diverse circuits run via the same equipment, thus any issues like another 100ms outage will cause an impact)
- Hikikomori 6y ago
  >I have 2 ethernet circuits from the UK to Washington DC, to me it looks like two layer 2 1500MTU circuit. Only by talking to the provier can I work out which circuits it actually travels on trans atlanticly. It's supposed to be separate, but latency changed by 2ms a few days ago. Asked them about it, and there was a failure in their network, they rerouted in a few hundered milliseconds (which isn't good as now both diverse circuits run via the same equipment, thus any issues like another 100ms outage will cause an impact) Buying two mpls circuits (assuming they are that) from the same provider is a single point of faliure, your only real redundancy is your handover at each site, if they are separate. You're better of buying an optical link, then you are also not sharing bandwidth with anyone else. The cost isn't that different in my experience, but might be for an trans-Atlantic circuit. IPSEC over normal internet connections with multiple isp's is a better choice.

exikyut 6y ago

Do these have googleable model numbers, if I wanted to see pictures of them ("oh that's what my packets go through")?

snuxoll 6y ago
Look at Junipers PTX line, they are fairly common in carrier grade networks AFAICT.
walrus01 6y ago
At the small end, a juniper MX204 is a good example of a current-gen router that has enough RAM to take multiple full BGP tables, and has a sufficient number of 10 and 100GbE interfaces.
kitteh 6y ago
Juniper MX series. Juniper PTX. Cisco ASR 9000. That's basically a good portion of the internet.

Hikikomori 6y ago

ISP core networks typically have two components, routers and dwdm/opto equipment. The routers are not really special, just your regular ISP core/edge router, adding roughly the same delay for each hop (invisible mpls hops or not) on the order of microseconds. Opto equipment adds practically no delay as they just forward the light without looking at it or processing it. There might be redundant paths that it can switch between if there's a fiber cut.