3 ms·

I've seen ISP routers massively increasing latency in conjunction with packet loss -- had a problem with one provider recently where packet loss would increase

I've seen ISP routers massively increasing latency in conjunction with packet loss -- had a problem with one provider recently where packet loss would increase every evening, and so would the rtt. Suggests large buffers on the congested part. The reverse DNS of the given IP suggested it was a gig-e connection (my side of the hop, which was also into LINX, is 100G) Of course you can hide a router by not decrementing the TTL as it passes through your network at layer 3, you can hide the IP by not responding with ICMP expired messages An ICMP could well return on a different path to the direction it was sent, with a path like this Host -> R1 -> R2 -> R3 // R3 -> R4 -> R5 -> R1 -> Host Traceroute will only show the outbound route, so you should traceroute from both ends The latency will also be affected by ICMP generation on the router, which could be delayed, rate limited, dropping, etc. Doing a quick traceroute to a host of mine in Sydney, from London, shows 5ms to i-91.ulco-core02.telstraglobal.net 202.40.148.33 82ms to i-10104.unse-core01.telstraglobal.net 202.84.141.145 132ms to i-10601.1wlt-core02.telstraglobal.net 202.40.148.106 277ms to i-10406.sydo-core04.telstraglobal.net 202.84.141.226 sydo will be sydney To get the exact map I could talk to Telstra (in this case I peer with telstra directly in London) Or I could look at telstra's map, which ddg helpfully tells me is at https://www.telstraglobal.com/network-infrastructure-map/ https://www.telstraglobal.com/network-infrastructure-map/ Sadly the map doesn't work very well, more form over function ddging 1wlt-core02.telstraglobal.net returns this though https://crowdsupport.telstra.com.au/t5/home-broadband/bad-routing-to-hong-kong-high-ping-latency/td-p/847009 https://crowdsupport.telstra.com.au/t5/home-broadband/bad-ro... Which tells me 1wlt is LA. unse will thus be east coast U.S. I'd have expected routing via Singapore. There's no way to know which way the traffic is actually going without asking Telstra. I have 2 ethernet circuits from the UK to Washington DC, to me it looks like two layer 2 1500MTU circuit. Only by talking to the provier can I work out which circuits it actually travels on trans atlanticly. It's supposed to be separate, but latency changed by 2ms a few days ago. Asked them about it, and there was a failure in their network, they rerouted in a few hundered milliseconds (which isn't good as now both diverse circuits run via the same equipment, thus any issues like another 100ms outage will cause an impact)

Hikikomori 6y ago
>I have 2 ethernet circuits from the UK to Washington DC, to me it looks like two layer 2 1500MTU circuit. Only by talking to the provier can I work out which circuits it actually travels on trans atlanticly. It's supposed to be separate, but latency changed by 2ms a few days ago. Asked them about it, and there was a failure in their network, they rerouted in a few hundered milliseconds (which isn't good as now both diverse circuits run via the same equipment, thus any issues like another 100ms outage will cause an impact) Buying two mpls circuits (assuming they are that) from the same provider is a single point of faliure, your only real redundancy is your handover at each site, if they are separate. You're better of buying an optical link, then you are also not sharing bandwidth with anyone else. The cost isn't that different in my experience, but might be for an trans-Atlantic circuit. IPSEC over normal internet connections with multiple isp's is a better choice.
- iso1631 6y ago
  Don't get me started on that. However the two circuits are sold as a fully resilient pair (different tails from different DCs arrive into the building in Washington different directions with guaranteed different paths to the DCs, then in contract it's guaranteed neither circuit will run via the same equipment or on the same submarine circuit. In the UK they arrive in two separate cities) However I know reality and contracts never meet, unfortunately in a large disfunctional organization there are other considerations in circuit procurement than technical requirements. As for internet circuits, I had two ISPs in NY on two separate paths, which is great. Something went wrong about a month ago, and the routing from one of the ISPs changed, meaning that we were back to a single point of failure who we have no business relationship with
  - Hikikomori 6y ago
    We had a partner that bought a redundant mpls service from their DC to another DC (not their own) and to AWS direct connect via that DC. Their mpls provider was a SPOF, the provider for the second DC was a SPOF, then the direct connect circuits terminated in the same AWS router, another SPOF. All 3 had SPOF's had caused its own significant downtime within a few months. This was designed by the partner, contract signed by us, the whole solution paid by us, without consulting any network engineer on our side until 1 week before it was supposed to be implemented. We proposed two different solutions that would cost less than 1/20 of the cost their solution was costing us ($20k/month), while providing real redundancy (yet to be implemented, though some SPOFs solved by now). Never enjoyed outages as much as this as we got to explain why their solution was so bad each time.

Hikikomori 6y ago

>I have 2 ethernet circuits from the UK to Washington DC, to me it looks like two layer 2 1500MTU circuit. Only by talking to the provier can I work out which circuits it actually travels on trans atlanticly. It's supposed to be separate, but latency changed by 2ms a few days ago. Asked them about it, and there was a failure in their network, they rerouted in a few hundered milliseconds (which isn't good as now both diverse circuits run via the same equipment, thus any issues like another 100ms outage will cause an impact) Buying two mpls circuits (assuming they are that) from the same provider is a single point of faliure, your only real redundancy is your handover at each site, if they are separate. You're better of buying an optical link, then you are also not sharing bandwidth with anyone else. The cost isn't that different in my experience, but might be for an trans-Atlantic circuit. IPSEC over normal internet connections with multiple isp's is a better choice.

iso1631 6y ago
Don't get me started on that. However the two circuits are sold as a fully resilient pair (different tails from different DCs arrive into the building in Washington different directions with guaranteed different paths to the DCs, then in contract it's guaranteed neither circuit will run via the same equipment or on the same submarine circuit. In the UK they arrive in two separate cities) However I know reality and contracts never meet, unfortunately in a large disfunctional organization there are other considerations in circuit procurement than technical requirements. As for internet circuits, I had two ISPs in NY on two separate paths, which is great. Something went wrong about a month ago, and the routing from one of the ISPs changed, meaning that we were back to a single point of failure who we have no business relationship with
- Hikikomori 6y ago
  We had a partner that bought a redundant mpls service from their DC to another DC (not their own) and to AWS direct connect via that DC. Their mpls provider was a SPOF, the provider for the second DC was a SPOF, then the direct connect circuits terminated in the same AWS router, another SPOF. All 3 had SPOF's had caused its own significant downtime within a few months. This was designed by the partner, contract signed by us, the whole solution paid by us, without consulting any network engineer on our side until 1 week before it was supposed to be implemented. We proposed two different solutions that would cost less than 1/20 of the cost their solution was costing us ($20k/month), while providing real redundancy (yet to be implemented, though some SPOFs solved by now). Never enjoyed outages as much as this as we got to explain why their solution was so bad each time.