6 ms·

Take a traceroute from everyone experiencing the problem, look for the common hops among them all. Then compare that list against people not experiancing the p

Take a traceroute from everyone experiencing the problem, look for the common hops among them all. Then compare that list against people not experiancing the problem to find the differences. The finsl set there are good places to start looking. Sweitches and routers along that way could be the cause

toast0 6y ago
It might be better (although harder!) to take the traceroute from example.org, instead of from the clients. Forward and reverse paths often diverge, so it's important to find the path with the error.
grayfaced 6y ago
If you ping the hops with a large icmp payload, you might be able to observe the flipped bit in the echo reply. That could help isolate which hop it is.
- toast0 6y ago
  You get some amount of your traceroute packet back too, could have flipped bits in there.
joshcaron 6y ago
Some people on Twitter have started collecting IPs: https://twitter.com/alexstamos/status/1336100299841314817 https://twitter.com/alexstamos/status/1336100299841314817
- mitchs 6y ago
  They need to be capturing src/dest IPs as well as ports for AT&T to have any hope of using that data. Edit to make the comment more useful: If anyone is curious, look up "ECMP hashing." There are probably tons of parallel paths through AT&T's network, and to narrow down to the hardware causing problems, they will need to identify which specific path was chosen. Hardware switches packets out equally viable pathways by hashing some of the attributes of the packet. Hash output % number of pathways selects which pathway at every hop. Hardware does this because everyone wants all packets involved in the same "flow" (all packets with the same src/dest IP and port and protocol (TCP)) to deterministically go through the same set of pipes to avoid packet re-ordering. If you randomly sprayed packets, the various buffer depths of routers (or even speed of light and slightly different length fibers along the way) could cause packets to swap ordering. While TCP "copes" with reordering, it doesn't like it and and older implementations slowed way down when it happened.

toast0 6y ago

It might be better (although harder!) to take the traceroute from example.org, instead of from the clients. Forward and reverse paths often diverge, so it's important to find the path with the error.

grayfaced 6y ago

If you ping the hops with a large icmp payload, you might be able to observe the flipped bit in the echo reply. That could help isolate which hop it is.

toast0 6y ago
You get some amount of your traceroute packet back too, could have flipped bits in there.

joshcaron 6y ago

Some people on Twitter have started collecting IPs: https://twitter.com/alexstamos/status/1336100299841314817 https://twitter.com/alexstamos/status/1336100299841314817

mitchs 6y ago
They need to be capturing src/dest IPs as well as ports for AT&T to have any hope of using that data. Edit to make the comment more useful: If anyone is curious, look up "ECMP hashing." There are probably tons of parallel paths through AT&T's network, and to narrow down to the hardware causing problems, they will need to identify which specific path was chosen. Hardware switches packets out equally viable pathways by hashing some of the attributes of the packet. Hash output % number of pathways selects which pathway at every hop. Hardware does this because everyone wants all packets involved in the same "flow" (all packets with the same src/dest IP and port and protocol (TCP)) to deterministically go through the same set of pipes to avoid packet re-ordering. If you randomly sprayed packets, the various buffer depths of routers (or even speed of light and slightly different length fibers along the way) could cause packets to swap ordering. While TCP "copes" with reordering, it doesn't like it and and older implementations slowed way down when it happened.