4 ms·

Not really...The issue is more using TTL in your ECMP hash algo. It’s a weird default to have.

CKN23-ARIN 8y ago
That's what caused the issue to surface, but the root cause is offering a stateful service over anycast. There is no hard requirement that you ECMP flows consistently. Spraying may be sub-optimal, but it must be accepted. I do agree that including TTL by default is weird, though.
- deleted 8y ago
  [deleted]
- xkgt 8y ago
  Am I the only one who sees a different issue here? The problem is neither stateful service over anycast nor TTL based hashing. Being a DDoS service, one can imagine the need to have stateful POP since each edge needs to track the TCP state in order to provide DOS protection. At the same time, it is understandable that the state can't be replicated at scale. As for including TTL in hashing algorithm, it is aimed to solve link under-utilization so it is also a valid implementation. The real bug here is Arista CPE mangling the TTL bits for the Client Hello packets. I always hate it when networking gear meddles with the protocol stack. Sure it gives some flexibility but time and time again, it ends up breaking something somewhere in the path since much of internet networking is a pile of assumptions. Tampering with protocol fields unilaterally is going to break someone's assumptions somewhere down the path.
  - CKN23-ARIN 8y ago
    Again, there is no hard requirement that all packets in a flow take the same route. Keeping the TCP state machine POP-local when running anycast TCP is exposing a buggy TCP implementation to the Internet. I understand that it is desirable to do so, especially under calm routing conditions where it tends to work, but it is not correct. I agree that Arista shouldn't be mangling TTL here. However, TTL mangling shouldn't break TCP because TTLs can already change under normal operation, e.g. when routes change, which happens all the time on the Internet. In a non-anycast situation, TCP connections would stay open under these conditions.
  - frnkblk 8y ago
    To clarify, the Arista routers are the ISP's border routers. It was the residential/business customers' SOHO routers, of various makes and models, that were not decrementing the initial TCP SYN.
    - xkgt 8y ago
      Thanks for pointing out. Sorry I overlooked that part. Perhaps I got primed by the opening statements which implied that problem happened only after placing new Arista routers and hastily assumed that it was Arista's routers that mangled the bits.
      - frnkblk 8y ago
        You are correct, the problem started only after placing new Arista border routers. The previous border routers were not doing ECMP. The issue was a combination of the use of anycast, diverse Internet transit egress, this model of Arista defaulting to using the packet's TTL in its ECMP hash calculations, and the end-customer router CPE egressing packets that are part of the same TCP connection with variable TTL values. Change any one of those items and the issue would not have shown up.

That's what caused the issue to surface, but the root cause is offering a stateful service over anycast. There is no hard requirement that you ECMP flows consistently. Spraying may be sub-optimal, but it must be accepted. I do agree that including TTL by default is weird, though.

deleted 8y ago
[deleted]
xkgt 8y ago
Am I the only one who sees a different issue here? The problem is neither stateful service over anycast nor TTL based hashing. Being a DDoS service, one can imagine the need to have stateful POP since each edge needs to track the TCP state in order to provide DOS protection. At the same time, it is understandable that the state can't be replicated at scale. As for including TTL in hashing algorithm, it is aimed to solve link under-utilization so it is also a valid implementation. The real bug here is Arista CPE mangling the TTL bits for the Client Hello packets. I always hate it when networking gear meddles with the protocol stack. Sure it gives some flexibility but time and time again, it ends up breaking something somewhere in the path since much of internet networking is a pile of assumptions. Tampering with protocol fields unilaterally is going to break someone's assumptions somewhere down the path.
- CKN23-ARIN 8y ago
  Again, there is no hard requirement that all packets in a flow take the same route. Keeping the TCP state machine POP-local when running anycast TCP is exposing a buggy TCP implementation to the Internet. I understand that it is desirable to do so, especially under calm routing conditions where it tends to work, but it is not correct. I agree that Arista shouldn't be mangling TTL here. However, TTL mangling shouldn't break TCP because TTLs can already change under normal operation, e.g. when routes change, which happens all the time on the Internet. In a non-anycast situation, TCP connections would stay open under these conditions.
- frnkblk 8y ago
  To clarify, the Arista routers are the ISP's border routers. It was the residential/business customers' SOHO routers, of various makes and models, that were not decrementing the initial TCP SYN.
  - xkgt 8y ago
    Thanks for pointing out. Sorry I overlooked that part. Perhaps I got primed by the opening statements which implied that problem happened only after placing new Arista routers and hastily assumed that it was Arista's routers that mangled the bits.
    - frnkblk 8y ago
      You are correct, the problem started only after placing new Arista border routers. The previous border routers were not doing ECMP. The issue was a combination of the use of anycast, diverse Internet transit egress, this model of Arista defaulting to using the packet's TTL in its ECMP hash calculations, and the end-customer router CPE egressing packets that are part of the same TCP connection with variable TTL values. Change any one of those items and the issue would not have shown up.