8 ms·

Service provider story about tracking down TCP RSTs

91 points by cnst 8y ago

Animats 8y ago
A consistent route for every packet in a flow is considered desirable, but it's not an entitlement or a requirement of the IP protocol. An DDOS prevention tool that insists that TTL be the same for every packet is broken. Here's one that checks for inconsistent TTL, but it has some tolerance for variation.[1] The one mentioned in the original post didn't like a difference of 1 in TTL. [1] https://fortiguard.com/encyclopedia/ips/12934/tcp-ttl-evasion https://fortiguard.com/encyclopedia/ips/12934/tcp-ttl-evasio...
- _wmd 8y ago
  I'm not a networker, but sounds like from a correctness standpoint, a problem on Fastly's end -- they're reusing frontend IPs for distinct sets of machines, and traffic directed to the 'wrong' PoP is dropped hard rather than attempting any kind of internal routing Of course that kind of routing would create a potential bottleneck for an attacker to exploit ("simply" force traffic to the wrong IPs to the wrong PoP, assuming $attacker had this level of access to the backbone), but that's the problem Fastly are supposedly paid to deal with Their scheme is fine and dandy with a protocol like DNS where UDP retries are transparent and TCP is a tiny fraction of weird traffic, but for business applications handling credit cards, surely the occasional RST is already too many Or another way to look at it, basically they're saying their IP addresses are special snowflakes and actually the full address includes the route, and source networks are wrong for assuming things work the way they're supposed to everywhere else on the Internet
  - windowsworkstoo 8y ago
    Not really. Anycast is fairly standard and usable for stateful connections - the issue is again middleboxes fucking with stuff and a weird default of incorporating TTL in the ECMP hashing algo
    - toast0 8y ago
      Expecting all flows to have all packets on a flow delivered via the same path is extremely optimistic.
      - deleted 8y ago
        [deleted]
    - deleted 8y ago
      [deleted]
    - jiveturkey 8y ago
      > Anycast is fairly standard and usable for stateful connections please cite some sources to back this up. i find this statement surprising.
  - cnst 8y ago
    > Of course that kind of routing would create a potential bottleneck for an attacker to exploit ("simply" force traffic to the wrong IPs to the wrong PoP, assuming $attacker had this level of access to the backbone) It's actually simpler than that — the state of the TCP connection is controlled by the hosts to the TCP connection, so, all it takes is for a "client" host to pretend that the connection has already been established (sending a single packet alleging as such) — no need for any special access to any backbone.
- maz- 8y ago
  There's no DDoS prevention tool in play here (aside from that being Fastly's business). Arista's default of having the TTL as part of the hashing function to calculate packet path means that any TCP connection that does not have a consistent TTL when it reaches this (source) ISP's routers can potentially be bucketed onto multiple different egress paths (transit or peering connections) onto different upstream SP's. Because BGP works on the concept of AS Numbers rather than IP's (and because Fastly uses Anycast, which announces the same set of IP's from multiple POP's), it's possible that packets that leave the ISP network over different transit connections end up at different Fastly POP's, as each transit provider or peer will have a different route to Fastly's AS. Fastly does not share TCP session information between POP's as I imagine at that scale it's massively prohibitive to do so. The TCP RST happens because there is no established TCP session at the secondary POP which receives the TLS handshake packet, and because TTL is part of the hash function it will always end up following a different path (and therefore hitting a different datacenter, if routing stays consistent) if the TTL is reduced by 1. It's generally deemed 'acceptable' in Anycast terms to cause a TCP RST when traffic switches from one POP to another, as this usually happens rarely, in response to changes in the path between two AS numbers. For things like websites, browsers will usually just retry the connection, and as the path has changed and stays consistent, it starts to work and all you notice is a slight delay. It doesn't work in this situation because the second packet with the reduced TTL is consistently routed to an 'incorrect' datacenter and breaks the TCP session every time, effectively dropping service to 0.
  - londons_explore 8y ago
    >It's generally deemed 'acceptable' in Anycast terms to cause a TCP RST when traffic switches from one POP to another... >For things like websites, browsers will usually just retry the connection This is not the case. Every single TCP RST you send on an active http connection will lead to an http request failing in a way which causes application brokenness. No browser auto-retries in that case. The user always has to hit refresh to fix it.
    - cnst 8y ago
      At least with the TCP RST it'll fail fast. What's annoying is that some providers decide that you aren't worthy of the RST; then your browser has to wait for like 3 minutes until the timeout is reached — really annoying.
- spydum 8y ago
  i tend to agree: anycasting stateful services is always a gamble.. however, i do think including ttl in the ecmp hashing alg is prone to failure, and is probably a poor default.
  - alecbenzer 8y ago
    You mean 'stateful' at the IP layer, right? So anything using TCP counts as "stateful"? Like, an application which implemented "stateless" HTTP endpoints would still run into this issue?
CKN23-ARIN 8y ago
TL;DR: Incapsula, Imperva, Sucuri, Fastly, and likely others are using anycast incorrectly.
- windowsworkstoo 8y ago
  Not really...The issue is more using TTL in your ECMP hash algo. It’s a weird default to have.
  - CKN23-ARIN 8y ago
    That's what caused the issue to surface, but the root cause is offering a stateful service over anycast. There is no hard requirement that you ECMP flows consistently. Spraying may be sub-optimal, but it must be accepted. I do agree that including TTL by default is weird, though.
    - deleted 8y ago
      [deleted]
    - xkgt 8y ago
      Am I the only one who sees a different issue here? The problem is neither stateful service over anycast nor TTL based hashing. Being a DDoS service, one can imagine the need to have stateful POP since each edge needs to track the TCP state in order to provide DOS protection. At the same time, it is understandable that the state can't be replicated at scale. As for including TTL in hashing algorithm, it is aimed to solve link under-utilization so it is also a valid implementation. The real bug here is Arista CPE mangling the TTL bits for the Client Hello packets. I always hate it when networking gear meddles with the protocol stack. Sure it gives some flexibility but time and time again, it ends up breaking something somewhere in the path since much of internet networking is a pile of assumptions. Tampering with protocol fields unilaterally is going to break someone's assumptions somewhere down the path.
      - CKN23-ARIN 8y ago
        Again, there is no hard requirement that all packets in a flow take the same route. Keeping the TCP state machine POP-local when running anycast TCP is exposing a buggy TCP implementation to the Internet. I understand that it is desirable to do so, especially under calm routing conditions where it tends to work, but it is not correct. I agree that Arista shouldn't be mangling TTL here. However, TTL mangling shouldn't break TCP because TTLs can already change under normal operation, e.g. when routes change, which happens all the time on the Internet. In a non-anycast situation, TCP connections would stay open under these conditions.
sleepydog 8y ago
I've never heard of routers using the ttl field for hashing, why would that ever be useful? Also it seems an incorrect use of anycast to terminate the same flow at different machines. At Google, anycast traffic goes through clusters of maglevs that directs individual flows to the same endpoint consistently. https://ai.google/research/pubs/pub44824 https://ai.google/research/pubs/pub44824 Also I'd like to echo that Fastly's NOC is great. Super responsive and smart. Disclaimer: I work at Google
- mcpherrinm 8y ago
  If you're doing multi-tier ECMP with the same hashing algorithm at each tier, using TTL can ensure you don't get polarization issues. Though there's plenty of other ways to avoid polarization, this works out of the box. I have heard fastly competitors tut-tut their use of anycast as potentially unreliable in the face of issues like this.
  - greglindahl 8y ago
    "this works out of the box" seems a bit of an odd thing to say when it's not working for significant edge cases. Maybe "it worked when we first tested it" is better? :-)
    - p1necone 8y ago
      I dunno, in my experience anything that "works right out of the box" usually misses a bunch of edge cases (possibly necessarily/by definition). The description seems apt to me.
  - bogomipz 8y ago
    What is "polarization" here? I am not familiar with this term applied to routing? Oscillating between different available paths?
    - zkms 8y ago
      > Oscillating between different available paths? The term for unwanted oscillation between different paths is, IIRC, "flapping". However, when working with ECMP (equal-cost multi path), you actually do want to use all your paths (hence "equal cost") simultaneously (for load-balancing purposes, say). "Polarisation" refers to the unwanted condition when the hash algorithm that decides which path a flow/packet takes is not properly distributing flows/packets across links, leading to underutilisation and/or overutilisation of links/routes.
jdwithit 8y ago
We run Arista 7280SR's on our edge. I went looking, and as far as I can tell, the TTL is NOT part of the default ECMP algorithm on this platform (Jericho chipset)? We certainly haven't tuned this setting one way or another. I would love for someone more experienced with Arista kit to weigh in on this, since it seems like it may be platform dependent. We do have a support contract so I can reach out to them directly, too. These commands are the best I could find browsing the Arista docs/blog/forums. edit: fixed code formatting edit2: We're on the EOS 4.20.x version train, fwiw #show port-channel load-balance jericho fields | grep TTL IP TTL field hashing is OFF #show load-balance profile (output snipped for brevity) ---------- default (global) ---------- IPv4 hash fields: Source IPv4 Address is ON Protocol is ON Time-To-Live is OFF Destination IPv4 Address is ON
- frnkblk 8y ago
  The Arista units are 7504N which are running 4.20 or later, and I believe they use the Arad or Petra chpiset.
mirimir 8y ago
I wonder how such anycast setups would deal with MPTCP.
- toast0 8y ago
  I don't think MPTCP is going to interact very well with modern high load sites. Even using unicast, it's going to be hard (impossible?) to ensure that all the individual flows make it to the same nic queue, which is what you'd really want for performance. I suspect this is part of the reason why it hasn't caught on very much. (Also, Google doesn't seem to want to invest in a good tcp stack on Android, instead they put an additional tcp layer (http/2) and then built tcp on top of udp (quic) ).
  - mirimir 8y ago
    Sad. Apple seemed interested, at one point anyway. But damn, it does work amazingly well site-to-site. I've managed ~50 Mbps throughput using bbcp (four streams) between Tor .onion services via OnionCat. And ~190 Mbps total from one source transferring simultaneously to five target servers. Each peer had six .onion services. With six OnionCat interfaces per peer, in MPTCP full-mesh mode, there are up to ~36 subflows per TCP connection. So using bbcp with four streams, there are as many as ~150 tcp6 connections via Tor per bbcp transfer. And with five simultaneous transfers, the MPTCP kernel in the source VPS was managing up to ~750 tcp6 connections. That's impressive! https://ipfs.io/ipfs/QmUDV2KHrAgs84oUc7z9zQmZ3whx1NB6YDPv8ZRuf4dutN/ https://ipfs.io/ipfs/QmUDV2KHrAgs84oUc7z9zQmZ3whx1NB6YDPv8ZR...
    - toast0 8y ago
      Apple is still pushing it, which is great. I'm just not sure how well it's going to scale -- MPTCP adds an extra layer of indirection, and extra locking in the easy case where it's just one server handling an IP. In the load balancing case, people are going to have to teach their load balancers a lot of new tricks to get the subflows aligned. If using anycast, the client is likely to be using multiple networks, so using the same server address seems likely to get to a different pop; exposing a pop specific extra server ip seems like something people don't want to do, since exposing that may make it easier to do a single pop. I've been debugging an issue where incidentally I'm hitting 1gbps on a single tcp connection (server to server, with tls), so I'm not sure why MPTCP is required? ;) But I guess if we had it, I would probably hit 2gbps instead of being capped by the one nic.

91 points by cnst 8y ago

A consistent route for every packet in a flow is considered desirable, but it's not an entitlement or a requirement of the IP protocol. An DDOS prevention tool that insists that TTL be the same for every packet is broken. Here's one that checks for inconsistent TTL, but it has some tolerance for variation.[1] The one mentioned in the original post didn't like a difference of 1 in TTL. [1] https://fortiguard.com/encyclopedia/ips/12934/tcp-ttl-evasion https://fortiguard.com/encyclopedia/ips/12934/tcp-ttl-evasio...

_wmd 8y ago
I'm not a networker, but sounds like from a correctness standpoint, a problem on Fastly's end -- they're reusing frontend IPs for distinct sets of machines, and traffic directed to the 'wrong' PoP is dropped hard rather than attempting any kind of internal routing Of course that kind of routing would create a potential bottleneck for an attacker to exploit ("simply" force traffic to the wrong IPs to the wrong PoP, assuming $attacker had this level of access to the backbone), but that's the problem Fastly are supposedly paid to deal with Their scheme is fine and dandy with a protocol like DNS where UDP retries are transparent and TCP is a tiny fraction of weird traffic, but for business applications handling credit cards, surely the occasional RST is already too many Or another way to look at it, basically they're saying their IP addresses are special snowflakes and actually the full address includes the route, and source networks are wrong for assuming things work the way they're supposed to everywhere else on the Internet
- windowsworkstoo 8y ago
  Not really. Anycast is fairly standard and usable for stateful connections - the issue is again middleboxes fucking with stuff and a weird default of incorporating TTL in the ECMP hashing algo
  - toast0 8y ago
    Expecting all flows to have all packets on a flow delivered via the same path is extremely optimistic.
    - deleted 8y ago
      [deleted]
  - deleted 8y ago
    [deleted]
  - jiveturkey 8y ago
    > Anycast is fairly standard and usable for stateful connections please cite some sources to back this up. i find this statement surprising.
- cnst 8y ago
  > Of course that kind of routing would create a potential bottleneck for an attacker to exploit ("simply" force traffic to the wrong IPs to the wrong PoP, assuming $attacker had this level of access to the backbone) It's actually simpler than that — the state of the TCP connection is controlled by the hosts to the TCP connection, so, all it takes is for a "client" host to pretend that the connection has already been established (sending a single packet alleging as such) — no need for any special access to any backbone.
maz- 8y ago
There's no DDoS prevention tool in play here (aside from that being Fastly's business). Arista's default of having the TTL as part of the hashing function to calculate packet path means that any TCP connection that does not have a consistent TTL when it reaches this (source) ISP's routers can potentially be bucketed onto multiple different egress paths (transit or peering connections) onto different upstream SP's. Because BGP works on the concept of AS Numbers rather than IP's (and because Fastly uses Anycast, which announces the same set of IP's from multiple POP's), it's possible that packets that leave the ISP network over different transit connections end up at different Fastly POP's, as each transit provider or peer will have a different route to Fastly's AS. Fastly does not share TCP session information between POP's as I imagine at that scale it's massively prohibitive to do so. The TCP RST happens because there is no established TCP session at the secondary POP which receives the TLS handshake packet, and because TTL is part of the hash function it will always end up following a different path (and therefore hitting a different datacenter, if routing stays consistent) if the TTL is reduced by 1. It's generally deemed 'acceptable' in Anycast terms to cause a TCP RST when traffic switches from one POP to another, as this usually happens rarely, in response to changes in the path between two AS numbers. For things like websites, browsers will usually just retry the connection, and as the path has changed and stays consistent, it starts to work and all you notice is a slight delay. It doesn't work in this situation because the second packet with the reduced TTL is consistently routed to an 'incorrect' datacenter and breaks the TCP session every time, effectively dropping service to 0.
- londons_explore 8y ago
  >It's generally deemed 'acceptable' in Anycast terms to cause a TCP RST when traffic switches from one POP to another... >For things like websites, browsers will usually just retry the connection This is not the case. Every single TCP RST you send on an active http connection will lead to an http request failing in a way which causes application brokenness. No browser auto-retries in that case. The user always has to hit refresh to fix it.
  - cnst 8y ago
    At least with the TCP RST it'll fail fast. What's annoying is that some providers decide that you aren't worthy of the RST; then your browser has to wait for like 3 minutes until the timeout is reached — really annoying.
spydum 8y ago
i tend to agree: anycasting stateful services is always a gamble.. however, i do think including ttl in the ecmp hashing alg is prone to failure, and is probably a poor default.
- alecbenzer 8y ago
  You mean 'stateful' at the IP layer, right? So anything using TCP counts as "stateful"? Like, an application which implemented "stateless" HTTP endpoints would still run into this issue?

CKN23-ARIN 8y ago

TL;DR: Incapsula, Imperva, Sucuri, Fastly, and likely others are using anycast incorrectly.

windowsworkstoo 8y ago
Not really...The issue is more using TTL in your ECMP hash algo. It’s a weird default to have.
- CKN23-ARIN 8y ago
  That's what caused the issue to surface, but the root cause is offering a stateful service over anycast. There is no hard requirement that you ECMP flows consistently. Spraying may be sub-optimal, but it must be accepted. I do agree that including TTL by default is weird, though.
  - deleted 8y ago
    [deleted]
  - xkgt 8y ago
    Am I the only one who sees a different issue here? The problem is neither stateful service over anycast nor TTL based hashing. Being a DDoS service, one can imagine the need to have stateful POP since each edge needs to track the TCP state in order to provide DOS protection. At the same time, it is understandable that the state can't be replicated at scale. As for including TTL in hashing algorithm, it is aimed to solve link under-utilization so it is also a valid implementation. The real bug here is Arista CPE mangling the TTL bits for the Client Hello packets. I always hate it when networking gear meddles with the protocol stack. Sure it gives some flexibility but time and time again, it ends up breaking something somewhere in the path since much of internet networking is a pile of assumptions. Tampering with protocol fields unilaterally is going to break someone's assumptions somewhere down the path.
    - CKN23-ARIN 8y ago
      Again, there is no hard requirement that all packets in a flow take the same route. Keeping the TCP state machine POP-local when running anycast TCP is exposing a buggy TCP implementation to the Internet. I understand that it is desirable to do so, especially under calm routing conditions where it tends to work, but it is not correct. I agree that Arista shouldn't be mangling TTL here. However, TTL mangling shouldn't break TCP because TTLs can already change under normal operation, e.g. when routes change, which happens all the time on the Internet. In a non-anycast situation, TCP connections would stay open under these conditions.

sleepydog 8y ago

I've never heard of routers using the ttl field for hashing, why would that ever be useful? Also it seems an incorrect use of anycast to terminate the same flow at different machines. At Google, anycast traffic goes through clusters of maglevs that directs individual flows to the same endpoint consistently. https://ai.google/research/pubs/pub44824 https://ai.google/research/pubs/pub44824 Also I'd like to echo that Fastly's NOC is great. Super responsive and smart. Disclaimer: I work at Google

mcpherrinm 8y ago
If you're doing multi-tier ECMP with the same hashing algorithm at each tier, using TTL can ensure you don't get polarization issues. Though there's plenty of other ways to avoid polarization, this works out of the box. I have heard fastly competitors tut-tut their use of anycast as potentially unreliable in the face of issues like this.
- greglindahl 8y ago
  "this works out of the box" seems a bit of an odd thing to say when it's not working for significant edge cases. Maybe "it worked when we first tested it" is better? :-)
  - p1necone 8y ago
    I dunno, in my experience anything that "works right out of the box" usually misses a bunch of edge cases (possibly necessarily/by definition). The description seems apt to me.
- bogomipz 8y ago
  What is "polarization" here? I am not familiar with this term applied to routing? Oscillating between different available paths?
  - zkms 8y ago
    > Oscillating between different available paths? The term for unwanted oscillation between different paths is, IIRC, "flapping". However, when working with ECMP (equal-cost multi path), you actually do want to use all your paths (hence "equal cost") simultaneously (for load-balancing purposes, say). "Polarisation" refers to the unwanted condition when the hash algorithm that decides which path a flow/packet takes is not properly distributing flows/packets across links, leading to underutilisation and/or overutilisation of links/routes.

jdwithit 8y ago

We run Arista 7280SR's on our edge. I went looking, and as far as I can tell, the TTL is NOT part of the default ECMP algorithm on this platform (Jericho chipset)? We certainly haven't tuned this setting one way or another. I would love for someone more experienced with Arista kit to weigh in on this, since it seems like it may be platform dependent. We do have a support contract so I can reach out to them directly, too. These commands are the best I could find browsing the Arista docs/blog/forums. edit: fixed code formatting edit2: We're on the EOS 4.20.x version train, fwiw #show port-channel load-balance jericho fields | grep TTL IP TTL field hashing is OFF #show load-balance profile (output snipped for brevity) ---------- default (global) ---------- IPv4 hash fields: Source IPv4 Address is ON Protocol is ON Time-To-Live is OFF Destination IPv4 Address is ON

frnkblk 8y ago
The Arista units are 7504N which are running 4.20 or later, and I believe they use the Arad or Petra chpiset.

mirimir 8y ago

I wonder how such anycast setups would deal with MPTCP.

toast0 8y ago
I don't think MPTCP is going to interact very well with modern high load sites. Even using unicast, it's going to be hard (impossible?) to ensure that all the individual flows make it to the same nic queue, which is what you'd really want for performance. I suspect this is part of the reason why it hasn't caught on very much. (Also, Google doesn't seem to want to invest in a good tcp stack on Android, instead they put an additional tcp layer (http/2) and then built tcp on top of udp (quic) ).
- mirimir 8y ago
  Sad. Apple seemed interested, at one point anyway. But damn, it does work amazingly well site-to-site. I've managed ~50 Mbps throughput using bbcp (four streams) between Tor .onion services via OnionCat. And ~190 Mbps total from one source transferring simultaneously to five target servers. Each peer had six .onion services. With six OnionCat interfaces per peer, in MPTCP full-mesh mode, there are up to ~36 subflows per TCP connection. So using bbcp with four streams, there are as many as ~150 tcp6 connections via Tor per bbcp transfer. And with five simultaneous transfers, the MPTCP kernel in the source VPS was managing up to ~750 tcp6 connections. That's impressive! https://ipfs.io/ipfs/QmUDV2KHrAgs84oUc7z9zQmZ3whx1NB6YDPv8ZRuf4dutN/ https://ipfs.io/ipfs/QmUDV2KHrAgs84oUc7z9zQmZ3whx1NB6YDPv8ZR...
  - toast0 8y ago
    Apple is still pushing it, which is great. I'm just not sure how well it's going to scale -- MPTCP adds an extra layer of indirection, and extra locking in the easy case where it's just one server handling an IP. In the load balancing case, people are going to have to teach their load balancers a lot of new tricks to get the subflows aligned. If using anycast, the client is likely to be using multiple networks, so using the same server address seems likely to get to a different pop; exposing a pop specific extra server ip seems like something people don't want to do, since exposing that may make it easier to do a single pop. I've been debugging an issue where incidentally I'm hitting 1gbps on a single tcp connection (server to server, with tls), so I'm not sure why MPTCP is required? ;) But I guess if we had it, I would probably hit 2gbps instead of being capped by the one nic.