3 ms·

One curiosity: TCP Fast Open requires working path MTU discovery because MSS clamping no longer works as a hack advertise a lower MTU: the Fast Open cookie adds

One curiosity: TCP Fast Open requires working path MTU discovery because MSS clamping no longer works as a hack advertise a lower MTU: the Fast Open cookie adds an arbitrary amount of data which counts towards the packet length, but not towards the segment length.

toast0 2y ago
As described in the RFC, TCP Fast Open is supposed to be used from a client IP that connected to the server IP recently. The client could be expected to have discovered the effective MTU during that previous connection and limit it's SYN data to that size. The server gets the (presumably clamped) MSS option with the SYN+data, and could use that, or possibly include the previous MSS in its syn cookie and use the lesser of the two. Or more conservatively the size of the client SYN packet or 576/1280. MSS/MTU doesn't have to be the same in both directions, but it's usually a pretty good assumption to assume it is. I've seen much better results with broken networks when the syn+ack sends back min(server mss, client mss) rather than always sending back client mss; and the impact on working networks is very small. sending back min(server mss, client mss - X) where X is 8 (assume PPPoE) or 20 (assume IPIP tunnel) works even better at establishing working connections, although with some additional overhead where the client actually knows its mss. Some devices clamp MSS on outbound SYN but not on inbound SYN+ACK. sigh Personally, if I were to deploy something like TCP Fast Open, I'd dispense with the cookie... allow clients to speculatively include syn data, if they want. Cap to packets of 576/1280 length to be reasonable. Servers should consider recent results of Fast Open and local capacity to decide if they want to accept it or not. Server response should be limited to the same size as the client sent to avoid amplification. Publish some common heuristics --- if clients sending fast open get to connected in 90% of syn+acks sent, go ahead and process the fast open data, but when success falls below that, don't do it. Have some limit on how many fast opens you want to leave open. Server side done. Client side, try it, if it doesn't work try without. Every time the retry without works, double a skip counter; every time the try with it works, decrement the skip counter. Max out at trying once every 1024 connections. You can even do happy eyeballs stuff like send out a plain SYN after a short time, use whichever comes back first, but if the fast open does come back a bit after plain syn, you know that fast open is viable on this network / to that destination.

toast0 2y ago

As described in the RFC, TCP Fast Open is supposed to be used from a client IP that connected to the server IP recently. The client could be expected to have discovered the effective MTU during that previous connection and limit it's SYN data to that size. The server gets the (presumably clamped) MSS option with the SYN+data, and could use that, or possibly include the previous MSS in its syn cookie and use the lesser of the two. Or more conservatively the size of the client SYN packet or 576/1280. MSS/MTU doesn't have to be the same in both directions, but it's usually a pretty good assumption to assume it is. I've seen much better results with broken networks when the syn+ack sends back min(server mss, client mss) rather than always sending back client mss; and the impact on working networks is very small. sending back min(server mss, client mss - X) where X is 8 (assume PPPoE) or 20 (assume IPIP tunnel) works even better at establishing working connections, although with some additional overhead where the client actually knows its mss. Some devices clamp MSS on outbound SYN but not on inbound SYN+ACK. sigh Personally, if I were to deploy something like TCP Fast Open, I'd dispense with the cookie... allow clients to speculatively include syn data, if they want. Cap to packets of 576/1280 length to be reasonable. Servers should consider recent results of Fast Open and local capacity to decide if they want to accept it or not. Server response should be limited to the same size as the client sent to avoid amplification. Publish some common heuristics --- if clients sending fast open get to connected in 90% of syn+acks sent, go ahead and process the fast open data, but when success falls below that, don't do it. Have some limit on how many fast opens you want to leave open. Server side done. Client side, try it, if it doesn't work try without. Every time the retry without works, double a skip counter; every time the try with it works, decrement the skip counter. Max out at trying once every 1024 connections. You can even do happy eyeballs stuff like send out a plain SYN after a short time, use whichever comes back first, but if the fast open does come back a bit after plain syn, you know that fast open is viable on this network / to that destination.