4 ms·

This is excellent. I never thought about it that way. Indeed, with syn cookies the receiving party has no idea what was the original sequence number, so it will

by majke 7y ago

This is excellent. I never thought about it that way. Indeed, with syn cookies the receiving party has no idea what was the original sequence number, so it will treat _any_ delivered packet as first. Couple of caveats: - you can jam syn cookies enabled with tcp_syncookies=2 sysctl - syn cookies are generally bad because they prevent to negotiate window scaling. Window scaling is important unless you are doing low bandwidth like telnet :) - you can somewhat negotiate window scaling when tcp timestamps are enabled. But enabling tcp timestamps in general case brings little benefit and wastes 12 bytes of each packet for basically no gain. - for a bonus point, consider what happens when both syn cookies and TCP_DEFER_ACCEPT are enabled. More about syn packet handling in linux https://blog.cloudflare.com/syn-packet-handling-in-the-wild/ https://blog.cloudflare.com/syn-packet-handling-in-the-wild/

leetbulb 7y ago
I love write-ups on these types of issues. It really is helpful to share the processes involved to understand and debug a problem by experts. Thank you, OP. As suggested by majke, CloudFlare's blog is another great source of material regarding protocol wrangling at scale. If anyone else has similar blogs, bookmarks, etc please do share :)
the8472 7y ago
> But enabling tcp timestamps in general case brings little benefit and wastes 12 bytes of each packet for basically no gain. I think timestamps are used as a possible source to compute one-way delay in latency based congestion controllers.
Majromax 7y ago
> - you can somewhat negotiate window scaling when tcp timestamps are enabled. Reading the RFC, couldn't the requested window scale be jammed into the sequence number? Section 2.3 limits the window scale to <= 14 bits. This article suggests that the MSS category fits into 2 bits, so we take up 6 of the 32 sequence number bits for mandatory additional data. The syn cookie would then be (MSScat | WSCALE | HASH(key | saddr | daddr | sport | dport | sseq | MSScat | WSCALE)). The hash could then be 26 bits, leaving a very small probability of forgery of 2^-26. To re-address the problem of replay attacks, the server could rotate keys rather than include a timestamp in the data component. The cookie would need to verify with one of the (small) K most recent keys. If K were 2, then any ACKs younger than COOKIE_AGE would verify, ACKs up to twice COOKIE_AGE might verify, and older ACKs would be rejected. The probability of forgery would marginally increase to 2^-25. But, I'm no TCP expert. I'm sure there's something misguided or wrong in the above?
- toast0 7y ago
  FreeBSD uses a pretty similar approach to what you've described: MSS and Windows Scale are each turned into 3-bit table entries, SACK gets one bit, and one bit indicates which MAC secret was used; secrets are rotated every 15 seconds, two are kept, so client ACK needs to come in 15-30 seconds depending on where in the rotation cycle you are; 24-bits are used for the MAC. 15 second round trip covers the vast majority of internet connections. Details nicely commented here: https://github.com/freebsd/freebsd/blob/master/sys/netinet/tcp_syncache.c#L1967 https://github.com/freebsd/freebsd/blob/master/sys/netinet/t...
  - Majromax 7y ago
    Perfect, that's exactly the kind of thing I was looking for. Although I'm a little confused by the comment's description of the birthday attack -- it doesn't immediately seem useful to me for an attacker to find separate connections that generate the same cookie.
    - toast0 7y ago
      I'm not a cryptographer, but as I understand it, the birthday attack says if you have a valid hash for one thing, you can generate more things, and get a collision in a surpisingly small number of iterations. In this context, it's easy to get a valid hash --- when the system is in syncookie mode, send a SYN from an address where you have visibility of the SYN+ACK responses. Then, you could use that cookie (sequence number) to spoof ACK packets with other sources, and they've estimated the number of packets you need to spoof before you'll have probably generated a connection. That number of packets is significantly fewer than when the syncache has not overflowed recently, and you need to have sent a SYN, and have an exact match of the sequence number.
emmericp 7y ago
> But enabling tcp timestamps in general case brings little benefit and wastes 12 bytes of each packet for basically no gain. I disagree; TCP timestamps are awesome. Linux enables these by defaults. Quick search gives me some measurements from 2012 [1] that indicate that TCP timestamps are enabled on 83% of the top 100k web hosts. You can afford to waste 12 bytes; the bottleneck isn't these 12 bytes but how well you get congestion control to work. And congestion control relies on getting an accurate estimate of the round-trip time [1] https://link.springer.com/chapter/10.1007/978-3-642-36516-4_14 https://link.springer.com/chapter/10.1007/978-3-642-36516-4_... (paywall) Edit: typo Edit: Also, just because 83% of web hosts having it enabled does not imply that it is a good idea to do so in general. They could just all be running the linux defaults and these could be just wrong
- djoldman 7y ago
  One of the authors of the paper has it linked from his website: https://trammell.ch/publication/ecn-pam-2013/ https://trammell.ch/publication/ecn-pam-2013/
- jsnell 7y ago
  You can measure RTT continuously, accurately, and even in the presence of packet loss just with selective acks and a little bit of extra bookkeeping in the sender. It's hard to overstate how expensive TCP timestamps are. The thing is that they bloat every single packet including control packets. 2% of the world's bandwidth is being wasted on this. The only reason for anyone to implement TCP timestamps today is that iOS clients have horrible receive window scaling if timestamps are disabled. (Well, that was the only reason a few years ago when I was still in the game of keeping up with the quirks of different TCP stacks.) I wrote more on the subject at the time: https://www.snellman.net/blog/archive/2017-07-20-s3-mystery/ https://www.snellman.net/blog/archive/2017-07-20-s3-mystery/
  - toast0 7y ago
    I would guess that Apple could be convinced to fix this, if someone has the right contact. The iOS TCP stack shows a lot of care, generally: they do path MTU probing well, and they've deployed MP-TCP (requires apps to enable it though), among other things I can't remember. Fixing performance if timestamps are disabled seems like something they'd do.
jiveturkey 7y ago
> original initial the word 'initial [sequence number]' is a term of art, not an adjective that can be substituted. very commonly called ISN. odd that the article never used the acronym. > syn cookies are generally bad because they prevent to negotiate window scaling. this is wrong thinking. they degrade performance under "attack", yes. the alternative is instant death. "attacks" need not be from the big bad internet either. my experience is that in the last 10 years most synflood activations are internal buggy or poorly throttled clients. > enabling tcp timestamps in general case brings little benefit disagree, but sibling comment addresses this.
- toast0 7y ago
  > my experience is that in the last 10 years most synflood activations are internal buggy or poorly throttled clients. From running a large messaging service I agree; most of the perceived attacks (synflood or otherwise) were actually coming from our own clients. But there was a periodic stream of strictly abusive traffic coming from who knows where; mostly UDP reflection, but a SYN flood every once in a while.

by majke 7y ago

leetbulb 7y ago

I love write-ups on these types of issues. It really is helpful to share the processes involved to understand and debug a problem by experts. Thank you, OP. As suggested by majke, CloudFlare's blog is another great source of material regarding protocol wrangling at scale. If anyone else has similar blogs, bookmarks, etc please do share :)

the8472 7y ago

> But enabling tcp timestamps in general case brings little benefit and wastes 12 bytes of each packet for basically no gain. I think timestamps are used as a possible source to compute one-way delay in latency based congestion controllers.

Majromax 7y ago

> - you can somewhat negotiate window scaling when tcp timestamps are enabled. Reading the RFC, couldn't the requested window scale be jammed into the sequence number? Section 2.3 limits the window scale to <= 14 bits. This article suggests that the MSS category fits into 2 bits, so we take up 6 of the 32 sequence number bits for mandatory additional data. The syn cookie would then be (MSScat | WSCALE | HASH(key | saddr | daddr | sport | dport | sseq | MSScat | WSCALE)). The hash could then be 26 bits, leaving a very small probability of forgery of 2^-26. To re-address the problem of replay attacks, the server could rotate keys rather than include a timestamp in the data component. The cookie would need to verify with one of the (small) K most recent keys. If K were 2, then any ACKs younger than COOKIE_AGE would verify, ACKs up to twice COOKIE_AGE might verify, and older ACKs would be rejected. The probability of forgery would marginally increase to 2^-25. But, I'm no TCP expert. I'm sure there's something misguided or wrong in the above?

toast0 7y ago
FreeBSD uses a pretty similar approach to what you've described: MSS and Windows Scale are each turned into 3-bit table entries, SACK gets one bit, and one bit indicates which MAC secret was used; secrets are rotated every 15 seconds, two are kept, so client ACK needs to come in 15-30 seconds depending on where in the rotation cycle you are; 24-bits are used for the MAC. 15 second round trip covers the vast majority of internet connections. Details nicely commented here: https://github.com/freebsd/freebsd/blob/master/sys/netinet/tcp_syncache.c#L1967 https://github.com/freebsd/freebsd/blob/master/sys/netinet/t...
- Majromax 7y ago
  Perfect, that's exactly the kind of thing I was looking for. Although I'm a little confused by the comment's description of the birthday attack -- it doesn't immediately seem useful to me for an attacker to find separate connections that generate the same cookie.
  - toast0 7y ago
    I'm not a cryptographer, but as I understand it, the birthday attack says if you have a valid hash for one thing, you can generate more things, and get a collision in a surpisingly small number of iterations. In this context, it's easy to get a valid hash --- when the system is in syncookie mode, send a SYN from an address where you have visibility of the SYN+ACK responses. Then, you could use that cookie (sequence number) to spoof ACK packets with other sources, and they've estimated the number of packets you need to spoof before you'll have probably generated a connection. That number of packets is significantly fewer than when the syncache has not overflowed recently, and you need to have sent a SYN, and have an exact match of the sequence number.

emmericp 7y ago

> But enabling tcp timestamps in general case brings little benefit and wastes 12 bytes of each packet for basically no gain. I disagree; TCP timestamps are awesome. Linux enables these by defaults. Quick search gives me some measurements from 2012 [1] that indicate that TCP timestamps are enabled on 83% of the top 100k web hosts. You can afford to waste 12 bytes; the bottleneck isn't these 12 bytes but how well you get congestion control to work. And congestion control relies on getting an accurate estimate of the round-trip time [1] https://link.springer.com/chapter/10.1007/978-3-642-36516-4_14 https://link.springer.com/chapter/10.1007/978-3-642-36516-4_... (paywall) Edit: typo Edit: Also, just because 83% of web hosts having it enabled does not imply that it is a good idea to do so in general. They could just all be running the linux defaults and these could be just wrong

djoldman 7y ago
One of the authors of the paper has it linked from his website: https://trammell.ch/publication/ecn-pam-2013/ https://trammell.ch/publication/ecn-pam-2013/
jsnell 7y ago
You can measure RTT continuously, accurately, and even in the presence of packet loss just with selective acks and a little bit of extra bookkeeping in the sender. It's hard to overstate how expensive TCP timestamps are. The thing is that they bloat every single packet including control packets. 2% of the world's bandwidth is being wasted on this. The only reason for anyone to implement TCP timestamps today is that iOS clients have horrible receive window scaling if timestamps are disabled. (Well, that was the only reason a few years ago when I was still in the game of keeping up with the quirks of different TCP stacks.) I wrote more on the subject at the time: https://www.snellman.net/blog/archive/2017-07-20-s3-mystery/ https://www.snellman.net/blog/archive/2017-07-20-s3-mystery/
- toast0 7y ago
  I would guess that Apple could be convinced to fix this, if someone has the right contact. The iOS TCP stack shows a lot of care, generally: they do path MTU probing well, and they've deployed MP-TCP (requires apps to enable it though), among other things I can't remember. Fixing performance if timestamps are disabled seems like something they'd do.

jiveturkey 7y ago

> original initial the word 'initial [sequence number]' is a term of art, not an adjective that can be substituted. very commonly called ISN. odd that the article never used the acronym. > syn cookies are generally bad because they prevent to negotiate window scaling. this is wrong thinking. they degrade performance under "attack", yes. the alternative is instant death. "attacks" need not be from the big bad internet either. my experience is that in the last 10 years most synflood activations are internal buggy or poorly throttled clients. > enabling tcp timestamps in general case brings little benefit disagree, but sibling comment addresses this.

toast0 7y ago
> my experience is that in the last 10 years most synflood activations are internal buggy or poorly throttled clients. From running a large messaging service I agree; most of the perceived attacks (synflood or otherwise) were actually coming from our own clients. But there was a periodic stream of strictly abusive traffic coming from who knows where; mostly UDP reflection, but a SYN flood every once in a while.