10 ms·

TCP is harder than it looks

175 points by jsnell 12y ago

leonardinius 12y ago
I've to confess I've quite limited knowledge with TCP/IP stack internals, e.g. the way stack extensions work et cetera. Does anyone know of any available online visual materials/tutorials? I'm particularly searching for tools capable of recording and replaying TCP/IP stack packets with visual representation, references to RFCs and specifications.
- dgdsgdsg 12y ago
  http://www.pcvr.nl/tcpip/ http://www.pcvr.nl/tcpip/
- dgdsgdsg 12y ago
  https://www.wireshark.org/ https://www.wireshark.org/
- agumonkey 12y ago
  Coursera and stanford mooc helped me understand networking deeper, you might try to apply.
  - signa11 12y ago
    which ones if you could point them out please ?
    - 1945795 12y ago
      https://www.coursera.org/course/comnetworks https://www.coursera.org/course/comnetworks http://networking.class.stanford.edu/ http://networking.class.stanford.edu/
- mahmud 12y ago
  Doug Comer's "Internetworking with TCP IP" volumes 1&2. Walks you through the protocol suite along with the implementation of a simple stack in C. https://www.cs.purdue.edu/homes/comer/netbooks.html https://www.cs.purdue.edu/homes/comer/netbooks.html Richard Steven's "TCP/IP Illustrated" books are the original works, Unix network programming at its finest. More advanced than Comer's book, and much more applicable. Get used, older editions if cheaper, you will be fine.
acqq 12y ago
The author uses the customized settings for his TCP stack and then laments that some nodes on the internet which aren't under his control depend on more common settings. Honestly I don't see why he could have expected any different outcome.
- philh 12y ago
  The other node doesn't depend on the settings, it depends on not receiving a duplicate SYN.
  - acqq 12y ago
    Note what he actually writes: "Our TCP implementation has a variable base SYN retransmit timout, and in this case it was roughly 500ms. So most of the time the page load would fail with our TCP stack, but succeed with an off the shelf one that had a SYN retransmit timeout of 1 second." If his timeout were 1 second, his connection with the other node would work. It's his own choice to insist on "variable base SYN retransmit timeout" even with the nodes which weren't tested with such a setup. He also says: "The options are either to tell a customer that some traffic won't work (which is normally unacceptable, even if the root cause is undeniably on the other end), or to water down a useful feature a little bit to at least fail no more often than the "competition" does." Meaning, the way they use that less common setting is "all or nothing": either all clients or none. With the assumption that that way he will be "competitive." Again, why is then surprising to discover that "the whole world" is not perfect, once you implement your part with the assumption it were? Just because he knows what the "ideal case" is he can't expect that everything he confronts would be ideal.
    - viraptor 12y ago
      I don't think it was an issue with the timeout. The other side simply could not handle retransmitted SYN. At all. The 500ms comes up only because that happened to be their RTT. If they used 1 second, this host would work, but another one using the same stack but with 1 second delay would continue to fail. It's not about the other host not having an ideal implementation, or about testing ideal cases. The other host had a bug which cannot cope with some situations. There's nothing they could do about it, apart from tweaking the timeout to some bigger value.
longwave 12y ago
This reminded me of the best TCP timeout story I've ever heard, the case of the 500-mile email: http://www.ibiblio.org/harris/500milemail.html http://www.ibiblio.org/harris/500milemail.html
- michaelx386 12y ago
  Great story with the added bonus of showing me the GNU Units command. What a fantastic program, no more Googling when I need to convert stuff.
  - MrBuddyCasino 12y ago
    Indeed, I just checked OSX - sadly, units doesn't know about millilightseconds ("586 units, 56 prefixes").
    - abecedarius 12y ago
      "milli lightsecond" should do it.
      - gregsadetsky 12y ago
        "milli lightsecond" doesn't work on OSX unfortunately (under 10.6 at least). One value which does work is "3 c-millisecond". (I learned this by reading the /usr/share/misc/units.lib file and finding the lightyear definition: "c-yr")
        abecedarius 12y ago
        Ah, whoops, I assumed it was GNU Units. Thanks and c-millisecond is cute.
    - taejo 12y ago
      GNU Units (available in Homebrew) is much more complete than the OS X version.
jfuhrman 12y ago
Pretty much everything in technology is harder than it looks. Took me a day to configure Apache Solr for the first time the other day while my estimate was just one hour.
Jabbles 12y ago
Is there a history of these compatibility workarounds being the source of security bugs(e.g. DoS)?
- zwp 12y ago
  Yes. See, for example: https://en.wikipedia.org/wiki/Christmas_tree_packet https://en.wikipedia.org/wiki/Christmas_tree_packet https://en.wikipedia.org/wiki/Ping_of_death https://en.wikipedia.org/wiki/Ping_of_death
nabla9 12y ago
What is the general lesson we should learn from this? Postel's law aka robustness principle [1] can easily lead into accumulating complexity when implementations adapt to the bugs in other implementations. How could protocol designers mitigate this problem beforehand? [1]: https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle
- cesarb 12y ago
  The lesson is "be strict in what you accept, since the beginning". For instance, a lesson learned by the Linux kernel developers: when adding a new system call to an operating system, if you have a flags parameter (and you should, which is another lesson they learned), if any unknown flag is set, fail with -EINVAL (or equivalent). Otherwise, buggy programs will depend on these flags being ignored, and you won't be able to use them later for new features. But it has to be since the beginning; once the protocol is "in the field", you can't increase the strictness without pain.
  - sillysaurus3 12y ago
    Otherwise, buggy programs will depend on these flags being ignored, and you won't be able to use them later for new features. I don't understand why that's true. Just add the new flag and use it for your new feature. Aren't buggy programs who were already sending the flag responsible for their own bugs?
    - oranlooney 12y ago
      Yes and no. If you push an update and sysadmins all over the world dutifully upgrade and immediately notice programs breaking, their first thought is not going to be "oh, those programs must have had latent bugs." No, they're going to blame you. Besides, having stuff work correctly is always more fun than assigning blame, and validating flags is an easy way to avoid these scenarios.
      - sillysaurus3 12y ago
known 12y ago
I miss http://www.kohala.com/start/ http://www.kohala.com/start/
ay 12y ago
SYN-ACK which does not get retransmitted, advertises a zero window and does not have options looks like some implementation of SYN cookies. A nice (and useful in some cases) hack but a violation of the TCP spec, which is why it is disabled by default in most places that implement it.
markb139 12y ago
Electricity doesn't conduct at the speed of light. Its about 2/3 the speed of light.

175 points by jsnell 12y ago

I've to confess I've quite limited knowledge with TCP/IP stack internals, e.g. the way stack extensions work et cetera. Does anyone know of any available online visual materials/tutorials? I'm particularly searching for tools capable of recording and replaying TCP/IP stack packets with visual representation, references to RFCs and specifications.

dgdsgdsg 12y ago
http://www.pcvr.nl/tcpip/ http://www.pcvr.nl/tcpip/
dgdsgdsg 12y ago
https://www.wireshark.org/ https://www.wireshark.org/
agumonkey 12y ago
Coursera and stanford mooc helped me understand networking deeper, you might try to apply.
- signa11 12y ago
  which ones if you could point them out please ?
  - 1945795 12y ago
    https://www.coursera.org/course/comnetworks https://www.coursera.org/course/comnetworks http://networking.class.stanford.edu/ http://networking.class.stanford.edu/
mahmud 12y ago
Doug Comer's "Internetworking with TCP IP" volumes 1&2. Walks you through the protocol suite along with the implementation of a simple stack in C. https://www.cs.purdue.edu/homes/comer/netbooks.html https://www.cs.purdue.edu/homes/comer/netbooks.html Richard Steven's "TCP/IP Illustrated" books are the original works, Unix network programming at its finest. More advanced than Comer's book, and much more applicable. Get used, older editions if cheaper, you will be fine.

acqq 12y ago

The author uses the customized settings for his TCP stack and then laments that some nodes on the internet which aren't under his control depend on more common settings. Honestly I don't see why he could have expected any different outcome.

philh 12y ago
The other node doesn't depend on the settings, it depends on not receiving a duplicate SYN.
- acqq 12y ago
  Note what he actually writes: "Our TCP implementation has a variable base SYN retransmit timout, and in this case it was roughly 500ms. So most of the time the page load would fail with our TCP stack, but succeed with an off the shelf one that had a SYN retransmit timeout of 1 second." If his timeout were 1 second, his connection with the other node would work. It's his own choice to insist on "variable base SYN retransmit timeout" even with the nodes which weren't tested with such a setup. He also says: "The options are either to tell a customer that some traffic won't work (which is normally unacceptable, even if the root cause is undeniably on the other end), or to water down a useful feature a little bit to at least fail no more often than the "competition" does." Meaning, the way they use that less common setting is "all or nothing": either all clients or none. With the assumption that that way he will be "competitive." Again, why is then surprising to discover that "the whole world" is not perfect, once you implement your part with the assumption it were? Just because he knows what the "ideal case" is he can't expect that everything he confronts would be ideal.
  - viraptor 12y ago
    I don't think it was an issue with the timeout. The other side simply could not handle retransmitted SYN. At all. The 500ms comes up only because that happened to be their RTT. If they used 1 second, this host would work, but another one using the same stack but with 1 second delay would continue to fail. It's not about the other host not having an ideal implementation, or about testing ideal cases. The other host had a bug which cannot cope with some situations. There's nothing they could do about it, apart from tweaking the timeout to some bigger value.

longwave 12y ago

This reminded me of the best TCP timeout story I've ever heard, the case of the 500-mile email: http://www.ibiblio.org/harris/500milemail.html http://www.ibiblio.org/harris/500milemail.html

michaelx386 12y ago
Great story with the added bonus of showing me the GNU Units command. What a fantastic program, no more Googling when I need to convert stuff.
- MrBuddyCasino 12y ago
  Indeed, I just checked OSX - sadly, units doesn't know about millilightseconds ("586 units, 56 prefixes").
  - abecedarius 12y ago
    "milli lightsecond" should do it.
    - gregsadetsky 12y ago
      "milli lightsecond" doesn't work on OSX unfortunately (under 10.6 at least). One value which does work is "3 c-millisecond". (I learned this by reading the /usr/share/misc/units.lib file and finding the lightyear definition: "c-yr")
      - abecedarius 12y ago
        Ah, whoops, I assumed it was GNU Units. Thanks and c-millisecond is cute.
  - taejo 12y ago
    GNU Units (available in Homebrew) is much more complete than the OS X version.

jfuhrman 12y ago

Pretty much everything in technology is harder than it looks. Took me a day to configure Apache Solr for the first time the other day while my estimate was just one hour.

Jabbles 12y ago

Is there a history of these compatibility workarounds being the source of security bugs(e.g. DoS)?

zwp 12y ago
Yes. See, for example: https://en.wikipedia.org/wiki/Christmas_tree_packet https://en.wikipedia.org/wiki/Christmas_tree_packet https://en.wikipedia.org/wiki/Ping_of_death https://en.wikipedia.org/wiki/Ping_of_death

nabla9 12y ago

What is the general lesson we should learn from this? Postel's law aka robustness principle [1] can easily lead into accumulating complexity when implementations adapt to the bugs in other implementations. How could protocol designers mitigate this problem beforehand? [1]: https://en.wikipedia.org/wiki/Robustness_principle https://en.wikipedia.org/wiki/Robustness_principle

cesarb 12y ago
The lesson is "be strict in what you accept, since the beginning". For instance, a lesson learned by the Linux kernel developers: when adding a new system call to an operating system, if you have a flags parameter (and you should, which is another lesson they learned), if any unknown flag is set, fail with -EINVAL (or equivalent). Otherwise, buggy programs will depend on these flags being ignored, and you won't be able to use them later for new features. But it has to be since the beginning; once the protocol is "in the field", you can't increase the strictness without pain.
- sillysaurus3 12y ago
  Otherwise, buggy programs will depend on these flags being ignored, and you won't be able to use them later for new features. I don't understand why that's true. Just add the new flag and use it for your new feature. Aren't buggy programs who were already sending the flag responsible for their own bugs?
  - oranlooney 12y ago
    Yes and no. If you push an update and sysadmins all over the world dutifully upgrade and immediately notice programs breaking, their first thought is not going to be "oh, those programs must have had latent bugs." No, they're going to blame you. Besides, having stuff work correctly is always more fun than assigning blame, and validating flags is an easy way to avoid these scenarios.
    - sillysaurus3 12y ago

known 12y ago

I miss http://www.kohala.com/start/ http://www.kohala.com/start/

ay 12y ago

SYN-ACK which does not get retransmitted, advertises a zero window and does not have options looks like some implementation of SYN cookies. A nice (and useful in some cases) hack but a violation of the TCP spec, which is why it is disabled by default in most places that implement it.

markb139 12y ago

Electricity doesn't conduct at the speed of light. Its about 2/3 the speed of light.