10 ms·

What they mostly should know: TCP provides a bidirectional stream of bytes on the application level. It does NOT provide a stream of packets. That means whatev

What they mostly should know: TCP provides a bidirectional stream of bytes on the application level. It does NOT provide a stream of packets. That means whatever you pass to a send() call is not necessarily the same amount of data the receiver will observe in a single read() call. You might get more or less bytes, since the transport layer is free to buffer and to fragment data. I have seen the assumption of TCP having packet boundaries on application level being made too often - typically in stackoverflow questions like: „I don’t receive all data. Is my OS/library broken?“

fenwick67 6y ago
This probably bites lots of newbies, since when you're just sending traffic over localhost, the send()s and read()s tend to line up.
- yjftsjthsd-h 6y ago
  I have often wished for an "unhelpful testing environment" of sorts, to deal with these things before they get out of hand. It would feature a compiler that had creatively different interpretations of undefined behaviors, randomly compile against glibc and musl, have a base OS lovingly crafted from Ubuntu, but with most coreutils replaced with busybox and/or BSD versions. And, now, I suppose, it would have a customized network stack (kernel module?) that would randomly reorder/drop/duplicate packets, randomly reselect MTU on every boot, or maybe just randomly fragments things regardless of MTU. Ideally it would come with a FAQ of "my program broke on X; what did I do wrong?". The idea being that if your software is actually written to relevant standards, and actually handles things properly outside the golden path, then it should still work fine. If, however, you accidentally did something implementation-defined, or that only worked by coincidence, this system will break it.
  - Matthias247 6y ago
    I created such an environment for my unit-tests: Wrapping TCP sockets in a stream which only accepts 1 byte at a time in both directions and returns EAGAIN on every second read provides an easy way to make sure the code on top of the socket does perform all the correct retries. That will most likely not help newcomers which directly write their code agains the OS socket. But once you get a better understanding of the topic and start adding tests to your codebase it's rather easy to add.
    - nitwit005 6y ago
      I've done something similar of forcing the sends to be a single byte at a time. That's usually enough to find the obvious issues in parsing data.
  - jeroenhd 6y ago
    There are tools that intentionally insert failures into the network streams of applications. A few of them are described here: https://medium.com/@docler/network-issues-simulation-how-to-test-against-bad-network-conditions-b28f651d8a96 https://medium.com/@docler/network-issues-simulation-how-to-... The other linking/OS problems can probably be automated with some simple integration tests and a bunch of different docker containers to compile the code in. Should be possible to squeeze it into a CI/CD flow somewhere with some clever tricks.
anilakar 6y ago
In the fall of 2016 I had a lengthy email exchange with an industrial automation vendor who didn't understand this issue. I even mailed them a short Python proof-of-concept snippet that slept a few milliseconds between the write() calls and in response got back my code "fixed" with the sleep removed. In between the emails I googled a bit and found the changelogs for the RTOS they were using. Turned out that it was a bug in the upstream HTTP server. This also meant that the platform they were using had all the security holes from those five-plus years. The bug was later silently fixed when they acquired a newer release from upstream. Currently I'm having a similar issue with the very same vendor. This time they don't understand why client-side authentication means no authentication at all and why passwords must not be stored in plain text in the database that can be remotely backed up from the device.
- throwaway_pdp09 6y ago
  How do you not kill these people? How do you put up with it? How do vendors like this survive?
  - maartenh 6y ago
    Just like in nature, they survive because they are good enough, and don't experience enough competition to be eliminated by selection.
    - jolmg 6y ago
      Depending on what kind of vendor we're talking about, it might be that such aspects aren't even part of what makes them competitive. The average user is not going to know about these types of issues, and so they're not even going to consider such issues when evaluating the vendor.
    - the8472 6y ago
      full disclosure could put some selective pressure on them.
      - 6y ago
richardwhiuk 6y ago
If you do want that, then SCTP will provide it.
- deleted 6y ago
  [deleted]
austincheney 6y ago
Your terminology is a little off. TCP does not provide anything for the application layer as it is transport layer. The application layer rides on top of that. Examples of transport protocols are TCP and UDP while application protocols are things like http, ssh, irc, and all those things your applications use. The network layer on which the transport layer rides is packet switched. The TCP uses segments with each segment having its own header and sequence numbers. Streams are just a series of segments populating across a single established handshake without a prior defined termination segment.
- Matthias247 6y ago
  I didn't meant to talk about OSI terminologies. It was more about: [user-space] applications which use the TCP/IP stack do not observe packet boundaries, whereas the Kernel certainly does. Obviously this is a bit ambiguous, and you can even get packet boundaries in user-space by running a TCP stack there. But for most TCP/IP usages it holds true.
  - austincheney 6y ago
    > It was more about: [user-space] applications which use the TCP/IP stack do not observe packet boundaries That is still a bit imprecise. Userland applications won't directly see TCP as they are just looking at an application protocol. Typically it's the OS that packages and unpacks the application protocol data into a TCP segment, so of course the userland application won't see it since its not managing that part of the communication. https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Network_function https://en.wikipedia.org/wiki/Transmission_Control_Protocol#... There are some exceptions where some application platforms allow developers to write custom TCP protocols, such as Node.js, but these exceptions generally apply to network services and don't commonly apply to the end user application experiance. https://nodejs.org/dist/latest-v14.x/docs/api/net.html#net_net https://nodejs.org/dist/latest-v14.x/docs/api/net.html#net_n...
nicolaslem 6y ago
One way to stop falling into this trap is by knowing what happens behind the send syscall: the application is not sending bytes down the wire, it just fills a buffer in the OS. Once in the buffer there is no boundary between bytes from different send calls. Same thing for receiving, in reverse.
jes5199 6y ago
if you turn off Nagle's algorithm, it gets closer to this though
- jfkebwjsbx 6y ago
  No, it has nothing to do with that.
  - Animats 6y ago
    I suppose I should say something. The thing to turn off is delayed ACKs. See "TCP_QUICKACK". Delayed ACKs were a feature which is only useful for things like Telnet, where the payload in each packet is one character when the user is typing. The fixed timer for delayed ACKs is for keyboard typing speeds, and for networks so slow that human typing could congest them. There's a reasonably good explanation here.[1] As others said above, TCP is not a message protocol. It's a stream protocol. If you're sending messages over a stream, you need something that's reading data from the stream, and when it has a full message, it send that off to be processed. There is no set of TCP options which will reliably cause one write at the sending end to result in one read at the receiving end. If there were, it would be inefficient for small messages and would fail for large ones. [1] https://www.extrahop.com/company/blog/2016/tcp-nodelay-nagle-quickack-best-practices/#1 https://www.extrahop.com/company/blog/2016/tcp-nodelay-nagle...
twotwotwo 6y ago
Yeah. Fun problem for beginners, because 1) your incorrect code may work for a while when reads/writes are small or it's only run on a local network or such, 2) you might design a broken protocol if you don't understand fragmentation, etc., which will tend to be harder than (say) an isolated client bug to fix, 3) the implementation-dependent nature of fragmentation can make it look like you hit a language/library/OS issue, 4) your language/library may or may not offer tools to help a beginner to implement a delimited or framed wire format properly (ideally with things like record-size limits and timeouts). Not sure it says anything you haven't, but a StackOverflow answer on fragmentation (framed by asker as Go not behaving like C) is one of the more-read ones I've written: https://stackoverflow.com/questions/26999615/go-tcp-read-is-non-blocking/27003111 https://stackoverflow.com/questions/26999615/go-tcp-read-is-...
brlewis 6y ago
For me, at least in this decade, it would have been better if I didn't know that. I put off learning websockets longer than I should have because I don't find packet boundaries fun to deal with, and my interest in websockets was mainly for fun. Then when I finally picked websockets up I was pleasantly surprised that message framing is built in.
outworlder 6y ago
> What they mostly should know: TCP provides a bidirectional stream of bytes on the application level. It does NOT provide a stream of packets. > That means whatever you pass to a send() call is not necessarily the same amount of data the receiver will observe in a single read() call. Yes, this. For god's sake, listen to them. I had to fight a coworker on this. I had quickly created some client code just to validate that the server was working. Due to some quirk, all the messages were arriving in full in every read call. He told me to ship it. I said no! "I need to check if there's more data and if so add a loop to read again" "But it is working, release it". That went on for a while, to no avail. Wouldn't look at documentation either. Eventually he head to leave for the day, and I took the time to implement it correctly. I started including basic TCP questions on interviews. Not many people even get past the TCP handshake (if they even know about that).
- Ididntdothis 6y ago
  "But it is working, release it". Famous last words :-)
  - austincheney 6y ago
    Sounds like how most software handles security until it’s audited.
- scott_s 6y ago
  The problem here was not a lack of knowledge of a particular subject. The problem is that this person was unwilling to learn about a thing they thought they knew.
  - draw_down 6y ago
    That's correct.
- SilasX 6y ago
  Stupid question: why would you be writing code that works at the level of TCP? Don't you usually want to use the OS's (or some popular library's) TCP software stack?
Unklejoe 6y ago
> stream of bytes I've always wondered: What's the best/defacto way to delimit this back into packets at the application level on the receiving end? I would think the obvious approach would be to insert some magic word into the stream so that you can re-sync. Or is this not an issue since you know that once you're connected, you'll never drop a single byte, therefore, the only way to get out of sync would be a program error?
- jstanley 6y ago
  You will never drop a single byte. If you need some packet-oriented messaging, you could use something like http://jsonlines.org/ http://jsonlines.org/ (i.e. JSON messages separated by newline characters), or https://github.com/protocolbuffers/protobuf https://github.com/protocolbuffers/protobuf if it's more performance-critical.
  - timeinput 6y ago
    Protobuf isn't self delimiting so you still have to have some extra packet wrapper around it to say the length. I like zeromq to get to a packet based system.
- mytailorisrich 6y ago
  The standard way is to include explicit information on the length of the message that is following. For example if the message is x bytes long then you first send 'x' then you send the x bytes of the message. Or your messages have a defined header that contains the length of the message payload.
- genpfault 6y ago
  Netstrings[1] :) [1]: https://en.wikipedia.org/wiki/Netstring https://en.wikipedia.org/wiki/Netstring
- vasilvv 6y ago
  It will never get out-of-sync because TCP guarantees that the bytes will be delivered in the same order they've arrived. The best approach is typically put a length in front of every message. The good things about that approach are: 1. The receiver can allocate buffer that is exactly the size it needs to fit the message. 2. The receiver can check whether the message is too long before seeing the entire message. The only disadvantage is that you have to know the length of all messages in advance.
wahern 6y ago
A version of Microsoft Exchange had a bug in its SMTP implementation that was tickled when lines crossed packet boundaries. (EDIT: The issue was more likely a bug in Exchange's TLS record processing, breaking when a logical line crossed TLS records.) My async SMTP library used a simple fifo for buffering outbound data which didn't realign the write pointer to 0 except when it was completely drained, so when reading slices (iovec's) from the fifo for write-out it would occasionally call write/send with an incomplete line (i.e. part of a line that wrapped around from the end of the fifo buffer array to the front) even if the application had only written full lines. (At the time it didn't support writev/sendmsg, though I'm not sure it would have helped as the TLS record layer might still have been prone to splitting logical lines across packets.) There was no bug here on my end--everything would be sent correctly--but you can't tell the customer that he can't send e-mail to some third-party because that third-party is using a broken version of Exchange. The first quick fix was to unconditionally realign the fifo contents after every write (the fifo had a realign method), but that ran into a computational complexity problem when you had lots of small lines (e.g. the application caller dumped a huge message into the buffer and then flushed it out in one go) and a high-latency connection that resulted in many short writes; you were constantly memmove'ing the megabytes of remaining contents in the buffer for every tiny write you did. So then I ended up having to add a new interface to the fifo that returned a slice up to a limit but always ending with a specified delimiter (e.g. "\n") if the delimiter was within the maximum chunk size. Of course, none of these fixes would have completely remedied the issue as lower layers (the TLS stack, the kernel TCP stack) could have still potentially split logical lines, and I'm sure did on occasion. But it at least seemed to put us on equal footing with everybody else in terms of how often it happened, which is really the best anybody could have done. Complaints did die down.
keitmo 6y ago
I often tell people to assume the TCP stack's buffering is arbitrary & capricious and will do the most inconvenient thing for your code. That can mean ether a) dribbling data in one byte at a time per recv() call, or b) buffering multiple megabytes and returning it all in a single recv() call.

fenwick67 6y ago

This probably bites lots of newbies, since when you're just sending traffic over localhost, the send()s and read()s tend to line up.

yjftsjthsd-h 6y ago
I have often wished for an "unhelpful testing environment" of sorts, to deal with these things before they get out of hand. It would feature a compiler that had creatively different interpretations of undefined behaviors, randomly compile against glibc and musl, have a base OS lovingly crafted from Ubuntu, but with most coreutils replaced with busybox and/or BSD versions. And, now, I suppose, it would have a customized network stack (kernel module?) that would randomly reorder/drop/duplicate packets, randomly reselect MTU on every boot, or maybe just randomly fragments things regardless of MTU. Ideally it would come with a FAQ of "my program broke on X; what did I do wrong?". The idea being that if your software is actually written to relevant standards, and actually handles things properly outside the golden path, then it should still work fine. If, however, you accidentally did something implementation-defined, or that only worked by coincidence, this system will break it.
- Matthias247 6y ago
  I created such an environment for my unit-tests: Wrapping TCP sockets in a stream which only accepts 1 byte at a time in both directions and returns EAGAIN on every second read provides an easy way to make sure the code on top of the socket does perform all the correct retries. That will most likely not help newcomers which directly write their code agains the OS socket. But once you get a better understanding of the topic and start adding tests to your codebase it's rather easy to add.
  - nitwit005 6y ago
    I've done something similar of forcing the sends to be a single byte at a time. That's usually enough to find the obvious issues in parsing data.
- jeroenhd 6y ago
  There are tools that intentionally insert failures into the network streams of applications. A few of them are described here: https://medium.com/@docler/network-issues-simulation-how-to-test-against-bad-network-conditions-b28f651d8a96 https://medium.com/@docler/network-issues-simulation-how-to-... The other linking/OS problems can probably be automated with some simple integration tests and a bunch of different docker containers to compile the code in. Should be possible to squeeze it into a CI/CD flow somewhere with some clever tricks.

anilakar 6y ago

In the fall of 2016 I had a lengthy email exchange with an industrial automation vendor who didn't understand this issue. I even mailed them a short Python proof-of-concept snippet that slept a few milliseconds between the write() calls and in response got back my code "fixed" with the sleep removed. In between the emails I googled a bit and found the changelogs for the RTOS they were using. Turned out that it was a bug in the upstream HTTP server. This also meant that the platform they were using had all the security holes from those five-plus years. The bug was later silently fixed when they acquired a newer release from upstream. Currently I'm having a similar issue with the very same vendor. This time they don't understand why client-side authentication means no authentication at all and why passwords must not be stored in plain text in the database that can be remotely backed up from the device.

throwaway_pdp09 6y ago
How do you not kill these people? How do you put up with it? How do vendors like this survive?
- maartenh 6y ago
  Just like in nature, they survive because they are good enough, and don't experience enough competition to be eliminated by selection.
  - jolmg 6y ago
    Depending on what kind of vendor we're talking about, it might be that such aspects aren't even part of what makes them competitive. The average user is not going to know about these types of issues, and so they're not even going to consider such issues when evaluating the vendor.
  - the8472 6y ago
    full disclosure could put some selective pressure on them.
    - 6y ago

richardwhiuk 6y ago

If you do want that, then SCTP will provide it.

deleted 6y ago
[deleted]

austincheney 6y ago

Your terminology is a little off. TCP does not provide anything for the application layer as it is transport layer. The application layer rides on top of that. Examples of transport protocols are TCP and UDP while application protocols are things like http, ssh, irc, and all those things your applications use. The network layer on which the transport layer rides is packet switched. The TCP uses segments with each segment having its own header and sequence numbers. Streams are just a series of segments populating across a single established handshake without a prior defined termination segment.

Matthias247 6y ago
I didn't meant to talk about OSI terminologies. It was more about: [user-space] applications which use the TCP/IP stack do not observe packet boundaries, whereas the Kernel certainly does. Obviously this is a bit ambiguous, and you can even get packet boundaries in user-space by running a TCP stack there. But for most TCP/IP usages it holds true.
- austincheney 6y ago
  > It was more about: [user-space] applications which use the TCP/IP stack do not observe packet boundaries That is still a bit imprecise. Userland applications won't directly see TCP as they are just looking at an application protocol. Typically it's the OS that packages and unpacks the application protocol data into a TCP segment, so of course the userland application won't see it since its not managing that part of the communication. https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Network_function https://en.wikipedia.org/wiki/Transmission_Control_Protocol#... There are some exceptions where some application platforms allow developers to write custom TCP protocols, such as Node.js, but these exceptions generally apply to network services and don't commonly apply to the end user application experiance. https://nodejs.org/dist/latest-v14.x/docs/api/net.html#net_net https://nodejs.org/dist/latest-v14.x/docs/api/net.html#net_n...

nicolaslem 6y ago

One way to stop falling into this trap is by knowing what happens behind the send syscall: the application is not sending bytes down the wire, it just fills a buffer in the OS. Once in the buffer there is no boundary between bytes from different send calls. Same thing for receiving, in reverse.

jes5199 6y ago

if you turn off Nagle's algorithm, it gets closer to this though

jfkebwjsbx 6y ago
No, it has nothing to do with that.
- Animats 6y ago
  I suppose I should say something. The thing to turn off is delayed ACKs. See "TCP_QUICKACK". Delayed ACKs were a feature which is only useful for things like Telnet, where the payload in each packet is one character when the user is typing. The fixed timer for delayed ACKs is for keyboard typing speeds, and for networks so slow that human typing could congest them. There's a reasonably good explanation here.[1] As others said above, TCP is not a message protocol. It's a stream protocol. If you're sending messages over a stream, you need something that's reading data from the stream, and when it has a full message, it send that off to be processed. There is no set of TCP options which will reliably cause one write at the sending end to result in one read at the receiving end. If there were, it would be inefficient for small messages and would fail for large ones. [1] https://www.extrahop.com/company/blog/2016/tcp-nodelay-nagle-quickack-best-practices/#1 https://www.extrahop.com/company/blog/2016/tcp-nodelay-nagle...

twotwotwo 6y ago

Yeah. Fun problem for beginners, because 1) your incorrect code may work for a while when reads/writes are small or it's only run on a local network or such, 2) you might design a broken protocol if you don't understand fragmentation, etc., which will tend to be harder than (say) an isolated client bug to fix, 3) the implementation-dependent nature of fragmentation can make it look like you hit a language/library/OS issue, 4) your language/library may or may not offer tools to help a beginner to implement a delimited or framed wire format properly (ideally with things like record-size limits and timeouts). Not sure it says anything you haven't, but a StackOverflow answer on fragmentation (framed by asker as Go not behaving like C) is one of the more-read ones I've written: https://stackoverflow.com/questions/26999615/go-tcp-read-is-non-blocking/27003111 https://stackoverflow.com/questions/26999615/go-tcp-read-is-...

brlewis 6y ago

For me, at least in this decade, it would have been better if I didn't know that. I put off learning websockets longer than I should have because I don't find packet boundaries fun to deal with, and my interest in websockets was mainly for fun. Then when I finally picked websockets up I was pleasantly surprised that message framing is built in.

outworlder 6y ago

> What they mostly should know: TCP provides a bidirectional stream of bytes on the application level. It does NOT provide a stream of packets. > That means whatever you pass to a send() call is not necessarily the same amount of data the receiver will observe in a single read() call. Yes, this. For god's sake, listen to them. I had to fight a coworker on this. I had quickly created some client code just to validate that the server was working. Due to some quirk, all the messages were arriving in full in every read call. He told me to ship it. I said no! "I need to check if there's more data and if so add a loop to read again" "But it is working, release it". That went on for a while, to no avail. Wouldn't look at documentation either. Eventually he head to leave for the day, and I took the time to implement it correctly. I started including basic TCP questions on interviews. Not many people even get past the TCP handshake (if they even know about that).

Ididntdothis 6y ago
"But it is working, release it". Famous last words :-)
- austincheney 6y ago
  Sounds like how most software handles security until it’s audited.
scott_s 6y ago
The problem here was not a lack of knowledge of a particular subject. The problem is that this person was unwilling to learn about a thing they thought they knew.
- draw_down 6y ago
  That's correct.
SilasX 6y ago
Stupid question: why would you be writing code that works at the level of TCP? Don't you usually want to use the OS's (or some popular library's) TCP software stack?

Unklejoe 6y ago

> stream of bytes I've always wondered: What's the best/defacto way to delimit this back into packets at the application level on the receiving end? I would think the obvious approach would be to insert some magic word into the stream so that you can re-sync. Or is this not an issue since you know that once you're connected, you'll never drop a single byte, therefore, the only way to get out of sync would be a program error?

jstanley 6y ago
You will never drop a single byte. If you need some packet-oriented messaging, you could use something like http://jsonlines.org/ http://jsonlines.org/ (i.e. JSON messages separated by newline characters), or https://github.com/protocolbuffers/protobuf https://github.com/protocolbuffers/protobuf if it's more performance-critical.
- timeinput 6y ago
  Protobuf isn't self delimiting so you still have to have some extra packet wrapper around it to say the length. I like zeromq to get to a packet based system.
mytailorisrich 6y ago
The standard way is to include explicit information on the length of the message that is following. For example if the message is x bytes long then you first send 'x' then you send the x bytes of the message. Or your messages have a defined header that contains the length of the message payload.
genpfault 6y ago
Netstrings[1] :) [1]: https://en.wikipedia.org/wiki/Netstring https://en.wikipedia.org/wiki/Netstring
vasilvv 6y ago
It will never get out-of-sync because TCP guarantees that the bytes will be delivered in the same order they've arrived. The best approach is typically put a length in front of every message. The good things about that approach are: 1. The receiver can allocate buffer that is exactly the size it needs to fit the message. 2. The receiver can check whether the message is too long before seeing the entire message. The only disadvantage is that you have to know the length of all messages in advance.

wahern 6y ago

A version of Microsoft Exchange had a bug in its SMTP implementation that was tickled when lines crossed packet boundaries. (EDIT: The issue was more likely a bug in Exchange's TLS record processing, breaking when a logical line crossed TLS records.) My async SMTP library used a simple fifo for buffering outbound data which didn't realign the write pointer to 0 except when it was completely drained, so when reading slices (iovec's) from the fifo for write-out it would occasionally call write/send with an incomplete line (i.e. part of a line that wrapped around from the end of the fifo buffer array to the front) even if the application had only written full lines. (At the time it didn't support writev/sendmsg, though I'm not sure it would have helped as the TLS record layer might still have been prone to splitting logical lines across packets.) There was no bug here on my end--everything would be sent correctly--but you can't tell the customer that he can't send e-mail to some third-party because that third-party is using a broken version of Exchange. The first quick fix was to unconditionally realign the fifo contents after every write (the fifo had a realign method), but that ran into a computational complexity problem when you had lots of small lines (e.g. the application caller dumped a huge message into the buffer and then flushed it out in one go) and a high-latency connection that resulted in many short writes; you were constantly memmove'ing the megabytes of remaining contents in the buffer for every tiny write you did. So then I ended up having to add a new interface to the fifo that returned a slice up to a limit but always ending with a specified delimiter (e.g. "\n") if the delimiter was within the maximum chunk size. Of course, none of these fixes would have completely remedied the issue as lower layers (the TLS stack, the kernel TCP stack) could have still potentially split logical lines, and I'm sure did on occasion. But it at least seemed to put us on equal footing with everybody else in terms of how often it happened, which is really the best anybody could have done. Complaints did die down.

keitmo 6y ago

I often tell people to assume the TCP stack's buffering is arbitrary & capricious and will do the most inconvenient thing for your code. That can mean ether a) dribbling data in one byte at a time per recv() call, or b) buffering multiple megabytes and returning it all in a single recv() call.