6 ms·

It seems to me GP is talking about using TCP, not implementing it.

SilasX 6y ago
Right, I mean I thought that the TCP protocol implementation itself handled that issue, and your calls to such a library abstracted away from that.
- jacobolus 6y ago
  No, it does not, by design. When you use any standard TCP implementation the abstraction provided at both ends is just a stream of bytes. The guarantee TCP makes is that the bytes received at one end will be in the same order as the bytes that were sent at the other end. If you want to use TCP to implement some (higher-level) message-based protocol, you need to parse those out for yourself.
  - SilasX 6y ago
    The original question didn't refer to such issues of parsing, just checking for whether the full message was received, which, I would think, would be handled by the TCP library read() (or whatever) call. It sounded like the OP (outworlder) was delving into lower-level TCP details that should have been abstracted away -- my thinking is that the TCP caller shouldn't have to concern itself with details about checking whether the full message has been received, at least not as a separate step. That is, it just wouldn't return anything until the full message is received, or would include some data structure that indicates it's not complete. Does that make sense? Edit: On second thought, I guess OP meant that all of the results were coming back "complete" which doesn't obviate the issue of needing to do a check that handles the "not done" case.
    - Nursie 6y ago
      > which, I would think, would be handled by the TCP library read() You would be wrong.
      - SilasX 6y ago
        Thanks for the explanation, but I don't see why a library utility function wouldn't do that.
        anonymoushn 6y ago
        Most TCP libraries simply do not come with a function that does "read, but only give me 0 bytes or n bytes, and if you get some other number please hang on to the leftovers until next time". I guess I could follow your advice, but the first step of doing that would be to write such a function again.
        userbinator 6y ago
        They do... with an option called MSG_WAITALL. That naturally leads to the question of why it's not the default, which can be answered by understanding the history of TCP and computer networking; and more interestingly, how might things have been different if MSG_WAITALL was the default from the beginning.
        anonymoushn 6y ago
        I don't think this gets you nonblocking all-or-nothing recv.
        spydum 6y ago
        But if the goal is to process or retrieve a string of bytes, how do you know when you've got them all? That is the root of the problem: tcp isn't built to exchange messages, it's just the stream transport layer. If you want messages you have to encode and decide that yourself.
        SilasX 6y ago
        Sorry, the rest of the conversation seemed to be assuming that "whether you have received the full message" is well-defined at the level of TCP, as suggested by the original comment where I joined[1]: >I had to fight a coworker on this. I had quickly created some client code just to validate that the server was working. Due to some quirk, all the messages were arriving in full in every read call. He told me to ship it. >I said no! "I need to check if there's more data and if so add a loop to read again" "But it is working, release it". That went on for a while, to no avail. Wouldn't look at documentation either. I was just going along with that. I didn't check up on TCP further to verify whether this was actually true; if you are saying that's not a well-defined concept, you might want to reply to that comment to say so. [1] https://news.ycombinator.com/item?id=23195230 https://news.ycombinator.com/item?id=23195230
    - jacobolus 6y ago
      > the TCP caller shouldn't have to concern itself with details about checking whether the full message has been received The caller absolutely must concern itself with this crucial “detail”. If you do otherwise, your code is broken, full stop. You can implement a higher-level protocol on top which handles this kind of thing internally and presents a higher-level interface (e.g. not passing any partial data along to its caller until a full message has been received), but if you are just working with TCP directly, what you get is just a stream of bytes. The guarantee you get is that the bytes will be in order and without any gaps. If you e.g. send UTF-8 encoded text, you must be prepared on the read side to have your stream of bytes cut off arbitrarily in the middle of a character.
      - SilasX 6y ago
        Let function A call TCP read() and pass back a struct that includes the bytes it's received and a flag that indicates whether it's read to the end of the message. Let function B call TCP read() and never return anything until it's received all the bytes of the message. Both of those seem (IMHO) like functions you could have in a TCP library. Neither seems (IMHO) like a higher level protocol.
        jayshua 6y ago
        That might be nice. But that's not how it works.
        Matthias247 6y ago
        There is no "message" in TCP. So no - you can not have this in a library.
        yuribro 6y ago
        The reason that this function doesn't exist is that even if the sender sent a "message" in a single API call, there is no guarantee that the networking code will send it in one IP packet (or one layer 2 frame). We don't want to couple the message size with the lower network protocol, the type of network equipment and so on, and also want to allow merging of messages for more efficient use of the network (If we have a large window size it allows better throughput). So the job of separating the stream into messages is left to the application layer. (unlike for example in UDP, but then you have to worry about dropped messages)
- bavell 6y ago
  If you're using a library, then sure. But if you're just reading from a raw TCP socket, it's just a stream of bytes. It's up to your application to parse those bytes (e.g. into a http request). The OS will buffer bytes received from TCP packets for you until you read from the socket again to drain the buffer. Your application needs to determine how to semantically chop those bytes up into the protocol it's expecting (e.g. http request). My low-level networking chops are a little rusty so please correct my understanding if I'm off-base somewhere.

Right, I mean I thought that the TCP protocol implementation itself handled that issue, and your calls to such a library abstracted away from that.

jacobolus 6y ago
No, it does not, by design. When you use any standard TCP implementation the abstraction provided at both ends is just a stream of bytes. The guarantee TCP makes is that the bytes received at one end will be in the same order as the bytes that were sent at the other end. If you want to use TCP to implement some (higher-level) message-based protocol, you need to parse those out for yourself.
- SilasX 6y ago
  The original question didn't refer to such issues of parsing, just checking for whether the full message was received, which, I would think, would be handled by the TCP library read() (or whatever) call. It sounded like the OP (outworlder) was delving into lower-level TCP details that should have been abstracted away -- my thinking is that the TCP caller shouldn't have to concern itself with details about checking whether the full message has been received, at least not as a separate step. That is, it just wouldn't return anything until the full message is received, or would include some data structure that indicates it's not complete. Does that make sense? Edit: On second thought, I guess OP meant that all of the results were coming back "complete" which doesn't obviate the issue of needing to do a check that handles the "not done" case.
  - Nursie 6y ago
    > which, I would think, would be handled by the TCP library read() You would be wrong.
    - SilasX 6y ago
      Thanks for the explanation, but I don't see why a library utility function wouldn't do that.
      - anonymoushn 6y ago
        Most TCP libraries simply do not come with a function that does "read, but only give me 0 bytes or n bytes, and if you get some other number please hang on to the leftovers until next time". I guess I could follow your advice, but the first step of doing that would be to write such a function again.
        userbinator 6y ago
        They do... with an option called MSG_WAITALL. That naturally leads to the question of why it's not the default, which can be answered by understanding the history of TCP and computer networking; and more interestingly, how might things have been different if MSG_WAITALL was the default from the beginning.
        anonymoushn 6y ago
        I don't think this gets you nonblocking all-or-nothing recv.
      - spydum 6y ago
        But if the goal is to process or retrieve a string of bytes, how do you know when you've got them all? That is the root of the problem: tcp isn't built to exchange messages, it's just the stream transport layer. If you want messages you have to encode and decide that yourself.
        SilasX 6y ago
        Sorry, the rest of the conversation seemed to be assuming that "whether you have received the full message" is well-defined at the level of TCP, as suggested by the original comment where I joined[1]: >I had to fight a coworker on this. I had quickly created some client code just to validate that the server was working. Due to some quirk, all the messages were arriving in full in every read call. He told me to ship it. >I said no! "I need to check if there's more data and if so add a loop to read again" "But it is working, release it". That went on for a while, to no avail. Wouldn't look at documentation either. I was just going along with that. I didn't check up on TCP further to verify whether this was actually true; if you are saying that's not a well-defined concept, you might want to reply to that comment to say so. [1] https://news.ycombinator.com/item?id=23195230 https://news.ycombinator.com/item?id=23195230
  - jacobolus 6y ago
    > the TCP caller shouldn't have to concern itself with details about checking whether the full message has been received The caller absolutely must concern itself with this crucial “detail”. If you do otherwise, your code is broken, full stop. You can implement a higher-level protocol on top which handles this kind of thing internally and presents a higher-level interface (e.g. not passing any partial data along to its caller until a full message has been received), but if you are just working with TCP directly, what you get is just a stream of bytes. The guarantee you get is that the bytes will be in order and without any gaps. If you e.g. send UTF-8 encoded text, you must be prepared on the read side to have your stream of bytes cut off arbitrarily in the middle of a character.
    - SilasX 6y ago
      Let function A call TCP read() and pass back a struct that includes the bytes it's received and a flag that indicates whether it's read to the end of the message. Let function B call TCP read() and never return anything until it's received all the bytes of the message. Both of those seem (IMHO) like functions you could have in a TCP library. Neither seems (IMHO) like a higher level protocol.
      - jayshua 6y ago
        That might be nice. But that's not how it works.
      - Matthias247 6y ago
        There is no "message" in TCP. So no - you can not have this in a library.
      - yuribro 6y ago
        The reason that this function doesn't exist is that even if the sender sent a "message" in a single API call, there is no guarantee that the networking code will send it in one IP packet (or one layer 2 frame). We don't want to couple the message size with the lower network protocol, the type of network equipment and so on, and also want to allow merging of messages for more efficient use of the network (If we have a large window size it allows better throughput). So the job of separating the stream into messages is left to the application layer. (unlike for example in UDP, but then you have to worry about dropped messages)
bavell 6y ago
If you're using a library, then sure. But if you're just reading from a raw TCP socket, it's just a stream of bytes. It's up to your application to parse those bytes (e.g. into a http request). The OS will buffer bytes received from TCP packets for you until you read from the socket again to drain the buffer. Your application needs to determine how to semantically chop those bytes up into the protocol it's expecting (e.g. http request). My low-level networking chops are a little rusty so please correct my understanding if I'm off-base somewhere.