5 ms·

> stream of bytes I've always wondered: What's the best/defacto way to delimit this back into packets at the application level on the receiving end? I would t

> stream of bytes I've always wondered: What's the best/defacto way to delimit this back into packets at the application level on the receiving end? I would think the obvious approach would be to insert some magic word into the stream so that you can re-sync. Or is this not an issue since you know that once you're connected, you'll never drop a single byte, therefore, the only way to get out of sync would be a program error?

jstanley 6y ago
You will never drop a single byte. If you need some packet-oriented messaging, you could use something like http://jsonlines.org/ http://jsonlines.org/ (i.e. JSON messages separated by newline characters), or https://github.com/protocolbuffers/protobuf https://github.com/protocolbuffers/protobuf if it's more performance-critical.
- timeinput 6y ago
  Protobuf isn't self delimiting so you still have to have some extra packet wrapper around it to say the length. I like zeromq to get to a packet based system.
mytailorisrich 6y ago
The standard way is to include explicit information on the length of the message that is following. For example if the message is x bytes long then you first send 'x' then you send the x bytes of the message. Or your messages have a defined header that contains the length of the message payload.
genpfault 6y ago
Netstrings[1] :) [1]: https://en.wikipedia.org/wiki/Netstring https://en.wikipedia.org/wiki/Netstring
vasilvv 6y ago
It will never get out-of-sync because TCP guarantees that the bytes will be delivered in the same order they've arrived. The best approach is typically put a length in front of every message. The good things about that approach are: 1. The receiver can allocate buffer that is exactly the size it needs to fit the message. 2. The receiver can check whether the message is too long before seeing the entire message. The only disadvantage is that you have to know the length of all messages in advance.
- kccqzy 6y ago
  Definitely be sure to check the length though. Imagine a mistaken client trying to send HTTP, but of course the first four bytes "HTTP" when interpreted as a 32-bit integer, whichever endian, is an absurdly large buffer.
  - nucleardog 6y ago
    I mean, at this point you're effectively defining a new network protocol (or, you will be shortly once you implement ways to work around all the other issues you're going to run into). I'd go all-in from the start and start every packet with a magic string/byte sequence of your own, a length, and probably a version code just to make it extensible. Or see if there's an existing protocol you can abuse for what you want. If it's transactional, you get a pretty big ecosystem of battle-tested clients/servers/proxies/etc if you use HTTP.
  - userbinator 6y ago
    A 16-bit length (64k max message size) is usually sufficient, or even 24-bit (16M max) if you really feel the need, but 32 bits is far more than should be needed for parsing messages in memory; it would be fine for a streaming application, however (in which case a 64-bit length wouldn't be a bad idea either.)
    - kccqzy 6y ago
      Good advice. I was actually referring to this: https://rachelbythebay.com/w/2016/02/21/malloc/ https://rachelbythebay.com/w/2016/02/21/malloc/ I read this article a long time ago, and yet I made a similar mistake in my own code.
patrickmcmanus 6y ago
preamble of chunk length and 1 bit for end-of-message indicator.. if you only do chunk length you will eventually find you can't stream but want to. or just use http.

jstanley 6y ago

You will never drop a single byte. If you need some packet-oriented messaging, you could use something like http://jsonlines.org/ http://jsonlines.org/ (i.e. JSON messages separated by newline characters), or https://github.com/protocolbuffers/protobuf https://github.com/protocolbuffers/protobuf if it's more performance-critical.

timeinput 6y ago
Protobuf isn't self delimiting so you still have to have some extra packet wrapper around it to say the length. I like zeromq to get to a packet based system.

mytailorisrich 6y ago

The standard way is to include explicit information on the length of the message that is following. For example if the message is x bytes long then you first send 'x' then you send the x bytes of the message. Or your messages have a defined header that contains the length of the message payload.

genpfault 6y ago

Netstrings[1] :) [1]: https://en.wikipedia.org/wiki/Netstring https://en.wikipedia.org/wiki/Netstring

vasilvv 6y ago

It will never get out-of-sync because TCP guarantees that the bytes will be delivered in the same order they've arrived. The best approach is typically put a length in front of every message. The good things about that approach are: 1. The receiver can allocate buffer that is exactly the size it needs to fit the message. 2. The receiver can check whether the message is too long before seeing the entire message. The only disadvantage is that you have to know the length of all messages in advance.

kccqzy 6y ago
Definitely be sure to check the length though. Imagine a mistaken client trying to send HTTP, but of course the first four bytes "HTTP" when interpreted as a 32-bit integer, whichever endian, is an absurdly large buffer.
- nucleardog 6y ago
  I mean, at this point you're effectively defining a new network protocol (or, you will be shortly once you implement ways to work around all the other issues you're going to run into). I'd go all-in from the start and start every packet with a magic string/byte sequence of your own, a length, and probably a version code just to make it extensible. Or see if there's an existing protocol you can abuse for what you want. If it's transactional, you get a pretty big ecosystem of battle-tested clients/servers/proxies/etc if you use HTTP.
- userbinator 6y ago
  A 16-bit length (64k max message size) is usually sufficient, or even 24-bit (16M max) if you really feel the need, but 32 bits is far more than should be needed for parsing messages in memory; it would be fine for a streaming application, however (in which case a 64-bit length wouldn't be a bad idea either.)
  - kccqzy 6y ago
    Good advice. I was actually referring to this: https://rachelbythebay.com/w/2016/02/21/malloc/ https://rachelbythebay.com/w/2016/02/21/malloc/ I read this article a long time ago, and yet I made a similar mistake in my own code.

patrickmcmanus 6y ago

preamble of chunk length and 1 bit for end-of-message indicator.. if you only do chunk length you will eventually find you can't stream but want to. or just use http.