3 ms·
If you use 4 bytes for size, which is fairly convenient, you get 16 megabytes before any 1s show up in the most significant byte. If you expect that it might b
by jerf 10y ago
If you use 4 bytes for size, which is fairly convenient, you get 16 megabytes before any 1s show up in the most significant byte.
If you expect that it might be an accidental ASCII character, the smallest one you are likely to accidentally receive is space. 32 x 16MB = 512MB.
If that's still not a large enough atomic message to satisfy, allocate 8 bytes for the size. If sneaky ASCII creeps in to the most significant byte, it's going to at least be 2^57, or 512 exabytes.
Plus, this is just the first layer of defense. It's an easy and cheap one that keeps you from overallocating, but it's not the end of the story. You still need to examine the rest of the message for validity, and throw it out as soon as possible if you find it's invalid.
And, finally, if you are taking the "allocate a buffer to hold the message" approach, and you might be receiving multiple messages at a time, you have to consider the possibility of receiving multiple spurious messages at once. The only thing worse than a process spuriously allocating a gigabyte and trying to parse stuff into it is a process spuriously trying to do that a few thousand times per second.
- chrisseaton 10y agoI can just imagine a HN article 'Check out the crazy reason that this protocol is limited to 512MB!'
- Dylan16807 10y agoThere's barely any difference between a limit of 512MB vs. a limit of 2GB/4GB. Almost nobody is going to split that hair. You either design for reasonably-small messages, or you design for 'unlimited' size and go with a 64 bit size field. Either way shaving a few bits off the top is harmless. But you shouldn't be allocating 100MB+ upfront anyway. No matter how big of a message you allow.
- kccqzy 10y agoThis reminds me of the other article on HN recently. Using calloc makes it cheap because it's using the kernel's CoW. Of course you probably want to use that CoW functionality explicitly in this case.
- PhasmaFelis 10y agoYeah, everybody goes "640K OUGHT TO BE ENOUGH FOR ANYBODY, LOL." But if you're designing (say) a custom in-house protocol for transmitting telemetry, then "what happens in 20 years when someone tries use it for bulk data transfer for some reason" really shouldn't be higher on your list of priorities than "what happens when it receives literally any data besides what it was expecting." Hypothetical future expandability is good to consider, but not as important as present-day intended use.