3 ms·

One approach I've seen to writing big endian safe software is to not load integers directly, and apply the same logic regardless of endianness. Eg. Instead of:

One approach I've seen to writing big endian safe software is to not load integers directly, and apply the same logic regardless of endianness. Eg. Instead of: uint16_t x; #ifdef BIG_ENDIAN x = swap(*p); #else x = *p; #endif You can write: uint16_t x; unsigned char *q = (unsigned char*)p; x = q[0] | (((uint16_t)q[1]) << 8); (Pardon the rushed job of writing C in an HN form on mobile.) This exercises the same deserialization from little endian regardless of the hardware representation of integers. This means you are testing how it will act on big endian even when running on little endian.

plorkyeran 4y ago
The problem is that it's hard to be certain that you actually did it correctly everywhere if you aren't testing it. Writing endian-neutral code isn't terribly difficult, but you can easily accidentally write endian-dependent code here and there.
uluyol 4y ago
This works when your serialization format is designed to be either big or little endian. But for some programs, the serialization format is memcpy the structure to disk. This isn't a good strategy, but unfortunately it can be difficult to undo historical choices. So the padding, alignment, endianness, all leak. And it sounds like X does something like this: clients expect the endianness of serialized data to match the local machine and the server can optionally account for this.
- asveikau 4y ago
  I've done things like this. First, the structs need the nonstandard pragmas and attributes to not pad. Then, all accesses of 16, 32, or 64 bit quantities need to go through a helper function. Even with that baggage, the syntax of having struct member access is can still feel more convenient than more verbose serialization code.
zzo38computer 4y ago
Code like that second one (using shifts) is generally what I do always (dealing with small-endian, big-endian, or PDP-endian data), anyways. Hopefully, the compiler will be able to optimize the code so that it works like the first one. (I don't know if the compiler actually does optimize it or not, though. However, it seems to me that a code like that should be common enough that it would make sense to implement such an optimization.)
lallysingh 4y ago
So.. ntohl?
- asveikau 4y ago
  If you put that in a helper function, then kind of.. except ntohl and ntohs deal with big endian. And is a no-op on big endian. And there is no ntohl for 64 bit quantities.
wmf 4y ago
Neither of those are best practice; you should use some kind of uint16_from_le() macro.
- josephg 4y ago
  Yep. Rust's APIs do this right. Eg u32::to_be_bytes() / u32::to_le_bytes() and u32::from_be_bytes() / u32::from_le_bytes(). There's also methods for native byte order, but thats almost never the right thing to use. I think Go does the same thing. I heard about this approach first from Rob Pike, who would rant about this to anyone who would listen. He said to think of "bytes on disk/network" and "an integer in RAM" as different types that you need to convert between. When you do, the API should specify the endian-ness of the bytes on disk / in the network. He argued you shouldn't do "byte swapping" at all based on endianness. Just parse the int, byte at a time and use the same code on every computer. The real problem is that C makes it easy to convert between a pointer to an integer and a pointer to some bytes in RAM. Just never do that using pointer casting, and everything works out great. https://doc.rust-lang.org/std/primitive.u32.html#method.to_be_bytes https://doc.rust-lang.org/std/primitive.u32.html#method.to_b...
- asveikau 4y ago
  We agree. I'm writing it out as an example. In real life you would put it in some kind of helper function. (Or macro.) I'm not suggesting writing it out inline on every access.
  - wmf 4y ago
    In that case I disagree. You can't guarantee that the shift-and-or will be optimized out by the compiler in cases where no swapping is needed so you're making the code potentially slower.
froh 4y ago
why not just use POSIX hton{,s,l} ? https://linux.die.net/man/3/htons https://linux.die.net/man/3/htons

plorkyeran 4y ago

The problem is that it's hard to be certain that you actually did it correctly everywhere if you aren't testing it. Writing endian-neutral code isn't terribly difficult, but you can easily accidentally write endian-dependent code here and there.

uluyol 4y ago

This works when your serialization format is designed to be either big or little endian. But for some programs, the serialization format is memcpy the structure to disk. This isn't a good strategy, but unfortunately it can be difficult to undo historical choices. So the padding, alignment, endianness, all leak. And it sounds like X does something like this: clients expect the endianness of serialized data to match the local machine and the server can optionally account for this.

asveikau 4y ago
I've done things like this. First, the structs need the nonstandard pragmas and attributes to not pad. Then, all accesses of 16, 32, or 64 bit quantities need to go through a helper function. Even with that baggage, the syntax of having struct member access is can still feel more convenient than more verbose serialization code.

zzo38computer 4y ago

Code like that second one (using shifts) is generally what I do always (dealing with small-endian, big-endian, or PDP-endian data), anyways. Hopefully, the compiler will be able to optimize the code so that it works like the first one. (I don't know if the compiler actually does optimize it or not, though. However, it seems to me that a code like that should be common enough that it would make sense to implement such an optimization.)

lallysingh 4y ago

So.. ntohl?

asveikau 4y ago
If you put that in a helper function, then kind of.. except ntohl and ntohs deal with big endian. And is a no-op on big endian. And there is no ntohl for 64 bit quantities.

wmf 4y ago

Neither of those are best practice; you should use some kind of uint16_from_le() macro.

josephg 4y ago
Yep. Rust's APIs do this right. Eg u32::to_be_bytes() / u32::to_le_bytes() and u32::from_be_bytes() / u32::from_le_bytes(). There's also methods for native byte order, but thats almost never the right thing to use. I think Go does the same thing. I heard about this approach first from Rob Pike, who would rant about this to anyone who would listen. He said to think of "bytes on disk/network" and "an integer in RAM" as different types that you need to convert between. When you do, the API should specify the endian-ness of the bytes on disk / in the network. He argued you shouldn't do "byte swapping" at all based on endianness. Just parse the int, byte at a time and use the same code on every computer. The real problem is that C makes it easy to convert between a pointer to an integer and a pointer to some bytes in RAM. Just never do that using pointer casting, and everything works out great. https://doc.rust-lang.org/std/primitive.u32.html#method.to_be_bytes https://doc.rust-lang.org/std/primitive.u32.html#method.to_b...
asveikau 4y ago
We agree. I'm writing it out as an example. In real life you would put it in some kind of helper function. (Or macro.) I'm not suggesting writing it out inline on every access.
- wmf 4y ago
  In that case I disagree. You can't guarantee that the shift-and-or will be optimized out by the compiler in cases where no swapping is needed so you're making the code potentially slower.

froh 4y ago

why not just use POSIX hton{,s,l} ? https://linux.die.net/man/3/htons https://linux.die.net/man/3/htons