27 ms·
The Byte Order Fiasco
- the_real_sparky 5y agoThis problem is it’s own special horror in Canbus data. Between endianness and sign it’s a nightmare of en/decoding possibilities and the associated mistakes that come with that.
- rwmj 5y agoTIFF is another one. The only endian-switchable image format that I'm aware of. Fun fact: CD-ROM superblocks have both-endian fields. Each integer is stored twice in big and little endian format. I assume this was to allow underpowered 80s hardware which didn't have enough resource to do byte swapping.
- ipython 5y agoIt’s not every day you can write a blog post that calls out rob pike… ;)
- amelius 5y agoByte order is one of the great unnecessary historical fuck ups in computing. A similar one is that signedness of char is machine dependent. It's typically signed on Intel and unsigned on ARM. Sigh!
- bregma 5y agoAnd which is the correct byte ordering, pray tell?
- rwmj 5y agoBig Endian of course :-) However the one which has won is Little Endian. Even IBM admitted this when it switched the default in POWER 7 to little endian. s390x is the only significant architecture that is still big endian.
- bonzini 5y agoLittle endian has the advantage that you can read the low bits of data without having to adjust the address. So you can for example do long addition in memory order rather than having to go backwards, or (with an appropriate representation such as ULEB128) in one pass without knowing the size.
- js8 5y agoMaybe I am biased working on mainframes, but I would personally take big endian over little endian. The reason is when reading a hex dump, I can easily read the binary integers from left to right.
- bonzini 5y agoThat's the only thing that BE has over LE. But for example bitmaps in BE are a huge source of bugs, as readers and writers need to agree on the size to use for memory operations. "SIMD in a word" (e.g. doing strlen or strcmp with 32- or 64-bit memory accesses) might have mostly fallen out of fashion these days, but it's also more efficient in LE.
- kstenerud 5y agoBig endian is easier for humans to read when looking at a memory dump, but little endian has many useful features in binary encoding schemes due to the low byte being first. I used to like big endian more, but after deep investigation I now prefer little endian for any encoding schemes.
- jcadam 5y agoA while back I was on a project to port a satellite simulator from SPARC/Solaris to RHEL/x64. The compressed telemetry stream that came from the satellite needed to be in big endian (and that's what the ground station software expected), and the simulator needed to mimic the behavior. This was not a problem for the old SPARC system, which naturally put everything in the correct order without any fuss, but one of the biggest sticking points in porting over to x64 was having to now manually pack all of that binary data. Using Ada, (what else!) of course.
- metiscus 5y agoIf memory serves correctly, ada 2012 and beyond has language level support for this. I was working on porting some code from an aviation platform to run on PC and it was all in ada 2005 so we didn't have the benefit of that available.
- jcadam 5y agoSame here, Ada2005 for the port. The simulator was originally written in Ada95. Part of what made it even less fun was the data was highly packed and individual fields crossed byte boundaries (these 5 bits are X, the next 4 bits are Y, etc.) :(
- tails4e 5y agoUbsan should default on. If people don't like it, then they should be made turn it off with a switch, so at least it's more likely to be run than not run. Could save a huge amount of time debugging when compilers or architecture changes. Without it, I'd say many a programmer would be caught by these subtleties in the standard. Coming from a HW background (Verilog) I'd more naturally default to masking and shifting when building up larger variables from smaller ones, but I can imagine many would not.
- jedisct1 5y agoSanitizers may introduce side channels. This is an issue for crypto code.
- MauranKilom 5y ago> Ubsan should default on > Could save a huge amount of time debugging when compilers or architecture changes. I'm assuming we come from very different backgrounds, but it's not clear to me how switching compilers or architectures is so common that hardening code against it by default is appropriate. I would think that switching compilers or architectures is generally done very deliberately, so instrumenting code with UBsan for that transition would be the right thing to do?
- tails4e 5y agoChanging gcc version could cause your code with undefined behaviour to change. If you rely UB, whether you know you are or not, you are in for a bad time. Ubsan at least let's you know if your code is robust, or a ticking time bomb...
- toast0 5y agoChanging compilers is a pretty regular thing IMHO; I use the compiler that comes with the OS and let's assume a yearly OS release cycle. Most of those will contain at least some changes to the compiler. I don't really want to have to take that yearly update to go through and review (and presumablu fix) all the UB that has managed to sneak in over the year. It would be better to have avoided putting it in.
- rwmj 5y agoIf you can assume GCC or Clang then __builtin_bswap{16,32,64} functions are provided which will be considerably more efficient, less error-prone, and easier to use than anything you can homebrew.
- dataflow 5y ago_byteswap_{ushort,ulong,uint64} for MSVC. Together with yours on x86 these should take care of the three major compilers.
- jart 5y agoThat's not true. If you write the byte swap in ANSI C using the gigantic mask+shift expression it'll optimize down to the bswap instruction under both GCC and Clang, as the blog post points out.
- rwmj 5y agoAssuming the macros or your giant expression are correct. But you might as well use the compiler intrinsics which you know are both correct and the most efficient possible, and get on with your life.
- jart 5y agoSorry I'd rather place my faith in arithmetic rather than someone's API provided the compiler is smart enough to understand the arithmetic and optimize accordingly.
- deleted 5y ago[deleted]
- loeg 5y ago"Someone" here is the same compiler you're trusting to optimize your giant arithmetic expression of the same idea. Your statement is internally inconsistent.
- londons_explore 5y agoRemember how we used to have machines with a 7 bit byte? And everything was written to handle either 6, 7, or 8 bit bytes? And now we've settled on all machines being 8 bit bytes, and programmers no longer have to worry about such details? Is it time to do the same for big endian machines? Is it time to accept that all machines that matter are little endian, and the extra effort keeping everything portable to big endian is no longer worth the mental effort?
- bumbada 5y agoWhat happens is that all machines that matter are little endian but network works always in Big Endian.
- londons_explore 5y agoWe'll have to keep it as a quirk of history... A bit like the electron has a negative charge...
- PixelOfDeath 5y agoThey hat a 50/50 chance at getting the technical electricity direction right... and the fucked it up!
- bombcar 5y agoIsn’t big endian a bit more natural considered on a bit level? The bits start from highest to lowest on a serial connection.
- ben509 5y agoBig-endian is natural when you're comparing numbers, which is probably why people represent numbers in a big-endian fashion. Little-endian is natural with casts because the address doesn't change, and it's the order in which addition takes place.
- 5y ago
- pabs3 5y agoI wonder if those macros work with middle-endian systems.
- dataflow 5y agoIs this a joke or am I just unaware of any systems out there that are "middle-endian"..?!
- hvdijk 5y agoThere are no current middle-endian systems but they used to exist. The PDP-11 is the most famous one. The macros would work on all systems, but as only very old systems are middle-endian, they also have old compilers so may not be able to optimise it as well.
- mannschott 5y agoSadly not a joke, but thankfully quite obscure: https://en.wikipedia.org/wiki/Endianness#Middle-endian https://en.wikipedia.org/wiki/Endianness#Middle-endian
- froh 5y agono. but hton(3)/ntoh(3) from inet.h do.
- nly 5y agoThis is why, in 2021, the mantra that C is a good language for these low level byte twiddling tasks needs to die. Dealing with alignment and endianness properly requires a language that allows you to build abstractions. The following is perfectly well defined in C++, despite looking like almost the same as the original unsafe C: #include <boost/endian.hpp> #include <cstdio> using namespace boost::endian; unsigned char b[5] = {0x80,0x01,0x02,0x03,0x04}; int main() { uint32_t x = *((big_uint32_t*)(b+1)); printf("%08x\n", x); } Note that I deliberately misaligned the pointer by adding 1. https://gcc.godbolt.org/z/5416oefjx https://gcc.godbolt.org/z/5416oefjx [Edit] Fun twist: the above code doesn't work where the intermediate variable x is removed because printf itself is not type safe, so no type conversion (which is when the bswap is deferred to) happens. In pure C++ when using a type safe formatting function (like fmt or iostreams) this wouldn't happen. printf will let you throw any garbage in to it. tl;dr outside embedded use cases writing C in 2021 is fucking nuts.
- vladharbuz 5y agoCorrect me if I'm wrong, but your example is just using a library to do the same task, rather than illustrating any difference between C and C++. If you want to pull boost in to do this, that's great, but that hardly seems like a fair comparison to the OP, since instead of implementing code to solve this problem yourself you're just importing someone else's code.
- nly 5y agoNo, the fact that this can be done in a library and looks like a native language feature demonstrates the power of C++ as a language. This example is demonstrating: - First class treatment of user (or library) defined types - Operator overloading - The fact that it produces fast machine code. Try changing big_uint32_t to regular uint32_t to see how this changes. When you use the later ubsan will introduce a trap for runtime checks, but it doesn't need to in this case.
- simias 5y agoOperator overloading is a mixed blessing though, it can be very convenient but it's also very good at obfuscating what's going on. For instance I'm not familiar with this boost library so I'd have a lot of trouble piecing out what your snippet does, especially since there's no explicit function call besides the printf. Personally if we're going the OOP route I'd much prefer something like Rust's `var.to_be()`, `var.to_le` etc... At least it's very explicit. My hot take is that operator overloading should only ever be used for mathematical operators (multiplying vectors etc...), everything else is almost invariably a bad idea.
- mrlonglong 5y agoIn an ideal world which endian format would one go for?
- tempodox 5y agoI for one would go for big-endian, simply because reading memory dumps and byte blocks in assembly or elsewhere works without mental byte-swapping arithmetics for multi-byte entities. Just out of curiosity, I would be interested in learning why so many CPUs today are little-endian. Is it because it is cheaper / more efficient for processor implementations or is it because “the others do it, so we do it the same way”?
- bombcar 5y agohttps://stackoverflow.com/questions/5185551/why-is-x86-little-endian https://stackoverflow.com/questions/5185551/why-is-x86-littl... It simplifies certain instructions internally. Practically everything is little endian because x86 won. > And if you think about a serial machine, you have to process all the addresses and data one-bit at a time, and the rational way to do that is: low-bit to high-bit because that’s the way that carry would propagate. So it means that [in] the jump instruction itself, the way the 14-bit address would be put in a serial machine is bit-backwards, as you look at it, because that’s the way you’d want to process it. Well, we were gonna built a byte-parallel machine, not bit-serial and our compromise (in the spirit of the customer and just for him), we put the bytes in backwards. We put the low- byte [first] and then the high-byte. This has since been dubbed “Little Endian” format and it’s sort of contrary to what you’d think would be natural. Well, we did it for Datapoint. As you’ll see, they never did use the [8008] chip and so it was in some sense “a mistake”, but that [Little Endian format] has lived on to the 8080 and 8086 and [is] one of the marks of this family.
- mrlonglong 5y agoAnd does middle endian even exist?
- 5y ago
- cygx 5y ago> So the solution is simple right? Let's just use unsigned char instead. Sadly no. Because unsigned char in C expressions gets type promoted to the signed type int. If you do use unsigned char, an alternative to masking would be performing the cast to uint32_t before instead of after the shift. edit: For reference, this is what it would look like when implemented as a function instead of a macro: static inline uint32_t read32be(const uint8_t *p) { return (uint32_t)p[0] << 24 | (uint32_t)p[1] << 16 | (uint32_t)p[2] << 8 | (uint32_t)p[3]; }
- captainmuon 5y agoIt is a ridiculous feature of modern C that you have to write the super verbose "mask and shift" code, which then gets compiled to a simple `mov` and maybe a `bswap`. Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. There is a huge mismatch between the assumptions of the C spec and actual machine code. One of the few reasons I ever even reached to C is the ability to slurp in data and reinterpret it as a struct, or the ability to reason in which registers things will show up and mix in some `asm` with my C. I think there should really be a dialect of C(++) where the machine model is exactly the physical machine. That doesn't mean the compiler can't do optimizations, but it shouldn't do things like prove code as UB and fold everything to a no-op. (Like when you defensively compare a pointer to NULL that according to spec must not be NULL, but practically could be...) `-fno-strict-overflow -fno-strict-aliasing -fno-delete-null-pointer-checks` gets you halfway there, but it would really only be viable if you had a blessed `-std=high-level-assembler` or `-std=friendly-c` flag.
- nly 5y ago> I think there should really be a dialect of C(++) where the machine model is exactly the physical machine. Sounds great, until you have to rewrite all your software to go from x86-64 to ARM
- pjmlp 5y agoQuite common when coding games back in the 8 and 16 bit days. :) However for the case in hand, it would suffice to just write the key routines in Assembly, not everything.
- simias 5y ago> Wheras, the direct equivalent in C, an assignment with a (type changing) cast, is illegal. I don't understand what you mean by that. The direct equivalent of what? Endianess is not part of the type system in C so I'm not sure I follow. > I think there should really be a dialect of C(++) where the machine model is exactly the physical machine. Linus agrees with you here, and I disagree with both of you. Some UBs could certainly be relaxed, but as a rule I want my code to be portable and for the compiler to have enough leeway to correctly optimize my code for different targets without having to tweak my code. I want strict aliasing and I want the compiler to delete extraneous NULL pointer checks. Strict overflow I'm willing to concede, at the very least the standard should mandate wrap-on-overflow ever for signed integers IMO.
- commandersaki 5y agoIt wasn't clear to me but what was the undefined behaviour in the naive approach?
- cygx 5y agoViolation of the effective typing rules ('strict aliasing') and a potential violation of alignment requirements of your platform.
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- ttt0 5y agohttps://twitter.com/m13253/status/1371615680068526081 https://twitter.com/m13253/status/1371615680068526081 Would it hurt anyone to define this undefined behavior and do exactly what the source code says?
- MauranKilom 5y agoNot sure what you think the source code "says". I mean, I know what you want it to mean, but just because integer wrapping is intuitive to you doesn't imply that that is what the code means. C++ abstract machine and all. But to answer the actual question: For C++20, integer types were revisited. It is now (finally) guaranteed that signed integers are two's complement, along with a list of other changes. See http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0907r4.html http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p090... also for how the committee voted on the individual issues. Note in particular: > The main change between [P0907r0] and the subsequent revision is to maintain undefined behavior when signed integer overflow occurs, instead of defining wrapping behavior. This direction was motivated by: > - Performance concerns, whereby defining the behavior prevents optimizers from assuming that overflow never occurs; > - Implementation leeway for tools such as sanitizers; > - Data from Google suggesting that over 90% of all overflow is a bug, and defining wrapping behavior would not have solved the bug. So yes, the committee very recently revisited this specific issue, and re-affirmed that signed integer overflow should be UB.
- ttt0 5y agoI haven't noticed the signed integer overflow, which does indeed complicate things, and I thought it was just the infinite loop UB. > Data from Google suggesting that over 90% of all overflow is a bug, and defining wrapping behavior would not have solved the bug. Of all overflow? Including unsigned integers where the behavior is defined?
- aliceryhl 5y agoThat 90% of all overflows are bugs doesn't surprise me at all, even if you include unsigned integers.
- st_goliath 5y agoFWIW there is a <sys/endian.h> on various BSDs that contains "beXXtoh", "leXXtoh", "htobeXX", "htoleXX" where XX is a number of bits (16, 32, 64). That header is also available on Linux, but glibc (and compatible libraries) named it <endian.h> instead. See: man 3 endian (https://linux.die.net/man/3/endian https://linux.die.net/man/3/endian) Of course it gets a bit hairier if the code is also supposed to run on other systems. MacOS has OSSwapHostToLittleIntXX, OSSwapLittleToHostIntXX, OSSwapHostToBigIntXX and OSSwapBigToHostIntXX in <libkern/OSByteOrder.h>. I'm not sure if Windows has something similar, or if it even supports running on big endian machines (if you know, please tell). My solution for achieving some portability currently entails cobbling together a "compat.h" header that defines macros for the MacOS functions and including the right headers. Something like this: https://github.com/AgentD/squashfs-tools-ng/blob/master/include/compat.h#L64 https://github.com/AgentD/squashfs-tools-ng/blob/master/incl... This is usually my go-to-solution for working with low level on-disk or on-the-wire binary data structures that demand a specific endianness. In C I use "load/store" style functions that memcpy the data from a buffer into a struct instance and do the endian swapping (or reverse for the store). The copying is also necessary because the struct in the buffer may not have proper alignment. Technically, the giant macro of doom in the article takes care of all of this as well. But unlike the article, I would very much not recommend hacking up your own stuff if there are systems libraries readily available that take care of doing the same thing in an efficient manner. In C++ code, all of this can of course be neatly stowed away in a special class with overloaded operators that transparently takes care of everything and "decays" into a single integer and exactly the above code after compilation, but is IMO somewhat cleaner to read and adds much needed type safety.
- tjoff 5y agoAt least historically windows have had big-endian versions as both SPARC and Itanium use big endian.
- electroly 5y agoItanium can be configured to run in either endianness (it's "bi-endian"). Windows on Itanium always ran in little-endian mode and did not support big-endian mode. The same was true of PowerPC. Windows never ran in big-endian mode on any architecture.
- secondcoming 5y agoIsn't the 'modern' solution to memcpy into a temp and swap the bytes in that? C++ has added/will add std::launder and std::bless to deal with this issue
- loeg 5y ago> Isn't the 'modern' solution to memcpy into a temp and swap the bytes in that? Or just use the endian.h / sys/endian.h routines, which do the right thing (be32dec / be32enc / whatever). memcpy+swap is fine, and easier to get right than the author's giant expressions, but you might as well use the named routines that do exactly what you want already.
- lanstin 5y agoNo, it is to read a byte at a time and turn it into the semantic value for the data structure you are filling in. Like read 128 and then 1 and set the variable to 32769. If u are the author of protobufs then you may run profiling and write the best assembly etc but otherwise no, don’t do it.
- xeeeeeeeeeeenu 5y ago>C++ has added/will add std::launder and std::bless to deal with this issue You're thinking of std::bit_cast. std::launder solves a different, much more obscure problem: https://miyuki.github.io/2016/10/21/std-launder.html https://miyuki.github.io/2016/10/21/std-launder.html
- secondcoming 5y agooops, my mistake
- deleted 5y ago[deleted]
- themulticaster 5y agoThe first example in the article is flawed (or at least misleading). 1) They define a char array (which defaults to signed char, as mentioned in the post), including the value 0x80 which can't be represented in char, resulting in a compiler warning (e.g. in GCC 11.1). The mentioned reason against using unsigned char (that shifting 128 left by 24 places results in UB) is also misleading: I could not reproduce the UB when changing the array to unsigned char. Perhaps the author meant leaving the array defined as signed char, but casting the signed chars to unsigned before shifting. That indeed results in UB, but I don't see why you would define the array as signed in the first place. 2) The cause for the undefined behavior isn't the bswap_32, rather it's because they try reading an uint32_t value from a char array, where b[0] is not aligned on a word boundary. There is no need at all do redefine bswap. The simple solution would be to use an unsigned char array instead of a char array and just reading the values byte-wise. Of course C has its footguns and warts and so on, but there is no need to dramatize it this much in my opinion. I've prepared a Godbolt example to better explain the arguments mentioned above: https://godbolt.org/z/Y1EWK6e17 https://godbolt.org/z/Y1EWK6e17 Edit: To add to point 2) above: Another way to avoid the UB (in this specific case) would be to add __attribute__ ((aligned (4))) to the definition of b. In that case, even reading the array as a single uint32_t works as expected since the access is aligned to a word boundary. Obviously, you can't expect any random (unsigned char) pointer to be aligned on a word boundary. Therefore, it is still necessary to read the uint32_t byte by byte.
- cygx 5y ago> The mentioned reason against using unsigned char (that shifting 128 left by 24 places results in UB) is also misleading No, that reasoning is correct. Integer promotions are performed on the operands of a shift expression, meaning the left operand will be promoted to signed int even if it starts out as unsigned char. Trying to shift a byte value with highest bit set by 24 will results in a value not representable as signed int, leading to UB.
- themulticaster 5y agoThanks, I just noticed a small mistake in my example (I don't trigger the UB because I access b[0] containing 0x80 without shifting, however I meant to do it the other way around). Still, adding an explicit cast to the left operand seems to be enough to avoid this, e.g.: uint32_t x = ((uint32_t)b[0]) << 24; In summary, I think my point that using unsigned char would be appropriate in this case still stands.
- Sebb767 5y agoIn case anyone else wonders how the code in the linked tweet [0] would format your hard drive, it's the missing return on f1. Therefore, f1 is empty as well (no ret) and calling it will result in f2 being run. The commented out code is irrelevant. EDIT: Reading the bug report [1], the actual cause for the missing ret is that the for loop will overflow, which is UB and causes clang to not emit any code for the function. [0] https://twitter.com/m13253/status/1371615680068526081 https://twitter.com/m13253/status/1371615680068526081 [1] https://bugs.llvm.org/show_bug.cgi?id=49599 https://bugs.llvm.org/show_bug.cgi?id=49599
- vlmutolo 5y ago> If you program in C long enough, stuff like this becomes second nature, and it starts to almost feel inappropriate to even have macros like the above, since it might be more appropriately inlined into the specific code. Since there have simply been too many APIs introduced over the years for solving this problem. To name a few for 32-bit byte swapping alone: bswap_32, htobe32, htole32, be32toh, le32toh, ntohl, and htonl which all have pretty much the same meaning. > Now you don't need to use those APIs because you know the secret. This sentiment seems problematic. The solution shouldn't be "we just have to educate the masses of C programmers on how to properly deal with endianness". That will never happen. The solution should be "It's in the standard library. Go look there and don't think too hard." C is sufficiently low-level, and endianness problems sufficiently common, that I would expect that kind of routine to be available.
- pjmlp 5y agoTypical C culture, you would also expect that by now something like SDS would be part of the standard as well. https://github.com/antirez/sds https://github.com/antirez/sds
- lanstin 5y agoThe point is that keeping the distinction clear in your head between numeric semantics and sequence of octets semantics makes the problem universally tractible. You have a data structure where with a numeric value. Here you have a sequence of octets described by some protocol formalism, BNF in the old days. The mapping from one to the other occurs in the math between octets and numeric values and the various network protocols for representing numbers. There are many more choices than just big endian or little endian. Could be ASN infinite precision ints. Could be 32 bit IEEE floats or 64 bit IEEE floats. The distinction is universal between language semantics and external representations. This is why people that memcpy structs right into the buf get such derision, even if it’s faster and written for a mono-Implementation of a language semantics. It is sloppy thought made manifest.
- kingsuper20 5y agoI've never been very satisfied with these approaches for C where you hope the compiler does the right thing. It makes sense to provide some C implementation for portability's sake but any sizeable reordering cries out for a handtuned, processor specific, approach (and the non-sizeable probably doesn't require high speed). I would expect any SIMD instruction set to include a shuffle.
- phkahler 5y agoIt can also be a good idea to swap recursively. First swap the upper and lower half, then swap the upper and lower quarters (bytes for a 32bit) which can be done with only 2 masks. Then if its 64bit value swap alternate bytes, again with only 2 masks. This can be extended all the way to full bit reverse in 3 more lines each with 2 masks and shifts.
- bigbillheck 5y agoI just use ntohl/htonl like a civilized person. (Yes, the article mentions those, but they've been standard for decades).
- froh 5y agowhat's the best practice for 64bit values these days? is htonll ntohll widely available yet?
- amluto 5y agoWhy mask and then shift instead of casting to the correct type and then shifting, like this: (uint32_t)x[0] << 24 | ... Of course, this requires that x[0] be unsigned.
- syockit 5y agoIf this is for deserialisation then it's okay for x[0] to be signed. You just need to recast the result as int32_t (or simply assign to an int32_t variable without any cast) and it is not UB.
- gumby 5y agoIn her first sentence, the phrase “the C / C++ programming language” is no longer correct: C++20 requires two’s complement signed integers. C++ 20 is quite new so I would assume that very few people know this yet. C and C++ obviously differ a lot, but by that phrase she clearly means “the part where then two languages overlap”. The C++ committee has been willing to break C compatibility in a few ways (not every valid C program is a valid C++ program), and this has been true for a while.
- loeg 5y agoIt hasn't been true since C99, at least -- C++ didn't adopt C99 designated initializers.
- hctaw 5y agoWhat chips can be targeted by C compilers today that don't use 2's complement?
- gumby 5y agoI haven’t seen a one’s complement machine in decades but at the time C was standardized here were still quite a few (afaik none had a single-chip CPU, to get to your question). But since they existed, the language definition didn’t require it and some optimizations were technically UB. The C++ committee decided that everyone had figured this out by now and so made this breaking change.
- klyrs 5y ago"the c/c++ language" exists insofar as you can import this c code into your c++, and this is something that c++ programmers need to know how to do, so they'd better learn enough of the differences between c and c++ or they'll be stumped when they crack open somebody else's old code.
- baby 5y agoI suggest this article: https://www.cryptologie.net/article/474/bits-and-bytes-ordering-in-5-minutes/ https://www.cryptologie.net/article/474/bits-and-bytes-order... (shameless plug)
- Animats 5y agoRust gets this right. These primitives are available for all the numeric types. u32::from_le_byte(bytes) // u32 from 4 bytes, little endian u32::from_be_byte(bytes) // u32 from 4 bytes, big endian u32::to_le_bytes(num) // u32 to 4 bytes, little endian u32::to_be_bytes(num) // u32 to 4 bytes, big endian This was very useful to me recently as I had to write the marshaling and un-marshaling for a game networking format with hundreds of messages. With primitives like this, you can see what's going on.
- Negitivefrags 5y agoUnless you are planning on running your game on a mainframe, just don’t bother with endianness for the networking. Big endian is dead for game developers. Copy entire arrays of structs onto the wire without fear! (Just #pragma pack them first)
- musicale 5y ago> game on a mainframe Maybe your program isn't a game. Maybe you have to deal a server that uses Power, or an embedded system that uses PowerPC (or ARM or MIPS in big-endian mode). Maybe you're running on an older architecture (SPARC, PowerPC, 68K.) Maybe you have to deal with a pre-defined data format (e.g. TCP/IP packet headers) that uses big-endian byte ordering for some of its components.
- Aeolun 5y agoThat’s theoretically possible. But I’d be very interested in why. Especially if you are doing anything involving networking.
- Animats 5y agoBecause it makes it clear what's going on. Most of those functions just generate a move, but it's the correct move. I had to read through excessively-clever C++ code that did the same thing to figure out what conversions were happening, then re-express it in Rust. I'm re-implementing a legacy mess that people are afraid to work on. As it happens, in this message system, some items, mainly packet sequence numbers, are big-endian, because they were following what IP and UDP do, and everything else is little endian. I know how to do this with shifts and masks, and I've done things like that when programming in assembly. That was a long time ago. There's been progress in how to write programs.
- mitchs 5y agoOr just cast the pointer to uint##_t and use be##toh and htobe## from <endian.h>? I think this is making a mountain out of a mole hill. I've spent tons of time doing wire (de)serialization in C for network protocols and endian swaps are far from the most pressing issue I see. The big problem imo is the unsafe practices around buffer handling allowing buffer over runs.
- stkdump 5y agoHistorical and obscure machines aside, there are a few things modern C++ code should take for granted, because even new systems will probably not bother breaking them: Text is encoded in UTF-8. Negative integers are twos-complement. Float is 32 bit ieee 754, double and long double are 64 bit ieee 754. Char is 8 bit, short is 16 bit, int is 32 bit, long long is 64 bit.
- einpoklum 5y agoThis is valid code in C++20: if constexpr (std::endian::native == std::endian::big) { std::cout << "big-endian" << '\n'; } else if constexpr (std::endian::native == std::endian::little) { std::cout << "little-endian" << '\n'; } else { std::cout << "mixed-endian" << '\n'; } Doesn't solve everything, but it's saner even if what you're writing is C-style low-level code.
- jsrcout 5y agoOf course, the canonical work on this subject is Danny Cohen's On Holy Wars And A Plea For Peace [0]. It's an informative and highly readable article. My favorite quote, from the conclusion, is: The "Be reasonable, do it my way" approach does not work. Neither does the Esperanto approach of "let's all switch to yet a new language". His bottom line conclusion being It is more important to agree upon an order than which order is agreed upon. [0] https://www.rfc-editor.org/ien/ien137.txt https://www.rfc-editor.org/ien/ien137.txt
- pajko 5y agoThat huge macro appears to be wrong, as there are little endian PowerPC systems, where __ppc__ and _powerpc__ macros are also defined, making the outcome of the detection invalid.
- loup-vaillant 5y agoHere's how I implement little endian parsing: static uint32_t load32_le(const uint8_t s[4]) { return (uint32_t)s[0] | ((uint32_t)s[1] << 8) | ((uint32_t)s[2] << 16) | ((uint32_t)s[3] << 24); } I start with unsigned char to begin with (well `uint8_t` to be precise, which has the advantage of not compiling at all if you happen to use a DSP that uses 32-bit chars). Then I convert those chars to unsigned 32-bit integers. Only then do I shift them. There is no need to mask anything here. Note that modern compilers translate this whole thing into a single unaligned load operation. Even better, I've noticed that using a macro instead of a function tends to make performance worse with modern compilers.
- account42 5y agoI agree that in an ideal world we should just write load code using byte loads and shifts. But in the world we live in, compilers only got the ability to recognize that and emit a bswap instead in relatively recent [0] versions (compared to the age of C). And the recognition can still depend on the exact pattern used. Also, debug builds will still emit the whole shift mess, which in some cases can be annoying. [0] https://godbolt.org/z/jMbqT86jo https://godbolt.org/z/jMbqT86jo