4 ms·

In my opinion bugs related to endianness expose a design flaw in the C/C++ programming languages, or at least in the way people tend to use those languages. It

• alfanick 6y ago
  s/compiles to/may or may not compile to/
  • amelius 6y ago
    Exactly. You might not be so lucky on a different platform. Imho a better way to handle this would be to use a macro or special function that comes with the compiler, assuming it exists of course.
• cbmuser 6y ago
  That looks like it could result in unaligned access and hence a crash on sparc64.
  • kazinator 6y ago
    Thus, it can't compile that way for sparc64.
    • giovannibajo1 6y ago
      That’s fine, GCC on sparc will generate the sequence to do an unaligned read with the right endianess. If that means reading bytes and shifting them, so be it.
      • EdSchouten 6y ago
        Exactly. And even if we want to optimize access for those architectures, there surely must be ways to do that. Maybe compilers already do that if you use alignas(int32_t) on the byte array at the call site and ensure that read_int32() function of mine gets inlined?
        • giovannibajo1 6y ago
          No they won’t, if not in very basic cases that can’t be generalized as a rule of thumb solution. Your suggestion bargains correctness with performance. If you know you are reading aligned memory and you need to do it with a single word memory access, then you need to use intNN_t* and use htonl or similar functions to byteswap. There’s no way around it.
          • kazinator 6y ago
            The compiler can generate a test for alignment, and then a branch between two pieces of code. However, if the test and branch branch causes pipeline stalls due to bad prediction, it could be worse than just running the byte-banging code unconditionally.
      • kazinator 6y ago
        I.e. "if that means doing what the programmer literally expressed, so be it."
  • EdSchouten 6y ago
    How? We’re just reading from a byte array.
    • josefx 6y ago
      The optimized code assumes it can read a full 32 bit value at once. So at least that is only valid on a platform that doesn't assume that 32 bit loads are aligned and the actual compiled code would look a lot worse on platforms that do.
• pm215 6y ago
  Yes, it's totally possible to write good abstractions for this sort of basic operation, which handle both endianness and alignment. (QEMU's version of them uses memcpy(), again relying on the compiler to optimise this to a simple load where the host CPU supports unaligned accesses.) I agree that doing that kind of thing so code can be endian-agnostic and keeping the ifdefs to a minimum is better style and easier to understand. But if the big-endian case isn't run through CI then bugs will still creep in, where people forget to use the abstraction and just cast a buffer pointer to do a 32-bit read, or where they don't keep the distinction between 'a 32 bit value in host endianness' and 'a 32-bit value in host-independent endianness (eg network data, or pixel data)' straight and forget a conversion step. QEMU's testing on s390 and ppc64 catches a steady trickle of this kind of bug before it goes into the codebase. (You can catch unaligned-access bugs on x86 with clang/gcc address-sanitizer, I think, but it can't help with endianness bugs.)
• conistonwater 6y ago
  I clicked around a little, and it seems "icc 21.1.9", "x64 msvc x19.28", "risc-v clang (trunk)", "armv7-a clang (trunk)" compile it to four separate byte loads, it looks like. It's not that I don't trust my compiler (usually clang) to optimize it this way, it's just that the level of variability between compilers makes me think I shouldn't really rely on this behaving this way. I wonder what the arch difference is between the clangs that do it and the ones that don't, I don't know much about arm and risc-v.
  • user-the-name 6y ago
    ARM does not support non-aligned reads, and can not safely change four byte loads into a one 32-bit load unless it can guarantee the address is aligned to 4 bytes. Not sure about RISC-V but I would not be surprised if the same applies there.
    • shakna 6y ago
      > Not sure about RISC-V but I would not be surprised if the same applies there. It's probably a case of lowest common denominator. Unalligned access is discouraged but allowed by the spec... But it's allowed to raise an exception. But not required. Cheaper to optimise to avoid that circumstance.
    • pm215 6y ago
      It used to be the case that Arm could not handle unaligned loads, but that functionality was added in v6 of the architecture (many years ago now) and so is present in most Arm cores around today including all the 64-bit ones. (Some instructions insist on aligned data, like atomic read-modify-write operations and some types of SIMD insn, but x86 has a few needs-alignment special cases too. Basic "just load/store data" insns that the compiler will emit are fine.)
• jcelerier 6y ago
  > For example, if I wanted to read a little endian 32-bit integer value from a byte stream, I could write a function like this But for the longest time people would basically dump the raw bytes of their structs from / to the network whenever possible, e.g. struct my_data { int foo, bar; float x,y,z; long dontdothat; } data; write_bytes_to_the_network((const unsigned char*) &data, sizeof(my_data));
  • johannes1234321 6y ago
    This not only leads to the endianess issues, but also alignment/packing. But yes, people often start that way and regret only when it is too late and the files are in the wild.
• captainmuon 6y ago
  For a long time, I struggled with endianness, because I thought that bitshifting and masking would be different on BE vs. LE: // wrong uint32_t num = ...; // num is set to 0x10203040 num >> 24 == 0x10; // On little endian num >> 24 == 0x10; // On big endian Of course that is wrong, and the second one is always the case. In a way, you could say C/C++ is big endian - the higher valued byte is first (left in English reading order). In fact, if you avoid reinterpreting via casting / punning / memcpy, you don't have to think about it at all [1]. One thing I have to do frustratingly often is to implement a bunch of functions like "read_uint32_le", it is a pity that there is no comprehensive standard. ([1] On the other hand, I miss the good old days where you could treat C like "high level assembler", a pointer was just a number that indexed memory, and you could reinterpret bytes with impunity - sometimes had to to get decent performance!)
• alxlaz 6y ago
  It certainly doesn't help that the "wrong" way is featured in almost every textbook, tutorial and article I've seen in the last 15 years. Your example is actually for the most favourable case because there are native 32-bit types. I've seen countless attempts to implement three-byte records via custom Uint24 classes that overload operators and abstract endianness (in various buggy ways) instead of just reading three damn bytes in a uint32_t. Supposedly in the name of readability ("I want to be explicit that this stores a 24-bit variable, not a 32-bit variable, so I'm going to copy-paste these 200 lines of either template magic or line noise, I'm not sure, but see how readable it is now? Oh yeah it's still 32-bit underneath, yeah, I'm not sure what happens if you write a 32-bit variable there, I think it breaks. This is way better than a typedef though, 'cause, uh, C++-17?)
• smcameron 6y ago
  But that code isn't portable for a different endian machine. To write portable endian clean code, you want to write functions/macros named like: cpu_to_le32(), cpu_to_be32(), be32_to_cpu(), le32_to_cpu(), etc. (and similar for 16, and 64 bit quantities) similar to what's described here: https://kernelnewbies.org/EndianIssues https://kernelnewbies.org/EndianIssues) These functions will either swap, or not, depending on the endianness of the target machine, and this decision is made at compile time, so the "no swap" case imposes no performance penalty.) That way, the intent is clear, and the code works on big endian and little endian. So instead of what you propose, which I imagine you might use via: x = some_value; /* say this is big endian, and we want it little endian, because we're on Intel. */ uint32_t y = read_int((char *) &x); you'd have: y = be32_to_cpu(x); Now the intent is clear, and the code is portable, x is big endian, and we want native CPU (whether that be big endian or little endian). Likewise, you could write: x = cpu_to_be32(y); say before writing it out to disk or network for some purpose that requires big endian. This is similar to htonl(), ntoh(), htons(), ntohs(). (host to network/network to host long/short).
  • garaetjjte 6y ago
    >But that code isn't portable for a different endian machine. But it is. That's the whole point, to not bother with weird endian conversion macros.
    • smcameron 6y ago
      It isn't. It always swaps. Most times you're dealing with endianness, you want to convert to or from native CPU to or from big or little endian. Code that always swaps regardless of native endianness will silently do the wrong thing if compiled on a machine of different endianness. You're telling me that what the linux kernel guys do is "weird endian conversion macros?" I don't believe you.
      • messe 6y ago
        You are wrong. It doesn't always swap. Look at the assembly I posted above comparing mips64le to mips64be. The code takes the byte at b[n] and shifts it (8.n)-bits. These are the same bytes on both little endian and big endian machines. They read the bytes in the same order (little endian), and convert them to the native order. You can also see very clearly that it doesn't always swap by looking at the simple x86_64 assembly output that the was posted where no shifts or swaps are performed. Just a mov instruction. Read the assembly. Or better yet, try the code out for yourself.