4 ms·

I expected this to be an aliasing violation and was not disappointed. Dereferencing a uint8_t through a uint64_t is undefined behaviour whatever the target arch

I expected this to be an aliasing violation and was not disappointed. Dereferencing a uint8_t through a uint64_t is undefined behaviour whatever the target architecture is. That it works for the author on x64 just means the compiler didn't break it this time around. In particular, an 8 byte aligned uint8_t also can't be dereferenced via a uint64_t. It isn't UB because the alignment is wrong, it's UB by definition and also the alignment might be wrong. This is a woeful state of affairs propagated by an ill founded and widespread confidence that the C and C++ standards probably aren't as hostile to type aliasing as they actually are.

gpderetta 3y ago
If uint8_t is a char type (and invariably it is), then it is fully defined. Similarly, it is perfectly safe to cast a correctly aligned uint8_t* to uint64_t* and dereference it as long as you originally stored an uint64_t into it (or the compiler can't prove you didn't). Remember aliasing UB is always on derefencing with the wrong dynamic type, not on casting pointers.
- Chabsff 3y ago
  You are correct in principle, but the code in the original post definitely exhibits strict aliasing violations. Not only that, it didn't need to.
  - gpderetta 3y ago
    The original code computing the hash by casting uint8_t* to uint64_t* for wide leads is generally wrong, but the memcpy one is fine.
- tom_ 3y ago
  I'm surprised the gcc and clang people have never noticed that uint8_t is an integer type, and therefore does not need to be a character type.
  - gpderetta 3y ago
    Well, if we were starting from scratch {u,}int8_t should definitely not be character type; instead we would have byte_t (plus an explicitly sized variant, octect_t?) as the general universally aliasable byte type, and all of these would be guaranteed to be distinct types. But today, uint8_t is in practice always a char type. You can static_assert if you want to be completely sure. Or just use std::byte.
- JonChesterfield 3y ago
  There's an old gcc bug report about whether uint8_t should have the magic aliasing properties of char or not. I don't remember what they concluded. You can cast any pointer you like to any other pointer. When you dereference it, if the declared type is not the type of the underlying object, bad times for you. An interface which takes a uint8_t* and immediately invokes UB if you pass it anything other than a pointer to a uint64_t can indeed be written, but one should expect people to pass it things other than a cast uint64_t*. Further, the magic aliasing char doesn't work in both directions. If this code was changed to take a char instead of a uint8_t it would be exactly as undefined as it is currently. You can aim a char* into a uint64_t and deref the char, but you can't aim a uint64_t* into a char and deref the uint64_t.*
  - quietbritishjim 3y ago
    They ended up making uint8_t a typedef for unsigned char rather than a distinct type (which is permitted but not required by the standard) so the semantics are now clear. Edit: That's what I remember but I can't find evidence of it. The docs say this is ABI dependent so perhaps the above statement is true on some specific ABI (if I haven't misremembered it entirely). But I can't track down wheere a relevant ABI would be documented. https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Architecture-implementation.html https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Architecture-i... Edit 2: this header has a typedef of uint8_t to unsigned char, but it's part of glibc rather than gcc https://android.googlesource.com/platform/prebuilts/gcc/linux-x86/host/i686-linux-glibc2.7-4.6/+/refs/heads/tools_r20/sysroot/usr/include/stdint.h https://android.googlesource.com/platform/prebuilts/gcc/linu...
    - kzrdude 3y ago
      I thought if uint8_t exists, it logically must be the same as unsigned char? You put some of the statements of the standards together and that's their combined consequence.
      - quietbritishjim 3y ago
        No it doesn't have to be, and it wasn't. That was the whole problem. They were both unsigned integer types with 8 bit widths but they were distinct and, in particular, uint8_t didn't have the aliasing exception that unsigned char does.
- leni536 3y ago
  No, the relationship is directional. You can reinterpret a pointer to an integral type to a pointer to unsigned char and read the underlying bytes. The other way around is UB.
  - gpderetta 3y ago
    What I mean is that this is perfectly valid code: uint64_t x=0; uint8_t * p = (uint8_t*)&x; uint64_t y = *(uint64_t*)p; It is not the casting that is the problem, it is the dereferencing. And as long as that matches the dynamic type, it is ok.
AnimalMuppet 3y ago
Well, yes, if you're playing this kind of game, the compiler cannot type-check what you're doing (or sanity-check it in any other way). You're outside the area covered by the standard. You're coding without a net. But C and C++ are explicitly intended for this kind of situation. You know enough about how types are represented in memory to fiddle with their internals, and you have sufficient reason to do so? Go for it. Just don't expect the type checking to save you, because it won't.
- JonChesterfield 3y ago
  I'd be totally happy with C or C++ here if the attitude to type checking was "you've done weird things so you're on your own, be careful". Things like mutating a vtable could be fair game, just do it carefully. However that's not the setup. C++ in particular, and increasingly C copying from it, optimises assuming you never do anything weird with pointers. This makes application code that doesn't do such things faster in a separate compilation world. It makes system code that does do such things undefined, forcing either compiler flags (fno-strict-aliasing and friends, aka writing in a different language that looks like the nominal one) or taking care around separate compilation to make sure the compiler never sees both sides of the boundary at the same time.
  - trealira 3y ago
    If I had the skill, I would want to create my own C compiler that tried not to optimize things like this, e.g. don't optimize away infinite loops, don't optimize away signed overflow checks or do incorrect algebraic simplifications because signed overflow is undefined behavior, and by default act as though -fno-strict-aliasing were on. Add options for trapping on overflow or optimizing away overflow checks, signed or unsigned. Just do register allocation and minor constant folding (but again, don't act like signed overflow doesn't exist, and don't do constant folding across pointer writes if its address has been taken). There exist open source C compilers that aren't GCC or Clang, such as TCC and Zig, but to my knowledge they all produce really, really bad assembly. There existed proprietary C compilers in the 1980s and 1990s that were written by one or two people that still produced solid assembly, so it's possible, just no one has done it, so Clang and GCC are the only free and open source options for decent assembly for C. Those two also have the downside of being C++ compilers, making them more complex than they need to be. Maybe another option would be to improve an already existing open source C compiler's codegen than to write a new one from scratch again, assuming they'd agree with the above ideals.
pkkm 3y ago
> This is a woeful state of affairs propagated by an ill founded and widespread confidence that the C and C++ standards probably aren't as hostile to type aliasing as they actually are. Yep. Honestly, while I get the historical reasons for strict aliasing, I wish it hadn't been created. Just make -fno-strict-aliasing the default and normalize the abundant use of restrict, maybe shorten it to res or rest just like we use const instead of typing out constant. These implicit rules that usually, but not always, do what you want can be a bigger footgun than having to do things explicitly.
- tialaramex 3y ago
  > just like we use const instead of typing out constant. Those aren't constants†, they're immutable variables which are quite different. Because of the "as if" rule and provenance in some cases you get the same benefit as with a real constant, but that's not what this is. † In C or C++. In some other languages const does, logically, get you an actual constant.
  - pkkm 3y ago
    Unless you think that the abbreviation const stands for some other word than constant, I'm not sure how that's relevant here?
    - tialaramex 3y ago
      Because these aren't actually constants it would actually be even worse if the keyword was "constant" rather than "const". Jargon that's identical to ordinary words but is just plain wrong if taken that way makes it harder to be understood. In a language where the const keyword gets you an actual constant this condition wouldn't arise.
- wahern 3y ago
  How historical is the importance of strict aliasing? AFAIU, strict aliasing survives not because of alignment issues when type punning, but because optimizing compilers heavily rely on non-aliasing to guarantee semantic correctness when reordering code, especially in and around loops. And the performance benefit of reordering has grown exponentially in lock-step with the emergence and increasing sophistication of superscalar architectures, pipelining, and cache hierarchies. Strict aliasing, IOW, is as a practical matter mostly about when and to what extent compilers (and sometimes processors) can optimize code through automatic reordering, much like wrt parallelism and memory barriers. In this sense, the role of strict aliasing is more important than it ever has been, and while technically "merely" a matter of performance (correctness could be obtained by completely abstaining from reordering), the performance of modern processors plummets precipitously when implicit hardware parallelism can't be fully leveraged.
xscott 3y ago
I think we're getting to the point where C++ users should define two classes: class Integer { // uses unsigned integers for all operations // but converts to signed when you need output }; class Pointer { // uses void* and memcpy for all loads or stores }; Put all of the correct operator overloading on those to get the ergonomics we had in the 90s. Then we can safely ignore all the undefined behavior which has been so eagerly embraced by the compiler writers, and we can go back to treating the hardware as though it is sane. (Which the hardware is desperately trying to fake being anyways) After that, we can create some sane container types which don't offer up dangling references or pointers, and Bob's your uncle.
- celrod 3y ago
  If my signed integers overflow, that is wrong. I want to be able to detect if it ever happens, not have that be defined as perfectly valid wrapping behavior. If it's undefined, it's perfectly legal for the compiler to make overflow trap, so I can catch it, or confirm it never happens when running tests our debugging. `-ftrapv` or ``-fsanitize=undefined` do this.
  - xscott 3y ago
    To each their own. I want my 64 bit integers to do what the CPU does, and I could bore you with useful examples where two's complement wrap around is exactly the right behavior. But more importantly, I don't ever want to worry about stumbling into undefined behavior, and compiler extensions like -ftrapv or -fwrapv are not part of the standard. I can't count on those being there when I switch compilers, and if I write a library for others, I can't count on the application developer using the flags I need.

gpderetta 3y ago

If uint8_t is a char type (and invariably it is), then it is fully defined. Similarly, it is perfectly safe to cast a correctly aligned uint8_t* to uint64_t* and dereference it as long as you originally stored an uint64_t into it (or the compiler can't prove you didn't). Remember aliasing UB is always on derefencing with the wrong dynamic type, not on casting pointers.

Chabsff 3y ago
You are correct in principle, but the code in the original post definitely exhibits strict aliasing violations. Not only that, it didn't need to.
- gpderetta 3y ago
  The original code computing the hash by casting uint8_t* to uint64_t* for wide leads is generally wrong, but the memcpy one is fine.
tom_ 3y ago
I'm surprised the gcc and clang people have never noticed that uint8_t is an integer type, and therefore does not need to be a character type.
- gpderetta 3y ago
  Well, if we were starting from scratch {u,}int8_t should definitely not be character type; instead we would have byte_t (plus an explicitly sized variant, octect_t?) as the general universally aliasable byte type, and all of these would be guaranteed to be distinct types. But today, uint8_t is in practice always a char type. You can static_assert if you want to be completely sure. Or just use std::byte.
JonChesterfield 3y ago
There's an old gcc bug report about whether uint8_t should have the magic aliasing properties of char or not. I don't remember what they concluded. You can cast any pointer you like to any other pointer. When you dereference it, if the declared type is not the type of the underlying object, bad times for you. An interface which takes a uint8_t* and immediately invokes UB if you pass it anything other than a pointer to a uint64_t can indeed be written, but one should expect people to pass it things other than a cast uint64_t*. Further, the magic aliasing char doesn't work in both directions. If this code was changed to take a char instead of a uint8_t it would be exactly as undefined as it is currently. You can aim a char* into a uint64_t and deref the char, but you can't aim a uint64_t* into a char and deref the uint64_t.*
- quietbritishjim 3y ago
  They ended up making uint8_t a typedef for unsigned char rather than a distinct type (which is permitted but not required by the standard) so the semantics are now clear. Edit: That's what I remember but I can't find evidence of it. The docs say this is ABI dependent so perhaps the above statement is true on some specific ABI (if I haven't misremembered it entirely). But I can't track down wheere a relevant ABI would be documented. https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Architecture-implementation.html https://gcc.gnu.org/onlinedocs/gcc-13.2.0/gcc/Architecture-i... Edit 2: this header has a typedef of uint8_t to unsigned char, but it's part of glibc rather than gcc https://android.googlesource.com/platform/prebuilts/gcc/linux-x86/host/i686-linux-glibc2.7-4.6/+/refs/heads/tools_r20/sysroot/usr/include/stdint.h https://android.googlesource.com/platform/prebuilts/gcc/linu...
  - kzrdude 3y ago
    I thought if uint8_t exists, it logically must be the same as unsigned char? You put some of the statements of the standards together and that's their combined consequence.
    - quietbritishjim 3y ago
      No it doesn't have to be, and it wasn't. That was the whole problem. They were both unsigned integer types with 8 bit widths but they were distinct and, in particular, uint8_t didn't have the aliasing exception that unsigned char does.
leni536 3y ago
No, the relationship is directional. You can reinterpret a pointer to an integral type to a pointer to unsigned char and read the underlying bytes. The other way around is UB.
- gpderetta 3y ago
  What I mean is that this is perfectly valid code: uint64_t x=0; uint8_t * p = (uint8_t*)&x; uint64_t y = *(uint64_t*)p; It is not the casting that is the problem, it is the dereferencing. And as long as that matches the dynamic type, it is ok.

AnimalMuppet 3y ago

Well, yes, if you're playing this kind of game, the compiler cannot type-check what you're doing (or sanity-check it in any other way). You're outside the area covered by the standard. You're coding without a net. But C and C++ are explicitly intended for this kind of situation. You know enough about how types are represented in memory to fiddle with their internals, and you have sufficient reason to do so? Go for it. Just don't expect the type checking to save you, because it won't.

JonChesterfield 3y ago
I'd be totally happy with C or C++ here if the attitude to type checking was "you've done weird things so you're on your own, be careful". Things like mutating a vtable could be fair game, just do it carefully. However that's not the setup. C++ in particular, and increasingly C copying from it, optimises assuming you never do anything weird with pointers. This makes application code that doesn't do such things faster in a separate compilation world. It makes system code that does do such things undefined, forcing either compiler flags (fno-strict-aliasing and friends, aka writing in a different language that looks like the nominal one) or taking care around separate compilation to make sure the compiler never sees both sides of the boundary at the same time.
- trealira 3y ago
  If I had the skill, I would want to create my own C compiler that tried not to optimize things like this, e.g. don't optimize away infinite loops, don't optimize away signed overflow checks or do incorrect algebraic simplifications because signed overflow is undefined behavior, and by default act as though -fno-strict-aliasing were on. Add options for trapping on overflow or optimizing away overflow checks, signed or unsigned. Just do register allocation and minor constant folding (but again, don't act like signed overflow doesn't exist, and don't do constant folding across pointer writes if its address has been taken). There exist open source C compilers that aren't GCC or Clang, such as TCC and Zig, but to my knowledge they all produce really, really bad assembly. There existed proprietary C compilers in the 1980s and 1990s that were written by one or two people that still produced solid assembly, so it's possible, just no one has done it, so Clang and GCC are the only free and open source options for decent assembly for C. Those two also have the downside of being C++ compilers, making them more complex than they need to be. Maybe another option would be to improve an already existing open source C compiler's codegen than to write a new one from scratch again, assuming they'd agree with the above ideals.

pkkm 3y ago

> This is a woeful state of affairs propagated by an ill founded and widespread confidence that the C and C++ standards probably aren't as hostile to type aliasing as they actually are. Yep. Honestly, while I get the historical reasons for strict aliasing, I wish it hadn't been created. Just make -fno-strict-aliasing the default and normalize the abundant use of restrict, maybe shorten it to res or rest just like we use const instead of typing out constant. These implicit rules that usually, but not always, do what you want can be a bigger footgun than having to do things explicitly.

tialaramex 3y ago
> just like we use const instead of typing out constant. Those aren't constants†, they're immutable variables which are quite different. Because of the "as if" rule and provenance in some cases you get the same benefit as with a real constant, but that's not what this is. † In C or C++. In some other languages const does, logically, get you an actual constant.
- pkkm 3y ago
  Unless you think that the abbreviation const stands for some other word than constant, I'm not sure how that's relevant here?
  - tialaramex 3y ago
    Because these aren't actually constants it would actually be even worse if the keyword was "constant" rather than "const". Jargon that's identical to ordinary words but is just plain wrong if taken that way makes it harder to be understood. In a language where the const keyword gets you an actual constant this condition wouldn't arise.
wahern 3y ago
How historical is the importance of strict aliasing? AFAIU, strict aliasing survives not because of alignment issues when type punning, but because optimizing compilers heavily rely on non-aliasing to guarantee semantic correctness when reordering code, especially in and around loops. And the performance benefit of reordering has grown exponentially in lock-step with the emergence and increasing sophistication of superscalar architectures, pipelining, and cache hierarchies. Strict aliasing, IOW, is as a practical matter mostly about when and to what extent compilers (and sometimes processors) can optimize code through automatic reordering, much like wrt parallelism and memory barriers. In this sense, the role of strict aliasing is more important than it ever has been, and while technically "merely" a matter of performance (correctness could be obtained by completely abstaining from reordering), the performance of modern processors plummets precipitously when implicit hardware parallelism can't be fully leveraged.

xscott 3y ago

I think we're getting to the point where C++ users should define two classes: class Integer { // uses unsigned integers for all operations // but converts to signed when you need output }; class Pointer { // uses void* and memcpy for all loads or stores }; Put all of the correct operator overloading on those to get the ergonomics we had in the 90s. Then we can safely ignore all the undefined behavior which has been so eagerly embraced by the compiler writers, and we can go back to treating the hardware as though it is sane. (Which the hardware is desperately trying to fake being anyways) After that, we can create some sane container types which don't offer up dangling references or pointers, and Bob's your uncle.

celrod 3y ago
If my signed integers overflow, that is wrong. I want to be able to detect if it ever happens, not have that be defined as perfectly valid wrapping behavior. If it's undefined, it's perfectly legal for the compiler to make overflow trap, so I can catch it, or confirm it never happens when running tests our debugging. `-ftrapv` or ``-fsanitize=undefined` do this.
- xscott 3y ago
  To each their own. I want my 64 bit integers to do what the CPU does, and I could bore you with useful examples where two's complement wrap around is exactly the right behavior. But more importantly, I don't ever want to worry about stumbling into undefined behavior, and compiler extensions like -ftrapv or -fwrapv are not part of the standard. I can't count on those being there when I switch compilers, and if I write a library for others, I can't count on the application developer using the flags I need.