6 ms·

Summarizing the conclusion of the article: GCC correctly interprets the spec as saying that all integers reads must be aligned in memory, and when vectorizing

by nkurz 7y ago

Summarizing the conclusion of the article: GCC correctly interprets the spec as saying that all integers reads must be aligned in memory, and when vectorizing the code chooses to use an "aligned" instruction that fails on unaligned data (MOVDQA). On modern x64 processors, the unaligned version of this instruction (MOVDQU) is just as fast on both aligned an unaligned data, and has the advantage of not causing a segfault when run. Is this a bug in GCC that should be fixed, or is GCC justified in its behavior? Or is there another interpretation? Having been bitten by this in the past, my conclusion is that while this is not a bug, it is (slight) evidence that GCC will not act in its users' best interests unless required to do so by the spec. Given the same inputs, Intel's ICC generates working fast code. If both compilers were equally available, I would usually prefer ICC over GCC for code that is going to run on modern Intel processors.

nly 7y ago
GCC is doing its best to give you the best performance on a broad set of CPUs, and its behaviour is 100% correct. Saying you want GCC to not exploit this contract (that the char* actually points to an aligned uint32_t) is asking for inferior performance, which is unacceptable to most people still using C. A warning would be nice though.
- nkurz 7y ago
  I should add that my comment above is dependent on compiling with options that target a specific processor, when the performance on that processor is equally good with aligned and unaligned data. That is, when compiling with "-march=native" on a modern Intel processor, I strongly prefer a compiler that generates fast and working code over one that segfaults because the spec allows it to. I'm open to arguments that it should behave differently when asked to generate generic code that can work on generic processors. Are there x64 processors in common use today that are significantly slower with MOVDQU than MOVDQA?
  - gpderetta 7y ago
    Movdqu vs movdqa is just one of the possible bad things that can happen when misaligned data is erroneously used. And it will segfault consistently which is a good thing which means that will be caught early in testing. You want compilers to produce binaries that will catch mistakes early in testing, not binaries that work 99.99% of the time and fail in obscure ways.
    - MaulingMonkey 7y ago
      To expand on the 0.01% cases - atomic operations (be they used for lock-free algorithms, or to implement spinlocks) generally require alignment. If you're lucky, unaligned access will segfault. I have been so lucky, when unaligned global char[] buffers were recast to types containing pthread locks. Operations on said locks were kind enough to segfault when unit testing a port to ARM hardware. If you're unlucky, your "atomic" x64 instructions will silently revert to non-atomic behavior when straddling cachelines but otherwise "succeed". This will introduce one of the absolute nastiest kinds of heisenbugs into your codebase, in one of the hardest to understand and debug parts of your program. Speaking frankly - my coworkers won't figure it out. I won't figure it out. Instead, we'll ship unstable software, and maybe blame the hardware and suggest memtest. A movdqa segfault by comparison is a tame, easily understood, easily fixed, and may well help me catch and fix said accidentally unatomic behavior.
      - gmueckl 7y ago
        Citation needed that lock-prefixed instructions require alignment or must not span cache lines on x86. I don't see that in amy documentation.
        MaulingMonkey 7y ago
        > lock-prefixed instructions Are not the only instructions used for atomic operations. > Citation needed EDIT: memory_order_acq_rel like I've shown bellow isn't actually a valid value to pass to store (noticed this when trying to spot sane disassembly in MSVC for yet another datapoint, which was turning into a noop!) memory_order_seq_cst is, however, which gets compiled to xchg in both GCC, Clang, and MSVC... but memory_order_release and memory_order_relaxed are both legal too, and get compiled down to vanilla movs in GCC, Clang, and MSVC as well: https://clang.godbolt.org/z/QfnxPJ https://clang.godbolt.org/z/QfnxPJ void foo(std::atomic<int> & i) { i.store(42, std::memory_order_acq_rel); } Gets compiled down into a no-lock-prefix mov by clang (gcc emits xchg which is implicitly lock and will be atomic even straddling cacheline boundaries): https://clang.godbolt.org/z/87x98C https://clang.godbolt.org/z/87x98C Mov can't even take a lock prefix (https://software.intel.com/sites/default/files/managed/39/c5/325462-sdm-vol-1-2abcd-3abcd.pdf https://software.intel.com/sites/default/files/managed/39/c5... page 1158): > [...] The LOCK prefix can be prepended only to the following instructions and only to those forms of the instructions where the destination operand is a memory operand: ADD, ADC, AND, BTC, BTR, BTS, CMPXCHG, CMPXCH8B, CMPXCHG16B, DEC, INC, NEG, NOT, OR, SBB, SUB, XOR, XADD, and XCHG [...] And mov is only atomic when within a single cacheline on modern processors: (https://software.intel.com/sites/default/files/managed/39/c5/325462-sdm-vol-1-2abcd-3abcd.pdf https://software.intel.com/sites/default/files/managed/39/c5... page 3052): > The Intel486 processor (and newer processors since) guarantees that the following basic memory operations will always be carried out atomically: > [...] > • Reading or writing a doubleword aligned on a 32-bit boundary > The Pentium processor (and newer processors since) guarantees that the following additional memory operations will always be carried out atomically: > [...] > • Unaligned 16-, 32-, and 64-bit accesses to cached memory that fit within a cache line > Accesses to cacheable memory that are split across cache lines and page boundaries are not guaranteed to be atomic by the Intel Core 2 Duo, Intel® Atom™, Intel Core Duo, Pentium M, Pentium 4, Intel Xeon, P6 family, Pentium, and Intel486 processors. The Intel Core 2 Duo, Intel Atom, Intel Core Duo, Pentium M, Pentium 4, Intel Xeon, and P6 family processors provide bus control signals that permit external memory subsystems to make split accesses atomic; however, nonaligned data accesses will seriously impact the performance of the processor and should be avoided. (PDF is the latest version of "Intel® 64 and IA-32 Architectures Software Developer’s Manual, Combined Volumes: 1, 2A, 2B, 2C, 2D, 3A, 3B, 3C, 3D and 4" I found at https://software.intel.com/en-us/download/intel-64-and-ia-32-architectures-sdm-combined-volumes-1-2a-2b-2c-2d-3a-3b-3c-3d-and-4 https://software.intel.com/en-us/download/intel-64-and-ia-32... )
userbinator 7y ago
unless required to do so by the spec Which IMHO is an absolutely stupid point of view, because compilers don't exist in a void. As the old saying goes, "what's right isn't always legal, and what's legal isn't always right", and behaving to the letter of the law is not the same as behaving to the spirit of the law. Thus I think it is absolutely a bug. The standard even says undefined behaviour may result in something like "behaving in a manner characteristic of the environment", which is absolutely what programmers expect from the language. I also suspect the fact that GCC has become a de-facto monopoly (duopoly if you count Clang/LLVM) among C compilers for Linux platforms makes them more likely to dismiss such complaints. My experience agrees with yours that ICC and MSVC are nowhere near as aggressive and hostile with UB, yet still generate very good code.
- MaulingMonkey 7y ago
  > Thus I think it is absolutely a bug. While I agree with your rationale, I disagree with it's application in this case. Changing this compiler behavior won't get rid of alignment-related crashes - it'll just make them harder for me to reproduce / debug because I'll have to get coredumps and symbols off of my ARM hardware instead. > [...] MSVC [...] Has it's own history of alignment issues as well FWIW, although I'm happy to see that v19 seems to error out on __declspec(align(...))ments it won't guarantee for parameters - whereas that used to be a mere warning. A warning that was missing and wouldn't trigger in at least one MSVC version, which caused it's share of debugging sessions for me: https://clang.godbolt.org/z/hykGEm https://clang.godbolt.org/z/hykGEm
- __s 7y ago
  Given that it's gcc is generating instructions which require alignment "behaving in a manner characteristic of the environment" is one way of describing this situation
  - userbinator 7y ago
    Then perhaps Intel is ultimately to blame for this mess, since requiring alignment is completely at odds with how x86 normally behaves, and as shown in the article, there's a version of the instruction not requiring alignment and not really slower at all.
jlokier 7y ago
I've just tried it and for me GCC 9.2.1 produced either movdqa or movdqu depending on which x64 CPU it is asked to tune for. That seems like it's doing what it's supposed to. I also tried a tiny change to the C code using __attribute__((packed)), and it made the compiler output movdqu on the CPU where it was previously using movdqa, admittedly also producing slightly less efficient code. So again, seems like GCC is doing what it's supposed to. I'm surprised the author of the article went into obscure arch-specific GCC attributes, and tried things like memcpy(), but didn't try __attribute__((packed)) on a single-element struct, which is arch-independent.

by nkurz 7y ago

nly 7y ago

GCC is doing its best to give you the best performance on a broad set of CPUs, and its behaviour is 100% correct. Saying you want GCC to not exploit this contract (that the char* actually points to an aligned uint32_t) is asking for inferior performance, which is unacceptable to most people still using C. A warning would be nice though.

nkurz 7y ago
I should add that my comment above is dependent on compiling with options that target a specific processor, when the performance on that processor is equally good with aligned and unaligned data. That is, when compiling with "-march=native" on a modern Intel processor, I strongly prefer a compiler that generates fast and working code over one that segfaults because the spec allows it to. I'm open to arguments that it should behave differently when asked to generate generic code that can work on generic processors. Are there x64 processors in common use today that are significantly slower with MOVDQU than MOVDQA?
- gpderetta 7y ago
  Movdqu vs movdqa is just one of the possible bad things that can happen when misaligned data is erroneously used. And it will segfault consistently which is a good thing which means that will be caught early in testing. You want compilers to produce binaries that will catch mistakes early in testing, not binaries that work 99.99% of the time and fail in obscure ways.
  - MaulingMonkey 7y ago
    To expand on the 0.01% cases - atomic operations (be they used for lock-free algorithms, or to implement spinlocks) generally require alignment. If you're lucky, unaligned access will segfault. I have been so lucky, when unaligned global char[] buffers were recast to types containing pthread locks. Operations on said locks were kind enough to segfault when unit testing a port to ARM hardware. If you're unlucky, your "atomic" x64 instructions will silently revert to non-atomic behavior when straddling cachelines but otherwise "succeed". This will introduce one of the absolute nastiest kinds of heisenbugs into your codebase, in one of the hardest to understand and debug parts of your program. Speaking frankly - my coworkers won't figure it out. I won't figure it out. Instead, we'll ship unstable software, and maybe blame the hardware and suggest memtest. A movdqa segfault by comparison is a tame, easily understood, easily fixed, and may well help me catch and fix said accidentally unatomic behavior.
    - gmueckl 7y ago
      Citation needed that lock-prefixed instructions require alignment or must not span cache lines on x86. I don't see that in amy documentation.
      - MaulingMonkey 7y ago
        > lock-prefixed instructions Are not the only instructions used for atomic operations. > Citation needed EDIT: memory_order_acq_rel like I've shown bellow isn't actually a valid value to pass to store (noticed this when trying to spot sane disassembly in MSVC for yet another datapoint, which was turning into a noop!) memory_order_seq_cst is, however, which gets compiled to xchg in both GCC, Clang, and MSVC... but memory_order_release and memory_order_relaxed are both legal too, and get compiled down to vanilla movs in GCC, Clang, and MSVC as well: https://clang.godbolt.org/z/QfnxPJ https://clang.godbolt.org/z/QfnxPJ void foo(std::atomic<int> & i) { i.store(42, std::memory_order_acq_rel); } Gets compiled down into a no-lock-prefix mov by clang (gcc emits xchg which is implicitly lock and will be atomic even straddling cacheline boundaries): https://clang.godbolt.org/z/87x98C https://clang.godbolt.org/z/87x98C Mov can't even take a lock prefix (https://software.intel.com/sites/default/files/managed/39/c5/325462-sdm-vol-1-2abcd-3abcd.pdf https://software.intel.com/sites/default/files/managed/39/c5... page 1158): > [...] The LOCK prefix can be prepended only to the following instructions and only to those forms of the instructions where the destination operand is a memory operand: ADD, ADC, AND, BTC, BTR, BTS, CMPXCHG, CMPXCH8B, CMPXCHG16B, DEC, INC, NEG, NOT, OR, SBB, SUB, XOR, XADD, and XCHG [...] And mov is only atomic when within a single cacheline on modern processors: (https://software.intel.com/sites/default/files/managed/39/c5/325462-sdm-vol-1-2abcd-3abcd.pdf https://software.intel.com/sites/default/files/managed/39/c5... page 3052): > The Intel486 processor (and newer processors since) guarantees that the following basic memory operations will always be carried out atomically: > [...] > • Reading or writing a doubleword aligned on a 32-bit boundary > The Pentium processor (and newer processors since) guarantees that the following additional memory operations will always be carried out atomically: > [...] > • Unaligned 16-, 32-, and 64-bit accesses to cached memory that fit within a cache line > Accesses to cacheable memory that are split across cache lines and page boundaries are not guaranteed to be atomic by the Intel Core 2 Duo, Intel® Atom™, Intel Core Duo, Pentium M, Pentium 4, Intel Xeon, P6 family, Pentium, and Intel486 processors. The Intel Core 2 Duo, Intel Atom, Intel Core Duo, Pentium M, Pentium 4, Intel Xeon, and P6 family processors provide bus control signals that permit external memory subsystems to make split accesses atomic; however, nonaligned data accesses will seriously impact the performance of the processor and should be avoided. (PDF is the latest version of "Intel® 64 and IA-32 Architectures Software Developer’s Manual, Combined Volumes: 1, 2A, 2B, 2C, 2D, 3A, 3B, 3C, 3D and 4" I found at https://software.intel.com/en-us/download/intel-64-and-ia-32-architectures-sdm-combined-volumes-1-2a-2b-2c-2d-3a-3b-3c-3d-and-4 https://software.intel.com/en-us/download/intel-64-and-ia-32... )

userbinator 7y ago

unless required to do so by the spec Which IMHO is an absolutely stupid point of view, because compilers don't exist in a void. As the old saying goes, "what's right isn't always legal, and what's legal isn't always right", and behaving to the letter of the law is not the same as behaving to the spirit of the law. Thus I think it is absolutely a bug. The standard even says undefined behaviour may result in something like "behaving in a manner characteristic of the environment", which is absolutely what programmers expect from the language. I also suspect the fact that GCC has become a de-facto monopoly (duopoly if you count Clang/LLVM) among C compilers for Linux platforms makes them more likely to dismiss such complaints. My experience agrees with yours that ICC and MSVC are nowhere near as aggressive and hostile with UB, yet still generate very good code.

MaulingMonkey 7y ago
> Thus I think it is absolutely a bug. While I agree with your rationale, I disagree with it's application in this case. Changing this compiler behavior won't get rid of alignment-related crashes - it'll just make them harder for me to reproduce / debug because I'll have to get coredumps and symbols off of my ARM hardware instead. > [...] MSVC [...] Has it's own history of alignment issues as well FWIW, although I'm happy to see that v19 seems to error out on __declspec(align(...))ments it won't guarantee for parameters - whereas that used to be a mere warning. A warning that was missing and wouldn't trigger in at least one MSVC version, which caused it's share of debugging sessions for me: https://clang.godbolt.org/z/hykGEm https://clang.godbolt.org/z/hykGEm
__s 7y ago
Given that it's gcc is generating instructions which require alignment "behaving in a manner characteristic of the environment" is one way of describing this situation
- userbinator 7y ago
  Then perhaps Intel is ultimately to blame for this mess, since requiring alignment is completely at odds with how x86 normally behaves, and as shown in the article, there's a version of the instruction not requiring alignment and not really slower at all.

jlokier 7y ago

I've just tried it and for me GCC 9.2.1 produced either movdqa or movdqu depending on which x64 CPU it is asked to tune for. That seems like it's doing what it's supposed to. I also tried a tiny change to the C code using __attribute__((packed)), and it made the compiler output movdqu on the CPU where it was previously using movdqa, admittedly also producing slightly less efficient code. So again, seems like GCC is doing what it's supposed to. I'm surprised the author of the article went into obscure arch-specific GCC attributes, and tried things like memcpy(), but didn't try __attribute__((packed)) on a single-element struct, which is arch-independent.