4 ms·

Hi, LLVM and Clang developer here. I don't see anyone being blasé about this. The issue comes down to something really simple: - The compiler can emit substan

Hi, LLVM and Clang developer here. I don't see anyone being blasé about this. The issue comes down to something really simple: - The compiler can emit substantially simpler code if it has a guarantee from the platform's implementation of memcpy. - The compiler authors thought they had such a guarantee and so chose to leverage it. Now, maybe they don't have that guarantee on all platforms. If that's the case, its flat out and simply a bug in the compiler. And you'll find folks on that bug arguing that it doesn't actually hold, which is good and important. But it's actually quite reasonable to have a guarantee from a platform's libc that this will work and leverage that. The compiler isn't generating C code or code bound by any standard. This is about two parts of the implementation of C agreeing about what internal invariants will hold. The problem this bug highlights (and it is a real problem, and not one anyone takes lightly) is that we have insufficient agreement about these invariants, both for certain platforms (glibc) and even within LLVM. Fixing this remains absolutely critical, but may only involve documenting the agreement.

haberman 10y ago
Hi Chandler, I should have clarified that the comments I was referring to were on this bug: https://llvm.org/bugs/show_bug.cgi?id=11763 https://llvm.org/bugs/show_bug.cgi?id=11763 I was surprised to see: "I don't think it's really worth warning about, considering I don't know of any real-world implementation where it doesn't work." That sounds a lot like someone knowingly writing UB in a C program because no compiler they know of breaks the intended behavior. I doubt any memcpy() has ever guaranteed in any kind of official way that overlapping regions are ok, even if it has historically worked in practice. I mean I'm sure it'll get fixed now, I was just surprised to see this kind of reasoning coming from someone who I presumes works on the compiler. btw. I'm pretty sure I heard that glibc broke this behavior sometime recently. There's a new memcpy() symbol in glibc these days with a different symbol version that no longer provides correct behavior in the case of overlapping memory regions. This is all second-hand from some other people I was working with, but if I got their meaning correctly I think this issue will actually have relatively widespread impact.
- chandlerc1024 10y ago
  That comment was actually a considered comment: when we look at what warnings we should produce, we consider cases where everyone has written code a particular way and the fact that it isn't defined is actually a language bug. We (the Clang community members, and I suspect the GCC community members as well) have also worked to change language standards to provide guarantees relied on consistently and where the lack of guarantee has no reasonable basis. Also that comment is very old, and predates us learning about all of why this is probably something actually worth warning on. That discussion ended up in a different forum, and we do actually warn on stuff like self assignment. Understand that it is often precisely the people working on a compiler that do deal with the cases of when we discover widespread UB in real code and whether or not the code is incorrect and must be changed or the standard is incorrect and must be fixed. But all of that is about a warning, and not about the actual question of how the compiler and the system library interact. Note that both Clang and GCC have at least some variants of this bug, and so I don't actually expect glibc to break this in the foreseeable future. But we're still actively working to defend against this on platforms for which we don't have a guarantee.
  - haberman 10y ago
    I can't see an argument here for why the standard should be changed. memcpy() does one thing and memmove() does another. Changing the standard so they are the same just because memcpy() gets misused in real-world programs doesn't make sense to me. If people misuse memcpy(), then have a compiler flag akin to -fstrict-aliasing where memcpy() gets rewritten to memmove() unless people compile with -fstrict-memcpy. That seems a separate issue though from what calls the compiler should emit. I am not following why LLVM would emit memcpy() when the semantics it actually requires from libc are memmove(). Am I missing something?
    - Joky 10y ago
      It is not about making memcpy behave the same as memmove: in this case the difference would still be between exact identity (src == dest) and partial overlap.
xorblurb 10y ago
Can't you see the pattern here? Even LLVM devs are incapable of doing what they are requiring from random regular developers. If that does not motivate to address that kind of craziness more in depth in a systematic way to restore safety and defense in depth, I don't know what will. - Programmers can and do emit substantially simpler code if they have a guarantee from the platform's implementation of <feature x>. - Programmers think they have such a guarantee. (I don't know a single programmer around me who knows the complete list of undefined behavior, myself included) - Compiler injects bug -- and LLVM/CLang seek excuse in the "technically not a bug" completely useless answer. - Sane projects actually disable all that kind of crazyness (adding -fno-<everything> options every now and then) so all that kind of cute optims are useless in the real world (either dangerous, or completely disabled) It's actually reasonable to have a platform that e.g. gives you 2's complement arithmetic or defines the result of shifts, or allows unaligned access, because mainstream computer hardware does provide those guarantee. And at least half of C programmers are not even aware those kind of stuff are technically forbidden, (historically often because of other architectures, some obscure, and not compiler optimizations, btw). More knowledgeable people who know about the spirit of the standard (1) can also be bitten by compiler revelation of bugs because compilers devs are doing literal and insane interpretation. Bug ensues, including security holes. (1): Let me remind you about the definition of undefined behavior: "behavior, upon use of a nonportable or erroneous program construct or of erroneous data, for which this International Standard imposes no requirements" One keyword is nonportable and it is clearly not considered enough. So yes, there are numerous constructs that are reasonable to expect on a given architecture when we do not learn the compiler manual by heart, but some people don't give a fuck about the principle of least astonishment when they can gain 1% on a synthetic benchmark.
- astrange 10y ago
  > It's actually reasonable to have a platform that e.g. gives you 2's complement arithmetic or defines the result of shifts, or allows unaligned access, because mainstream computer hardware does provide those guarantee. ARM, PPC, and x86 all define the results of some shifts differently (for shift amounts >32UL) and behave differently with unaligned pointers (only x86 tends to accept them). Which mainstream hardware are you using? Adding magical nice semantics to your language is another way to hide bugs in your program. Try building with ubsan and doing runtime testing instead.

haberman 10y ago

Hi Chandler, I should have clarified that the comments I was referring to were on this bug: https://llvm.org/bugs/show_bug.cgi?id=11763 https://llvm.org/bugs/show_bug.cgi?id=11763 I was surprised to see: "I don't think it's really worth warning about, considering I don't know of any real-world implementation where it doesn't work." That sounds a lot like someone knowingly writing UB in a C program because no compiler they know of breaks the intended behavior. I doubt any memcpy() has ever guaranteed in any kind of official way that overlapping regions are ok, even if it has historically worked in practice. I mean I'm sure it'll get fixed now, I was just surprised to see this kind of reasoning coming from someone who I presumes works on the compiler. btw. I'm pretty sure I heard that glibc broke this behavior sometime recently. There's a new memcpy() symbol in glibc these days with a different symbol version that no longer provides correct behavior in the case of overlapping memory regions. This is all second-hand from some other people I was working with, but if I got their meaning correctly I think this issue will actually have relatively widespread impact.

chandlerc1024 10y ago
That comment was actually a considered comment: when we look at what warnings we should produce, we consider cases where everyone has written code a particular way and the fact that it isn't defined is actually a language bug. We (the Clang community members, and I suspect the GCC community members as well) have also worked to change language standards to provide guarantees relied on consistently and where the lack of guarantee has no reasonable basis. Also that comment is very old, and predates us learning about all of why this is probably something actually worth warning on. That discussion ended up in a different forum, and we do actually warn on stuff like self assignment. Understand that it is often precisely the people working on a compiler that do deal with the cases of when we discover widespread UB in real code and whether or not the code is incorrect and must be changed or the standard is incorrect and must be fixed. But all of that is about a warning, and not about the actual question of how the compiler and the system library interact. Note that both Clang and GCC have at least some variants of this bug, and so I don't actually expect glibc to break this in the foreseeable future. But we're still actively working to defend against this on platforms for which we don't have a guarantee.
- haberman 10y ago
  I can't see an argument here for why the standard should be changed. memcpy() does one thing and memmove() does another. Changing the standard so they are the same just because memcpy() gets misused in real-world programs doesn't make sense to me. If people misuse memcpy(), then have a compiler flag akin to -fstrict-aliasing where memcpy() gets rewritten to memmove() unless people compile with -fstrict-memcpy. That seems a separate issue though from what calls the compiler should emit. I am not following why LLVM would emit memcpy() when the semantics it actually requires from libc are memmove(). Am I missing something?
  - Joky 10y ago
    It is not about making memcpy behave the same as memmove: in this case the difference would still be between exact identity (src == dest) and partial overlap.

xorblurb 10y ago

Can't you see the pattern here? Even LLVM devs are incapable of doing what they are requiring from random regular developers. If that does not motivate to address that kind of craziness more in depth in a systematic way to restore safety and defense in depth, I don't know what will. - Programmers can and do emit substantially simpler code if they have a guarantee from the platform's implementation of <feature x>. - Programmers think they have such a guarantee. (I don't know a single programmer around me who knows the complete list of undefined behavior, myself included) - Compiler injects bug -- and LLVM/CLang seek excuse in the "technically not a bug" completely useless answer. - Sane projects actually disable all that kind of crazyness (adding -fno-<everything> options every now and then) so all that kind of cute optims are useless in the real world (either dangerous, or completely disabled) It's actually reasonable to have a platform that e.g. gives you 2's complement arithmetic or defines the result of shifts, or allows unaligned access, because mainstream computer hardware does provide those guarantee. And at least half of C programmers are not even aware those kind of stuff are technically forbidden, (historically often because of other architectures, some obscure, and not compiler optimizations, btw). More knowledgeable people who know about the spirit of the standard (1) can also be bitten by compiler revelation of bugs because compilers devs are doing literal and insane interpretation. Bug ensues, including security holes. (1): Let me remind you about the definition of undefined behavior: "behavior, upon use of a nonportable or erroneous program construct or of erroneous data, for which this International Standard imposes no requirements" One keyword is nonportable and it is clearly not considered enough. So yes, there are numerous constructs that are reasonable to expect on a given architecture when we do not learn the compiler manual by heart, but some people don't give a fuck about the principle of least astonishment when they can gain 1% on a synthetic benchmark.

astrange 10y ago
> It's actually reasonable to have a platform that e.g. gives you 2's complement arithmetic or defines the result of shifts, or allows unaligned access, because mainstream computer hardware does provide those guarantee. ARM, PPC, and x86 all define the results of some shifts differently (for shift amounts >32UL) and behave differently with unaligned pointers (only x86 tends to accept them). Which mainstream hardware are you using? Adding magical nice semantics to your language is another way to hide bugs in your program. Try building with ubsan and doing runtime testing instead.