13 ms·

I feel like I've misunderstood something here... shouldn't memcpy(anything, anything, 0) just do nothing, because you're copying 0 bytes?

mjg59 2y ago
That's a reasonable intuitive interpretation of how it should behave, but according to the spec it's undefined behaviour and compilers have a great degree of freedom in what happens as a result.
- voidUpdate 2y ago
  Why didn't they just... define it, back when they wrote it?
  - deleted 2y ago
    [deleted]
  - deleted 2y ago
    [deleted]
  - frabert 2y ago
    Every time they leave something undefined, they do so to leave implementations free to use the underlying platform's default behavior, and to allow compilers to use it as an optimization point
    - jcelerier 2y ago
      Here it's more that it allows to assume that this is never the case, thus no need to have an additional check in it I assume ?
    - lucozade 2y ago
      > time they leave something undefined, they do so to leave implementations free to use the underlying platform's default behavior That's implementation defined (more or less) ie teh compiler can do whatever makes mst sense for its implementation. Undefined means (more or less) that the compiler can assume the behaviour never happens so can apply transforms without taking it into account. > to allow compilers to use it as an optimization point That's the main advantage of undefined behaviour ie if you can ignore the usage, you may be able to apply optimisations that you couldn't if you had to take it into account. In the article, for example, GCC eliminated what it considered dead code for a NULL check of a variable that couldn't be NULL according to the C spec. That's also probably the most frustrating thing about optimisations based on undefined behaviour ie checks that prevent undefined behaviour are removed because the compiler thinks that the check can't ever succeed because, if it did, there must have been undefined behaviour. But the way the developer was ensuring defined behaviour was with the check!
      - frabert 2y ago
        AFAIK, something having undefined behavior in the spec does not prevent an implementation- (platform-)specific behavior being defined. As to your point about checks being erased, that generally happens when the checks happen too late (according to the compiler), or in a wrong way. For example, checking that `src` is not NULL _after_ memcpy(sec, dst, 0) is called. Or, checking for overflow by doing `if(x+y<0) ...` when x and y are nonnegative signed ints.
  - nephanth 2y ago
    I mean, they might not have given thought to that particular corner case, they probably wrote something like > memcpy(void* ptr1, void* ptr2, int n) Copy n bytes from ptr1 to ptr2. UNDEFINED if ptr1 is NULL or ptr2 is NULL ‐------ It might also have come from a "explicit better than implicit" opinion, as in "it is better to have developers explicitly handle cases where the null pointer is involved
    - jbverschoor 2y ago
      I think it's more a strategy. C was not created to be safe. It's pretty much a tiny wrapper around assembler. Every limitation requires extra cycles, compile time or runtime, both of which were scarce. Of course, someone needs to check in the layers of abstraction. The user, programmer, compiler, cpu, architecture.. They chose for the programmer, who like to call themselves "engineers" these days.
      - wruza 2y ago
        Not sure what your last remark means wrt everything else.
      - poincaredisk 2y ago
        I disagree with your premise. C was designed to be a high level (for its time) language, abstracted from actual hardware >It's pretty much a tiny wrapper around assembler Assebler has zero problem with adding "null + 4" or computing "null-null". C does, because it's not actually a tiny wrapper.
        jbverschoor 2y ago
        Not high-level.. Portable. Portable layer above assembler/arch. NULL doesn't exist in assembler, and in C, NULL is only a defined as a macro. It's not something built-in. C doesn't have any problems adding 4 to NULL nor subtracting NULL from NULL.
        jbverschoor 2y ago
  - larschdk 2y ago
    When C was conceived, CPU architectures and platforms were more varied than what we see today. In order to remain portable and yet performant, some details were left as either implementation defined, or completely undefined (i.e. the responsibility of the programmer). Seems archaic today, but it was necessary when C compilers had to be two-pass and run in mere kilobytes of RAM. Even warnings for risky and undefined behavior is a relatively modern concept (last 10-20 years) compared to the age of C.
    - actionfromafar 2y ago
      When C was conceived, it was made for a specific DEC CPU, for making an operating system. The idea of a C standard was in the future. If you wanted to know what (for instance) memcpy actually did, you looked at the source code, or even more likely, the assembler or machine code output. That was "the standard".
      - anticensor 2y ago
        No, K&R's book was the standard.
        actionfromafar 2y ago
        First came the language, then a few years later they described it in a book.
      - da_chicken 2y ago
        I think it's reasonable to assume that GP clearly meant the C standard being conceived, as, obviously, K&R's C implementation of the language was ad hoc rather than exhibiting any prescribed specification.
    - scoutt 2y ago
      > Seems archaic today ... run in mere kilobytes of RAM There is an entire industry that does pretty much that... today. They might run in flash instead of RAM, but still, a few kilobytes. Probably there are more embedded devices out there than PCs. PIC, AVR, MSP, ARM, custom archs. There might be one of those right now under your hand, in that thing you use to move the cursor.
  - hyperman1 2y ago
    memcpy used to be a rep movsb on 8086 DOS compilers. I don't remember if rep movsb stops if cx=0 on entry, or decrements first and wraps around, copying 64K of data.
    - connicpu 2y ago
      I know at least MSVC's memcpy on x86_64 still results in a rep movsb if the cpuid flag that says rep movsb is fast is set, which it should be on all x86 chips from about 2011/2012 and onward ;)
    - dfox 2y ago
      The specification does not explicitly say that, but the clear intention is that REP with CX=0 should be no-op (you get exactly that situation when REP gets interrupted during the last iteration, in that case CX is zero and IP points to the REP, not the following instruction).
      - bonzini 2y ago
        Rep movsb copies 64K if CX=0 (that's actually very useful), but memcpy could be implemented as two instructions: jcxz skip rep movsb skip:
  - wat10000 2y ago
    The original C standard was more descriptive than prescriptive. There was probably an implementation where it crashed or misbehaved.
  - menaerus 2y ago
    Charitable interpretation may be: Back then when the contract of this function was standardized, presumably in C89 which is ~35 years ago, CPUs but also C compilers were not as powerful so wasting an extra couple of CPU cycles to check this condition was much more expensive than it is today. Because of that contract, and which can be seen in the example in the below comments, the compiler is also free to eliminate the dead code which also has the effect of shaving off some extra CPU cycles.
  - lmm 2y ago
    Back when they wrote it they were trying to accommodate existing compilers, including those who did useful things to help people catch errors in their programs (e.g. making memcpy trap and send a signal if you called it with NULL). The current generation of compilers that use undefined behaviour as an excuse to do horrible things that screw over regular programmers but increase performance on microbenchmarks postdates the standard.
  - FartyMcFarter 2y ago
    Because the benefit was probably seen as very little, and the cost significant. When you're writing a compiler for an architecture where every byte counts you don't make it write extra code for little benefit. Programmers were routinely counting bytes (both in code size and data) when writing Assembly code back then, and I mean that literally. Some of that carried into higher-level languages, and rightly so.
  - killerstorm 2y ago
    From what I understand: 1. Initially, they just wanted to give compiler makers more freedom: both in the sense "do whatever is simplest" and "do something platform-specific which dev wants". 2. Compiler devs found that they can use UB for optimization: e.g. if we assume that a branch with UB is unreachable we can generate more efficient code. 3. Sadly, compiler devs started to exploit every opportunity for optimization, e.g. removing code with a potential segfault. I.e. people who made a standard thought that compiler would remove no-op call to memcpy, but GCC removes the whole branch which makes the call as it considers the whole branch impossible. Standard makers thought that compiler devs would be more reasonable
    - kllrnohj 2y ago
      > Standard makers thought that compiler devs would be more reasonable This is a bit of a terrible take? Compiler devs never did anything "unreasonable", they didn't sit down and go "mwahahaha we can exploit the heck out of UB to break everything!!!!" Rather, repeatedly applying a series of targeted optimizations, each one in isolation being "reasonable", results in an eventual "unreasonable" total transformation. But this is more an emergent property of modern compilers having hundreds of optimization passes. At the time the standards were created, the idea of compilers applying so many optimization passes was just not conceivable. Compilers struggled to just do basic compilation. The assumption was a near 1:1 mapping between code & assembly, and that just didn't age well at all.
      - LegionMammal978 2y ago
        One could argue that "optimizing based on signed overflow" was an unreasonable step to take, since any given platform will have some sane, consistent behavior when the underlying instructions cause an overflow. A developer using signed operations without poring over the standard might have easily expected incorrect values (or maybe a trap if the platform likes to use those), but not big changes in control flow. In my experience, signed overflow is generally the biggest cause of "they're putting UB in my reasonable C code!", followed by the rules against type punning, which are violated every day by ordinary usage of the POSIX socket functions.
  - ynik 2y ago
    Probably because they did not think of this special case when writing the standard, or did not find it important enough to consider complicating the standard text for. In C89, there's just a general provision for all standard library functions: > Each of the following statements applies unless explicitly stated otherwise in the detailed descriptions that follow. If an argument to a function has an invalid value (such as a value outside the domain of the function, or a pointer outside the address space of the program, or a null pointer), the behavior is undefined. [...] And then there isn't anything on `memcpy` that would explicitly state otherwise. Later versions of the standard explicitly clarified that this requirement applies even to size 0, but at that point it was only a clarification of an existing requirement from the earlier standard. People like to read a lot more intention into the standard than is reasonable. Lots of it is just historical accident, really.
- david-gpu 2y ago
  More information on this behavior in the link below. > Note that, apart from contrived examples with deleted null checks, the current rules do not actually help the compiler meaningfully optimize code. A memcpy implementation cannot rely on pointer validity to speculatively read because, even though memcpy(NULL, NULL, 0) is undefined, slices at the end of a buffer are fine. [And if the end of the buffer] were at the end of a page with nothing allocated afterwards, a speculative read from memcpy would break https://davidben.net/2024/01/15/empty-slices.html https://davidben.net/2024/01/15/empty-slices.html
  - Someone 2y ago
    > [And if the end of the buffer] were at the end of a page with nothing allocated afterwards, a speculative read from memcpy would break ‘Only’ on platforms that have memory protection hardware. Even there, the platform can always allocate an overflow page for a process, or have the page fault handler check whether the page fault happened due to a speculative read, and repair things (I think the latter is hugely, hugely, hugely impractical, but the standard cannot rule it out)
    - immibis 2y ago
      Platforms without memory protection hardware also have no problem reading NULL.
      - Someone 2y ago
        My comment is a reply to (part of) a comment that isn’t talking about reading from NULL. That’s what the [And if the end of the buffer] part implies. Even if it didn’t, I don’t think the standard should assume that “Platforms without memory protection hardware also have no problem reading NULL” An OS could, for example, have a very simple memory protection feature where the bottom half of the memory address range is reserved for the OS, the top half for user processes, and any read from an address with the high bit clear by code in the top half of the address range traps and makes the OS kill the process doing the read.
- xbar 2y ago
  Upon which some people may rely...
  - int_19h 2y ago
    People will only rely on UB when it is well defined by a particular implementation, either explicitly or because of a long history of past use. E.g. using unions for type punning in gcc, or allowing methods to be called on null pointers in MSVC. But there's nothing like that here.
    - pjmlp 2y ago
      Until a compiler version comes out and since it was UB anyway, the compiler sundenly now behaves in a different way.
- jancsika 2y ago
  I get that for the library. But I'm a bit puzzled about the optimizations done by a compiler based on this behavior. E.g., suppose we patch GCC to preserve any conditional containing the string 'NULL' in it. Would that have a measurable performance impact on Linux/Chromium/Firefox?
- captainmuon 2y ago
  I feel strongly they should split undefined behavior in behavior that is not defined, and things that the compiler is allowed to assume. The former basically already exists as "implementation defined behavior". The latter should be written out explicitly in the documentation: > memcpy(dest, src, count) > Copies count bytes from src to dest. [...] Note this is not a plain function, but a special form that applies the constraints dest != NULL and src != NULL to the surrounding scope. Equivalent to: assume(dest != NULL) assume(src != NULL) actual_memcpy(dest, src, count) The conflation of both concepts breaks the mental model of many programmers, especially ones who learned C/C++ in the 90s where it was common to write very different code, with all kinds of now illegal things like type punning and checking this != NULL. I'd love to have a flag "-fno-surprizing-ub" or "-fhighlevel-assembler" combined with the above `assume` function or some other syntax to let me help the compiler, so that I can write C like in the 90s - close to metal but with less surprizes.
  - Thorrez 2y ago
    >Note this is not a plain function, but a special form that applies the constraints dest != NULL and src != NULL to the surrounding scope. Plain functions can apply constraints to the surrounding code: https://godbolt.org/z/fP58WGz9f https://godbolt.org/z/fP58WGz9f
  - tialaramex 2y ago
    > I'd love to have a flag "-fno-surprizing-ub" or "-fhighlevel-assembler" combined with the above `assume` function or some other syntax to let me help the compiler, so that I can write C like in the 90s - close to metal but with less surprizes. The problem, which you may realise with some more introspection is that "surprising" is actually a property of you, not of the compiler, so you're asking for mind-reading and that's not one of the options. You want not to experience surprise. You can of course still get 1990s compilers and you're welcome to them. I cannot promise you won't still feel surprised despite your compiler nostalgia, but I can pretty much guarantee that the 1990s compiler results in slower and buggier software, so that's nice, remember only to charge 1990s rates for the work.
rcxdude 2y ago
Purely mechanically, yes, but in terms of the definition of the behaviour in the C abstract machine, no, because certain operations on null pointers are undefined, even if the obvious low-level compilation turns into nothing.
- codedokode 2y ago
  Maybe we should get rid of "abstract machine" and treat pointers as memory addresses?
  - davidt84 2y ago
    Congratulations, you've invented an entirely new language. Now, who's going to write the compiler for it?
    - anticensor 2y ago
      No, it's C at -O0.
      - davidt84 2y ago
        No, it's not. Undefined behaviour is undefined behaviour whatever optimisation level you use. Some -f flags may extend the C standard and remove undefined behaviour in some cases (e.g. strict aliasing, signed integer overflow, writable string constants, etc.)
  - gpderetta 2y ago
    int* oracle(); int foo() { int x = 1; *oracle() = 42; return x; } Is the above program allowed to return anything other than 1 in your language?
    - kibwen 2y ago
      To elaborate, we treat pointers as more than just integers because it gives optimizers the latitude to reorder and eliminate pointer operations. In the example above we cannot do this, because we cannot prove at compile time that x doesn't live at the address returned by oracle. For some high-quality further discussion, see Ralf Jung's series of blog posts starting with https://www.ralfj.de/blog/2018/07/24/pointers-and-bytes.html https://www.ralfj.de/blog/2018/07/24/pointers-and-bytes.html
IcePic 2y ago
"man bcopy" on BSD: 'If len is zero, no bytes are copied.' Seems reasonable.
- crest 2y ago
  As I understand that doesn't imply that it's not undefined to pass NULL pointers. While not what most users expect/want it's possible to this is just a wrapper around an memcpy() which will only be correct to call with valid destination and source pointers even if the length is zero.
pkhuong 2y ago
It does nothing, but is only defined when the pointers point into or one past the end of valid objects (live allocations), because that's how the standard defines the C VM, in terms of objects, not a flat byte array.
- whytevuhuni 2y ago
  What if the objects are non-NULL, but invalid (not actually allocated)? For example, Rust will use address 1 with length 0 for static empty strings, because 1 is a properly aligned non-null pointer. I would imagine such strings end up being passed to C code sometimes, which may end up calling memcpy with a length of 0 on them.
  - pkhuong 2y ago
    also UB according to the spec, but LLVM is free to define it. e.g., clang often converts trivial C++ copy constructors to memcpy, which is UB for self-assignment, but I assume that's fine because the C++ front-end only targets LLVM, and LLVM presumably defines the behaviour to do what you'd expect.
    - whytevuhuni 2y ago
      Where I work, it is quite normal to link together C code compiled with GCC and Rust code compiled with LLVM, due to how the build system is set up. As far as I know that disables LTO, but the build system is so complex, and the C code so large, that nobody bothers switching the C side to Clang/LLVM as well.
  - creshal 2y ago
    > What if the objects are non-NULL, but invalid (not actually allocated)? Still UB, since they're restricted pointers that must be valid to begin with.
    - bonzini 2y ago
      This is wrong. If you do p=malloc(256), p+256 is valid even though it does not point to a valid address (it might be in an unmapped page; check out ElectricFence). Rust's non-null aligned other pointer is the same, memcpy can't assume it can be dereferenced if the size is zero. The standard text in the linked paper says the same.
bluetomcat 2y ago
A trivial implementation wouldn't dereference dest or src in case the length is 0. That's how a student would write it with a for loop (byte-by-byte copy). A non-trivial implementation might do something with the pointers before entering the copy loop.
ryukoposting 2y ago
Yes and no. No, because ISO never said it must behave this way. Yes, because every libc I've personally encountered acts this way. At a glance, glibc's x86 implementation[1, 2], musl, and picolibc all handle 0-length memcpy as you'd expect. I'm sure other folks could dig up the code for Newlib, uclibc, and others, and they'd see the same thing. On a related note, ISO C has THREE different things that most people tend to lump together as "undefined behavior." They are: Implementation-defined behavior: ISO doesn't require any particular behavior, but they do require implementations to consistently apply a particular behavior, and document that behavior. Unspecified behavior: ISO doesn't require any particular behavior, but they do require implementations to consistently use a particular behavior, but they don't require that behavior to be documented. Undefined behavior: ISO doesn't require any particular behavior, and they don't require implementations to define any particular behavior either. [1]: https://github.com/lattera/glibc/blob/master/string/memcpy.c https://github.com/lattera/glibc/blob/master/string/memcpy.c [2]: https://github.com/lattera/glibc/blob/895ef79e04a953cac1493863bcae29ad85657ee1/sysdeps/i386/memcopy.h#L26 https://github.com/lattera/glibc/blob/895ef79e04a953cac14938...
ryao 2y ago
I have asked this question in the past and was told that memcpy() is allowed to preemptively read before it has determined it needs to write to make it faster on some CPUs. The presumption is that if you are going to be copying data, there is at least one cache line there already, so reading can start early.
- deleted 2y ago
  [deleted]

mjg59 2y ago

That's a reasonable intuitive interpretation of how it should behave, but according to the spec it's undefined behaviour and compilers have a great degree of freedom in what happens as a result.

voidUpdate 2y ago
Why didn't they just... define it, back when they wrote it?
- deleted 2y ago
  [deleted]
- deleted 2y ago
  [deleted]
- frabert 2y ago
  Every time they leave something undefined, they do so to leave implementations free to use the underlying platform's default behavior, and to allow compilers to use it as an optimization point
  - jcelerier 2y ago
    Here it's more that it allows to assume that this is never the case, thus no need to have an additional check in it I assume ?
  - lucozade 2y ago
    > time they leave something undefined, they do so to leave implementations free to use the underlying platform's default behavior That's implementation defined (more or less) ie teh compiler can do whatever makes mst sense for its implementation. Undefined means (more or less) that the compiler can assume the behaviour never happens so can apply transforms without taking it into account. > to allow compilers to use it as an optimization point That's the main advantage of undefined behaviour ie if you can ignore the usage, you may be able to apply optimisations that you couldn't if you had to take it into account. In the article, for example, GCC eliminated what it considered dead code for a NULL check of a variable that couldn't be NULL according to the C spec. That's also probably the most frustrating thing about optimisations based on undefined behaviour ie checks that prevent undefined behaviour are removed because the compiler thinks that the check can't ever succeed because, if it did, there must have been undefined behaviour. But the way the developer was ensuring defined behaviour was with the check!
    - frabert 2y ago
      AFAIK, something having undefined behavior in the spec does not prevent an implementation- (platform-)specific behavior being defined. As to your point about checks being erased, that generally happens when the checks happen too late (according to the compiler), or in a wrong way. For example, checking that `src` is not NULL _after_ memcpy(sec, dst, 0) is called. Or, checking for overflow by doing `if(x+y<0) ...` when x and y are nonnegative signed ints.
- nephanth 2y ago
  I mean, they might not have given thought to that particular corner case, they probably wrote something like > memcpy(void* ptr1, void* ptr2, int n) Copy n bytes from ptr1 to ptr2. UNDEFINED if ptr1 is NULL or ptr2 is NULL ‐------ It might also have come from a "explicit better than implicit" opinion, as in "it is better to have developers explicitly handle cases where the null pointer is involved
  - jbverschoor 2y ago
    I think it's more a strategy. C was not created to be safe. It's pretty much a tiny wrapper around assembler. Every limitation requires extra cycles, compile time or runtime, both of which were scarce. Of course, someone needs to check in the layers of abstraction. The user, programmer, compiler, cpu, architecture.. They chose for the programmer, who like to call themselves "engineers" these days.
    - wruza 2y ago
      Not sure what your last remark means wrt everything else.
    - poincaredisk 2y ago
      I disagree with your premise. C was designed to be a high level (for its time) language, abstracted from actual hardware >It's pretty much a tiny wrapper around assembler Assebler has zero problem with adding "null + 4" or computing "null-null". C does, because it's not actually a tiny wrapper.
      - jbverschoor 2y ago
        Not high-level.. Portable. Portable layer above assembler/arch. NULL doesn't exist in assembler, and in C, NULL is only a defined as a macro. It's not something built-in. C doesn't have any problems adding 4 to NULL nor subtracting NULL from NULL.
        jbverschoor 2y ago
- larschdk 2y ago
  When C was conceived, CPU architectures and platforms were more varied than what we see today. In order to remain portable and yet performant, some details were left as either implementation defined, or completely undefined (i.e. the responsibility of the programmer). Seems archaic today, but it was necessary when C compilers had to be two-pass and run in mere kilobytes of RAM. Even warnings for risky and undefined behavior is a relatively modern concept (last 10-20 years) compared to the age of C.
  - actionfromafar 2y ago
    When C was conceived, it was made for a specific DEC CPU, for making an operating system. The idea of a C standard was in the future. If you wanted to know what (for instance) memcpy actually did, you looked at the source code, or even more likely, the assembler or machine code output. That was "the standard".
    - anticensor 2y ago
      No, K&R's book was the standard.
      - actionfromafar 2y ago
        First came the language, then a few years later they described it in a book.
    - da_chicken 2y ago
      I think it's reasonable to assume that GP clearly meant the C standard being conceived, as, obviously, K&R's C implementation of the language was ad hoc rather than exhibiting any prescribed specification.
  - scoutt 2y ago
    > Seems archaic today ... run in mere kilobytes of RAM There is an entire industry that does pretty much that... today. They might run in flash instead of RAM, but still, a few kilobytes. Probably there are more embedded devices out there than PCs. PIC, AVR, MSP, ARM, custom archs. There might be one of those right now under your hand, in that thing you use to move the cursor.
- hyperman1 2y ago
  memcpy used to be a rep movsb on 8086 DOS compilers. I don't remember if rep movsb stops if cx=0 on entry, or decrements first and wraps around, copying 64K of data.
  - connicpu 2y ago
    I know at least MSVC's memcpy on x86_64 still results in a rep movsb if the cpuid flag that says rep movsb is fast is set, which it should be on all x86 chips from about 2011/2012 and onward ;)
  - dfox 2y ago
    The specification does not explicitly say that, but the clear intention is that REP with CX=0 should be no-op (you get exactly that situation when REP gets interrupted during the last iteration, in that case CX is zero and IP points to the REP, not the following instruction).
    - bonzini 2y ago
      Rep movsb copies 64K if CX=0 (that's actually very useful), but memcpy could be implemented as two instructions: jcxz skip rep movsb skip:
- wat10000 2y ago
  The original C standard was more descriptive than prescriptive. There was probably an implementation where it crashed or misbehaved.
- menaerus 2y ago
  Charitable interpretation may be: Back then when the contract of this function was standardized, presumably in C89 which is ~35 years ago, CPUs but also C compilers were not as powerful so wasting an extra couple of CPU cycles to check this condition was much more expensive than it is today. Because of that contract, and which can be seen in the example in the below comments, the compiler is also free to eliminate the dead code which also has the effect of shaving off some extra CPU cycles.
- lmm 2y ago
  Back when they wrote it they were trying to accommodate existing compilers, including those who did useful things to help people catch errors in their programs (e.g. making memcpy trap and send a signal if you called it with NULL). The current generation of compilers that use undefined behaviour as an excuse to do horrible things that screw over regular programmers but increase performance on microbenchmarks postdates the standard.
- FartyMcFarter 2y ago
  Because the benefit was probably seen as very little, and the cost significant. When you're writing a compiler for an architecture where every byte counts you don't make it write extra code for little benefit. Programmers were routinely counting bytes (both in code size and data) when writing Assembly code back then, and I mean that literally. Some of that carried into higher-level languages, and rightly so.
- killerstorm 2y ago
  From what I understand: 1. Initially, they just wanted to give compiler makers more freedom: both in the sense "do whatever is simplest" and "do something platform-specific which dev wants". 2. Compiler devs found that they can use UB for optimization: e.g. if we assume that a branch with UB is unreachable we can generate more efficient code. 3. Sadly, compiler devs started to exploit every opportunity for optimization, e.g. removing code with a potential segfault. I.e. people who made a standard thought that compiler would remove no-op call to memcpy, but GCC removes the whole branch which makes the call as it considers the whole branch impossible. Standard makers thought that compiler devs would be more reasonable
  - kllrnohj 2y ago
    > Standard makers thought that compiler devs would be more reasonable This is a bit of a terrible take? Compiler devs never did anything "unreasonable", they didn't sit down and go "mwahahaha we can exploit the heck out of UB to break everything!!!!" Rather, repeatedly applying a series of targeted optimizations, each one in isolation being "reasonable", results in an eventual "unreasonable" total transformation. But this is more an emergent property of modern compilers having hundreds of optimization passes. At the time the standards were created, the idea of compilers applying so many optimization passes was just not conceivable. Compilers struggled to just do basic compilation. The assumption was a near 1:1 mapping between code & assembly, and that just didn't age well at all.
    - LegionMammal978 2y ago
      One could argue that "optimizing based on signed overflow" was an unreasonable step to take, since any given platform will have some sane, consistent behavior when the underlying instructions cause an overflow. A developer using signed operations without poring over the standard might have easily expected incorrect values (or maybe a trap if the platform likes to use those), but not big changes in control flow. In my experience, signed overflow is generally the biggest cause of "they're putting UB in my reasonable C code!", followed by the rules against type punning, which are violated every day by ordinary usage of the POSIX socket functions.
- ynik 2y ago
  Probably because they did not think of this special case when writing the standard, or did not find it important enough to consider complicating the standard text for. In C89, there's just a general provision for all standard library functions: > Each of the following statements applies unless explicitly stated otherwise in the detailed descriptions that follow. If an argument to a function has an invalid value (such as a value outside the domain of the function, or a pointer outside the address space of the program, or a null pointer), the behavior is undefined. [...] And then there isn't anything on `memcpy` that would explicitly state otherwise. Later versions of the standard explicitly clarified that this requirement applies even to size 0, but at that point it was only a clarification of an existing requirement from the earlier standard. People like to read a lot more intention into the standard than is reasonable. Lots of it is just historical accident, really.
david-gpu 2y ago
More information on this behavior in the link below. > Note that, apart from contrived examples with deleted null checks, the current rules do not actually help the compiler meaningfully optimize code. A memcpy implementation cannot rely on pointer validity to speculatively read because, even though memcpy(NULL, NULL, 0) is undefined, slices at the end of a buffer are fine. [And if the end of the buffer] were at the end of a page with nothing allocated afterwards, a speculative read from memcpy would break https://davidben.net/2024/01/15/empty-slices.html https://davidben.net/2024/01/15/empty-slices.html
- Someone 2y ago
  > [And if the end of the buffer] were at the end of a page with nothing allocated afterwards, a speculative read from memcpy would break ‘Only’ on platforms that have memory protection hardware. Even there, the platform can always allocate an overflow page for a process, or have the page fault handler check whether the page fault happened due to a speculative read, and repair things (I think the latter is hugely, hugely, hugely impractical, but the standard cannot rule it out)
  - immibis 2y ago
    Platforms without memory protection hardware also have no problem reading NULL.
    - Someone 2y ago
      My comment is a reply to (part of) a comment that isn’t talking about reading from NULL. That’s what the [And if the end of the buffer] part implies. Even if it didn’t, I don’t think the standard should assume that “Platforms without memory protection hardware also have no problem reading NULL” An OS could, for example, have a very simple memory protection feature where the bottom half of the memory address range is reserved for the OS, the top half for user processes, and any read from an address with the high bit clear by code in the top half of the address range traps and makes the OS kill the process doing the read.
xbar 2y ago
Upon which some people may rely...
- int_19h 2y ago
  People will only rely on UB when it is well defined by a particular implementation, either explicitly or because of a long history of past use. E.g. using unions for type punning in gcc, or allowing methods to be called on null pointers in MSVC. But there's nothing like that here.
  - pjmlp 2y ago
    Until a compiler version comes out and since it was UB anyway, the compiler sundenly now behaves in a different way.
jancsika 2y ago
I get that for the library. But I'm a bit puzzled about the optimizations done by a compiler based on this behavior. E.g., suppose we patch GCC to preserve any conditional containing the string 'NULL' in it. Would that have a measurable performance impact on Linux/Chromium/Firefox?
captainmuon 2y ago
I feel strongly they should split undefined behavior in behavior that is not defined, and things that the compiler is allowed to assume. The former basically already exists as "implementation defined behavior". The latter should be written out explicitly in the documentation: > memcpy(dest, src, count) > Copies count bytes from src to dest. [...] Note this is not a plain function, but a special form that applies the constraints dest != NULL and src != NULL to the surrounding scope. Equivalent to: assume(dest != NULL) assume(src != NULL) actual_memcpy(dest, src, count) The conflation of both concepts breaks the mental model of many programmers, especially ones who learned C/C++ in the 90s where it was common to write very different code, with all kinds of now illegal things like type punning and checking this != NULL. I'd love to have a flag "-fno-surprizing-ub" or "-fhighlevel-assembler" combined with the above `assume` function or some other syntax to let me help the compiler, so that I can write C like in the 90s - close to metal but with less surprizes.
- Thorrez 2y ago
  >Note this is not a plain function, but a special form that applies the constraints dest != NULL and src != NULL to the surrounding scope. Plain functions can apply constraints to the surrounding code: https://godbolt.org/z/fP58WGz9f https://godbolt.org/z/fP58WGz9f
- tialaramex 2y ago
  > I'd love to have a flag "-fno-surprizing-ub" or "-fhighlevel-assembler" combined with the above `assume` function or some other syntax to let me help the compiler, so that I can write C like in the 90s - close to metal but with less surprizes. The problem, which you may realise with some more introspection is that "surprising" is actually a property of you, not of the compiler, so you're asking for mind-reading and that's not one of the options. You want not to experience surprise. You can of course still get 1990s compilers and you're welcome to them. I cannot promise you won't still feel surprised despite your compiler nostalgia, but I can pretty much guarantee that the 1990s compiler results in slower and buggier software, so that's nice, remember only to charge 1990s rates for the work.

rcxdude 2y ago

Purely mechanically, yes, but in terms of the definition of the behaviour in the C abstract machine, no, because certain operations on null pointers are undefined, even if the obvious low-level compilation turns into nothing.

codedokode 2y ago
Maybe we should get rid of "abstract machine" and treat pointers as memory addresses?
- davidt84 2y ago
  Congratulations, you've invented an entirely new language. Now, who's going to write the compiler for it?
  - anticensor 2y ago
    No, it's C at -O0.
    - davidt84 2y ago
      No, it's not. Undefined behaviour is undefined behaviour whatever optimisation level you use. Some -f flags may extend the C standard and remove undefined behaviour in some cases (e.g. strict aliasing, signed integer overflow, writable string constants, etc.)
- gpderetta 2y ago
  int* oracle(); int foo() { int x = 1; *oracle() = 42; return x; } Is the above program allowed to return anything other than 1 in your language?
  - kibwen 2y ago
    To elaborate, we treat pointers as more than just integers because it gives optimizers the latitude to reorder and eliminate pointer operations. In the example above we cannot do this, because we cannot prove at compile time that x doesn't live at the address returned by oracle. For some high-quality further discussion, see Ralf Jung's series of blog posts starting with https://www.ralfj.de/blog/2018/07/24/pointers-and-bytes.html https://www.ralfj.de/blog/2018/07/24/pointers-and-bytes.html

IcePic 2y ago

"man bcopy" on BSD: 'If len is zero, no bytes are copied.' Seems reasonable.

crest 2y ago
As I understand that doesn't imply that it's not undefined to pass NULL pointers. While not what most users expect/want it's possible to this is just a wrapper around an memcpy() which will only be correct to call with valid destination and source pointers even if the length is zero.

pkhuong 2y ago

It does nothing, but is only defined when the pointers point into or one past the end of valid objects (live allocations), because that's how the standard defines the C VM, in terms of objects, not a flat byte array.

whytevuhuni 2y ago
What if the objects are non-NULL, but invalid (not actually allocated)? For example, Rust will use address 1 with length 0 for static empty strings, because 1 is a properly aligned non-null pointer. I would imagine such strings end up being passed to C code sometimes, which may end up calling memcpy with a length of 0 on them.
- pkhuong 2y ago
  also UB according to the spec, but LLVM is free to define it. e.g., clang often converts trivial C++ copy constructors to memcpy, which is UB for self-assignment, but I assume that's fine because the C++ front-end only targets LLVM, and LLVM presumably defines the behaviour to do what you'd expect.
  - whytevuhuni 2y ago
    Where I work, it is quite normal to link together C code compiled with GCC and Rust code compiled with LLVM, due to how the build system is set up. As far as I know that disables LTO, but the build system is so complex, and the C code so large, that nobody bothers switching the C side to Clang/LLVM as well.
- creshal 2y ago
  > What if the objects are non-NULL, but invalid (not actually allocated)? Still UB, since they're restricted pointers that must be valid to begin with.
  - bonzini 2y ago
    This is wrong. If you do p=malloc(256), p+256 is valid even though it does not point to a valid address (it might be in an unmapped page; check out ElectricFence). Rust's non-null aligned other pointer is the same, memcpy can't assume it can be dereferenced if the size is zero. The standard text in the linked paper says the same.

bluetomcat 2y ago

A trivial implementation wouldn't dereference dest or src in case the length is 0. That's how a student would write it with a for loop (byte-by-byte copy). A non-trivial implementation might do something with the pointers before entering the copy loop.

ryukoposting 2y ago

Yes and no. No, because ISO never said it must behave this way. Yes, because every libc I've personally encountered acts this way. At a glance, glibc's x86 implementation[1, 2], musl, and picolibc all handle 0-length memcpy as you'd expect. I'm sure other folks could dig up the code for Newlib, uclibc, and others, and they'd see the same thing. On a related note, ISO C has THREE different things that most people tend to lump together as "undefined behavior." They are: Implementation-defined behavior: ISO doesn't require any particular behavior, but they do require implementations to consistently apply a particular behavior, and document that behavior. Unspecified behavior: ISO doesn't require any particular behavior, but they do require implementations to consistently use a particular behavior, but they don't require that behavior to be documented. Undefined behavior: ISO doesn't require any particular behavior, and they don't require implementations to define any particular behavior either. [1]: https://github.com/lattera/glibc/blob/master/string/memcpy.c https://github.com/lattera/glibc/blob/master/string/memcpy.c [2]: https://github.com/lattera/glibc/blob/895ef79e04a953cac1493863bcae29ad85657ee1/sysdeps/i386/memcopy.h#L26 https://github.com/lattera/glibc/blob/895ef79e04a953cac14938...

ryao 2y ago

I have asked this question in the past and was told that memcpy() is allowed to preemptively read before it has determined it needs to write to make it faster on some CPUs. The presumption is that if you are going to be copying data, there is at least one cache line there already, so reading can start early.

deleted 2y ago
[deleted]