12 ms·

> The determined refusal of the specification to tie this abstract machine in concrete CPUs is the source of a significant amount of frustration in people who w

> The determined refusal of the specification to tie this abstract machine in concrete CPUs is the source of a significant amount of frustration in people who would like, for example, for there to be some semantics attached to what happens when you dereference an invalid pointer. They note that actual CPUs running C code all have defined semantics, so why can't C? Because while all CPUs running C code have defined semantics for any given construct, not all CPUs have the same defined semantics. Making C adhere to one CPU architecture for the sake of convenience would make implementations for other CPU architectures decidedly inconvenient. Rather than dereferencing an invalid pointer doing "whatever this CPU does", suddenly all C implementations now have to do whatever, for example, a VAX did. It would mean that C code running on simple CPUs without memory protection would have to look out for invalid pointer derefs (how?) and simulate access violations if one were mandated by the spec. Or, if the spec said that the address space should be treated as a flat memory area all of which is accessible by any pointer, modern implementations would have to simulate that.

kragen 4y ago
it would solve a lot of problems if leniency introduced into the ansi spec to permit c implementations on the burroughs large systems line was only used in cases like that, instead of compiler vendors abusing it to eke out 0.2% performance gains on amd64 hardware at the cost of introducing priority 1 security bugs into operating system kernels
torstenvl 4y ago
> Rather than dereferencing an invalid pointer doing "whatever this CPU does", suddenly all C implementations now have to do whatever, for example, a VAX did. Dereferencing an invalid pointer doesn't do "whatever this CPU does," it does completely unpredictable things. I think that's part of the problem (to the extent it's a problem)
- pavlov 4y ago
  It’s up to the platform in the end. I recall IBM’s AIX used to put a zeroed-out block at address zero, so this code was guaranteed to always return 0: *(int *)NULL The reason was that it allowed their optimizing compiler to speculatively dereference a pointer before executing a surrounding if condition.
  - josephg 4y ago
    > It’s up to the platform in the end. It’s not up to the platform. If it looks like a code path might dereference a null pointer, the compiler can and will make wild and bizarre optimizations in the surrounding code.
    - jstimpfle 4y ago
      The compiler doesn't have to come to your home and actively haunt you. The behaviour is indeed up to the platform (by "platform" I understand "compiler + host system" or whatever is used to run the program).
      - josephg 4y ago
        Ah - I suppose I don’t think of the compiler as part of the platform. Most platforms I work with tend to use gcc or clang, and as a result everyone is subject to the gcc / clang optimizations.
    - Karellen 4y ago
      If code dereferences a NULL pointer, the spec imposes no constraints whatsoever on what happens after that point. Yes, some compilers will make "wild and bizarre" optimisations (they are neither) based on the assumption that any pointer which is dereferenced is not NULL - because if it were, the code could have done the exact same thing anyway, because there are no constraints in that case. But compilers don't have to do that. There are no constraints. A compiler can emit code to do anything at all in the event of a NULL pointer being dereferenced, including emitting the machine code you'd naively expect - which may trap on some CPUs, or always return 0 on reads and swallow writes on other CPUs. It may also emit code that wipes your hard disk (and still counts as a conforming implementation). Or make demons fly out of your nose. A compiler vendor may choose to guarantee a specific behaviour for some invalid constructs. They are allowed to do this because there are no constraints on what they can do. It's just that most don't, because 99% of people comparing compilers care more about benchmarks than they do about what happens with invalid code. So in some ways, it is up to the compiler/platform/CPU what actually happens with code that has invalid behaviour. But as a C author, you can't assume any particular behaviour, because it might change from one CPU to another. Or from one OS to another. Or from one compiler to another. Or from one version of one compiler to another.
- bluGill 4y ago
  CPUs do different things when they know it is invalid. Often what happens is it send some sort of trap/signal to the operating system - on unix this would send a segfault and dump core. Not all invalid pointers can be detected, and not all CPUs have the required hardware (MMU) to detect any invalid pointer.
  - jstimpfle 4y ago
    Exactly. Even though you could specify what happens mechanically on any given system, the results are kind of uncontrollable on most real computers. Whether things should be UB or implementation defined requires a good value judgement. Per se there is not a real need of UB, (assuming that the behaviour of a machine can be specified), but on the other hand there isn't much value in specifying a situation where all control is lost to the point where implementation details leak (runtime structures are overwritten etc.). Since there is (I assume?) an expectation for "implementation-defined" to have an actual definition, pedantically an implementation leak would require all implementation details to be codified, and thus set in stone.
    - chalst 4y ago
      This could be indicated in a comment, perhaps using a *RFI-like notation.
    - bluGill 4y ago
      It is more that while we can define everything, there are costs. I can track every allocation in a hidden table, then before following a pointer verify that the point being followed is in the table. However that is extremely expensive (don't forget that you might follow a pointer in one thread while a different thread deletes it, so there is a complex race condition logic needed here that is tricky to get right) By making things undefined we can avoid a large amount of complex code to detect a situation that shouldn't happen.
      - kaba0 4y ago
        Exactly. While it is definitely harder with a low-level language, it is absolutely doable in terms of costs with managed languages. In Java for example even data races are well-defined to a degree (and OCaml’s new multithreaded mode defines data races with even bigger guarantees)
- butlerm 4y ago
  I disagree that it is completely unpredictable. If it is a read operation, dereferencing an invalid pointer on every normal system will either return data with an unspecified value or will raise a low level exception that may be caught. Since a compiler cannot predict in all cases in advance whether a pointer is invalid it should be required to compile a pointer dereference the same way in all cases, i.e. emit the same code regardless of whether it has determined the pointer is invalid or not. Otherwise it effectively emits random code depending on how the optimizer is feeling. Non-deterministic behavior for the same operation on the same values in the same program is not very helpful behavior, whenever and to the degree it can be avoided.
- arnsholt 4y ago
  But that's the thing: dereferencing an invalid pointer is undefined behaviour, which means the compiler is allowed to assume it never happens; a C program executing undefined behaviour is _invalid C_. Thus, any time you dereference a pointer you also implicitly proimise the compiler that this pointer will _never_ be an invalid pointer. Same with signed arithmetic: you are telling the compiler that your arithmetic is guaranteed to never overflow. Whether this is a good or bad thing is of course a legitimate (and good!) question, but for writing C today that's how the language is specced and a reality the programmer needs to take care to avoid, just like a bunch of other things that C leaves to the programmer like remembering to clean up allocated resources when they're no longer needed.
  - ClumsyPilot 4y ago
    < you are telling the compiler that your arithmetic is guaranteed to never overflow. Any time your are dealing with data from real, physical sensors or third-party APIs, this is an impossible guarantee to give - they could literslly break.
    - compiler-guy 4y ago
      of course it is possible. One must validate the input before performing arithmetic on signed integers.
      - deleted 4y ago
        [deleted]
      - imtringued 4y ago
        The compiler is allowed to assume that the data will never overflow so it is also allowed to get rid of overflow checks. Now imagine writing a complex validation routine that if it were violated would result in undefined behavior for every value that the validation rejects, a sufficiently smart compiler is allowed to simply remove your validation code, leaving you with no validation whatsoever.
        deleted 4y ago
- kps 4y ago
  C did do ‘whatever this CPU does’, and then the ANSI C committee broke that (probably by accident). That's what the headline post is about: “Those origins of C were there first”.
  - uecker 4y ago
    The ANSI C committee certainly did not break this. UB always meant that ANSI C simply did not define what happens in this specific case. This could mean "whatever the CPU does" or "whatever the compiler decides". What caused problems then was that compilers started to aggressively exploit UB for optimization (and customers probably asked for such optimizations) treating programs with UB as invalid. When this went to far and caused problems, compiler vendors then blamed the user and the standard. But obviously it is completely at the compiler developer's discretion what the compiler does, so customers should just push back and request changes.
    - bluGill 4y ago
      In the vast majority of cases I'm glad that my programs run fast. sure I have to do some work to avoid undefined behavior, but most code is correctly defined, and the exceptions are bad code that I need to fix anyway as even if the behavior was 100% defined to C it would still be buggy as in not do what my users want.
butlerm 4y ago
The problem is that dereferencing a null pointer does not actually have undefined semantics, it has system defined semantics. The compiler should compile source code in such a way as to produce machine code that does whatever that system does when a null pointer is dereferenced. It should do this in part because very large volumes of code will be compiled that way due to the inability to detect in advance whether a pointer is null or not, and compiling it differently when it is known that a pointer is null makes for inconsistent behavior and vulnerabilities like this one. This is the way relatively sane simple-minded compilers have worked for a long time, and no compiler should be allowed to pretend that certain constructs do not have reliable or at least consistent system defined behavior that will ensue if they are used. Similarly if a compiler can tell that a signed addition will overflow in some cases it should be required to do whatever a signed addition overflow does on that architecture, and for the same reason. Issue the appropriate warnings about non-portable code, but do something reliable and reliably useful.
- jstimpfle 4y ago
  > The problem is that dereferencing a null pointer does not actually have undefined semantics, it has system defined semantics. No it does not. Look in the standard. Seriously though, I get what you mean but what is the point of having a systems-defined NULL dereference? The semantics of the NULL pointer (in C) is such that you are not meant to dereference it. If you dereference it, you have a bug no matter what the manifestation on a specific system is. And there are probably actual systems where it would be hard to specify the behaviour. For example MMU-less systems, where you can derefence byte 0 but it might not be statically clear what kind of value is stored there? Maybe C allows a systems-defined behaviour to be implemented as undefined behaviour? But then the distinction is kinda moot.
  - zajio1am 4y ago
    > Maybe C allows a systems-defined behaviour to be implemented as undefined behaviour C standard distinguishes undefined behavior, unspecified behavior and implementation-defined behavior. The first is invalid and inconsistent, the second is valid and inconsistent, the third is valid and consistent (in scope of implementation). It is rather hard to require invalid but consistent, as the platform itself may produce inconsistent behavior in invalid cases. So what some people want is something like - "be no more inconsistent that would be naive implementation on given platform", but that is both vague and not really useful. And while some of these undefined-based optimizations seems egregious, some are necessary in order to have reasonably performant code (e.g. keeping variable values in registers instead of propagating them from/to memory after each step).
xwolfi 4y ago
But to play the devil's advocate, where would we be if architectures followed higher level abstractions to a point they would be hard to differenciate ? They d be slower to run maybe, but so much nicer to program for!
marcosdumay 4y ago
So, another article supporting implementation dependent behavior when everybody is out complaining about undefined behavior. Of course implementation dependent behavior is a necessity. Undefined behavior too, on the very few cases that nobody complains about. This isn't one of them.
IshKebab 4y ago
Right but there definitely are things that were different in the past but are never really going to change in the future (because they are de facto standards). A byte is now always 8 bits. Integers are always 2s complement with wrapping arithmetic. Nobody is going to make a CPU that doesn't use those (at least nobody that wants people to actually use it) because no software would work on it. For example RISC-V set the cache line size to 64 bytes because that's what everyone else does and going against the grain is too difficult now (and maybe there was no reason to in that case but still...) The only thing I can think of that benefits from C's "nothing is well defined" approach is CHERI, but I bet a ton of code needs fixes to work with it.

kragen 4y ago

it would solve a lot of problems if leniency introduced into the ansi spec to permit c implementations on the burroughs large systems line was only used in cases like that, instead of compiler vendors abusing it to eke out 0.2% performance gains on amd64 hardware at the cost of introducing priority 1 security bugs into operating system kernels

torstenvl 4y ago

> Rather than dereferencing an invalid pointer doing "whatever this CPU does", suddenly all C implementations now have to do whatever, for example, a VAX did. Dereferencing an invalid pointer doesn't do "whatever this CPU does," it does completely unpredictable things. I think that's part of the problem (to the extent it's a problem)

pavlov 4y ago
It’s up to the platform in the end. I recall IBM’s AIX used to put a zeroed-out block at address zero, so this code was guaranteed to always return 0: *(int *)NULL The reason was that it allowed their optimizing compiler to speculatively dereference a pointer before executing a surrounding if condition.
- josephg 4y ago
  > It’s up to the platform in the end. It’s not up to the platform. If it looks like a code path might dereference a null pointer, the compiler can and will make wild and bizarre optimizations in the surrounding code.
  - jstimpfle 4y ago
    The compiler doesn't have to come to your home and actively haunt you. The behaviour is indeed up to the platform (by "platform" I understand "compiler + host system" or whatever is used to run the program).
    - josephg 4y ago
      Ah - I suppose I don’t think of the compiler as part of the platform. Most platforms I work with tend to use gcc or clang, and as a result everyone is subject to the gcc / clang optimizations.
  - Karellen 4y ago
    If code dereferences a NULL pointer, the spec imposes no constraints whatsoever on what happens after that point. Yes, some compilers will make "wild and bizarre" optimisations (they are neither) based on the assumption that any pointer which is dereferenced is not NULL - because if it were, the code could have done the exact same thing anyway, because there are no constraints in that case. But compilers don't have to do that. There are no constraints. A compiler can emit code to do anything at all in the event of a NULL pointer being dereferenced, including emitting the machine code you'd naively expect - which may trap on some CPUs, or always return 0 on reads and swallow writes on other CPUs. It may also emit code that wipes your hard disk (and still counts as a conforming implementation). Or make demons fly out of your nose. A compiler vendor may choose to guarantee a specific behaviour for some invalid constructs. They are allowed to do this because there are no constraints on what they can do. It's just that most don't, because 99% of people comparing compilers care more about benchmarks than they do about what happens with invalid code. So in some ways, it is up to the compiler/platform/CPU what actually happens with code that has invalid behaviour. But as a C author, you can't assume any particular behaviour, because it might change from one CPU to another. Or from one OS to another. Or from one compiler to another. Or from one version of one compiler to another.
bluGill 4y ago
CPUs do different things when they know it is invalid. Often what happens is it send some sort of trap/signal to the operating system - on unix this would send a segfault and dump core. Not all invalid pointers can be detected, and not all CPUs have the required hardware (MMU) to detect any invalid pointer.
- jstimpfle 4y ago
  Exactly. Even though you could specify what happens mechanically on any given system, the results are kind of uncontrollable on most real computers. Whether things should be UB or implementation defined requires a good value judgement. Per se there is not a real need of UB, (assuming that the behaviour of a machine can be specified), but on the other hand there isn't much value in specifying a situation where all control is lost to the point where implementation details leak (runtime structures are overwritten etc.). Since there is (I assume?) an expectation for "implementation-defined" to have an actual definition, pedantically an implementation leak would require all implementation details to be codified, and thus set in stone.
  - chalst 4y ago
    This could be indicated in a comment, perhaps using a *RFI-like notation.
  - bluGill 4y ago
    It is more that while we can define everything, there are costs. I can track every allocation in a hidden table, then before following a pointer verify that the point being followed is in the table. However that is extremely expensive (don't forget that you might follow a pointer in one thread while a different thread deletes it, so there is a complex race condition logic needed here that is tricky to get right) By making things undefined we can avoid a large amount of complex code to detect a situation that shouldn't happen.
    - kaba0 4y ago
      Exactly. While it is definitely harder with a low-level language, it is absolutely doable in terms of costs with managed languages. In Java for example even data races are well-defined to a degree (and OCaml’s new multithreaded mode defines data races with even bigger guarantees)
butlerm 4y ago
I disagree that it is completely unpredictable. If it is a read operation, dereferencing an invalid pointer on every normal system will either return data with an unspecified value or will raise a low level exception that may be caught. Since a compiler cannot predict in all cases in advance whether a pointer is invalid it should be required to compile a pointer dereference the same way in all cases, i.e. emit the same code regardless of whether it has determined the pointer is invalid or not. Otherwise it effectively emits random code depending on how the optimizer is feeling. Non-deterministic behavior for the same operation on the same values in the same program is not very helpful behavior, whenever and to the degree it can be avoided.
arnsholt 4y ago
But that's the thing: dereferencing an invalid pointer is undefined behaviour, which means the compiler is allowed to assume it never happens; a C program executing undefined behaviour is _invalid C_. Thus, any time you dereference a pointer you also implicitly proimise the compiler that this pointer will _never_ be an invalid pointer. Same with signed arithmetic: you are telling the compiler that your arithmetic is guaranteed to never overflow. Whether this is a good or bad thing is of course a legitimate (and good!) question, but for writing C today that's how the language is specced and a reality the programmer needs to take care to avoid, just like a bunch of other things that C leaves to the programmer like remembering to clean up allocated resources when they're no longer needed.
- ClumsyPilot 4y ago
  < you are telling the compiler that your arithmetic is guaranteed to never overflow. Any time your are dealing with data from real, physical sensors or third-party APIs, this is an impossible guarantee to give - they could literslly break.
  - compiler-guy 4y ago
    of course it is possible. One must validate the input before performing arithmetic on signed integers.
    - deleted 4y ago
      [deleted]
    - imtringued 4y ago
      The compiler is allowed to assume that the data will never overflow so it is also allowed to get rid of overflow checks. Now imagine writing a complex validation routine that if it were violated would result in undefined behavior for every value that the validation rejects, a sufficiently smart compiler is allowed to simply remove your validation code, leaving you with no validation whatsoever.
      - deleted 4y ago
kps 4y ago
C did do ‘whatever this CPU does’, and then the ANSI C committee broke that (probably by accident). That's what the headline post is about: “Those origins of C were there first”.
- uecker 4y ago
  The ANSI C committee certainly did not break this. UB always meant that ANSI C simply did not define what happens in this specific case. This could mean "whatever the CPU does" or "whatever the compiler decides". What caused problems then was that compilers started to aggressively exploit UB for optimization (and customers probably asked for such optimizations) treating programs with UB as invalid. When this went to far and caused problems, compiler vendors then blamed the user and the standard. But obviously it is completely at the compiler developer's discretion what the compiler does, so customers should just push back and request changes.
  - bluGill 4y ago
    In the vast majority of cases I'm glad that my programs run fast. sure I have to do some work to avoid undefined behavior, but most code is correctly defined, and the exceptions are bad code that I need to fix anyway as even if the behavior was 100% defined to C it would still be buggy as in not do what my users want.

butlerm 4y ago

The problem is that dereferencing a null pointer does not actually have undefined semantics, it has system defined semantics. The compiler should compile source code in such a way as to produce machine code that does whatever that system does when a null pointer is dereferenced. It should do this in part because very large volumes of code will be compiled that way due to the inability to detect in advance whether a pointer is null or not, and compiling it differently when it is known that a pointer is null makes for inconsistent behavior and vulnerabilities like this one. This is the way relatively sane simple-minded compilers have worked for a long time, and no compiler should be allowed to pretend that certain constructs do not have reliable or at least consistent system defined behavior that will ensue if they are used. Similarly if a compiler can tell that a signed addition will overflow in some cases it should be required to do whatever a signed addition overflow does on that architecture, and for the same reason. Issue the appropriate warnings about non-portable code, but do something reliable and reliably useful.

jstimpfle 4y ago
> The problem is that dereferencing a null pointer does not actually have undefined semantics, it has system defined semantics. No it does not. Look in the standard. Seriously though, I get what you mean but what is the point of having a systems-defined NULL dereference? The semantics of the NULL pointer (in C) is such that you are not meant to dereference it. If you dereference it, you have a bug no matter what the manifestation on a specific system is. And there are probably actual systems where it would be hard to specify the behaviour. For example MMU-less systems, where you can derefence byte 0 but it might not be statically clear what kind of value is stored there? Maybe C allows a systems-defined behaviour to be implemented as undefined behaviour? But then the distinction is kinda moot.
- zajio1am 4y ago
  > Maybe C allows a systems-defined behaviour to be implemented as undefined behaviour C standard distinguishes undefined behavior, unspecified behavior and implementation-defined behavior. The first is invalid and inconsistent, the second is valid and inconsistent, the third is valid and consistent (in scope of implementation). It is rather hard to require invalid but consistent, as the platform itself may produce inconsistent behavior in invalid cases. So what some people want is something like - "be no more inconsistent that would be naive implementation on given platform", but that is both vague and not really useful. And while some of these undefined-based optimizations seems egregious, some are necessary in order to have reasonably performant code (e.g. keeping variable values in registers instead of propagating them from/to memory after each step).

xwolfi 4y ago

But to play the devil's advocate, where would we be if architectures followed higher level abstractions to a point they would be hard to differenciate ? They d be slower to run maybe, but so much nicer to program for!

marcosdumay 4y ago

So, another article supporting implementation dependent behavior when everybody is out complaining about undefined behavior. Of course implementation dependent behavior is a necessity. Undefined behavior too, on the very few cases that nobody complains about. This isn't one of them.

IshKebab 4y ago

Right but there definitely are things that were different in the past but are never really going to change in the future (because they are de facto standards). A byte is now always 8 bits. Integers are always 2s complement with wrapping arithmetic. Nobody is going to make a CPU that doesn't use those (at least nobody that wants people to actually use it) because no software would work on it. For example RISC-V set the cache line size to 64 bytes because that's what everyone else does and going against the grain is too difficult now (and maybe there was no reason to in that case but still...) The only thing I can think of that benefits from C's "nothing is well defined" approach is CHERI, but I bet a ton of code needs fixes to work with it.