11 ms·

> Rather than dereferencing an invalid pointer doing "whatever this CPU does", suddenly all C implementations now have to do whatever, for example, a VAX did.

> Rather than dereferencing an invalid pointer doing "whatever this CPU does", suddenly all C implementations now have to do whatever, for example, a VAX did. Dereferencing an invalid pointer doesn't do "whatever this CPU does," it does completely unpredictable things. I think that's part of the problem (to the extent it's a problem)

pavlov 4y ago
It’s up to the platform in the end. I recall IBM’s AIX used to put a zeroed-out block at address zero, so this code was guaranteed to always return 0: *(int *)NULL The reason was that it allowed their optimizing compiler to speculatively dereference a pointer before executing a surrounding if condition.
- josephg 4y ago
  > It’s up to the platform in the end. It’s not up to the platform. If it looks like a code path might dereference a null pointer, the compiler can and will make wild and bizarre optimizations in the surrounding code.
  - jstimpfle 4y ago
    The compiler doesn't have to come to your home and actively haunt you. The behaviour is indeed up to the platform (by "platform" I understand "compiler + host system" or whatever is used to run the program).
    - josephg 4y ago
      Ah - I suppose I don’t think of the compiler as part of the platform. Most platforms I work with tend to use gcc or clang, and as a result everyone is subject to the gcc / clang optimizations.
  - Karellen 4y ago
    If code dereferences a NULL pointer, the spec imposes no constraints whatsoever on what happens after that point. Yes, some compilers will make "wild and bizarre" optimisations (they are neither) based on the assumption that any pointer which is dereferenced is not NULL - because if it were, the code could have done the exact same thing anyway, because there are no constraints in that case. But compilers don't have to do that. There are no constraints. A compiler can emit code to do anything at all in the event of a NULL pointer being dereferenced, including emitting the machine code you'd naively expect - which may trap on some CPUs, or always return 0 on reads and swallow writes on other CPUs. It may also emit code that wipes your hard disk (and still counts as a conforming implementation). Or make demons fly out of your nose. A compiler vendor may choose to guarantee a specific behaviour for some invalid constructs. They are allowed to do this because there are no constraints on what they can do. It's just that most don't, because 99% of people comparing compilers care more about benchmarks than they do about what happens with invalid code. So in some ways, it is up to the compiler/platform/CPU what actually happens with code that has invalid behaviour. But as a C author, you can't assume any particular behaviour, because it might change from one CPU to another. Or from one OS to another. Or from one compiler to another. Or from one version of one compiler to another.
    - pavlov 4y ago
      This. In the case of AIX, IBM designed every component of the platform: the CPU, the operating system and the compiler. When they tell you that it’s safe to read up to 4K (or whatever) of zero bytes from NULL, they can make that promise. But obviously such code is not portable.
    - imtringued 4y ago
      But this means the vast majority of C code isn't portable since you can't know what it will do with a given platform and compiler combination and the presence of even a single line of UBI invalidates the whole program. It basically means that the whole portability argument that is supposed to be in favor of C is just wrong because every compiler and platform actually brings its own C dialect and it is just sheer coincidence that it works at all.
  - amalcon 4y ago
    So it is up to the platform, since the compiler is part of the platform.
bluGill 4y ago
CPUs do different things when they know it is invalid. Often what happens is it send some sort of trap/signal to the operating system - on unix this would send a segfault and dump core. Not all invalid pointers can be detected, and not all CPUs have the required hardware (MMU) to detect any invalid pointer.
- jstimpfle 4y ago
  Exactly. Even though you could specify what happens mechanically on any given system, the results are kind of uncontrollable on most real computers. Whether things should be UB or implementation defined requires a good value judgement. Per se there is not a real need of UB, (assuming that the behaviour of a machine can be specified), but on the other hand there isn't much value in specifying a situation where all control is lost to the point where implementation details leak (runtime structures are overwritten etc.). Since there is (I assume?) an expectation for "implementation-defined" to have an actual definition, pedantically an implementation leak would require all implementation details to be codified, and thus set in stone.
  - chalst 4y ago
    This could be indicated in a comment, perhaps using a *RFI-like notation.
  - bluGill 4y ago
    It is more that while we can define everything, there are costs. I can track every allocation in a hidden table, then before following a pointer verify that the point being followed is in the table. However that is extremely expensive (don't forget that you might follow a pointer in one thread while a different thread deletes it, so there is a complex race condition logic needed here that is tricky to get right) By making things undefined we can avoid a large amount of complex code to detect a situation that shouldn't happen.
    - kaba0 4y ago
      Exactly. While it is definitely harder with a low-level language, it is absolutely doable in terms of costs with managed languages. In Java for example even data races are well-defined to a degree (and OCaml’s new multithreaded mode defines data races with even bigger guarantees)
butlerm 4y ago
I disagree that it is completely unpredictable. If it is a read operation, dereferencing an invalid pointer on every normal system will either return data with an unspecified value or will raise a low level exception that may be caught. Since a compiler cannot predict in all cases in advance whether a pointer is invalid it should be required to compile a pointer dereference the same way in all cases, i.e. emit the same code regardless of whether it has determined the pointer is invalid or not. Otherwise it effectively emits random code depending on how the optimizer is feeling. Non-deterministic behavior for the same operation on the same values in the same program is not very helpful behavior, whenever and to the degree it can be avoided.
arnsholt 4y ago
But that's the thing: dereferencing an invalid pointer is undefined behaviour, which means the compiler is allowed to assume it never happens; a C program executing undefined behaviour is _invalid C_. Thus, any time you dereference a pointer you also implicitly proimise the compiler that this pointer will _never_ be an invalid pointer. Same with signed arithmetic: you are telling the compiler that your arithmetic is guaranteed to never overflow. Whether this is a good or bad thing is of course a legitimate (and good!) question, but for writing C today that's how the language is specced and a reality the programmer needs to take care to avoid, just like a bunch of other things that C leaves to the programmer like remembering to clean up allocated resources when they're no longer needed.
- ClumsyPilot 4y ago
  < you are telling the compiler that your arithmetic is guaranteed to never overflow. Any time your are dealing with data from real, physical sensors or third-party APIs, this is an impossible guarantee to give - they could literslly break.
  - compiler-guy 4y ago
    of course it is possible. One must validate the input before performing arithmetic on signed integers.
    - deleted 4y ago
      [deleted]
    - imtringued 4y ago
      The compiler is allowed to assume that the data will never overflow so it is also allowed to get rid of overflow checks. Now imagine writing a complex validation routine that if it were violated would result in undefined behavior for every value that the validation rejects, a sufficiently smart compiler is allowed to simply remove your validation code, leaving you with no validation whatsoever.
      - deleted 4y ago
kps 4y ago
C did do ‘whatever this CPU does’, and then the ANSI C committee broke that (probably by accident). That's what the headline post is about: “Those origins of C were there first”.
- uecker 4y ago
  The ANSI C committee certainly did not break this. UB always meant that ANSI C simply did not define what happens in this specific case. This could mean "whatever the CPU does" or "whatever the compiler decides". What caused problems then was that compilers started to aggressively exploit UB for optimization (and customers probably asked for such optimizations) treating programs with UB as invalid. When this went to far and caused problems, compiler vendors then blamed the user and the standard. But obviously it is completely at the compiler developer's discretion what the compiler does, so customers should just push back and request changes.
  - bluGill 4y ago
    In the vast majority of cases I'm glad that my programs run fast. sure I have to do some work to avoid undefined behavior, but most code is correctly defined, and the exceptions are bad code that I need to fix anyway as even if the behavior was 100% defined to C it would still be buggy as in not do what my users want.

pavlov 4y ago

It’s up to the platform in the end. I recall IBM’s AIX used to put a zeroed-out block at address zero, so this code was guaranteed to always return 0: *(int *)NULL The reason was that it allowed their optimizing compiler to speculatively dereference a pointer before executing a surrounding if condition.

josephg 4y ago
> It’s up to the platform in the end. It’s not up to the platform. If it looks like a code path might dereference a null pointer, the compiler can and will make wild and bizarre optimizations in the surrounding code.
- jstimpfle 4y ago
  The compiler doesn't have to come to your home and actively haunt you. The behaviour is indeed up to the platform (by "platform" I understand "compiler + host system" or whatever is used to run the program).
  - josephg 4y ago
    Ah - I suppose I don’t think of the compiler as part of the platform. Most platforms I work with tend to use gcc or clang, and as a result everyone is subject to the gcc / clang optimizations.
- Karellen 4y ago
  If code dereferences a NULL pointer, the spec imposes no constraints whatsoever on what happens after that point. Yes, some compilers will make "wild and bizarre" optimisations (they are neither) based on the assumption that any pointer which is dereferenced is not NULL - because if it were, the code could have done the exact same thing anyway, because there are no constraints in that case. But compilers don't have to do that. There are no constraints. A compiler can emit code to do anything at all in the event of a NULL pointer being dereferenced, including emitting the machine code you'd naively expect - which may trap on some CPUs, or always return 0 on reads and swallow writes on other CPUs. It may also emit code that wipes your hard disk (and still counts as a conforming implementation). Or make demons fly out of your nose. A compiler vendor may choose to guarantee a specific behaviour for some invalid constructs. They are allowed to do this because there are no constraints on what they can do. It's just that most don't, because 99% of people comparing compilers care more about benchmarks than they do about what happens with invalid code. So in some ways, it is up to the compiler/platform/CPU what actually happens with code that has invalid behaviour. But as a C author, you can't assume any particular behaviour, because it might change from one CPU to another. Or from one OS to another. Or from one compiler to another. Or from one version of one compiler to another.
  - pavlov 4y ago
    This. In the case of AIX, IBM designed every component of the platform: the CPU, the operating system and the compiler. When they tell you that it’s safe to read up to 4K (or whatever) of zero bytes from NULL, they can make that promise. But obviously such code is not portable.
  - imtringued 4y ago
    But this means the vast majority of C code isn't portable since you can't know what it will do with a given platform and compiler combination and the presence of even a single line of UBI invalidates the whole program. It basically means that the whole portability argument that is supposed to be in favor of C is just wrong because every compiler and platform actually brings its own C dialect and it is just sheer coincidence that it works at all.
- amalcon 4y ago
  So it is up to the platform, since the compiler is part of the platform.

bluGill 4y ago

CPUs do different things when they know it is invalid. Often what happens is it send some sort of trap/signal to the operating system - on unix this would send a segfault and dump core. Not all invalid pointers can be detected, and not all CPUs have the required hardware (MMU) to detect any invalid pointer.

jstimpfle 4y ago
Exactly. Even though you could specify what happens mechanically on any given system, the results are kind of uncontrollable on most real computers. Whether things should be UB or implementation defined requires a good value judgement. Per se there is not a real need of UB, (assuming that the behaviour of a machine can be specified), but on the other hand there isn't much value in specifying a situation where all control is lost to the point where implementation details leak (runtime structures are overwritten etc.). Since there is (I assume?) an expectation for "implementation-defined" to have an actual definition, pedantically an implementation leak would require all implementation details to be codified, and thus set in stone.
- chalst 4y ago
  This could be indicated in a comment, perhaps using a *RFI-like notation.
- bluGill 4y ago
  It is more that while we can define everything, there are costs. I can track every allocation in a hidden table, then before following a pointer verify that the point being followed is in the table. However that is extremely expensive (don't forget that you might follow a pointer in one thread while a different thread deletes it, so there is a complex race condition logic needed here that is tricky to get right) By making things undefined we can avoid a large amount of complex code to detect a situation that shouldn't happen.
  - kaba0 4y ago
    Exactly. While it is definitely harder with a low-level language, it is absolutely doable in terms of costs with managed languages. In Java for example even data races are well-defined to a degree (and OCaml’s new multithreaded mode defines data races with even bigger guarantees)

butlerm 4y ago

I disagree that it is completely unpredictable. If it is a read operation, dereferencing an invalid pointer on every normal system will either return data with an unspecified value or will raise a low level exception that may be caught. Since a compiler cannot predict in all cases in advance whether a pointer is invalid it should be required to compile a pointer dereference the same way in all cases, i.e. emit the same code regardless of whether it has determined the pointer is invalid or not. Otherwise it effectively emits random code depending on how the optimizer is feeling. Non-deterministic behavior for the same operation on the same values in the same program is not very helpful behavior, whenever and to the degree it can be avoided.

arnsholt 4y ago

But that's the thing: dereferencing an invalid pointer is undefined behaviour, which means the compiler is allowed to assume it never happens; a C program executing undefined behaviour is _invalid C_. Thus, any time you dereference a pointer you also implicitly proimise the compiler that this pointer will _never_ be an invalid pointer. Same with signed arithmetic: you are telling the compiler that your arithmetic is guaranteed to never overflow. Whether this is a good or bad thing is of course a legitimate (and good!) question, but for writing C today that's how the language is specced and a reality the programmer needs to take care to avoid, just like a bunch of other things that C leaves to the programmer like remembering to clean up allocated resources when they're no longer needed.

ClumsyPilot 4y ago
< you are telling the compiler that your arithmetic is guaranteed to never overflow. Any time your are dealing with data from real, physical sensors or third-party APIs, this is an impossible guarantee to give - they could literslly break.
- compiler-guy 4y ago
  of course it is possible. One must validate the input before performing arithmetic on signed integers.
  - deleted 4y ago
    [deleted]
  - imtringued 4y ago
    The compiler is allowed to assume that the data will never overflow so it is also allowed to get rid of overflow checks. Now imagine writing a complex validation routine that if it were violated would result in undefined behavior for every value that the validation rejects, a sufficiently smart compiler is allowed to simply remove your validation code, leaving you with no validation whatsoever.
    - deleted 4y ago

kps 4y ago

C did do ‘whatever this CPU does’, and then the ANSI C committee broke that (probably by accident). That's what the headline post is about: “Those origins of C were there first”.

uecker 4y ago
The ANSI C committee certainly did not break this. UB always meant that ANSI C simply did not define what happens in this specific case. This could mean "whatever the CPU does" or "whatever the compiler decides". What caused problems then was that compilers started to aggressively exploit UB for optimization (and customers probably asked for such optimizations) treating programs with UB as invalid. When this went to far and caused problems, compiler vendors then blamed the user and the standard. But obviously it is completely at the compiler developer's discretion what the compiler does, so customers should just push back and request changes.
- bluGill 4y ago
  In the vast majority of cases I'm glad that my programs run fast. sure I have to do some work to avoid undefined behavior, but most code is correctly defined, and the exceptions are bad code that I need to fix anyway as even if the behavior was 100% defined to C it would still be buggy as in not do what my users want.