4 ms·

Having written a compiler for a subset of the C99 standard, I'm going to disagree here. Array bounds are not being checked on every array access not because it

Having written a compiler for a subset of the C99 standard, I'm going to disagree here. Array bounds are not being checked on every array access not because it would make compilers too complex. Correctness is being sacrificed mostly for speed or portability on future CPUs. There are examples of language features that simplify compiler writing, however. For example, type promotion from char to int is a feature that reduces the number of cases one would have to deal with when implementing the type system in a compiler, but it's there because it doesn't sacrifice neither performance nor portability.

eru 2y ago
> Array bounds are not being checked on every array access not because it would make compilers too complex. That might be true, but you could still specify something slightly less exploit heavy than 'undefined behaviour'. Eg you could make out-of-bounds access into implementation defined behaviour.
- jstimpfle 2y ago
  There is no way to predict what will happen if your program is accessing random memory at runtime, especially if it's a write access. To specify what would happen on a write to random memory would fill books that basically lay out most of the internals of the compiler and also the host OS.
  - eru 2y ago
    You could at least define it not to travel backwards in time. Undefined behaviour in C infects the whole execution, not just what comes afterwards.
    - jstimpfle 2y ago
      Can you clarify what you mean? Is it defined to "travel backwards in time"? I suspect not.
      - tialaramex 2y ago
        Is the situation here that you're unaware of time travel UB optimisations in C and C++? https://devblogs.microsoft.com/oldnewthing/20140627-00/?p=633 https://devblogs.microsoft.com/oldnewthing/20140627-00/?p=63...
        eru 2y ago
        Thanks for digging out the link, so I don't have to!
        jstimpfle 2y ago
        No I'm aware of examples like these, I just asked for clarification what they mean by "define it to not travel backwards in time". To me this sounds nonsensical. I'm not deep into compiler construction, but to me these examples seem just like a logical consequence of what UB is -- it's a (runtime) situation that the compiler is not required to take into consideration. It can opt to not emit code to treat these situations at all, etc -- effectively assuming they don't happen. The point is to allow the compiler to blindly dereference a pointer even when it can't prove that the pointer is valid. Or to allow it to implement arithmetic on a register of bigger size (assuming the computation doesn't overflow), etc. Now, depending on how optimizers are written, the compiler may end up inadvertently detecting UB and optimizing out entire branches of code, just by virtue of how the optimizer works internally. You can bet that the compiler doesn't think much of e.g. what is earlier or later in time, when doing optimizing transformations. Of course a "miscompilation" (of code that is buggy in the first place) is an unfortunate situation and a diagnostic would be better. Compilers should improve (and they probably do). Compilers should be friendly and give unsurprising results and good diagnostics as much as possible. But to "define it to not travel backwards in time" right in the spec would probably be very hard and might negate the point of UB in the first place. It would require doing the work of compiler authors, which are the people responsible to figure out how to make _their_ compiler solid and ergonomic while also offering the optimizations people want. This is already a hard task for the authors of a specific compiler, and probably not something that you can easily define in a language spec! And for balance, I've never consciously had to deal with a miscompilation like this, and I write C and C++ in professional capacity almost every day. Instead, most bugs I deal with are of the most trivial kind, you hit a segmentation fault, quickly navigate to the piece of code where there is still some initialization stuff missing, and fill it in. Or there is a logic bug that is entirely unrelated to UB, those are in fact, typically, more difficult to find and fix. Note that while I'm by no means an exceptional programmer (not that I think you think that of me). I simply want to solve a problem. And while developping I introduce bugs and even UB sometimes (even though it seems to be quite rare if I can trust sanitizers). I'm actually sophisticated enough to develop in debug mode, with most optimizations turned off, and this might be one explanation why I've never hit an annoying situation like this. To me, these stories are fascinating, and I think they should be taken seriously. But their effect on online forums is mostly to heat up discussions.
pjmlp 2y ago
Yet every other systems programming language never had any issues with enabling bounds checks, their only failure was not having a free beer OS to come along for the ride.
- rfoo 2y ago
  I have to constantly fight against rustc and LLVM to convince it to eliminate bounds check in hot loop when I'm writing high performance Rust and it's a cursed experience I hope nobody has to. Other replies in this thread mentioned they have similar problems writing Go, I don't know to what extent it applies, in my limited experience working on Go codebases I never see such issues.
  - pjmlp 2y ago
    I am writing code since 1986, in my experience most of those cases are mostly a I feel good kind of thing, and have contributed zero to the project delivery acceptance criteria. When it does in fact cause an issue with project delivery acceptance testing, the issue is solved by making use of profiling tools, and cirugically disable bounds checking, which most systems languages since the dawn of time also support.
    - rfoo 2y ago
      Well, I would have no idea if a bounds check is eliminated at all (and who wants to care??), if it does not show up in profiling results. Unfortunately for what I do I had to do this a lot. I guess that's also why I'm not seeing it in Go, never tried to write a query engine in Go.
      - pjmlp 2y ago
        Well, does something like CERN TDAQ/HLT count? The algorithms, networking protocols and thread scheduling are much more relevant, that the bounds checking done in the C++ data structures. As for writing query engines with bounds checking languages, there are several examples.

eru 2y ago

> Array bounds are not being checked on every array access not because it would make compilers too complex. That might be true, but you could still specify something slightly less exploit heavy than 'undefined behaviour'. Eg you could make out-of-bounds access into implementation defined behaviour.

jstimpfle 2y ago
There is no way to predict what will happen if your program is accessing random memory at runtime, especially if it's a write access. To specify what would happen on a write to random memory would fill books that basically lay out most of the internals of the compiler and also the host OS.
- eru 2y ago
  You could at least define it not to travel backwards in time. Undefined behaviour in C infects the whole execution, not just what comes afterwards.
  - jstimpfle 2y ago
    Can you clarify what you mean? Is it defined to "travel backwards in time"? I suspect not.
    - tialaramex 2y ago
      Is the situation here that you're unaware of time travel UB optimisations in C and C++? https://devblogs.microsoft.com/oldnewthing/20140627-00/?p=633 https://devblogs.microsoft.com/oldnewthing/20140627-00/?p=63...
      - eru 2y ago
        Thanks for digging out the link, so I don't have to!
      - jstimpfle 2y ago
        No I'm aware of examples like these, I just asked for clarification what they mean by "define it to not travel backwards in time". To me this sounds nonsensical. I'm not deep into compiler construction, but to me these examples seem just like a logical consequence of what UB is -- it's a (runtime) situation that the compiler is not required to take into consideration. It can opt to not emit code to treat these situations at all, etc -- effectively assuming they don't happen. The point is to allow the compiler to blindly dereference a pointer even when it can't prove that the pointer is valid. Or to allow it to implement arithmetic on a register of bigger size (assuming the computation doesn't overflow), etc. Now, depending on how optimizers are written, the compiler may end up inadvertently detecting UB and optimizing out entire branches of code, just by virtue of how the optimizer works internally. You can bet that the compiler doesn't think much of e.g. what is earlier or later in time, when doing optimizing transformations. Of course a "miscompilation" (of code that is buggy in the first place) is an unfortunate situation and a diagnostic would be better. Compilers should improve (and they probably do). Compilers should be friendly and give unsurprising results and good diagnostics as much as possible. But to "define it to not travel backwards in time" right in the spec would probably be very hard and might negate the point of UB in the first place. It would require doing the work of compiler authors, which are the people responsible to figure out how to make _their_ compiler solid and ergonomic while also offering the optimizations people want. This is already a hard task for the authors of a specific compiler, and probably not something that you can easily define in a language spec! And for balance, I've never consciously had to deal with a miscompilation like this, and I write C and C++ in professional capacity almost every day. Instead, most bugs I deal with are of the most trivial kind, you hit a segmentation fault, quickly navigate to the piece of code where there is still some initialization stuff missing, and fill it in. Or there is a logic bug that is entirely unrelated to UB, those are in fact, typically, more difficult to find and fix. Note that while I'm by no means an exceptional programmer (not that I think you think that of me). I simply want to solve a problem. And while developping I introduce bugs and even UB sometimes (even though it seems to be quite rare if I can trust sanitizers). I'm actually sophisticated enough to develop in debug mode, with most optimizations turned off, and this might be one explanation why I've never hit an annoying situation like this. To me, these stories are fascinating, and I think they should be taken seriously. But their effect on online forums is mostly to heat up discussions.

pjmlp 2y ago

Yet every other systems programming language never had any issues with enabling bounds checks, their only failure was not having a free beer OS to come along for the ride.

rfoo 2y ago
I have to constantly fight against rustc and LLVM to convince it to eliminate bounds check in hot loop when I'm writing high performance Rust and it's a cursed experience I hope nobody has to. Other replies in this thread mentioned they have similar problems writing Go, I don't know to what extent it applies, in my limited experience working on Go codebases I never see such issues.
- pjmlp 2y ago
  I am writing code since 1986, in my experience most of those cases are mostly a I feel good kind of thing, and have contributed zero to the project delivery acceptance criteria. When it does in fact cause an issue with project delivery acceptance testing, the issue is solved by making use of profiling tools, and cirugically disable bounds checking, which most systems languages since the dawn of time also support.
  - rfoo 2y ago
    Well, I would have no idea if a bounds check is eliminated at all (and who wants to care??), if it does not show up in profiling results. Unfortunately for what I do I had to do this a lot. I guess that's also why I'm not seeing it in Go, never tried to write a query engine in Go.
    - pjmlp 2y ago
      Well, does something like CERN TDAQ/HLT count? The algorithms, networking protocols and thread scheduling are much more relevant, that the bounds checking done in the C++ data structures. As for writing query engines with bounds checking languages, there are several examples.