18 ms·

One Word Broke C

147 points by quelsolaar 7y ago

lmm 7y ago
Where is the constituency that would want this "sensible C"? People who care about reasonable behaviour have already moved on to better languages, a la https://www.lesswrong.com/posts/ZQG9cwKbct2LtmL3p/evaporative-cooling-of-group-beliefs https://www.lesswrong.com/posts/ZQG9cwKbct2LtmL3p/evaporativ... ; the people still using C are those who want "maximum performance" and don't think undefined behaviour is a problem, and so C compiler writers have (understandably) gone ever further in their exploitation of undefined behaviour to get that "maximum performance". I don't think this can be reversed, and I don't think we should necessarily want to reverse it. We have better alternatives now. Let the C people do their thing, and get on with your life in a language that ensures that programs, even "incorrect" ones, have reasonable behaviour.
- lone_haxx0r 7y ago
  What are these "better languages" you speak of? Because I've been looking for a good C replacement but to no avail. C++, D, Nim, Go, Rust, Zig, Odin, etc; I've looked into them but none convinced me.
  - admax88q 7y ago
    Stop looking for another C. Move on to something really different, something dynamic like JS, Clojure, Python, or Smalltalk
    - voldacar 7y ago
      Yeah what C programmers really want is dynamic languages
      - fulafel 7y ago
        You can also move to a static language of course. With C's mostly absent type system, Rust or Haskell might take some getting used to. But Go could be an easy transition, or Java.
      - admax88q 7y ago
        People still programming in C want C, its why they're still doing it. But you can't claim there are "no better languages" when all you're really looking for is C.
  - worthless-trash 7y ago
    If none of the above convinced you, you'd need to specify why each of them didn't convince you (to do what) to get a qualified answer here. Every design choice has a repercussion, specify what repercusison that you would find acceptable and then your ideal language can be defined.
  - lmm 7y ago
    What were your issues with those languages? Several of them sound like reasonable options to me. For a mature, general-purpose language my default suggestion is OCaml. If you really absolutely can't make garbage collection work for your case (something I've never seen happen to anyone who actually tried) then your choices are more limited (and I'd probably favour Rust, despite its relative immaturity). There are any number of good languages out there, and I could happily go into the details of which I think offers the best combination of tradeoffs (Scala). But the bigger picture is that a memory-safe language with the ML featureset (in particular first-class functions, parametric polymorphism, type inference, and sum types) should be the minimum baseline these days, and represents a substantial step up from C (at the most basic and pervasive level, being able to do error handling with a result type vastly improves your defect rate). Within that category you have plenty of reasonable choices offering their own particular selling points.
    - indymike 7y ago
      For me: libraries not available or are just wrappers on a C library anyway, inflexible memory management, bad support for OS features, etc... A lot of it is just maturity. C is ancient.
- adrianN 7y ago
  I'm pretty sure that a large part of the people working on legacy code bases would like to have a safer language. You don't move on to a better language if that means rewriting your whole project.
  - lmm 7y ago
    "Sensible C" would have to be a distinct language from C, at least as the latter is currently implemented, so it would still represent a change of languages for legacy codebases. It could probably interop with the C ABI, but we already have other languages that can do that; it's not at all clear that converting your codebase piecemeal to "sensible C" would be substantially easier than doing the same with, say, Rust.
    - naniwaduni 7y ago
      It would also have source-level interop with C _APIs_, for which the only other language we have is C++, and which for that purpose has the same issues as C.
  - pjmlp 7y ago
    I pretty much doubt it, given that ISO C WG doesn't care one second about making C safer. It is Oracle with SPARC ADI, Google/ARM/Apple with ARM MTE, Microsoft with Checked C, AT&T with Cyclone that actually bother to change the status quo.
    - saagarjha 7y ago
      Is anyone actually shipping with memory tagging yet?
      - pjmlp 7y ago
        Solaris on SPARC, iOS and it is a requirement for Android 11 ARM based devices.
        7y ago
- dooglius 7y ago
  raises hand The sort of optimizations that compilers abuse UB for aren't that useful, actually, and they don't actually require a compiler anyway; it's more about compiler writers overfitting for benchmarks.
- justinmeiners 7y ago
  I regularly use other "modern" languages and still enjoy using C. Am I cult member?
  - lmm 7y ago
    Cultishness is a spectrum, not a binary. C has always had a certain proportion of adherents who believed that "performance" (of code that implements microbenchmarks with unlimited hours of manual tuning) is the most important aspect of a general-purpose programming language, and that the problems of undefined behaviour are insignificant. I would say that both those viewpoints are crazy, but they are increasingly becoming the mainstream in the C community, via the mechanism described in the link: more moderate people are driven out, rendering the community more extreme, which then drives more of the moderate folk out, and so on.
    - courtf 7y ago
      I see something similar happening with functional programming. Beyond measurable improvements (ie, in error rate, etc) there is often a mindset where if a language does not implement the latest syntactic patterns, then the language cannot be considered truly functional. HN seems a breeding ground for these sentiments, and adherents can be quite self-righteous. Out in the untamed wilds of the broader industry (and the further you move away from the Stanford sphere) I have found that fanaticism, of all sorts, tends to taper off, to be replaced mostly with the mundane tasks of churning out software for money. Of course, no matter where you hail from, the influence of C is inescapable. Yet, provincial priests are mostly concerned with budgetary constraints and deadlines. If switching to Scala will make hiring 10% easier, then by all means.
      - lmm 7y ago
        One "fun" thing about working in Scala is that you get attacked from all sides; there are people who say that the language is too weird or too purist, and then there are people who say that the problem is that it makes it too easy to have mutable variables or imperative code, or the fact that it allows you to do traditional OO inheritance. FWIW, my experience is that when you introduce a bit of mutable state or hidden side effects, you usually regret it later. I probably come across as fanatical at times, but that fanaticism is coming purely from painful experience; these academic, theoretical concerns become a lot less academic when you've had to deal with production bugs that they could have helped you avoid. I find it impossible to imagine the same being true for the C devotees. I've seen fast and slow programs in all sorts of languages. I've never seen a performance problem in anything other than a scripting language that wasn't either solved with a little bit of profiling, or just fundamentally impossible in any language. I've seen a C++ program rewritten in Haskell for a 5x performance speedup. And I've seen so many segfaults. FWIW, I've found it pretty easy to escape C. In JVM-land you really don't have to deal with it, at least 99.9% of the time; I've hit like two JVM bugs in my entire career. Scala is by no means the easiest to hire for, but it's a great language for getting on with solving business problems - whether you want to do that by churning out reams of code, or by abstracting out the mundane parts and writing only the parts that are specific to the problem. It makes the easy things easy and the hard things possible; very rarely have I felt that the language was limiting me or stopping me from doing what I wanted. There's plenty wrong with it, but it's the best choice I've found.
dejj 7y ago
Quite the converse. To me, “permissible” sounds as capricious as does “possible”. I read the article on “mass amateurization” before and am thinking how words must be sounding differently to me as a non-native English speaker and computer scientist. Maybe it had been best to insert “No running with scissors!” to that spec.
dependenttypes 7y ago
You might find this relevant https://news.ycombinator.com/item?id=10772841 https://news.ycombinator.com/item?id=10772841 What you want is not another language but a compiler that has a well defined behaviour and is not willing to change it between versions when UB in the C standard is triggered.
_kst_ 7y ago
The author claims that: > In C89, undefined behavior is interpreted as, “The C standard doesn’t have requirements for the behavior, so you must define what the behavior is in your implementation, and there are a few permissible options”. That's a serious misinterpretation. C89 says that "Permissible undefined behavior" includes "ignoring the situation completely with unpredictable results". There is absolutely no requirement to document what those "unpredictable results" might be. The standard joke is that one possible consequence of undefined behavior is making demons fly out of your nose -- not because that's actually possible, but because actually making demons fly out of your nose would not violate the standard. That's equally true in C89 and all later versions of C. The change in C99 from "Permissible" to "Possible" made no difference. The phrase "imposes no requirements" has always meant exactly that. (And if you think the change from "Permissible" to "Posisble" is semantically significant, then it would have been a recognition of what was already accepted.) The idea of "nasal demons" goes back to 1992. http://catb.org/jargon/html/N/nasal-demons.html http://catb.org/jargon/html/N/nasal-demons.html
- mjd 7y ago
  I came here to say something similar. The author says: > In C89, undefined behavior is interpreted as, “The C standard doesn’t have requirements for the behavior, so you must define what the behavior is in your implementation, and there are a few permissible options”. This is wrong. That is the meaning of what C89 calls “implementation-defined” behavior. The implementation gets a choice; the standard may lay out what the choices are, and in any event the choice must be documented. But with C89's “undefined” behavior, all bets are off. Numerous examples are documented of cases where real compilers were forced to behave in utterly bizarre ways because of real hardware limitations.
  - eru 7y ago
    The C standard is quite liberal in its use of undefined behaviour. Some cases are due to conflicting existing implementation that predated the standard. (Though implementation defined might make more sense?) Some are for performance reasons. And some are utterly bizarre, like some things that should be syntax errors instead being declared undefined. If memory serves right, that includes eg not closing your string literals. (Thankfully, all implementations I know give you an error message.)
mjevans 7y ago
Any instance of undefined behavior should result in a Warning, if not an Error. Similarly, 'dead code paths' should also result in a Warning, if not an Error. (Possibly with a way of turning that off for specific functions.) I also fully agree with the author's statement about what a good compiler should be doing.
- millstone 7y ago
  > Any instance of undefined behavior should result in a Warning, if not an Error How would you implement this? For example, use-after-free is undefined, and difficult to detect.
  - vnorilo 7y ago
    Warning when optimization passes take advantage of UB is a reasonable desire. The compilers are getting increasingly good at this, but I believe it would be far too noisy to blanket enable all of that.
    - mjevans 7y ago
      The point is to push the info to the programmer so that the code can be re-written as only defined behavior. Yes this would eliminate the 'gains' of abusing UB, but it would actually fix the errors of abusing UB.
      - vnorilo 7y ago
        I guess signed indvars would be a prime example. Compilers famously ignore overflow of signed integers when computing loop trip counts. Would it be better to only ever use unsigned integers for indvars and loop limits? Maybe. But that would reach deep into the types in stdlib for example. And then there's https://software.intel.com/en-us/forums/intel-c-compiler/topic/698664 https://software.intel.com/en-us/forums/intel-c-compiler/top... So, all in all, not simple. Not that that invalidates your point: I just wanted to add some nuance.
      - marcan_42 7y ago
nneonneo 7y ago
My "favourite" recent compiler bug comes from Rust. Compile the following code in Release mode: fn main() { (|| loop {})() } This defines an anonymous function containing an infinite loop, and calls it. Any sane compiler would make this just loop forever, right? $ cargo run --release test Illegal instruction (core dumped) Yeah, that's right. In Rust, an ostensibly safe language, Clang's overeager optimizer chooses to emit an illegal instruction in lieu of an infinite loop. (Try it here: https://play.rust-lang.org/?version=stable&mode=release&edition=2018&gist=4233cbe4bf524b1154b2e7b560d58610 https://play.rust-lang.org/?version=stable&mode=release&edit...) This particular bug actually comes from C++. In C++, threads may be assumed to always make forward progress, i.e. side-effect-free infinite loops are UB. However, this is not true in C, and it's definitely not true in Rust, yet Clang makes this assumption anyway. It gets a lot worse; once Clang sees an infinite loop and decides it's UB, it gets to make all sorts of thoroughly invalid and silly "optimizations"; see this SO question for some examples: https://stackoverflow.com/questions/59925618/how-do-i-make-an-infinite-empty-loop-that-wont-be-optimized-away https://stackoverflow.com/questions/59925618/how-do-i-make-a...
- lmm 7y ago
  > This particular bug actually comes from C++. In C++, threads may be assumed to always make forward progress, i.e. side-effect-free infinite loops are UB. However, this is not true in C That's very much disputed. The C standard requires programs' behaviour to be as-if executed on the C abstract machine when they terminate. It's not at all clear whether the standard imposes any requirements on programs that would not terminate when executed on the C abstract machine.
  - _kst_ 7y ago
    C11 does have a special case rule for infinite loops. Section 6.8.5 paragraph 6: "An iteration statement whose controlling expression is not a constant expression, that performs no input/output operations, does not access volatile objects, and performs no synchronization or atomic operations in its body, controlling expression, or (in the case of a for statement) its expression-3, may be assumed by the implementation to terminate." (I'm personally not happy that this is stated in terms of that the implementation may assume rather than what a program may do.) Note that this does not apply to infinite loops with constant conditions. For example, given `while (1) { /* ... */ }`, the above doesn't apply and the implementation may not assume the loop will terminate.
bigcheesegs 7y ago
Yet again someone ignoring the line: > for which the Standard imposes no requirements. C89 had no requirements, C99 clarified that the line that followed was non-normative. The entire premise of this article is flawed.
- quelsolaar 7y ago
  The premise is that just because the standard doesn't impose requirements, it doesn't mean that other known factors like ABI, Architecture, and other platform specifics don't matter.
- TwoBit 7y ago
  The point of the article is that the author wants compilers to tell users when they do unexpected things. I don't see how your point is relevant to that.
  - bigcheesegs 7y ago
    That may be the point, but it's not the premise. Even the title is based on this flawed idea. If the author wants to make a point, they can do so without trying to drive it with incorrect ideas.
  - clarry 7y ago
    And the problem is that users ask the compiler to optimize, and then they complain because it optimizes. See, optimization can do a lot of unexpected things, but it is very difficult in general for the compiler to tell whether a certain application of (say) range analysis to eliminate dead code in a bunch of inlined calls is going to be unexpected. That's exactly the kind of optimization I want a compiler to do; I can write small functions that are as generic as possible yet cover all the edge cases, and then the compiler can find out which edge cases cannot apply in this particular situation. If the compiler told the user every time it did something, you'd never finish reading the output. Might as well ask the compiler not to optimize at that point.
  - gpderetta 7y ago
    The compiler doesn't know it is doing unexpected things. The compiler is doing what's expected given the axioms it has to work with. It would be nice if the compiler added assertions in place to validate that the axioms hold, but that would often be very expensive. Recently I have been using the sanitizers extensively and has been a huge improvement, but of course they are too expensive to leave on in prod. Would be nice if a cheaper, although less exhaustive version were available.
userbinator 7y ago
This topic comes up semi-regularly here on HN but I think the underlying issue is that standards do not exist in a void, yet a certain group of people seem to think that they do --- the fact that a standard "imposes no requirements" should really be taken to mean "think about what really makes sense", not "do whatever the fuck you want". Hence the "but we still complies with the standard!" defense from compiler authors when faced with perplexing results of UB is totally ignoring the reality and practicality of what a compiler is for. Note that the standard writers have even tried to give that hint, with the "behaving during translation or program execution in a documented manner characteristic of the environment" phrasing which is precisely what programmers are usually expecting, but it seems few actually took the hint.
- mpweiher 7y ago
  IIRC the C standard explicitly states that conformance to the standard is insufficient for having a compiler that is fit for purpose.
  - pjmlp 7y ago
    The best example are embedded platforms, full with extensions that probably will never be part of ISO C.
- giomasce 7y ago
  Different people have different ideas on what "makes sense". The point of a standard is to not have to use common sense, which is usually not common al all. So there is no point is saying that when the standard does not say anything you should use common sense. If the standard says nothing, you should just avoid relying of that thing. The compiler is not responsible for the quality of the source code. The programmer is. The compiler is responsible for the quality of the machine code, assuming that the source code is correct. It is good to have tools to check the quality of source code, but they are a different thing from compilers. Personally, I am happy that the compiler optimizes the code for me when I write correct C/C++ programs. If I make a mistake and inadvertently do UB, I take the responsibility, without shifting it on the innocent compiler.
_martamoreno_ 7y ago
Really bad example... Memset to zero members of a structure this way is actually something that should trigger a compiler error, because it makes no sense. I have limited understanding of C these days, but I consider that a good thing, because it allows me to look with fresh eyes on the crap I wrote back in the days. And this is one of them. It's nonsensical and similar to the general linux sentiment of naming your variable like random garbage. It's brainfuck. The only thing I can say with some distance is that exploiting undefined behavior in pretty much all cases should yield a compiler error.
- saagarjha 7y ago
  > I have limited understanding of C these days, but I consider that a good thing I wouldn't.
tumult 7y ago
This is shown as an example of illegal C: struct{ char x char y; }a; memset(&a, 0, sizeof(char) * 2); But the only thing illegal about it is that there's no semicolon after 'char x'. Writing to (theoretical) padding in this way is fine. Though, you might want to check the manual to see if this could generate a trap representation on your platform. (This won't, on any platform I'm aware of.) Also, the sizeof operator always returns 1 for char and unsigned char (C99: 6.5.3.4), so there's no reason to do sizeof(char).
- hackcasual 7y ago
  Agreed, that's not undefined behavior. It's just not guaranteed to 0 out y
  - tumult 7y ago
    It actually would be guaranteed to 0 it if it were instead unsigned chars. For plain char, the implementation has to specify if it is signed, and if it is, what kind of padding it has. If it has none, it would also be guaranteed. Therefore, you can just check the compiler manual to see if this guarantees zeroing. (None of the mainstream compilers on mainstream platforms will have padding for signed char, either.) (C99: §6.2.5 and §6.2.6.2) (If you know a platform where this is not true, I'd be interested in hearing about it!) Edit: Oh, also, an implementation is allowed to have different padding in struct fields than the padding of the type itself. But it has to define this, so you would be able to look it up in the manual to see if it's different. (§6.7.2.1: "Each non-bit-field member of a structure or union object is aligned in an implementation-defined manner appropriate to its type.")
    - quelsolaar 7y ago
      This is my point, No platform I know of puts padding between bytes, so the behavior should be clear, but since a platform CAN put padding between bytes, and writing to said padding is UB, some compiler designers thinks this a license to do what ever.
      - 7y ago
aw1621107 7y ago
I'm not really convinced the author is correct in claiming that a one-word change opened the floodgates to optimizations on undefined behavior. In particular I think: > Careful reading will reveal that the word “Permissible” has been exchanged to “Possible”. In my opinion this change has lead C to go in a very problematic direction. is a red herring. In my opinion, the actual problematic phrase is this: > ignoring the situation completely with unpredictable results which didn't change between C89 and C99. It all comes down to what "ignoring the situation" should mean. Compiler vendors appear to interpret this to mean "ignore situations that invoke undefined behavior". Programmers who dislike optimizations based on undefined behavior appear to interpret this to mean "ignore the violation that leads to undefined behavior and treat it like conforming code". Who's right? It's ambiguous. > C compilers have taken the concept of undefined behavior even further by doing the mental acrobatics of thinking that “If undefined behavior happens, I can do want I want, So therefor I can assume that it will never happen”. I think this description is a bit uncharitable. I think a better description might be "If undefined behavior happens, there are no constraints on program behavior. Thus, if I assume that undefined behavior never happens, optimize the program based on that assumption, and that assumption is violated, the resulting program behavior is still OK because the standard imposes no rules on what the program must do when undefined behavior is invoked." > Lets take a look at signed integer overflow: Wasn't this useful for loop counters? I think this gist had more info: https://gist.github.com/rygorous/e0f055bfb74e3d5f0af20690759de5a7 https://gist.github.com/rygorous/e0f055bfb74e3d5f0af20690759... In addition, this bit: > In C89, undefined behavior is interpreted as, “The C standard doesn’t have requirements for the behavior, so you must define what the behavior is in your implementation, and there are a few permissible options”. is implementation-defined behavior, not undefined behavior. As the C89 standard notes, defining the implementation's behavior is only one of the things an implementation may do, not something it must do.
- a1369209993 7y ago
  > is implementation-defined behavior, not undefined behavior. They're (confusingly) misusing the word "define" there. Implementation-defined behaviour is required to be documented, but every behaviour is something that "you must define" in the sense that you have written down some (possibly implicit/emergent) definition of it as part of the compiler's source code. Ie, it's not "the standard requires you to define this", it's "you are not capable of not defining this, and the standard requires(C89)/doesn't require(C99+) you to pick from the following options".
deleted 7y ago
[deleted]
deleted 7y ago
[deleted]
mpweiher 7y ago
What a great idea! https://blog.metaobject.com/2018/07/a-one-word-change-to-c-standard-to-make.html https://blog.metaobject.com/2018/07/a-one-word-change-to-c-s... The comment section also has a lot of the same justifications for the current interpretations that I see here, with fairly thorough debunkings. Changing that one word is probably not quite enough, at the very least you need to also not make the remaining text a note or also change the status of notes back again. Then there is the conflict between "imposes no requirements" and "here is the range of permissible actions". These cannot both be true, and the current dogma is to resolve the conflict by pretending the second part dosen't exist (with, admittedly, some justification). But then why on earth is it in the standard at all?
- saagarjha 7y ago
  "Ranges from" does not necessarily enumerate the only possible choices: https://english.stackexchange.com/questions/148428/using-three-examples-with-range-from https://english.stackexchange.com/questions/148428/using-thr...
  - mpweiher 7y ago
    Never said it did... ?
    - saagarjha 7y ago
      Specifically, it would allow for more than just the three options you mention there.
      - naniwaduni 7y ago
        It allows additional options _in between_ those listed. I'm skeptical that such an ill-defined rationale-centric change would hold any purchase with modern compiler-writers (it is certainly arguable what points lie "between" qualitatively-described behaviors), but in general a range does not usually include literally all possibilities.
jancsika 7y ago
> If the compiler thinks that writing to NULL, is undefined, it can therefore assume that since you are writing to p, p can’t be NULL. If p can’t be NULL, the entire if statement can be removed and after optimization the code looks like this: Is that correct? I thought Linus' rant was about NULL checks that came after a dereference had already happened. In that case it at least makes sense that the compiler would assume the NULL check would be superfluous. But how could branching to spit out an error and exit before the dereference ever get optimized out? I don't see any undefined behavior in the author's example upon which an optimization could trigger. (Or if it could trigger, it ought to trigger on array bounds checking and many other situations where removing the code would clearly cause bugs.)
- quelsolaar 7y ago
  You get this bug because the compiler doesn't know that the function will not return and therefor assumes that the de-reference will happen even if the value is NULL. Some compilers have a (non standard) keyword to indicate that a function will not return. adding an "else" will "fix" the issue.
  - aw1621107 7y ago
    This is a corner of C I'm not familiar with, but does the standard say anything about functions that never return? If it doesn't say that implementations may assume functions return then that sounds more like a compiler bug than taking advantage of UB.
    - oconnor663 7y ago
      This seems contrary to how (I believe) the compiler works with atomics. If I call some opaque function foo(), the compiler has to assume that foo() could perform sequentially consistent atomic operations, and it cannot move other reads or writes across that function call. Why isn't it also required to assume that a function could terminate the program or longjump out?
      - 7y ago
olliej 7y ago
There are many people arguing specific semantics of the authors arguments, but I believe the core problem is C and C++ both dramatically overusing “undefined” vs “unspecified”. The difference is huge. Signed integer overflow is (per spec) undefined behaviour, so an obvious bounds check if UB and so can be removed. If it were unspecified the compiler would be required to be at least self-consistent. Eg it couldn’t do 2s complement in one place, but then treat arithmetic as not being 2s complement elsewhere (the overflow checks). E.g if the compiler emits code where MAX_INT+1 is MIN_INT, then the compiler can’t also pretend that that doesn’t happen. Undefined should be reserved solely for things that cannot have a specified behavior (UaF, OoB memory, IO weirdness, etc).
- quelsolaar 7y ago
  Some compilers even make it hard to test for overflow: if(a + 1 < a) printf("Overflow error!"); else a++; gets converted to: a++; because the compiler thinks that a + 1 can never be smaller then a since it doesn't have to consider signed overflow.
  - CJefferson 7y ago
    While that looks stupid in isolation (and I agree it's annoyingly hard to check for overflow, although gcc and clang have special builtins nowadays to do it), it turns out there are important reasons for that optimisation. In general, knowing that 'a+1' is '1 larger than a' allows for lots of optimisations, when writing to an array in order we can vectorise, do things in bigger chunks, all sorts of useful and important optimsiations. If every time those were used the compiler had to check for overflow, it would seriously effect performance.
    - pjmlp 7y ago
      And that is how many CVE entries end up being created, because performance above anything else is what matters.
      - saagarjha 7y ago
brianpgordon 7y ago
> The point of a compiler is not to try to show off that who ever implemented it knows more loop holes in the C standard, then the user, but to help the programmer write a program that does that the programmer wants. The author makes it sound like the people working on optimizing compilers are deliberately seeking out these weird corner cases and selecting some random surprising behavior for them out of a hat, gleefully imagining how confusing it will be for end users. That's not how it works. Optimizers can be extraordinarily complex and need to maximize this ill-defined thing called "performance" in a highly multi-dimensional solution space. They ping-pong around inside this space constrained only by the specific requirements of the standard, and it's not surprising that some of the techniques used would produce some counter-intuitive results if the programmer is breaking the rules and relying on undefined behavior. It's kind of like if you trained a neural network to classify cat and dog pictures, and then you showed it a picture of a fire truck and expected it to give you a useful result. The idea of a new version of the C standard that defines some of the most surprising undefined behavior is an interesting one though, and I'd be interested to see how much that really impacts the ability of the optimizer to do its work.
- SAI_Peregrinus 7y ago
  I'd love it if the C standard just removed undefined behavior, replaced explicit instances ("the behavior is undefined" to "the behavior is implementation defined") and put in a blanket "Any behavior not specified by this standard is implementation defined". Keep the rest the same, just document the footguns. Implementation defined is exactly as powerful as undefined, it just makes the compiler writer describe what will happen.
dirtydroog 7y ago
It's C++, but UB allows the compiler to call a function the programmer never called https://kristerw.blogspot.com/2017/09/why-undefined-behavior-may-call-never.html https://kristerw.blogspot.com/2017/09/why-undefined-behavior... I'm not sure how any compiler writer can condone this behaviour.
- aw1621107 7y ago
  That particular behavior (and some other examples of pathological optimizer behavior) is probably an emergent property of how several optimizations combine as opposed to being a specifically targeted optimization. I can imagine something the optimizer doing something like: 1. Determine possible function call targets 2. Eliminate invalid targets from call set 3. If only one target remains, replace indirect call with direct call That might be useful for something like devirtualization, but also could lead to the behavior exhibited in that blog post. Indeed, the second blog post [0] shows that if there are two possible call targets, LLVM could generate an indirect call like what you may expect. [0]: https://kristerw.blogspot.com/2017/09/follow-up-on-why-undefined-behavior-may.html https://kristerw.blogspot.com/2017/09/follow-up-on-why-undef...
- AshamedCaptain 7y ago
  That is not "never used" as the programmer argues, and I find it misleading that the title of the blogpost tries to sell it as "a never used function is getting called". The NeverCalled function has external visibility so it could be eventually called by another CU, and it definitely references (or takes the address of) the supposedly unused "EraseAll" function. In order to be a legal program, some other CU has to call the NeverCalled at some point before main(), and the compiler is just assuming that.
scoutt 7y ago
The article kind of lost me at the first example. struct{ char x char y; }a; memset(&a, 0, sizeof(char) * 2); What does the compiler know about memset? Why it should? It can be a function called heyLetsFillTheMemoryOverHere(&a, wrongSize); and still it's the programmer's fault. In fact, it's a good thing that C doesn't care about memset, and doesn't get in between me and my code. And that's I love about C. It minds its own business and does what I tell it to do. > If I use my compiler to compile a program on my machine, the compiler knows that I’m compiling it to the x64 instruction set You can be cross-compiling. I do it every day. The C language doesn't need to know about every little platform out there. If a compiler wants to go the extra mile and help about it, that's a compiler-thing and not a C-language-thing. If there is a MCU that uses 3 bytes words, why should C be aware of it, 40 years before? Don't blame the specifications, blame the compiler if you want. Edit: removed a couple of personal opinions.
- saagarjha 7y ago
  > What does the compiler know about memset? Why it should? It can be a function called heyLetsFillTheMemoryOverHere(&a, wrongSize); and still it's the programmer's fault. Because memset is defined by the ISO C standard to have specific semantics, which the compiler can leverage to make your code faster.
  - scoutt 7y ago
    > void memset(void s, int c, size_t n); > The memset() function shall copy c (converted to an unsigned char) into each of the first n bytes of the object pointed to by s. The standard just declares an interface, to be added to a C library (maybe?). It's not the job of the compiler to check for programmer's faults. The compiler doesn't care and it shouldn't.
    - deleted 7y ago
      [deleted]
    - gpderetta 7y ago
      The standard defines the behaviour, not just the signature. Memset has been treated as a builtin since forever.
OliverJones 7y ago
rms was all over this "undefined" stuff. Early gcc versions actually tried to launch the games Rogue or Nethack upon detecting #pragma directives. Old IBM "theory of operation" books used the word "unpredictable" for this kind of thing. "unpredictable" should frighten programmers. It's worse than "random," because random results have a chance of being caught during unit or system test.
lonelappde 7y ago
Why is a completely wrong blog post based on a misunderstanding of a 30 year old standard on the front page with 110 votes?
- saagarjha 7y ago
  It's a perennial Hacker News topic.
- throwaway2048 7y ago
  because a lot of users here think that a compiler somehow detects undefined behavior, then emits wacky code just to spite them.
imtringued 7y ago
I often see people still choose C even in 2020 for some projects. The usual arguments are that C has stuck with us for several decades and that it is now rock solid. Articles like this make me think otherwise. Why do modern compilers break old code and then claim that it was broken from the start?
- saagarjha 7y ago
  Because the code was broken from the start, and previous compilers just didn't have the machinery to recognize that or do anything useful with it.
- aw1621107 7y ago
  > Why do modern compilers break old code and then claim that it was broken from the start? Because strictly speaking that old code is broken in that it violates the standard. Older compilers either didn't use the more aggressive interpretation of undefined behavior or didn't implement the optimizations that resulted in bad runtime behavior.
- jcranmer 7y ago
  The code was broken from the start. Most of the undefined behavior in C arises from traps: if your code, naively translated, would have resulted in a trap, then it is undefined behavior in C. Preserving when--and even if--a trap occurs is generally not desired, even by most end users. The instructions most likely to cause traps (memory operations and division operations) are expensive, and hoisting them outside of loops, or dead-code eliminating them is incredibly desirable. When people complain about "broken" undefined behavior, what they usually mean is that they expected to get a very particular kind of trap and the compiler didn't give them that trap.
- kps 7y ago
  Contrary to your other replies — old code, written in K&R C, was not broken from the start. ANSI C broke it with undefined behavior (by accident, I think) and ‘volatile’ (not by accident, just not fully thought through).
- madmax96 7y ago
einpoklum 7y ago
The author is encouraged to look into undefined behavior sanitizers: https://duckduckgo.com/?q=ubsan+sanitizer&t=ffsb&ia=web https://duckduckgo.com/?q=ubsan+sanitizer&t=ffsb&ia=web clang and gcc have those. Other than that - @_kst is correct, it was the same in C89 and C99, and the intention is simply to let compilers make the optimization they like ignoring what happens for UB cases.

147 points by quelsolaar 7y ago

lmm 7y ago

Where is the constituency that would want this "sensible C"? People who care about reasonable behaviour have already moved on to better languages, a la https://www.lesswrong.com/posts/ZQG9cwKbct2LtmL3p/evaporative-cooling-of-group-beliefs https://www.lesswrong.com/posts/ZQG9cwKbct2LtmL3p/evaporativ... ; the people still using C are those who want "maximum performance" and don't think undefined behaviour is a problem, and so C compiler writers have (understandably) gone ever further in their exploitation of undefined behaviour to get that "maximum performance". I don't think this can be reversed, and I don't think we should necessarily want to reverse it. We have better alternatives now. Let the C people do their thing, and get on with your life in a language that ensures that programs, even "incorrect" ones, have reasonable behaviour.

lone_haxx0r 7y ago
What are these "better languages" you speak of? Because I've been looking for a good C replacement but to no avail. C++, D, Nim, Go, Rust, Zig, Odin, etc; I've looked into them but none convinced me.
- admax88q 7y ago
  Stop looking for another C. Move on to something really different, something dynamic like JS, Clojure, Python, or Smalltalk
  - voldacar 7y ago
    Yeah what C programmers really want is dynamic languages
    - fulafel 7y ago
      You can also move to a static language of course. With C's mostly absent type system, Rust or Haskell might take some getting used to. But Go could be an easy transition, or Java.
    - admax88q 7y ago
      People still programming in C want C, its why they're still doing it. But you can't claim there are "no better languages" when all you're really looking for is C.
- worthless-trash 7y ago
  If none of the above convinced you, you'd need to specify why each of them didn't convince you (to do what) to get a qualified answer here. Every design choice has a repercussion, specify what repercusison that you would find acceptable and then your ideal language can be defined.
- lmm 7y ago
  What were your issues with those languages? Several of them sound like reasonable options to me. For a mature, general-purpose language my default suggestion is OCaml. If you really absolutely can't make garbage collection work for your case (something I've never seen happen to anyone who actually tried) then your choices are more limited (and I'd probably favour Rust, despite its relative immaturity). There are any number of good languages out there, and I could happily go into the details of which I think offers the best combination of tradeoffs (Scala). But the bigger picture is that a memory-safe language with the ML featureset (in particular first-class functions, parametric polymorphism, type inference, and sum types) should be the minimum baseline these days, and represents a substantial step up from C (at the most basic and pervasive level, being able to do error handling with a result type vastly improves your defect rate). Within that category you have plenty of reasonable choices offering their own particular selling points.
  - indymike 7y ago
    For me: libraries not available or are just wrappers on a C library anyway, inflexible memory management, bad support for OS features, etc... A lot of it is just maturity. C is ancient.
adrianN 7y ago
I'm pretty sure that a large part of the people working on legacy code bases would like to have a safer language. You don't move on to a better language if that means rewriting your whole project.
- lmm 7y ago
  "Sensible C" would have to be a distinct language from C, at least as the latter is currently implemented, so it would still represent a change of languages for legacy codebases. It could probably interop with the C ABI, but we already have other languages that can do that; it's not at all clear that converting your codebase piecemeal to "sensible C" would be substantially easier than doing the same with, say, Rust.
  - naniwaduni 7y ago
    It would also have source-level interop with C _APIs_, for which the only other language we have is C++, and which for that purpose has the same issues as C.
- pjmlp 7y ago
  I pretty much doubt it, given that ISO C WG doesn't care one second about making C safer. It is Oracle with SPARC ADI, Google/ARM/Apple with ARM MTE, Microsoft with Checked C, AT&T with Cyclone that actually bother to change the status quo.
  - saagarjha 7y ago
    Is anyone actually shipping with memory tagging yet?
    - pjmlp 7y ago
      Solaris on SPARC, iOS and it is a requirement for Android 11 ARM based devices.
      - 7y ago
dooglius 7y ago
raises hand The sort of optimizations that compilers abuse UB for aren't that useful, actually, and they don't actually require a compiler anyway; it's more about compiler writers overfitting for benchmarks.
justinmeiners 7y ago
I regularly use other "modern" languages and still enjoy using C. Am I cult member?
- lmm 7y ago
  Cultishness is a spectrum, not a binary. C has always had a certain proportion of adherents who believed that "performance" (of code that implements microbenchmarks with unlimited hours of manual tuning) is the most important aspect of a general-purpose programming language, and that the problems of undefined behaviour are insignificant. I would say that both those viewpoints are crazy, but they are increasingly becoming the mainstream in the C community, via the mechanism described in the link: more moderate people are driven out, rendering the community more extreme, which then drives more of the moderate folk out, and so on.
  - courtf 7y ago
    I see something similar happening with functional programming. Beyond measurable improvements (ie, in error rate, etc) there is often a mindset where if a language does not implement the latest syntactic patterns, then the language cannot be considered truly functional. HN seems a breeding ground for these sentiments, and adherents can be quite self-righteous. Out in the untamed wilds of the broader industry (and the further you move away from the Stanford sphere) I have found that fanaticism, of all sorts, tends to taper off, to be replaced mostly with the mundane tasks of churning out software for money. Of course, no matter where you hail from, the influence of C is inescapable. Yet, provincial priests are mostly concerned with budgetary constraints and deadlines. If switching to Scala will make hiring 10% easier, then by all means.
    - lmm 7y ago
      One "fun" thing about working in Scala is that you get attacked from all sides; there are people who say that the language is too weird or too purist, and then there are people who say that the problem is that it makes it too easy to have mutable variables or imperative code, or the fact that it allows you to do traditional OO inheritance. FWIW, my experience is that when you introduce a bit of mutable state or hidden side effects, you usually regret it later. I probably come across as fanatical at times, but that fanaticism is coming purely from painful experience; these academic, theoretical concerns become a lot less academic when you've had to deal with production bugs that they could have helped you avoid. I find it impossible to imagine the same being true for the C devotees. I've seen fast and slow programs in all sorts of languages. I've never seen a performance problem in anything other than a scripting language that wasn't either solved with a little bit of profiling, or just fundamentally impossible in any language. I've seen a C++ program rewritten in Haskell for a 5x performance speedup. And I've seen so many segfaults. FWIW, I've found it pretty easy to escape C. In JVM-land you really don't have to deal with it, at least 99.9% of the time; I've hit like two JVM bugs in my entire career. Scala is by no means the easiest to hire for, but it's a great language for getting on with solving business problems - whether you want to do that by churning out reams of code, or by abstracting out the mundane parts and writing only the parts that are specific to the problem. It makes the easy things easy and the hard things possible; very rarely have I felt that the language was limiting me or stopping me from doing what I wanted. There's plenty wrong with it, but it's the best choice I've found.

dejj 7y ago

Quite the converse. To me, “permissible” sounds as capricious as does “possible”. I read the article on “mass amateurization” before and am thinking how words must be sounding differently to me as a non-native English speaker and computer scientist. Maybe it had been best to insert “No running with scissors!” to that spec.

dependenttypes 7y ago

You might find this relevant https://news.ycombinator.com/item?id=10772841 https://news.ycombinator.com/item?id=10772841 What you want is not another language but a compiler that has a well defined behaviour and is not willing to change it between versions when UB in the C standard is triggered.

_kst_ 7y ago

The author claims that: > In C89, undefined behavior is interpreted as, “The C standard doesn’t have requirements for the behavior, so you must define what the behavior is in your implementation, and there are a few permissible options”. That's a serious misinterpretation. C89 says that "Permissible undefined behavior" includes "ignoring the situation completely with unpredictable results". There is absolutely no requirement to document what those "unpredictable results" might be. The standard joke is that one possible consequence of undefined behavior is making demons fly out of your nose -- not because that's actually possible, but because actually making demons fly out of your nose would not violate the standard. That's equally true in C89 and all later versions of C. The change in C99 from "Permissible" to "Possible" made no difference. The phrase "imposes no requirements" has always meant exactly that. (And if you think the change from "Permissible" to "Posisble" is semantically significant, then it would have been a recognition of what was already accepted.) The idea of "nasal demons" goes back to 1992. http://catb.org/jargon/html/N/nasal-demons.html http://catb.org/jargon/html/N/nasal-demons.html

mjd 7y ago
I came here to say something similar. The author says: > In C89, undefined behavior is interpreted as, “The C standard doesn’t have requirements for the behavior, so you must define what the behavior is in your implementation, and there are a few permissible options”. This is wrong. That is the meaning of what C89 calls “implementation-defined” behavior. The implementation gets a choice; the standard may lay out what the choices are, and in any event the choice must be documented. But with C89's “undefined” behavior, all bets are off. Numerous examples are documented of cases where real compilers were forced to behave in utterly bizarre ways because of real hardware limitations.
- eru 7y ago
  The C standard is quite liberal in its use of undefined behaviour. Some cases are due to conflicting existing implementation that predated the standard. (Though implementation defined might make more sense?) Some are for performance reasons. And some are utterly bizarre, like some things that should be syntax errors instead being declared undefined. If memory serves right, that includes eg not closing your string literals. (Thankfully, all implementations I know give you an error message.)

mjevans 7y ago

Any instance of undefined behavior should result in a Warning, if not an Error. Similarly, 'dead code paths' should also result in a Warning, if not an Error. (Possibly with a way of turning that off for specific functions.) I also fully agree with the author's statement about what a good compiler should be doing.

millstone 7y ago
> Any instance of undefined behavior should result in a Warning, if not an Error How would you implement this? For example, use-after-free is undefined, and difficult to detect.
- vnorilo 7y ago
  Warning when optimization passes take advantage of UB is a reasonable desire. The compilers are getting increasingly good at this, but I believe it would be far too noisy to blanket enable all of that.
  - mjevans 7y ago
    The point is to push the info to the programmer so that the code can be re-written as only defined behavior. Yes this would eliminate the 'gains' of abusing UB, but it would actually fix the errors of abusing UB.
    - vnorilo 7y ago
      I guess signed indvars would be a prime example. Compilers famously ignore overflow of signed integers when computing loop trip counts. Would it be better to only ever use unsigned integers for indvars and loop limits? Maybe. But that would reach deep into the types in stdlib for example. And then there's https://software.intel.com/en-us/forums/intel-c-compiler/topic/698664 https://software.intel.com/en-us/forums/intel-c-compiler/top... So, all in all, not simple. Not that that invalidates your point: I just wanted to add some nuance.
    - marcan_42 7y ago

nneonneo 7y ago

My "favourite" recent compiler bug comes from Rust. Compile the following code in Release mode: fn main() { (|| loop {})() } This defines an anonymous function containing an infinite loop, and calls it. Any sane compiler would make this just loop forever, right? $ cargo run --release test Illegal instruction (core dumped) Yeah, that's right. In Rust, an ostensibly safe language, Clang's overeager optimizer chooses to emit an illegal instruction in lieu of an infinite loop. (Try it here: https://play.rust-lang.org/?version=stable&mode=release&edition=2018&gist=4233cbe4bf524b1154b2e7b560d58610 https://play.rust-lang.org/?version=stable&mode=release&edit...) This particular bug actually comes from C++. In C++, threads may be assumed to always make forward progress, i.e. side-effect-free infinite loops are UB. However, this is not true in C, and it's definitely not true in Rust, yet Clang makes this assumption anyway. It gets a lot worse; once Clang sees an infinite loop and decides it's UB, it gets to make all sorts of thoroughly invalid and silly "optimizations"; see this SO question for some examples: https://stackoverflow.com/questions/59925618/how-do-i-make-an-infinite-empty-loop-that-wont-be-optimized-away https://stackoverflow.com/questions/59925618/how-do-i-make-a...

lmm 7y ago
> This particular bug actually comes from C++. In C++, threads may be assumed to always make forward progress, i.e. side-effect-free infinite loops are UB. However, this is not true in C That's very much disputed. The C standard requires programs' behaviour to be as-if executed on the C abstract machine when they terminate. It's not at all clear whether the standard imposes any requirements on programs that would not terminate when executed on the C abstract machine.
- _kst_ 7y ago
  C11 does have a special case rule for infinite loops. Section 6.8.5 paragraph 6: "An iteration statement whose controlling expression is not a constant expression, that performs no input/output operations, does not access volatile objects, and performs no synchronization or atomic operations in its body, controlling expression, or (in the case of a for statement) its expression-3, may be assumed by the implementation to terminate." (I'm personally not happy that this is stated in terms of that the implementation may assume rather than what a program may do.) Note that this does not apply to infinite loops with constant conditions. For example, given `while (1) { /* ... */ }`, the above doesn't apply and the implementation may not assume the loop will terminate.

bigcheesegs 7y ago

Yet again someone ignoring the line: > for which the Standard imposes no requirements. C89 had no requirements, C99 clarified that the line that followed was non-normative. The entire premise of this article is flawed.

quelsolaar 7y ago
The premise is that just because the standard doesn't impose requirements, it doesn't mean that other known factors like ABI, Architecture, and other platform specifics don't matter.
TwoBit 7y ago
The point of the article is that the author wants compilers to tell users when they do unexpected things. I don't see how your point is relevant to that.
- bigcheesegs 7y ago
  That may be the point, but it's not the premise. Even the title is based on this flawed idea. If the author wants to make a point, they can do so without trying to drive it with incorrect ideas.
- clarry 7y ago
  And the problem is that users ask the compiler to optimize, and then they complain because it optimizes. See, optimization can do a lot of unexpected things, but it is very difficult in general for the compiler to tell whether a certain application of (say) range analysis to eliminate dead code in a bunch of inlined calls is going to be unexpected. That's exactly the kind of optimization I want a compiler to do; I can write small functions that are as generic as possible yet cover all the edge cases, and then the compiler can find out which edge cases cannot apply in this particular situation. If the compiler told the user every time it did something, you'd never finish reading the output. Might as well ask the compiler not to optimize at that point.
- gpderetta 7y ago
  The compiler doesn't know it is doing unexpected things. The compiler is doing what's expected given the axioms it has to work with. It would be nice if the compiler added assertions in place to validate that the axioms hold, but that would often be very expensive. Recently I have been using the sanitizers extensively and has been a huge improvement, but of course they are too expensive to leave on in prod. Would be nice if a cheaper, although less exhaustive version were available.

userbinator 7y ago

This topic comes up semi-regularly here on HN but I think the underlying issue is that standards do not exist in a void, yet a certain group of people seem to think that they do --- the fact that a standard "imposes no requirements" should really be taken to mean "think about what really makes sense", not "do whatever the fuck you want". Hence the "but we still complies with the standard!" defense from compiler authors when faced with perplexing results of UB is totally ignoring the reality and practicality of what a compiler is for. Note that the standard writers have even tried to give that hint, with the "behaving during translation or program execution in a documented manner characteristic of the environment" phrasing which is precisely what programmers are usually expecting, but it seems few actually took the hint.

mpweiher 7y ago
IIRC the C standard explicitly states that conformance to the standard is insufficient for having a compiler that is fit for purpose.
- pjmlp 7y ago
  The best example are embedded platforms, full with extensions that probably will never be part of ISO C.
giomasce 7y ago
Different people have different ideas on what "makes sense". The point of a standard is to not have to use common sense, which is usually not common al all. So there is no point is saying that when the standard does not say anything you should use common sense. If the standard says nothing, you should just avoid relying of that thing. The compiler is not responsible for the quality of the source code. The programmer is. The compiler is responsible for the quality of the machine code, assuming that the source code is correct. It is good to have tools to check the quality of source code, but they are a different thing from compilers. Personally, I am happy that the compiler optimizes the code for me when I write correct C/C++ programs. If I make a mistake and inadvertently do UB, I take the responsibility, without shifting it on the innocent compiler.

_martamoreno_ 7y ago

Really bad example... Memset to zero members of a structure this way is actually something that should trigger a compiler error, because it makes no sense. I have limited understanding of C these days, but I consider that a good thing, because it allows me to look with fresh eyes on the crap I wrote back in the days. And this is one of them. It's nonsensical and similar to the general linux sentiment of naming your variable like random garbage. It's brainfuck. The only thing I can say with some distance is that exploiting undefined behavior in pretty much all cases should yield a compiler error.

saagarjha 7y ago
> I have limited understanding of C these days, but I consider that a good thing I wouldn't.

tumult 7y ago

This is shown as an example of illegal C: struct{ char x char y; }a; memset(&a, 0, sizeof(char) * 2); But the only thing illegal about it is that there's no semicolon after 'char x'. Writing to (theoretical) padding in this way is fine. Though, you might want to check the manual to see if this could generate a trap representation on your platform. (This won't, on any platform I'm aware of.) Also, the sizeof operator always returns 1 for char and unsigned char (C99: 6.5.3.4), so there's no reason to do sizeof(char).

hackcasual 7y ago
Agreed, that's not undefined behavior. It's just not guaranteed to 0 out y
- tumult 7y ago
  It actually would be guaranteed to 0 it if it were instead unsigned chars. For plain char, the implementation has to specify if it is signed, and if it is, what kind of padding it has. If it has none, it would also be guaranteed. Therefore, you can just check the compiler manual to see if this guarantees zeroing. (None of the mainstream compilers on mainstream platforms will have padding for signed char, either.) (C99: §6.2.5 and §6.2.6.2) (If you know a platform where this is not true, I'd be interested in hearing about it!) Edit: Oh, also, an implementation is allowed to have different padding in struct fields than the padding of the type itself. But it has to define this, so you would be able to look it up in the manual to see if it's different. (§6.7.2.1: "Each non-bit-field member of a structure or union object is aligned in an implementation-defined manner appropriate to its type.")
  - quelsolaar 7y ago
    This is my point, No platform I know of puts padding between bytes, so the behavior should be clear, but since a platform CAN put padding between bytes, and writing to said padding is UB, some compiler designers thinks this a license to do what ever.
    - 7y ago

aw1621107 7y ago

I'm not really convinced the author is correct in claiming that a one-word change opened the floodgates to optimizations on undefined behavior. In particular I think: > Careful reading will reveal that the word “Permissible” has been exchanged to “Possible”. In my opinion this change has lead C to go in a very problematic direction. is a red herring. In my opinion, the actual problematic phrase is this: > ignoring the situation completely with unpredictable results which didn't change between C89 and C99. It all comes down to what "ignoring the situation" should mean. Compiler vendors appear to interpret this to mean "ignore situations that invoke undefined behavior". Programmers who dislike optimizations based on undefined behavior appear to interpret this to mean "ignore the violation that leads to undefined behavior and treat it like conforming code". Who's right? It's ambiguous. > C compilers have taken the concept of undefined behavior even further by doing the mental acrobatics of thinking that “If undefined behavior happens, I can do want I want, So therefor I can assume that it will never happen”. I think this description is a bit uncharitable. I think a better description might be "If undefined behavior happens, there are no constraints on program behavior. Thus, if I assume that undefined behavior never happens, optimize the program based on that assumption, and that assumption is violated, the resulting program behavior is still OK because the standard imposes no rules on what the program must do when undefined behavior is invoked." > Lets take a look at signed integer overflow: Wasn't this useful for loop counters? I think this gist had more info: https://gist.github.com/rygorous/e0f055bfb74e3d5f0af20690759de5a7 https://gist.github.com/rygorous/e0f055bfb74e3d5f0af20690759... In addition, this bit: > In C89, undefined behavior is interpreted as, “The C standard doesn’t have requirements for the behavior, so you must define what the behavior is in your implementation, and there are a few permissible options”. is implementation-defined behavior, not undefined behavior. As the C89 standard notes, defining the implementation's behavior is only one of the things an implementation may do, not something it must do.

a1369209993 7y ago
> is implementation-defined behavior, not undefined behavior. They're (confusingly) misusing the word "define" there. Implementation-defined behaviour is required to be documented, but every behaviour is something that "you must define" in the sense that you have written down some (possibly implicit/emergent) definition of it as part of the compiler's source code. Ie, it's not "the standard requires you to define this", it's "you are not capable of not defining this, and the standard requires(C89)/doesn't require(C99+) you to pick from the following options".

deleted 7y ago

[deleted]

deleted 7y ago

[deleted]

mpweiher 7y ago

What a great idea! https://blog.metaobject.com/2018/07/a-one-word-change-to-c-standard-to-make.html https://blog.metaobject.com/2018/07/a-one-word-change-to-c-s... The comment section also has a lot of the same justifications for the current interpretations that I see here, with fairly thorough debunkings. Changing that one word is probably not quite enough, at the very least you need to also not make the remaining text a note or also change the status of notes back again. Then there is the conflict between "imposes no requirements" and "here is the range of permissible actions". These cannot both be true, and the current dogma is to resolve the conflict by pretending the second part dosen't exist (with, admittedly, some justification). But then why on earth is it in the standard at all?

saagarjha 7y ago
"Ranges from" does not necessarily enumerate the only possible choices: https://english.stackexchange.com/questions/148428/using-three-examples-with-range-from https://english.stackexchange.com/questions/148428/using-thr...
- mpweiher 7y ago
  Never said it did... ?
  - saagarjha 7y ago
    Specifically, it would allow for more than just the three options you mention there.
    - naniwaduni 7y ago
      It allows additional options _in between_ those listed. I'm skeptical that such an ill-defined rationale-centric change would hold any purchase with modern compiler-writers (it is certainly arguable what points lie "between" qualitatively-described behaviors), but in general a range does not usually include literally all possibilities.

jancsika 7y ago

> If the compiler thinks that writing to NULL, is undefined, it can therefore assume that since you are writing to p, p can’t be NULL. If p can’t be NULL, the entire if statement can be removed and after optimization the code looks like this: Is that correct? I thought Linus' rant was about NULL checks that came after a dereference had already happened. In that case it at least makes sense that the compiler would assume the NULL check would be superfluous. But how could branching to spit out an error and exit before the dereference ever get optimized out? I don't see any undefined behavior in the author's example upon which an optimization could trigger. (Or if it could trigger, it ought to trigger on array bounds checking and many other situations where removing the code would clearly cause bugs.)

quelsolaar 7y ago
You get this bug because the compiler doesn't know that the function will not return and therefor assumes that the de-reference will happen even if the value is NULL. Some compilers have a (non standard) keyword to indicate that a function will not return. adding an "else" will "fix" the issue.
- aw1621107 7y ago
  This is a corner of C I'm not familiar with, but does the standard say anything about functions that never return? If it doesn't say that implementations may assume functions return then that sounds more like a compiler bug than taking advantage of UB.
  - oconnor663 7y ago
    This seems contrary to how (I believe) the compiler works with atomics. If I call some opaque function foo(), the compiler has to assume that foo() could perform sequentially consistent atomic operations, and it cannot move other reads or writes across that function call. Why isn't it also required to assume that a function could terminate the program or longjump out?
    - 7y ago

olliej 7y ago

There are many people arguing specific semantics of the authors arguments, but I believe the core problem is C and C++ both dramatically overusing “undefined” vs “unspecified”. The difference is huge. Signed integer overflow is (per spec) undefined behaviour, so an obvious bounds check if UB and so can be removed. If it were unspecified the compiler would be required to be at least self-consistent. Eg it couldn’t do 2s complement in one place, but then treat arithmetic as not being 2s complement elsewhere (the overflow checks). E.g if the compiler emits code where MAX_INT+1 is MIN_INT, then the compiler can’t also pretend that that doesn’t happen. Undefined should be reserved solely for things that cannot have a specified behavior (UaF, OoB memory, IO weirdness, etc).

quelsolaar 7y ago
Some compilers even make it hard to test for overflow: if(a + 1 < a) printf("Overflow error!"); else a++; gets converted to: a++; because the compiler thinks that a + 1 can never be smaller then a since it doesn't have to consider signed overflow.
- CJefferson 7y ago
  While that looks stupid in isolation (and I agree it's annoyingly hard to check for overflow, although gcc and clang have special builtins nowadays to do it), it turns out there are important reasons for that optimisation. In general, knowing that 'a+1' is '1 larger than a' allows for lots of optimisations, when writing to an array in order we can vectorise, do things in bigger chunks, all sorts of useful and important optimsiations. If every time those were used the compiler had to check for overflow, it would seriously effect performance.
  - pjmlp 7y ago
    And that is how many CVE entries end up being created, because performance above anything else is what matters.
    - saagarjha 7y ago

brianpgordon 7y ago

> The point of a compiler is not to try to show off that who ever implemented it knows more loop holes in the C standard, then the user, but to help the programmer write a program that does that the programmer wants. The author makes it sound like the people working on optimizing compilers are deliberately seeking out these weird corner cases and selecting some random surprising behavior for them out of a hat, gleefully imagining how confusing it will be for end users. That's not how it works. Optimizers can be extraordinarily complex and need to maximize this ill-defined thing called "performance" in a highly multi-dimensional solution space. They ping-pong around inside this space constrained only by the specific requirements of the standard, and it's not surprising that some of the techniques used would produce some counter-intuitive results if the programmer is breaking the rules and relying on undefined behavior. It's kind of like if you trained a neural network to classify cat and dog pictures, and then you showed it a picture of a fire truck and expected it to give you a useful result. The idea of a new version of the C standard that defines some of the most surprising undefined behavior is an interesting one though, and I'd be interested to see how much that really impacts the ability of the optimizer to do its work.

SAI_Peregrinus 7y ago
I'd love it if the C standard just removed undefined behavior, replaced explicit instances ("the behavior is undefined" to "the behavior is implementation defined") and put in a blanket "Any behavior not specified by this standard is implementation defined". Keep the rest the same, just document the footguns. Implementation defined is exactly as powerful as undefined, it just makes the compiler writer describe what will happen.

dirtydroog 7y ago

It's C++, but UB allows the compiler to call a function the programmer never called https://kristerw.blogspot.com/2017/09/why-undefined-behavior-may-call-never.html https://kristerw.blogspot.com/2017/09/why-undefined-behavior... I'm not sure how any compiler writer can condone this behaviour.

aw1621107 7y ago
That particular behavior (and some other examples of pathological optimizer behavior) is probably an emergent property of how several optimizations combine as opposed to being a specifically targeted optimization. I can imagine something the optimizer doing something like: 1. Determine possible function call targets 2. Eliminate invalid targets from call set 3. If only one target remains, replace indirect call with direct call That might be useful for something like devirtualization, but also could lead to the behavior exhibited in that blog post. Indeed, the second blog post [0] shows that if there are two possible call targets, LLVM could generate an indirect call like what you may expect. [0]: https://kristerw.blogspot.com/2017/09/follow-up-on-why-undefined-behavior-may.html https://kristerw.blogspot.com/2017/09/follow-up-on-why-undef...
AshamedCaptain 7y ago
That is not "never used" as the programmer argues, and I find it misleading that the title of the blogpost tries to sell it as "a never used function is getting called". The NeverCalled function has external visibility so it could be eventually called by another CU, and it definitely references (or takes the address of) the supposedly unused "EraseAll" function. In order to be a legal program, some other CU has to call the NeverCalled at some point before main(), and the compiler is just assuming that.

scoutt 7y ago

The article kind of lost me at the first example. struct{ char x char y; }a; memset(&a, 0, sizeof(char) * 2); What does the compiler know about memset? Why it should? It can be a function called heyLetsFillTheMemoryOverHere(&a, wrongSize); and still it's the programmer's fault. In fact, it's a good thing that C doesn't care about memset, and doesn't get in between me and my code. And that's I love about C. It minds its own business and does what I tell it to do. > If I use my compiler to compile a program on my machine, the compiler knows that I’m compiling it to the x64 instruction set You can be cross-compiling. I do it every day. The C language doesn't need to know about every little platform out there. If a compiler wants to go the extra mile and help about it, that's a compiler-thing and not a C-language-thing. If there is a MCU that uses 3 bytes words, why should C be aware of it, 40 years before? Don't blame the specifications, blame the compiler if you want. Edit: removed a couple of personal opinions.

saagarjha 7y ago
> What does the compiler know about memset? Why it should? It can be a function called heyLetsFillTheMemoryOverHere(&a, wrongSize); and still it's the programmer's fault. Because memset is defined by the ISO C standard to have specific semantics, which the compiler can leverage to make your code faster.
- scoutt 7y ago
  > void memset(void s, int c, size_t n); > The memset() function shall copy c (converted to an unsigned char) into each of the first n bytes of the object pointed to by s. The standard just declares an interface, to be added to a C library (maybe?). It's not the job of the compiler to check for programmer's faults. The compiler doesn't care and it shouldn't.
  - deleted 7y ago
    [deleted]
  - gpderetta 7y ago
    The standard defines the behaviour, not just the signature. Memset has been treated as a builtin since forever.

OliverJones 7y ago

rms was all over this "undefined" stuff. Early gcc versions actually tried to launch the games Rogue or Nethack upon detecting #pragma directives. Old IBM "theory of operation" books used the word "unpredictable" for this kind of thing. "unpredictable" should frighten programmers. It's worse than "random," because random results have a chance of being caught during unit or system test.

lonelappde 7y ago

Why is a completely wrong blog post based on a misunderstanding of a 30 year old standard on the front page with 110 votes?

saagarjha 7y ago
It's a perennial Hacker News topic.
throwaway2048 7y ago
because a lot of users here think that a compiler somehow detects undefined behavior, then emits wacky code just to spite them.

imtringued 7y ago

I often see people still choose C even in 2020 for some projects. The usual arguments are that C has stuck with us for several decades and that it is now rock solid. Articles like this make me think otherwise. Why do modern compilers break old code and then claim that it was broken from the start?

saagarjha 7y ago
Because the code was broken from the start, and previous compilers just didn't have the machinery to recognize that or do anything useful with it.
aw1621107 7y ago
> Why do modern compilers break old code and then claim that it was broken from the start? Because strictly speaking that old code is broken in that it violates the standard. Older compilers either didn't use the more aggressive interpretation of undefined behavior or didn't implement the optimizations that resulted in bad runtime behavior.
jcranmer 7y ago
The code was broken from the start. Most of the undefined behavior in C arises from traps: if your code, naively translated, would have resulted in a trap, then it is undefined behavior in C. Preserving when--and even if--a trap occurs is generally not desired, even by most end users. The instructions most likely to cause traps (memory operations and division operations) are expensive, and hoisting them outside of loops, or dead-code eliminating them is incredibly desirable. When people complain about "broken" undefined behavior, what they usually mean is that they expected to get a very particular kind of trap and the compiler didn't give them that trap.
kps 7y ago
Contrary to your other replies — old code, written in K&R C, was not broken from the start. ANSI C broke it with undefined behavior (by accident, I think) and ‘volatile’ (not by accident, just not fully thought through).
madmax96 7y ago

einpoklum 7y ago

The author is encouraged to look into undefined behavior sanitizers: https://duckduckgo.com/?q=ubsan+sanitizer&t=ffsb&ia=web https://duckduckgo.com/?q=ubsan+sanitizer&t=ffsb&ia=web clang and gcc have those. Other than that - @_kst is correct, it was the same in C89 and C99, and the intention is simply to let compilers make the optimization they like ignoring what happens for UB cases.