10 ms·

Null Pointer Dereferencing Causes Undefined Behavior

82 points by DmitryNovikov 12y ago

notacoward 12y ago
The problem comes from the assumption that NULL is the same as zero. Even the author makes that assumption, when he gives an example that explicitly uses zero and then talks about it as if it had used NULL. In fact, the two have never been the same. I've worked on machines that represented NULL with a different value/bitpattern, and yes, it smoked out a lot of these bad assumptions. NULL can never point to valid memory, by language definition. An explicit zero can, from a language standpoint, even if the runtime/OS might disagree. ;) Thus, uses like that in offsetof are quite valid.
- pcwalton 12y ago
  But NULL must, broadly speaking, be the same as zero. C standard 3.2.2.3: "An integral constant expression with the value 0, or such an expression cast to type void * , is called a null pointer constant." C standard 4.1.5: "The macros are... NULL, which expands to an implementation-defined null pointer constant." See [1] for more info. [1]: http://stackoverflow.com/questions/2599207/can-a-conforming-c-implementation-define-null-to-be-something-wacky http://stackoverflow.com/questions/2599207/can-a-conforming-...
  - comex 12y ago
    The constant 0 has to convert to NULL, but nonconstant zeroes don't, and NULL doesn't have to be 0 when converted back to an integer. This would break code like the example offsetof implementation. See also http://stackoverflow.com/questions/2597142/when-was-the-null-macro-not-0 http://stackoverflow.com/questions/2597142/when-was-the-null...
  - notacoward 12y ago
    Casting zero to a pointer yields a null pointer. That doesn't preclude other-valued null pointers, or ensure that a cast the other way will yield zero. That's where the "undefined" part comes from.
    - mpyne 12y ago
      The way I like to think about it is that integer zero is the name in the C/C++ language (and runtime) for the actual null pointer, nothing more or less. In particular you can't assume anything about the integer representation of the actual null pointer.
    - _kst_ 12y ago
      Casting a constant zero to a pointer type yields a null pointer. The conversion is done at compile time, so the compiler knows that a null pointer is the required result. There is no requirement for an integer-to-pointer conversion to retain the bit pattern of the integer value, any more than an integer-to-floating-point conversion is required to do so. In most modern implementations, conversions among integer and pointer types of the same size are trivial, simply reinterpreting the bits as a value of a different type. A conforming C or C++ compiler can use, for example, the bit pattern 0xFFFFFFFF to represent a null pointer. Converting a constant 0 to a pointer type would yield a pointer with the representation 0xFFFFFFFF. Converting a non-constant 0 to a pointer type does not necessarily yield a null pointer; the result is implementation-defined, and may be indeterminate garbage.
  - EpicEng 12y ago
    NULL must compare equal to 0. That doesn't mean that it is represented by all 0 bits. NULL can have a value of anything add long as NULL == 0 is true.
  - deleted 12y ago
    [deleted]
  - deleted 12y ago
    [deleted]
- kstenerud 12y ago
  No, the problem comes from the assumption that you can legally fetch the "address" of a structure member via pointer dereference without checking for null so long as you don't actually try to access the resulting address. struct usb_line6 *line6 = &podhd->line6; The assumption is that this is OK so long as you don't try to access the contents of line6 without doing some upstream null checking first. This assumption is false. The very act of dereferencing is undefined, regardless of whether you access the resulting address or not. The assumption fails whether null is implemented as 0, 200, 0x8000000000000000, 0xffffffffffffffff.
  - ploxiln 12y ago
    There was no actual dereferencing, this is just pointer arithmetic. it's "line6 = podhd + offset". There isn't really a better way to do it.
    - ehmmm 12y ago
      There is no pointer arithmetic in the line: struct usb_line6 *line6 = &podhd->line6; Offset is calculated "below the hood". Pointer arithmetic is defined with arithmetic operands: +,- -> and . are not arithmetic operands.
      - jwatte 12y ago
        You sound very sure of yourself. If you look at the definition of the -> and [], you might learn something!
        ehmmm 12y ago
        Clearly, you don't have anything constructive to say, so you have to resort to ad hominem. [] operator doesn't have any place in this debate.
        spoiler 12y ago
        Well, as far as I understand it, `->` works in a similar fashion to `[]`; the difference being that one behaves on arrays while the other on structs. What @jwatte tried to say is that `->` works like this: // foo is struct thingity* &foo->bar == foo + offsetof(struct thingity, bar); The `[]` does a similar thing: // s is char* *(s+i) == s[i]
amelius 12y ago
How does Rust deal with this type of situation?
- lmm 12y ago
  By not having null pointers. If you want to have a pointer that might not point to anything valid, you do that explicitly with an option type. Of course if you really need to use a possibly-invalid pointer for performance you can drop into unsafe mode. (At least, the above is what I'd expect from a language like Rust)
  - yonran 12y ago
    But when you do need unsafe code (e.g. for defining a structure that interacts with a C API), is behavior undefined? I’m just looking at a bit of my own offsetof calculation and hoping it won’t break once I turn optimizations on. unsafe extern "stdcall" fn callback(arg0: *const IUnknown) { let myself: &MyStruct = &*(arg0.offset(-(&((*std::ptr::null::<MyStruct>()).as_event_handler) as isize)) as *const MyStruct); }
    - dbaupp 12y ago
      Yes, it generally is. Rust has similar (but not the same) undefined behaviours to C/C++, the compiler just ensures that they can't happen in safe code.
- alricb 12y ago
  Currently, by having a single implementation, so that there isn't really a "Rust standard" besides what this or that version of the Rust compiler does and what the documentation says, and if there's a discrepancy, it's a documentation or compiler bug. There might one day be a "Rust standard", but then let's all wish the standardization process leads to something like the Ada Language Reference Manual instead of the mess that is ANSI C.
- jerven 12y ago
  Rust does not have undefined behaviours in the sense of C. C and C++ are rather unique in the very large amount of undefined behaviour that is in common use. A nice comment pcwalton https://news.ycombinator.com/item?id=7601944 https://news.ycombinator.com/item?id=7601944 gives a better explanation than I can. The undefined behaviour in C is in my opinion a bad design that unfortunately poisoned a whole lot of software for minor gains in speed somewhere. A mistake that did not need to have become popular but for chance in history.
dmethvin 12y ago
The author may be correct, which is of course the best kind of correct. Yet I have written code for more than 25 years that depends on the behavior exemplified in the `offsetof()` macro referenced there. When writing low-level code it's really handy to know the offset of a member in a struct.
- mpyne 12y ago
  GCC advances frequently break code exploiting undefined behavior that had been working in a certain way for years. "Works as you'd expect" is certainly one of the possibilities of undefined behavior, after all, but that's no guarantee the next compiler version will continue to work as you'd expect. In fact I'm pretty sure a GCC update broke this exact type of code in the Linux kernel, by effectively removing null pointer checks after the optimizer used its knowledge of a pointer use later in a function to assume that the pointer must be valid and non-null (after all, if it had been null, the behavior would be undefined, and the compiler is allowed to assume no undefined behavior happens, which means the earlier null pointer check had been redundant). I believe that got fixed in GCC to do what you'd expect, but that's just one example of the awesomeness of undefined behavior in C-based languages.
- kps 12y ago
  > When writing low-level code it's really handy to know > the offset of a member in a struct. C provides offsetof() in <stddef.h> for that purpose, precisely because it's not expressible in C without compiler support.
- madmoose 12y ago
  Your compiler probably has a built-in method to achieve the equivalent of offsetof without invoking undefined behavior. GCC has __builtin_offsetof: https://gcc.gnu.org/onlinedocs/gcc/Offsetof.html https://gcc.gnu.org/onlinedocs/gcc/Offsetof.html
  - masklinn 12y ago
    offsetof is part of C89 stddef.h, why would you call the underlying GCC-specific __builtin_offsetof? Just call offsetof that's what it's here for, __builtin_offsetof is the GCC implementation detail.
quotemstr 12y ago
This article highlights yet more "gotcha" optimizations that, IMHO, do more harm than good. There is no reason that any compiler must miscompile &x->y, where x == NULL. What's undefined in C can be well-defined in POSIX or in the documentation of a specific build environment. It would behoove higher-level standards to make this construct well-defined even if C itself doesn't.
- wyldfire 12y ago
  I think POSIX almost always specifies strict additions in the form of function declarations and expected behavior of those functions, and some generic system-wide features. Your suggested change would amount to a delta on the language spec. That would probably be unprecedented.
  - TheLoneWolfling 12y ago
    Defining undefined behavior doesn't change the language spec, only adds to it. All it's doing is saying that "my compiler uses the freedom to do anything that undefined behavior grants to do something sane". By definition, anything that would be broken by such a change was already undefined behavior.
    - Retra 12y ago
      That's a reasonable way to do things, but only if it is not a misappropriation of resources and effort. If someone is writing C code, it is generally inadvisable to rely on behavior that is implementation specific, even if your implementation does nice things. That would violate portability and readability, both things that are nearly unanimously considered to be ideals of programming practice. So really, the "something sane" that you are talking about should be issuing an error or warning.
      - TheLoneWolfling 12y ago
        I don't see the difference between saying "this is valid C99 code" and "this is valid 'paranoid C' code". At least not if your 'paranoid C' compiler is working to a specification.
Animats 12y ago
Of course it's undefined behavior. It's also a famous issue with GCC optimization. This is another reason I look forward to the day when C is a legacy language.
- spoiler 12y ago
  As it stands now, C might outlive most, if not everyone in this thread. :-) Although, I don't mind C. I work in it every day and I've comfortable with it. A colleague of mine (who works primarily in Python) teased me it's because I have developed a Stockholm syndrome.
- pjmlp 12y ago
  At least Microsoft is helping achieving that. However it doesn't help that C++ allows for a C subset. We are several decades away from achieving that goal.
scott_s 12y ago
If you want to read more than can handle about undefined behavior in C, check out John Regehr's blog (http://blog.regehr.org/ http://blog.regehr.org/) and academic papers (http://www.cs.utah.edu/~regehr/papers/ http://www.cs.utah.edu/~regehr/papers/). He also posts here on HN.
_kst_ 12y ago
The standard offsetof macro is sometimes defined as: #define offsetof(st, m) ((size_t)(&((st *)0)->m)) That expression has undefined behavior, since it dereferences a null pointer. That doesn't mean it's an invalid definition of offsetof for a given implementation. Saying that it has "undefined behavior" means that the C language standard says nothing about how it behaves. If its behavior for the current compiler happens to satisfy the language requirements for offsetof, then it's a legitimate implementation. Code that implements the standard library is free to do whatever it likes, as long as the resulting behavior is correct. Part of the problem, I think, is the way the question is phrased. Quoting the linked article: > The programmers' community divided into two camps. The first claimed with confidence that it wasn't legal while the others were as sure saying that it was. The word "legal" doesn't even occur in the C standard. Some program constructs violate constraints or syntax rules; any such violation requires a compile-time diagnostic (which may be a non-fatal error). Other constructs have "undefined behavior"; the standard places no restrictions on what compilers can do for such constructs. Only a `#error` directive actually requires a program to be rejected.
- Taniwha 12y ago
  to be pedantic the pointer is not being dereferenced (the '&' inhibits that part of the operation)
  - _kst_ 12y ago
    Strictly speaking, yes, it is, at least in the abstract machine. If a unary "&" is applied directly to the result of a unary "" operator, then neither is evaluated. But in this case, the "&" is being applied to "((st )0)->m". "x->y" is syntactic sugar for "(x).y", so this is equivalent to If a unary "&" is applied directly to the result of a unary "" operator, then neither is evaluated. But in this case, the "&" is being applied to "((st )0)->m". "x->y" is syntactic sugar for "(x).y", so this is equivalent to "((st )0)).m)". A compiler will very likely optimize the expression so it doesn't generate machine code to dereference the null pointer. And in fact using this definition for the offsetof macro depends on that optimization, and on the compiler treating a null pointer as address 0.
- 12y ago
adamtj 12y ago
I'm sure he's right, but I'm not so sure he actually proved it. The real question is what happens when you deference a null pointer with '->'. He doesn't show us what the spec says on that, meaning there's a gap in his logic, leaving the quoted sentence in bold unsupported. I don't know but would bet that bolded sentence isn't even necessary. The undefinedness probably comes from the silence of the spec on what happens when you dereference a null pointer with '->'. As he showed, a null pointer cannot point to any object, and since a lvalue that does not designate an object gives undefined behavior, saying '* podhd' would clearly give undefined behavior when 'podhd' is a null pointer. However, instead of '* podhd' evaluating to a non-existant object, we have 'podhd->line6' asking for the object some particular offset from a non-existant object. It's entirely possible that a sufficiently insane spec might actually define such an operation. What the spec says on that point is the key. That's where the undefinedness must come from, but he doesn't talk about it or quote from the spec on '->'. He just assumes it, which leaves his original point unproven, even if it is correct. So why would anybody want to define legal behavior for dereferencing a null pointer with '->'? Well, some people start college funds for children who have yet to be conceived. If non-existant people can have account balances, why can't non-existant objects have properties? It's crazy, but it's not crazy like Intecal, where syntax errors are legal statements that you actually use for productive purposes. http://catb.org/~esr/intercal/ick.htm#Syntax-Error http://catb.org/~esr/intercal/ick.htm#Syntax-Error See also the statement "COME FROM".
- numeromancer 12y ago
  > As he showed, a null pointer cannot point to any object From my reading, he didn't even show that: >> If a null pointer constant is converted to a pointer type, the resulting pointer, called a null pointer, is guaranteed to compare unequal to a pointer to any object or function. So a null pointer could point to an object, it just wouldn't compare equal to any pointer, even itself.
  - ben0x539 12y ago
    Null pointers comparing equal to null pointers is kind of important though.
kragen 12y ago
This is ridiculous. The C standard was apparently written in order to allow compilers to introduce subtle security holes into previously-working code, all in the name of a fraction of a percent of performance increase. It’s totally irrational, especially in an epoch where C is no longer the language you’d use to get maximum bit-twiddling performance anyway — because it won’t run on your GPU! See http://blog.regehr.org/archives/761 http://blog.regehr.org/archives/761 for a sarcastic take on the problem, http://blog.regehr.org/archives/880 http://blog.regehr.org/archives/880 for a more serious description of the problem, and http://blog.regehr.org/archives/1180 http://blog.regehr.org/archives/1180 for the "Proposal for a Friendly Dialect of C" that proposes to actually solve it.
- CHY872 12y ago
  You've absolutely misrepresented the problem, presumably for effect. It's a far more subtle problem; the question is whether compiler writers should not only make their analyses and transformations be safe in the presence of valid code, but also safe in the presence of invalid code. This is ironically mainly a problem because older compilers acted normally on some undefined code. So more modern optimising compilers will expect defined behaviour and so forth, leading to new bugs reported in old code. These bugs were always present, they just appear only now. It's not the compiler trying to exploit the undefined behaviour deliberately; it's that the compiler is expecting the behaviour to be defined and behaving accordingly. So if you have int x = *foo; if (foo == NULL) { ERROR(); } then clearly in valid C, the if statement will never be taken, so it's safe to remove it. This is something you'd want to happen in most cases, it just so happens that in the above case you might not want it to. In terms of whether the fraction of a percent is worth it, well the people who do most of the development of the compilers is large companies like Google, and they don't do it out of the goodness of their hearts, they do it because the benefit of having high performance compilers is worth the expenditure. In terms of whether other languages are better, of course they are for many tasks, but C is king of the embedded world, where implementing a full Rust/Java/Node/language of day compiler is too much work. Then for many high performance tasks it's very difficult to beat the sheer performance of C or C++. If a task written in C or C++ runs just 20% faster than Java for Google/Microsoft/etc's 40,000 computer cluster, then that's 7000 computers that you don't need to buy. For the bit about GPUs, well that's kinda an aside but GPUs are useful for some things and not others and appealing to their use suggests ignorance. People use C for lots of things, and 'high performance bit-twiddling' isn't a standard one. GPUs are terrible at most tasks. The proposal for a friendly dialect of C is useful, but I suspect that the smarter way of doing it is the [0] undefined behaviour sanitiser, which produces an instrumented binary that errors when it reaches undefined behaviour. The developer, upon finding a new bug in their code, compiles, runs, gets an error, fixes, rinses, repeats. [0] - https://developerblog.redhat.com/2014/10/16/gcc-undefined-behavior-sanitizer-ubsan/ https://developerblog.redhat.com/2014/10/16/gcc-undefined-be...
TheLoneWolfling 12y ago
Ah, the wonders of C, where the obvious solution to "we can't guarantee <x> because it may cause problems on weird hardware" is "let's cause problems on mainstream hardware instead".
nraynaud 12y ago
When you have a bunch of experts arguing on the interpretation of the language spec to know if a construction in the most fundamental piece of software on the computer is valid like a herd of clerics around a vulgate deciding if dead newborns can go to heaven, you know you have a cultural problem.
- jwatte 12y ago
  Are those who argue experts, or are they "experts" ? I know of no expert who would age with the interpretation of the post.
  - nraynaud 12y ago
    they are strawmen, you get to pick
jwatte 12y ago
The post hinges on the faulty assumption that calculating the address of an lvalue "evaluates" that lvalue. This is not so. Any compiler that generates anything other than an add instruction (or lea or similar) is broken, for rather unsubtle reasons. Or to put it another way: dereferencing a null pointer may be undefined, but arithmetic on any pointer (including null) is well defined, and the code in question is arithmetic, not dereference. And, yet another broken blog post will make the rounds on the internet every three years and confuse even more people. It's sad.
- pcwalton 12y ago
  The post cites the spec. The C standard clearly seems to state that & must reference "an object", and NULL does not reference an object. A "pointer to any object" is distinguished from a null pointer. Do you have links stating otherwise?
  - Gibbon1 12y ago
    An of course plain C does not have 'objects' and a _lot_ of architectures have stuff mapped at address 0x0
    - aidenn0 12y ago
      C absolutely does have objects.
    - rcfox 12y ago
      While C is not an "object-oriented" language, it does have a concept of an object: http://port70.net/~nsz/c/c99/n1256.html#3.14p1 http://port70.net/~nsz/c/c99/n1256.html#3.14p1
    - nullc 12y ago
      You may be mistaking C for a fancy macro assembler. It's not-- It's a high level language, even though a fairly low level looking one. The obligation of the language is to behave as the spec says, and there may not be a close mapping between your program and what runs on the machine, though there often is. An "Object" is a concept defined and used extensively in the language specification; "a region of data storage in the execution environment,".
shultays 12y ago
I don't get it, there is no actual dereferencing going on here, It is just basic addition arithmetic between two values, one of them happens to be 0. Why It is illegal or "The program running well is pure luck"?
- cygx 12y ago
  The C standard does not specify the representation of null pointers, nor does it allow arithmetics with invalid pointers. Eg the conversion (void *)0 might result in a value with all bits set and member access will overflow. Another example would be hardware that raises a signal when encountering an invalid value during address calculations. In practice, the address calculation will work as expected (ie I'm unaware of machines where it doesn't). However, an aggressively optimizing compiler migh treat an expression like foo->bar as assert(foo != NULL), which might lead to unintended consequences in worst-case scenarios.
  - masklinn 12y ago
    > However, an aggressively optimizing compiler migh treat an expression like foo->bar as assert(foo != NULL) In fact most modern optimising compilers do exactly that, and the article does refer to such a behaviour having had security implications in the past.
DominikD 12y ago
I find this entire debate absurd. Pragmatic approach would be to refrain from using construct that are known to be controversial. By controversial I mean "there are reasons to believe that under certain conditions this may be dangerous, even though most people agree on what reasonable behavior is in this case". Don't write code that can bite you. The "struct foo *bar = &baz->qux;" is not your child. Kill it by refactoring it into something that's known to be obviously correct.
- pjmlp 12y ago
  The problem is that "correctness before performance", traditionally hasn't been part the C culture.
  - DominikD 12y ago
    There's no performance penalty here. The only difference is that you're on the safe side if you don't dereference before the null check. Whether discussed case is UB or not, compilers I've tried (MSVC 2013, GCC 4.9.something) produce the same code in both cases.
- eps 12y ago
  Refactoring for the sake of C pedantry rather than legitimate portability concerns is not a pragmatic approach. If this code were a part of some abstract "spherical horse in vacuum" project that may potentially build on esoteric platforms using random C compilers, then, yes, it might be worth re-factoring. In the context of Linux kernel project given the toolchain used the code is perfectly fine as is. The debate is silly though, can't argue with that.
  - DominikD 12y ago
    Linux kernel has official support for over 20 different ISAs. Some of the cores supported are pretty esoteric: BlackFin, SuperH, Etrax. Most of the problems with optimizing compilers - around UBs and others - come from architecture-specific optimizations. Problem discussed here is not abstract - people writing kernel code encounter these issues from time to time. I did, my friends did, we will be bitten by UBs in the future. Code my team is working on at the moment was written for a completely different architecture than it is run on right now. Sudden platform shifts happen, like when tsunami affected Renesas' ability to manufacture chips used in automotive. It IS pragmatic to avoid code that can potentially introduce problems in the future. Perhaps not for every piece of software, but low level one? Definitely.
ascotan 12y ago
That's why you should be passing input params by reference and not as a pointer. The compiler should be be used to check that inputs are valid references so you don't have to null check everything.
tempodox 12y ago
This is to be taken with a grain of salt. sizeof(&P->m_foo) does NOT return the size of `m_foo` but the size of a pointer to an `m_foo` on the target architecture.

82 points by DmitryNovikov 12y ago

The problem comes from the assumption that NULL is the same as zero. Even the author makes that assumption, when he gives an example that explicitly uses zero and then talks about it as if it had used NULL. In fact, the two have never been the same. I've worked on machines that represented NULL with a different value/bitpattern, and yes, it smoked out a lot of these bad assumptions. NULL can never point to valid memory, by language definition. An explicit zero can, from a language standpoint, even if the runtime/OS might disagree. ;) Thus, uses like that in offsetof are quite valid.

pcwalton 12y ago
But NULL must, broadly speaking, be the same as zero. C standard 3.2.2.3: "An integral constant expression with the value 0, or such an expression cast to type void * , is called a null pointer constant." C standard 4.1.5: "The macros are... NULL, which expands to an implementation-defined null pointer constant." See [1] for more info. [1]: http://stackoverflow.com/questions/2599207/can-a-conforming-c-implementation-define-null-to-be-something-wacky http://stackoverflow.com/questions/2599207/can-a-conforming-...
- comex 12y ago
  The constant 0 has to convert to NULL, but nonconstant zeroes don't, and NULL doesn't have to be 0 when converted back to an integer. This would break code like the example offsetof implementation. See also http://stackoverflow.com/questions/2597142/when-was-the-null-macro-not-0 http://stackoverflow.com/questions/2597142/when-was-the-null...
- notacoward 12y ago
  Casting zero to a pointer yields a null pointer. That doesn't preclude other-valued null pointers, or ensure that a cast the other way will yield zero. That's where the "undefined" part comes from.
  - mpyne 12y ago
    The way I like to think about it is that integer zero is the name in the C/C++ language (and runtime) for the actual null pointer, nothing more or less. In particular you can't assume anything about the integer representation of the actual null pointer.
  - _kst_ 12y ago
    Casting a constant zero to a pointer type yields a null pointer. The conversion is done at compile time, so the compiler knows that a null pointer is the required result. There is no requirement for an integer-to-pointer conversion to retain the bit pattern of the integer value, any more than an integer-to-floating-point conversion is required to do so. In most modern implementations, conversions among integer and pointer types of the same size are trivial, simply reinterpreting the bits as a value of a different type. A conforming C or C++ compiler can use, for example, the bit pattern 0xFFFFFFFF to represent a null pointer. Converting a constant 0 to a pointer type would yield a pointer with the representation 0xFFFFFFFF. Converting a non-constant 0 to a pointer type does not necessarily yield a null pointer; the result is implementation-defined, and may be indeterminate garbage.
- EpicEng 12y ago
  NULL must compare equal to 0. That doesn't mean that it is represented by all 0 bits. NULL can have a value of anything add long as NULL == 0 is true.
- deleted 12y ago
  [deleted]
- deleted 12y ago
  [deleted]
kstenerud 12y ago
No, the problem comes from the assumption that you can legally fetch the "address" of a structure member via pointer dereference without checking for null so long as you don't actually try to access the resulting address. struct usb_line6 *line6 = &podhd->line6; The assumption is that this is OK so long as you don't try to access the contents of line6 without doing some upstream null checking first. This assumption is false. The very act of dereferencing is undefined, regardless of whether you access the resulting address or not. The assumption fails whether null is implemented as 0, 200, 0x8000000000000000, 0xffffffffffffffff.
- ploxiln 12y ago
  There was no actual dereferencing, this is just pointer arithmetic. it's "line6 = podhd + offset". There isn't really a better way to do it.
  - ehmmm 12y ago
    There is no pointer arithmetic in the line: struct usb_line6 *line6 = &podhd->line6; Offset is calculated "below the hood". Pointer arithmetic is defined with arithmetic operands: +,- -> and . are not arithmetic operands.
    - jwatte 12y ago
      You sound very sure of yourself. If you look at the definition of the -> and [], you might learn something!
      - ehmmm 12y ago
        Clearly, you don't have anything constructive to say, so you have to resort to ad hominem. [] operator doesn't have any place in this debate.
        spoiler 12y ago
        Well, as far as I understand it, `->` works in a similar fashion to `[]`; the difference being that one behaves on arrays while the other on structs. What @jwatte tried to say is that `->` works like this: // foo is struct thingity* &foo->bar == foo + offsetof(struct thingity, bar); The `[]` does a similar thing: // s is char* *(s+i) == s[i]

amelius 12y ago

How does Rust deal with this type of situation?

lmm 12y ago
By not having null pointers. If you want to have a pointer that might not point to anything valid, you do that explicitly with an option type. Of course if you really need to use a possibly-invalid pointer for performance you can drop into unsafe mode. (At least, the above is what I'd expect from a language like Rust)
- yonran 12y ago
  But when you do need unsafe code (e.g. for defining a structure that interacts with a C API), is behavior undefined? I’m just looking at a bit of my own offsetof calculation and hoping it won’t break once I turn optimizations on. unsafe extern "stdcall" fn callback(arg0: *const IUnknown) { let myself: &MyStruct = &*(arg0.offset(-(&((*std::ptr::null::<MyStruct>()).as_event_handler) as isize)) as *const MyStruct); }
  - dbaupp 12y ago
    Yes, it generally is. Rust has similar (but not the same) undefined behaviours to C/C++, the compiler just ensures that they can't happen in safe code.
alricb 12y ago
Currently, by having a single implementation, so that there isn't really a "Rust standard" besides what this or that version of the Rust compiler does and what the documentation says, and if there's a discrepancy, it's a documentation or compiler bug. There might one day be a "Rust standard", but then let's all wish the standardization process leads to something like the Ada Language Reference Manual instead of the mess that is ANSI C.
jerven 12y ago
Rust does not have undefined behaviours in the sense of C. C and C++ are rather unique in the very large amount of undefined behaviour that is in common use. A nice comment pcwalton https://news.ycombinator.com/item?id=7601944 https://news.ycombinator.com/item?id=7601944 gives a better explanation than I can. The undefined behaviour in C is in my opinion a bad design that unfortunately poisoned a whole lot of software for minor gains in speed somewhere. A mistake that did not need to have become popular but for chance in history.

dmethvin 12y ago

The author may be correct, which is of course the best kind of correct. Yet I have written code for more than 25 years that depends on the behavior exemplified in the `offsetof()` macro referenced there. When writing low-level code it's really handy to know the offset of a member in a struct.

mpyne 12y ago
GCC advances frequently break code exploiting undefined behavior that had been working in a certain way for years. "Works as you'd expect" is certainly one of the possibilities of undefined behavior, after all, but that's no guarantee the next compiler version will continue to work as you'd expect. In fact I'm pretty sure a GCC update broke this exact type of code in the Linux kernel, by effectively removing null pointer checks after the optimizer used its knowledge of a pointer use later in a function to assume that the pointer must be valid and non-null (after all, if it had been null, the behavior would be undefined, and the compiler is allowed to assume no undefined behavior happens, which means the earlier null pointer check had been redundant). I believe that got fixed in GCC to do what you'd expect, but that's just one example of the awesomeness of undefined behavior in C-based languages.
kps 12y ago
> When writing low-level code it's really handy to know > the offset of a member in a struct. C provides offsetof() in <stddef.h> for that purpose, precisely because it's not expressible in C without compiler support.
madmoose 12y ago
Your compiler probably has a built-in method to achieve the equivalent of offsetof without invoking undefined behavior. GCC has __builtin_offsetof: https://gcc.gnu.org/onlinedocs/gcc/Offsetof.html https://gcc.gnu.org/onlinedocs/gcc/Offsetof.html
- masklinn 12y ago
  offsetof is part of C89 stddef.h, why would you call the underlying GCC-specific __builtin_offsetof? Just call offsetof that's what it's here for, __builtin_offsetof is the GCC implementation detail.

quotemstr 12y ago

This article highlights yet more "gotcha" optimizations that, IMHO, do more harm than good. There is no reason that any compiler must miscompile &x->y, where x == NULL. What's undefined in C can be well-defined in POSIX or in the documentation of a specific build environment. It would behoove higher-level standards to make this construct well-defined even if C itself doesn't.

wyldfire 12y ago
I think POSIX almost always specifies strict additions in the form of function declarations and expected behavior of those functions, and some generic system-wide features. Your suggested change would amount to a delta on the language spec. That would probably be unprecedented.
- TheLoneWolfling 12y ago
  Defining undefined behavior doesn't change the language spec, only adds to it. All it's doing is saying that "my compiler uses the freedom to do anything that undefined behavior grants to do something sane". By definition, anything that would be broken by such a change was already undefined behavior.
  - Retra 12y ago
    That's a reasonable way to do things, but only if it is not a misappropriation of resources and effort. If someone is writing C code, it is generally inadvisable to rely on behavior that is implementation specific, even if your implementation does nice things. That would violate portability and readability, both things that are nearly unanimously considered to be ideals of programming practice. So really, the "something sane" that you are talking about should be issuing an error or warning.
    - TheLoneWolfling 12y ago
      I don't see the difference between saying "this is valid C99 code" and "this is valid 'paranoid C' code". At least not if your 'paranoid C' compiler is working to a specification.

Animats 12y ago

Of course it's undefined behavior. It's also a famous issue with GCC optimization. This is another reason I look forward to the day when C is a legacy language.

spoiler 12y ago
As it stands now, C might outlive most, if not everyone in this thread. :-) Although, I don't mind C. I work in it every day and I've comfortable with it. A colleague of mine (who works primarily in Python) teased me it's because I have developed a Stockholm syndrome.
pjmlp 12y ago
At least Microsoft is helping achieving that. However it doesn't help that C++ allows for a C subset. We are several decades away from achieving that goal.

scott_s 12y ago

If you want to read more than can handle about undefined behavior in C, check out John Regehr's blog (http://blog.regehr.org/ http://blog.regehr.org/) and academic papers (http://www.cs.utah.edu/~regehr/papers/ http://www.cs.utah.edu/~regehr/papers/). He also posts here on HN.

_kst_ 12y ago

The standard offsetof macro is sometimes defined as: #define offsetof(st, m) ((size_t)(&((st *)0)->m)) That expression has undefined behavior, since it dereferences a null pointer. That doesn't mean it's an invalid definition of offsetof for a given implementation. Saying that it has "undefined behavior" means that the C language standard says nothing about how it behaves. If its behavior for the current compiler happens to satisfy the language requirements for offsetof, then it's a legitimate implementation. Code that implements the standard library is free to do whatever it likes, as long as the resulting behavior is correct. Part of the problem, I think, is the way the question is phrased. Quoting the linked article: > The programmers' community divided into two camps. The first claimed with confidence that it wasn't legal while the others were as sure saying that it was. The word "legal" doesn't even occur in the C standard. Some program constructs violate constraints or syntax rules; any such violation requires a compile-time diagnostic (which may be a non-fatal error). Other constructs have "undefined behavior"; the standard places no restrictions on what compilers can do for such constructs. Only a `#error` directive actually requires a program to be rejected.

Taniwha 12y ago
to be pedantic the pointer is not being dereferenced (the '&' inhibits that part of the operation)
- _kst_ 12y ago
  Strictly speaking, yes, it is, at least in the abstract machine. If a unary "&" is applied directly to the result of a unary "" operator, then neither is evaluated. But in this case, the "&" is being applied to "((st )0)->m". "x->y" is syntactic sugar for "(x).y", so this is equivalent to If a unary "&" is applied directly to the result of a unary "" operator, then neither is evaluated. But in this case, the "&" is being applied to "((st )0)->m". "x->y" is syntactic sugar for "(x).y", so this is equivalent to "((st )0)).m)". A compiler will very likely optimize the expression so it doesn't generate machine code to dereference the null pointer. And in fact using this definition for the offsetof macro depends on that optimization, and on the compiler treating a null pointer as address 0.
12y ago

adamtj 12y ago

I'm sure he's right, but I'm not so sure he actually proved it. The real question is what happens when you deference a null pointer with '->'. He doesn't show us what the spec says on that, meaning there's a gap in his logic, leaving the quoted sentence in bold unsupported. I don't know but would bet that bolded sentence isn't even necessary. The undefinedness probably comes from the silence of the spec on what happens when you dereference a null pointer with '->'. As he showed, a null pointer cannot point to any object, and since a lvalue that does not designate an object gives undefined behavior, saying '* podhd' would clearly give undefined behavior when 'podhd' is a null pointer. However, instead of '* podhd' evaluating to a non-existant object, we have 'podhd->line6' asking for the object some particular offset from a non-existant object. It's entirely possible that a sufficiently insane spec might actually define such an operation. What the spec says on that point is the key. That's where the undefinedness must come from, but he doesn't talk about it or quote from the spec on '->'. He just assumes it, which leaves his original point unproven, even if it is correct. So why would anybody want to define legal behavior for dereferencing a null pointer with '->'? Well, some people start college funds for children who have yet to be conceived. If non-existant people can have account balances, why can't non-existant objects have properties? It's crazy, but it's not crazy like Intecal, where syntax errors are legal statements that you actually use for productive purposes. http://catb.org/~esr/intercal/ick.htm#Syntax-Error http://catb.org/~esr/intercal/ick.htm#Syntax-Error See also the statement "COME FROM".

numeromancer 12y ago
> As he showed, a null pointer cannot point to any object From my reading, he didn't even show that: >> If a null pointer constant is converted to a pointer type, the resulting pointer, called a null pointer, is guaranteed to compare unequal to a pointer to any object or function. So a null pointer could point to an object, it just wouldn't compare equal to any pointer, even itself.
- ben0x539 12y ago
  Null pointers comparing equal to null pointers is kind of important though.

kragen 12y ago

This is ridiculous. The C standard was apparently written in order to allow compilers to introduce subtle security holes into previously-working code, all in the name of a fraction of a percent of performance increase. It’s totally irrational, especially in an epoch where C is no longer the language you’d use to get maximum bit-twiddling performance anyway — because it won’t run on your GPU! See http://blog.regehr.org/archives/761 http://blog.regehr.org/archives/761 for a sarcastic take on the problem, http://blog.regehr.org/archives/880 http://blog.regehr.org/archives/880 for a more serious description of the problem, and http://blog.regehr.org/archives/1180 http://blog.regehr.org/archives/1180 for the "Proposal for a Friendly Dialect of C" that proposes to actually solve it.

CHY872 12y ago
You've absolutely misrepresented the problem, presumably for effect. It's a far more subtle problem; the question is whether compiler writers should not only make their analyses and transformations be safe in the presence of valid code, but also safe in the presence of invalid code. This is ironically mainly a problem because older compilers acted normally on some undefined code. So more modern optimising compilers will expect defined behaviour and so forth, leading to new bugs reported in old code. These bugs were always present, they just appear only now. It's not the compiler trying to exploit the undefined behaviour deliberately; it's that the compiler is expecting the behaviour to be defined and behaving accordingly. So if you have int x = *foo; if (foo == NULL) { ERROR(); } then clearly in valid C, the if statement will never be taken, so it's safe to remove it. This is something you'd want to happen in most cases, it just so happens that in the above case you might not want it to. In terms of whether the fraction of a percent is worth it, well the people who do most of the development of the compilers is large companies like Google, and they don't do it out of the goodness of their hearts, they do it because the benefit of having high performance compilers is worth the expenditure. In terms of whether other languages are better, of course they are for many tasks, but C is king of the embedded world, where implementing a full Rust/Java/Node/language of day compiler is too much work. Then for many high performance tasks it's very difficult to beat the sheer performance of C or C++. If a task written in C or C++ runs just 20% faster than Java for Google/Microsoft/etc's 40,000 computer cluster, then that's 7000 computers that you don't need to buy. For the bit about GPUs, well that's kinda an aside but GPUs are useful for some things and not others and appealing to their use suggests ignorance. People use C for lots of things, and 'high performance bit-twiddling' isn't a standard one. GPUs are terrible at most tasks. The proposal for a friendly dialect of C is useful, but I suspect that the smarter way of doing it is the [0] undefined behaviour sanitiser, which produces an instrumented binary that errors when it reaches undefined behaviour. The developer, upon finding a new bug in their code, compiles, runs, gets an error, fixes, rinses, repeats. [0] - https://developerblog.redhat.com/2014/10/16/gcc-undefined-behavior-sanitizer-ubsan/ https://developerblog.redhat.com/2014/10/16/gcc-undefined-be...

TheLoneWolfling 12y ago

Ah, the wonders of C, where the obvious solution to "we can't guarantee <x> because it may cause problems on weird hardware" is "let's cause problems on mainstream hardware instead".

nraynaud 12y ago

When you have a bunch of experts arguing on the interpretation of the language spec to know if a construction in the most fundamental piece of software on the computer is valid like a herd of clerics around a vulgate deciding if dead newborns can go to heaven, you know you have a cultural problem.

jwatte 12y ago
Are those who argue experts, or are they "experts" ? I know of no expert who would age with the interpretation of the post.
- nraynaud 12y ago
  they are strawmen, you get to pick

jwatte 12y ago

The post hinges on the faulty assumption that calculating the address of an lvalue "evaluates" that lvalue. This is not so. Any compiler that generates anything other than an add instruction (or lea or similar) is broken, for rather unsubtle reasons. Or to put it another way: dereferencing a null pointer may be undefined, but arithmetic on any pointer (including null) is well defined, and the code in question is arithmetic, not dereference. And, yet another broken blog post will make the rounds on the internet every three years and confuse even more people. It's sad.

pcwalton 12y ago
The post cites the spec. The C standard clearly seems to state that & must reference "an object", and NULL does not reference an object. A "pointer to any object" is distinguished from a null pointer. Do you have links stating otherwise?
- Gibbon1 12y ago
  An of course plain C does not have 'objects' and a _lot_ of architectures have stuff mapped at address 0x0
  - aidenn0 12y ago
    C absolutely does have objects.
  - rcfox 12y ago
    While C is not an "object-oriented" language, it does have a concept of an object: http://port70.net/~nsz/c/c99/n1256.html#3.14p1 http://port70.net/~nsz/c/c99/n1256.html#3.14p1
  - nullc 12y ago
    You may be mistaking C for a fancy macro assembler. It's not-- It's a high level language, even though a fairly low level looking one. The obligation of the language is to behave as the spec says, and there may not be a close mapping between your program and what runs on the machine, though there often is. An "Object" is a concept defined and used extensively in the language specification; "a region of data storage in the execution environment,".

shultays 12y ago

I don't get it, there is no actual dereferencing going on here, It is just basic addition arithmetic between two values, one of them happens to be 0. Why It is illegal or "The program running well is pure luck"?

cygx 12y ago
The C standard does not specify the representation of null pointers, nor does it allow arithmetics with invalid pointers. Eg the conversion (void *)0 might result in a value with all bits set and member access will overflow. Another example would be hardware that raises a signal when encountering an invalid value during address calculations. In practice, the address calculation will work as expected (ie I'm unaware of machines where it doesn't). However, an aggressively optimizing compiler migh treat an expression like foo->bar as assert(foo != NULL), which might lead to unintended consequences in worst-case scenarios.
- masklinn 12y ago
  > However, an aggressively optimizing compiler migh treat an expression like foo->bar as assert(foo != NULL) In fact most modern optimising compilers do exactly that, and the article does refer to such a behaviour having had security implications in the past.

DominikD 12y ago

I find this entire debate absurd. Pragmatic approach would be to refrain from using construct that are known to be controversial. By controversial I mean "there are reasons to believe that under certain conditions this may be dangerous, even though most people agree on what reasonable behavior is in this case". Don't write code that can bite you. The "struct foo *bar = &baz->qux;" is not your child. Kill it by refactoring it into something that's known to be obviously correct.

pjmlp 12y ago
The problem is that "correctness before performance", traditionally hasn't been part the C culture.
- DominikD 12y ago
  There's no performance penalty here. The only difference is that you're on the safe side if you don't dereference before the null check. Whether discussed case is UB or not, compilers I've tried (MSVC 2013, GCC 4.9.something) produce the same code in both cases.
eps 12y ago
Refactoring for the sake of C pedantry rather than legitimate portability concerns is not a pragmatic approach. If this code were a part of some abstract "spherical horse in vacuum" project that may potentially build on esoteric platforms using random C compilers, then, yes, it might be worth re-factoring. In the context of Linux kernel project given the toolchain used the code is perfectly fine as is. The debate is silly though, can't argue with that.
- DominikD 12y ago
  Linux kernel has official support for over 20 different ISAs. Some of the cores supported are pretty esoteric: BlackFin, SuperH, Etrax. Most of the problems with optimizing compilers - around UBs and others - come from architecture-specific optimizations. Problem discussed here is not abstract - people writing kernel code encounter these issues from time to time. I did, my friends did, we will be bitten by UBs in the future. Code my team is working on at the moment was written for a completely different architecture than it is run on right now. Sudden platform shifts happen, like when tsunami affected Renesas' ability to manufacture chips used in automotive. It IS pragmatic to avoid code that can potentially introduce problems in the future. Perhaps not for every piece of software, but low level one? Definitely.

ascotan 12y ago

That's why you should be passing input params by reference and not as a pointer. The compiler should be be used to check that inputs are valid references so you don't have to null check everything.

tempodox 12y ago

This is to be taken with a grain of salt. sizeof(&P->m_foo) does NOT return the size of `m_foo` but the size of a pointer to an `m_foo` on the target architecture.