4 ms·

I drives me slightly crazy that an article about a fantastically technical subject such as "NULL pointer dereferences in the (Linux) kernel" immediately uses in

by unwind 5y ago

I drives me slightly crazy that an article about a fantastically technical subject such as "NULL pointer dereferences in the (Linux) kernel" immediately uses incorrect C code with undefined behavior to illustrate something. This: printf("The argv pointer = %d\n", argv); is not okay, you cannot pass a pointer-sized value when you tell printf() you're passing an integer. It needs to be: printf("The argv pointer = %p\n", (void *) argv); You should use %p which is for printing pointers, and cast to void pointer. Edit1: Oh, and I realize it "worked" for the author, no nasal demons [1] appeared and the code printed a number. That's just being lucky, and it's not a very nice foundation to base an argument on. Edit2: Okay, now I read a little bit further and I'm pretty sure the author lives in the "pointers are just integers, and NULL is zero" world, which is of course not a world that co-exists with our own. There is no guarantee that the in-memory representation of a NULL pointer is "all bits zero", that is implementation-defined and completely up the the architecture you're running on (and the compiler). I realize that in practice, on common hardware, NULL is typically implemented as all-bits-zero, but that is not defined by the language. I don't even know if I make sense, obviously I'm a C buff and take these things rather seriously. [1]: http://catb.org/jargon/html/N/nasal-demons.html http://catb.org/jargon/html/N/nasal-demons.html

saagarjha 5y ago
In the context of vulnerability analysis it is often more convenient to mark undefined behavior with something implementation-defined. This is why you can say "a buffer overflow will overwrite the return address" or "this integer will wrap to be negative", or, in this context, NULL is represented with the zero address/bit pattern.
vlovich123 5y ago
The article is from 2010 (missing tag) so the author likely is talking about 32 bit x86 machines. Also compilers from that era were less likely to exploit UB if at all. Certainly knowledge about it had only started trickling out in the general industry at the time as compilers started exploiting it with Chris Partner’s blog about it following about a year later [1]. [1] http://blog.llvm.org/2011/05/what-every-c-programmer-should-know_14.html http://blog.llvm.org/2011/05/what-every-c-programmer-should-...
- guerrilla 5y ago
  > Certainly knowledge about it had only started trickling out in the general industry at the time as compilers started exploiting it It was common knowledge among C programmers long before then, for both security and portability reasons but optimization was also done. Old MacOS even used the higher 8-bits of pointers for MM flags. Other popular systems did equally weird things (think VAX.) That discussion linked to by the Jargon file is from 1992.
  - Taniwha 5y ago
    I've programmed all these, old Macs certainly did weird stuff with the upper bits of pointers, Vaxen though have quite ordinary virtual address spaces, they do use upper bits to choose between page tables but that's about it
- tialaramex 5y ago
  > Also compilers from that era were less likely to exploit UB if at all No. Here's an LWN discussion of a CERT advisory in 2008 trying this nonsense, claiming it's a problem that GCC 4.2 elides futile late NULL checks. https://lwn.net/Articles/278137/ https://lwn.net/Articles/278137/ In fact several other compilers already had this optimisation, and CERT had to go back and revise the advisory because in practice all that's going on is, as now Real Programmers think their undefined program ought to do whatever it was they intended, and that's not how programming works.
- masklinn 5y ago
  > The article is from 2010 compilers from that era Funny. tcc was created in 2001 as a specifically non-optimising compiler.
pedrocr 5y ago
> There is no guarantee that the in-memory representation of a NULL pointer is "all bits zero", that is implementation-defined and completely up the the architecture you're running on (and the compiler). I always wondered if all the C codebases that do "if (somepointer)" to check for NULL were actually wrong of if the standard guaranteed that NULL always evaluates to false even if it's not implemented as zero. Apparently it's valid and everything works even in a theoretical architecture where NULL was not implemented as 0: http://c-faq.com/null/ptrtest.html http://c-faq.com/null/ptrtest.html
- masklinn 5y ago
  Yep, NULL doesn’t have a defined value, but that value must be false in a boolean context.
- tialaramex 5y ago
  Under the hood, for this to work there's an implicit cast to boolean, even though at the time it was invented C doesn't have a formal boolean type. So, (almost?) every expression in C has an implied boolean value, and the implied boolean value of pointers is false if they're NULL or true otherwise. A better design is to have a boolean type, and require that the if condition tests only actual booleans, but this requires a substantial eco-system wide commitment, if you just insist on actual booleans you make programming needlessly tiresome and programmers say this language is bad because I have to write so much boilerplate for no gain. You need to actually provide the rest of the language features that make this pain worth it. That commitment seems relatively affordable in 2021, but it was probably too big an ask when C was invented and by the time C was standardised it's too late. Likewise for C++.
gpvos 5y ago
> which is of course not a world that co-exists with our own The "our own" world you are describing has recently been created by UB-exploiting compilers; it's a very strange and unintuitive place, is only described in inscrutable standards documents, and compilers don't properly warn you if you do something wrong even if you ask them to. A bit more understanding of people making mistakes seems appropriate.
- mpweiher 5y ago
  And in fact, most of "us" don't inhabit this world voluntarily. And the standards in question are actually have very clear and simple language disallowing those shenanigans. "Permissible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message)." Alas, that language was made non-binding and now the people who made it non-binding pretend it doesn't exist at all. https://blog.metaobject.com/2018/07/a-one-word-change-to-c-standard-to-make.html https://blog.metaobject.com/2018/07/a-one-word-change-to-c-s...
  - ChrisSD 5y ago
    Hm, I think "ignoring the situation completely with unpredictable results" pretty much covers everything. It's how optimizations generally work. Each optimization pass simply assumes there's no UB in the input and will optimize based on that assumption. Optimizing compilers don't intentionally break programs with UB, they just ignore the possibility... with very unpredictable results.
    - mpweiher 5y ago
      I don't think "optimising based on UB" is covered by "ignoring". And neither is "optimising assuming UB never happens", especially if said UB is explicitly written in the code, thus contradicting the assumption.
mpweiher 5y ago
> you cannot pass a pointer-sized value when you tell printf() you're passing an integer. Actually you can. You probably shouldn't, particularly now that many 64 bit C machine models have different sized pointers and ints. If they were going to use an int conversion, they should have used %ld. > You should use %p which is for printing pointers, Agreed. > [no nasal daemons] That's just being lucky, Actually it isn't "just lucky". Well, to be precise it wasn't just lucky back when C compiler writers didn't stretch the idea of UB way beyond the breaking point, and way beyond certainly the intent and arguably the letter of the standard. (The cop out is that the part of the standard that explicitly says all these shenanigans are not OK was made non-binding in later versions of the standard, it originally was binding, and thus people pretend that this part of the standard doesn't exist at all. But it does.) > There is no guarantee that the in-memory representation of a NULL pointer is "all bits zero", There is no guarantee of this in the C standard. However, the C standard is intentionally incomplete, i.e. conformance to the standard is not enough for an implementation to be "fit for purpose". I think it even used to say that somewhere in the standard. > I realize that in practice, on common hardware, NULL is typically implemented as all-bits-zero, For a very not to say overwhelmingly large value of "typically". I can't name a single implementation that does not, and my guess would be that total market share of such implementations would be less than 0.1%, and I believe I am being very conservative with that estimate. Which also means that if you are on such an architecture, you almost certainly know it. > but that is not defined by the language. ...because the language definition is intentionally incomplete. People talk about the "C abstract" machine as if it were a thing. It's not. C maps to real hardware, that's the point of the language, and so taking the real-world behavior of the hardware that you are going to run on into account is perfectly fine.
- SAI_Peregrinus 5y ago
  Android on 64-bit ARM uses the MSB of pointers for a tag. There are 2^8 NULL pointer values at the hardware level there.
  - mpweiher 5y ago
    Interesting! Do we actually need any of them apart from all zeros? Are the tags retained when converting?
mrslave 5y ago
There are 2 types of C programmers: those who have read Van der Linden, and those who have not.

by unwind 5y ago

saagarjha 5y ago

In the context of vulnerability analysis it is often more convenient to mark undefined behavior with something implementation-defined. This is why you can say "a buffer overflow will overwrite the return address" or "this integer will wrap to be negative", or, in this context, NULL is represented with the zero address/bit pattern.

vlovich123 5y ago

The article is from 2010 (missing tag) so the author likely is talking about 32 bit x86 machines. Also compilers from that era were less likely to exploit UB if at all. Certainly knowledge about it had only started trickling out in the general industry at the time as compilers started exploiting it with Chris Partner’s blog about it following about a year later [1]. [1] http://blog.llvm.org/2011/05/what-every-c-programmer-should-know_14.html http://blog.llvm.org/2011/05/what-every-c-programmer-should-...

guerrilla 5y ago
> Certainly knowledge about it had only started trickling out in the general industry at the time as compilers started exploiting it It was common knowledge among C programmers long before then, for both security and portability reasons but optimization was also done. Old MacOS even used the higher 8-bits of pointers for MM flags. Other popular systems did equally weird things (think VAX.) That discussion linked to by the Jargon file is from 1992.
- Taniwha 5y ago
  I've programmed all these, old Macs certainly did weird stuff with the upper bits of pointers, Vaxen though have quite ordinary virtual address spaces, they do use upper bits to choose between page tables but that's about it
tialaramex 5y ago
> Also compilers from that era were less likely to exploit UB if at all No. Here's an LWN discussion of a CERT advisory in 2008 trying this nonsense, claiming it's a problem that GCC 4.2 elides futile late NULL checks. https://lwn.net/Articles/278137/ https://lwn.net/Articles/278137/ In fact several other compilers already had this optimisation, and CERT had to go back and revise the advisory because in practice all that's going on is, as now Real Programmers think their undefined program ought to do whatever it was they intended, and that's not how programming works.
masklinn 5y ago
> The article is from 2010 compilers from that era Funny. tcc was created in 2001 as a specifically non-optimising compiler.

pedrocr 5y ago

> There is no guarantee that the in-memory representation of a NULL pointer is "all bits zero", that is implementation-defined and completely up the the architecture you're running on (and the compiler). I always wondered if all the C codebases that do "if (somepointer)" to check for NULL were actually wrong of if the standard guaranteed that NULL always evaluates to false even if it's not implemented as zero. Apparently it's valid and everything works even in a theoretical architecture where NULL was not implemented as 0: http://c-faq.com/null/ptrtest.html http://c-faq.com/null/ptrtest.html

masklinn 5y ago
Yep, NULL doesn’t have a defined value, but that value must be false in a boolean context.
tialaramex 5y ago
Under the hood, for this to work there's an implicit cast to boolean, even though at the time it was invented C doesn't have a formal boolean type. So, (almost?) every expression in C has an implied boolean value, and the implied boolean value of pointers is false if they're NULL or true otherwise. A better design is to have a boolean type, and require that the if condition tests only actual booleans, but this requires a substantial eco-system wide commitment, if you just insist on actual booleans you make programming needlessly tiresome and programmers say this language is bad because I have to write so much boilerplate for no gain. You need to actually provide the rest of the language features that make this pain worth it. That commitment seems relatively affordable in 2021, but it was probably too big an ask when C was invented and by the time C was standardised it's too late. Likewise for C++.

gpvos 5y ago

> which is of course not a world that co-exists with our own The "our own" world you are describing has recently been created by UB-exploiting compilers; it's a very strange and unintuitive place, is only described in inscrutable standards documents, and compilers don't properly warn you if you do something wrong even if you ask them to. A bit more understanding of people making mistakes seems appropriate.

mpweiher 5y ago
And in fact, most of "us" don't inhabit this world voluntarily. And the standards in question are actually have very clear and simple language disallowing those shenanigans. "Permissible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message)." Alas, that language was made non-binding and now the people who made it non-binding pretend it doesn't exist at all. https://blog.metaobject.com/2018/07/a-one-word-change-to-c-standard-to-make.html https://blog.metaobject.com/2018/07/a-one-word-change-to-c-s...
- ChrisSD 5y ago
  Hm, I think "ignoring the situation completely with unpredictable results" pretty much covers everything. It's how optimizations generally work. Each optimization pass simply assumes there's no UB in the input and will optimize based on that assumption. Optimizing compilers don't intentionally break programs with UB, they just ignore the possibility... with very unpredictable results.
  - mpweiher 5y ago
    I don't think "optimising based on UB" is covered by "ignoring". And neither is "optimising assuming UB never happens", especially if said UB is explicitly written in the code, thus contradicting the assumption.

mpweiher 5y ago

> you cannot pass a pointer-sized value when you tell printf() you're passing an integer. Actually you can. You probably shouldn't, particularly now that many 64 bit C machine models have different sized pointers and ints. If they were going to use an int conversion, they should have used %ld. > You should use %p which is for printing pointers, Agreed. > [no nasal daemons] That's just being lucky, Actually it isn't "just lucky". Well, to be precise it wasn't just lucky back when C compiler writers didn't stretch the idea of UB way beyond the breaking point, and way beyond certainly the intent and arguably the letter of the standard. (The cop out is that the part of the standard that explicitly says all these shenanigans are not OK was made non-binding in later versions of the standard, it originally was binding, and thus people pretend that this part of the standard doesn't exist at all. But it does.) > There is no guarantee that the in-memory representation of a NULL pointer is "all bits zero", There is no guarantee of this in the C standard. However, the C standard is intentionally incomplete, i.e. conformance to the standard is not enough for an implementation to be "fit for purpose". I think it even used to say that somewhere in the standard. > I realize that in practice, on common hardware, NULL is typically implemented as all-bits-zero, For a very not to say overwhelmingly large value of "typically". I can't name a single implementation that does not, and my guess would be that total market share of such implementations would be less than 0.1%, and I believe I am being very conservative with that estimate. Which also means that if you are on such an architecture, you almost certainly know it. > but that is not defined by the language. ...because the language definition is intentionally incomplete. People talk about the "C abstract" machine as if it were a thing. It's not. C maps to real hardware, that's the point of the language, and so taking the real-world behavior of the hardware that you are going to run on into account is perfectly fine.

SAI_Peregrinus 5y ago
Android on 64-bit ARM uses the MSB of pointers for a tag. There are 2^8 NULL pointer values at the hardware level there.
- mpweiher 5y ago
  Interesting! Do we actually need any of them apart from all zeros? Are the tags retained when converting?

mrslave 5y ago

There are 2 types of C programmers: those who have read Van der Linden, and those who have not.