9 ms·

What Every C Programmer Should Know About Undefined Behavior (2011)

92 points by arunc 12y ago

youngthugger 12y ago
This makes C look like it's full of land mines. The minesweeper language.
- nnhtzgf 12y ago
  The title is making it seem as the listed behavior is largely unknown to C programmers. It isn't. And it is in the manual.
  - ape4 12y ago
    Yeah, hopefully if you are programming in C you'll know using an undefined variable is bad.
    - nnhtzgf 12y ago
      Yep. :-) The article doesn't make much sense at times: Dereferencing a NULL Pointer: >>>contrary to popular belief<<<, dereferencing a null pointer in C is undefined. I guess C programmers are not counted in the "popular" group.
      - stormbrew 12y ago
        I don't think it's wrong at all. I suspect if you asked a large number of even moderately competent C programmers, you'd find quite a lot of them believe it's defined to trap. That's exactly what the rest of it is talking about. Never mind the issue raised of how it potentially affects optimization, which I think very few C programmers would actually be able to speak to.
      - bhaak 12y ago
        Are you really sure that most C programmers know about this? IMO, if you'd ask the question "what happens if you dereference a NULL pointer", most C programmers would answer "a segfault".
        nnhtzgf 12y ago
        If you are in a company that codes largely in C, do a pool and report back. I'm honestly curious.
        bhaak 12y ago
        Unfortunately, I don't work in such a company. A completely unrepresentative poll among two C++ programmers showed that 50% thought it was a segfault, whereas the other 50% suggested it could be undefined behavior, deducing that from the knowledge that the behavior is different in Linux userspace and kernel. Although I wouldn't call those two typical C++ programmers. They do security CTFs for fun, so they might have a broader knowledge about exploitable behaviors in C/C++ code.
        unwind 12y ago
        That, of course, would be a leading question. Most people would probably tend to assume that you're asking "in the environment in which we're programming, what happens if you dereference NULL?" which is not what you meant. I didn't poll my coworkers either, but at least I think the answers would be different if I asked "what does C say should happen when a NULL pointer is dereferenced?". Perhaps that's just wishful thinking on my behalf, though. :)
        ghswa 12y ago
        I don't think that's particularly wishful thinking, however, your alternative phrasing could be seen as equally, if not more, leading. When quizzed on the behaviour of a C program I would expect a careful and experienced C programmer to consider the behaviour described by the standard (which standard?); unspecified and implementation defined behaviours; and deviations from the standard in both the compiler and the environment. Often I'd expect the answer to be "it depends".
      - legulere 12y ago
        For a while you were actually able to mmap the memory at address NULL under linux, which is usable for exploiting (you can pass arbitrary data to kernel code that you can bring to dereference null pointers)
        orbifold 12y ago
        Wouldn't virtual address translation prevent this from happening?
- Lrigikithumer 12y ago
  Well I mean it kind of is. C's my favorite language but it's like a machete, simple enough, fairly easy to use and useful in a lot of places but if you don't know how to use it properly it's very easy to cut yourself with.
- FooBarWidget 12y ago
  In a certain way, it is. But for a lot of things C and (C++) is still necessary, and the fact that C is not safe is a requirement. Can you imagine the Linux kernel not being written in C? Just imagine all the overhead if the Linux kernel has to perform overflow checking after every internal integer increment. Maybe some day Rust will be a good candidate, but that day hasn't come yet.
  - pjmlp 12y ago
    > Can you imagine the Linux kernel not being written in C? No, only because the Linux kernel started as an UNIX clone and C is the UNIX's systems programming language. Other OS were written in another systems programming languages. Before and after C got invented. C only became widespread because a few successful startups in the 80's adopted UNIX for their workstation OS.
    - pmalynin 12y ago
      In the OS-programming world C/C++ is staple.
      - pjmlp 12y ago
        I am old enough to remember when it wasn't even an option.
    - ArkyBeagle 12y ago
      I'd say that in terms of seats sold, Borland and Microsoft had more to do with it. Maybe first one then the other. But 'C' was the semi-official language of the old DOS PC world from 1985 until Linux happened. It's somewhat tragic, because Modula-2 and Ada toolchains existed, but were harder to use than 'C' for performance and memory constraint reasons ( not to mention the toolchain costs ) Borland started out with Pascal, but went to 'C'.
      - pjmlp 12y ago
        >Borland started out with Pascal, but went to 'C'. As someone that used all their products before they became Imprise or how it was called, I never saw this happening. Even nowadays C++ Builder actually uses the Delphi framework. EDIT: Just wanted to add that in my part of the planet, it was all about Assembly for systems programming, games and real applications. Turbo Pascal for general purpose applications without much performace issues and CLIPPER for the typical CRUD applications.
- jokoon 12y ago
  It's not the 80s anymore, there area many cases where there are much better alternatives to C. But it's still a language you need for certain things, like responsiveness or real time applications. Like linus torvalds said it, C is like a more comprehensive assembly generator than anything else. Having such a language allows you to change every little boring details to do exactly what you want. It has its use, and it's surely not intended to be mainstream as of today. But there are many cases where C comes back as a constant common denominator. It's sure that it's kinda shitty that there's nothing really better than a language designed in the 70s, but I guess you could either blame language designers for not making anything better. There also are business/political issues and preferences for certain things that comes into play. All in all, C just seems to be pretty good when you want to have code that works well with systems. Maybe there's just inertia as to where computer development is headed, which might be heavy related to C. All I wish, is someday to be able to learn some more "virtuous" language like haskell, scheme, or smalltalk.
  - pjmlp 12y ago
    The 80's already had Mesa and Modula-2, but then those guys at Berkley had to create a few successful startups.
  - anon13839 12y ago
    You may want to look up the age of those "virtuous" languages you listed. While it's fun to bash C, you could drive a damn yacht through the undefined behavior in Scheme. High-level languages are not immune here.
- username223 12y ago
  I recommend reading the articles linked elsewhere in this thread: https://news.ycombinator.com/item?id=8879948 A lot of this is C compiler writers using the spec to perform counterintuitive "optimizations" (with dubious performance value), and forgetting that they're writing software for C programmers to use, not trying to win a pedantry war with them.
bhaak 12y ago
Some of the "performance penalties" look like FUD to me. At least to the extent implied in the article. For example "use of an unitialized variable". Given that the compiler already has to analyze the program flow to determine if the variable could be use unitialized to output that warning, the optimization step to not initialize those variables that are written to before used doesn't seem so far away. This happens at compile time. But, there's also the fact that a modern OS gives you heap memory zeroed mostly for free. A call to calloc is highly optimized nowadays. It needs to be for security reasons, as a program must never receive memory with data left from a different process. Likewise the signed integer overflow check. Modern CPUs are so superscalar internally that an overflow check (which doesn't get flagged most of the time anyway) doesn't add anything to the execution speed. And finally we are able to use that without resorting to assembler. GCC 5 has added support for built-in functions for arithmetics with overflow checking.
- haberman 12y ago
  > But, there's also the fact that a modern OS gives you heap memory zeroed mostly for free. A call to calloc is highly optimized nowadays. It needs to be for security reasons, as a program must never receive memory with data left from a different process. Memory often is coming from previously-freed heap space within the process. There is no security need to zero such memory. Zeroing it wouldn't even help, because the process is free to read the memory before malloc returns it. And calloc, no matter how optimized it is, will still be more expensive than malloc which doesn't have to zero the memory. > Modern CPUs are so superscalar internally that an overflow check (which doesn't get flagged most of the time anyway) doesn't add anything to the execution speed. "Doesn't add anything" is not correct. These checks are not free: http://danluu.com/integer-overflow/ http://danluu.com/integer-overflow/ https://news.ycombinator.com/item?id=8765714#up_8765979 https://news.ycombinator.com/item?id=8765714#up_8765979
  - barrkel 12y ago
    Zeroed memory should not be particularly expensive on modern machines, because we have a lot of parallelism available to zero it off working threads. There might be some constraint on memory bandwidth, but IMO memory latency is a bigger issue, usually preventing maximizing memory throughput, so there should be some spare. The danluu link doesn't really support your argument. Yes, the checks aren't free, but they aren't very costly either in practice, and much of the cost is down to C compilers not being optimized for generating range-checking code. I feel you're also possibly missing a longer term feedback loop issue. When software starts relying on things like overflow checks and zeroed memory, hardware manufacturers like Intel start building and optimizing hardware to make them faster. The "prefer performance over correctness" bias that's existed in the C world has been responsible for a lot of damage. We need to get out of it.
iopq 12y ago
Anyone know any other websites that talk about surprising things about C? I saw a really good one about syntax and semantics of C, but I lost it.
- GregBuchholz 12y ago
  You may like: http://www.gowrikumar.com/c/index.php http://www.gowrikumar.com/c/index.php ...and "Expert C Programming: Deep C Secrets": https://www.google.com/search?q=expert+c+programming+deep+c+secrets https://www.google.com/search?q=expert+c+programming+deep+c+...
- alecco 12y ago
  You might like SQLite's author take on C as a reliable language. http://www.sqlite.org/talks/wroclaw-20090310.pdf http://www.sqlite.org/talks/wroclaw-20090310.pdf If you forgive my plug, my bullet point summary plus copy-ready code for coverage testing examples: http://aleccolocco.blogspot.com.ar/2009/08/sqlite-lesson-in-low-defect-software.html http://aleccolocco.blogspot.com.ar/2009/08/sqlite-lesson-in-...
zvrba 12y ago
He forgot one: modifying & using a variable between sequence points (e.g., i = i++ + i++;). EDIT: not forgot, but only mentioned briefly at the end.
to3m 12y ago
Obligatory links: http://blog.metaobject.com/2014/04/cc-osmartass.html http://blog.metaobject.com/2014/04/cc-osmartass.html, http://robertoconcerto.blogspot.co.uk/2010/10/strict-aliasing.html http://robertoconcerto.blogspot.co.uk/2010/10/strict-aliasin... (I post these every time this subject crops up. TL;DR - stand up for yourself, and don't let your C compiler treat you like shit.)
- userbinator 12y ago
  See also the "maybe it's not the programmers who are being stupid, it's the language that is in need of change" proposal at http://blog.regehr.org/archives/1180 http://blog.regehr.org/archives/1180 with discussion https://news.ycombinator.com/item?id=8233484 https://news.ycombinator.com/item?id=8233484
  - lmm 12y ago
    There are better alternatives to C if a friendly language is what you want. C has its niche and I don't see C changing, and I've yet to see a "dialect" be really successful. If you're willing to switch to "friendly C", why wouldn't you already be switching to Rust, or D, or even Haskell?
    - coldtea 12y ago
      Because you might need the low-level, reasoning about memory/cpu/etc capabilities of C -- which Haskell doesn't offer -- but don't want all the undefined behavior and extra unneeded complexity. As of Rust, it's not even 1.0-stable out yet. Even if it was, it would take 4-5 years to mature, get enough support, tooling etc to be something you'd use instead of C/C++ for everything.
      - lmm 12y ago
        > Because you might need the low-level, reasoning about memory/cpu/etc capabilities of C -- which Haskell doesn't offer -- but don't want all the undefined behavior and extra unneeded complexity. Surely if you need that low-level reasoning then you do need unions and strict-aliasing, and for the kernel not to be doing extra zeroing of your memory when you didn't ask for it. If you were willing to accept a factor of 2 overhead, you wouldn't be using C at all.
spott 12y ago
I'm actually curious: is there anything in C that would prevent some sort of annotation being made part of the standard that essentially meant "Do what I say"? For example: [[dwis]] { if ( signed_int + another_signed_int < signed_int ) { // do stuff if there is an overflow } } Or something similar, which wouldn't be optimized away?

92 points by arunc 12y ago

youngthugger 12y ago

This makes C look like it's full of land mines. The minesweeper language.

nnhtzgf 12y ago
The title is making it seem as the listed behavior is largely unknown to C programmers. It isn't. And it is in the manual.
- ape4 12y ago
  Yeah, hopefully if you are programming in C you'll know using an undefined variable is bad.
  - nnhtzgf 12y ago
    Yep. :-) The article doesn't make much sense at times: Dereferencing a NULL Pointer: >>>contrary to popular belief<<<, dereferencing a null pointer in C is undefined. I guess C programmers are not counted in the "popular" group.
    - stormbrew 12y ago
      I don't think it's wrong at all. I suspect if you asked a large number of even moderately competent C programmers, you'd find quite a lot of them believe it's defined to trap. That's exactly what the rest of it is talking about. Never mind the issue raised of how it potentially affects optimization, which I think very few C programmers would actually be able to speak to.
    - bhaak 12y ago
      Are you really sure that most C programmers know about this? IMO, if you'd ask the question "what happens if you dereference a NULL pointer", most C programmers would answer "a segfault".
      - nnhtzgf 12y ago
        If you are in a company that codes largely in C, do a pool and report back. I'm honestly curious.
        bhaak 12y ago
        Unfortunately, I don't work in such a company. A completely unrepresentative poll among two C++ programmers showed that 50% thought it was a segfault, whereas the other 50% suggested it could be undefined behavior, deducing that from the knowledge that the behavior is different in Linux userspace and kernel. Although I wouldn't call those two typical C++ programmers. They do security CTFs for fun, so they might have a broader knowledge about exploitable behaviors in C/C++ code.
      - unwind 12y ago
        That, of course, would be a leading question. Most people would probably tend to assume that you're asking "in the environment in which we're programming, what happens if you dereference NULL?" which is not what you meant. I didn't poll my coworkers either, but at least I think the answers would be different if I asked "what does C say should happen when a NULL pointer is dereferenced?". Perhaps that's just wishful thinking on my behalf, though. :)
        ghswa 12y ago
        I don't think that's particularly wishful thinking, however, your alternative phrasing could be seen as equally, if not more, leading. When quizzed on the behaviour of a C program I would expect a careful and experienced C programmer to consider the behaviour described by the standard (which standard?); unspecified and implementation defined behaviours; and deviations from the standard in both the compiler and the environment. Often I'd expect the answer to be "it depends".
    - legulere 12y ago
      For a while you were actually able to mmap the memory at address NULL under linux, which is usable for exploiting (you can pass arbitrary data to kernel code that you can bring to dereference null pointers)
      - orbifold 12y ago
        Wouldn't virtual address translation prevent this from happening?
Lrigikithumer 12y ago
Well I mean it kind of is. C's my favorite language but it's like a machete, simple enough, fairly easy to use and useful in a lot of places but if you don't know how to use it properly it's very easy to cut yourself with.
FooBarWidget 12y ago
In a certain way, it is. But for a lot of things C and (C++) is still necessary, and the fact that C is not safe is a requirement. Can you imagine the Linux kernel not being written in C? Just imagine all the overhead if the Linux kernel has to perform overflow checking after every internal integer increment. Maybe some day Rust will be a good candidate, but that day hasn't come yet.
- pjmlp 12y ago
  > Can you imagine the Linux kernel not being written in C? No, only because the Linux kernel started as an UNIX clone and C is the UNIX's systems programming language. Other OS were written in another systems programming languages. Before and after C got invented. C only became widespread because a few successful startups in the 80's adopted UNIX for their workstation OS.
  - pmalynin 12y ago
    In the OS-programming world C/C++ is staple.
    - pjmlp 12y ago
      I am old enough to remember when it wasn't even an option.
  - ArkyBeagle 12y ago
    I'd say that in terms of seats sold, Borland and Microsoft had more to do with it. Maybe first one then the other. But 'C' was the semi-official language of the old DOS PC world from 1985 until Linux happened. It's somewhat tragic, because Modula-2 and Ada toolchains existed, but were harder to use than 'C' for performance and memory constraint reasons ( not to mention the toolchain costs ) Borland started out with Pascal, but went to 'C'.
    - pjmlp 12y ago
      >Borland started out with Pascal, but went to 'C'. As someone that used all their products before they became Imprise or how it was called, I never saw this happening. Even nowadays C++ Builder actually uses the Delphi framework. EDIT: Just wanted to add that in my part of the planet, it was all about Assembly for systems programming, games and real applications. Turbo Pascal for general purpose applications without much performace issues and CLIPPER for the typical CRUD applications.
jokoon 12y ago
It's not the 80s anymore, there area many cases where there are much better alternatives to C. But it's still a language you need for certain things, like responsiveness or real time applications. Like linus torvalds said it, C is like a more comprehensive assembly generator than anything else. Having such a language allows you to change every little boring details to do exactly what you want. It has its use, and it's surely not intended to be mainstream as of today. But there are many cases where C comes back as a constant common denominator. It's sure that it's kinda shitty that there's nothing really better than a language designed in the 70s, but I guess you could either blame language designers for not making anything better. There also are business/political issues and preferences for certain things that comes into play. All in all, C just seems to be pretty good when you want to have code that works well with systems. Maybe there's just inertia as to where computer development is headed, which might be heavy related to C. All I wish, is someday to be able to learn some more "virtuous" language like haskell, scheme, or smalltalk.
- pjmlp 12y ago
  The 80's already had Mesa and Modula-2, but then those guys at Berkley had to create a few successful startups.
- anon13839 12y ago
  You may want to look up the age of those "virtuous" languages you listed. While it's fun to bash C, you could drive a damn yacht through the undefined behavior in Scheme. High-level languages are not immune here.
username223 12y ago
I recommend reading the articles linked elsewhere in this thread: https://news.ycombinator.com/item?id=8879948 A lot of this is C compiler writers using the spec to perform counterintuitive "optimizations" (with dubious performance value), and forgetting that they're writing software for C programmers to use, not trying to win a pedantry war with them.

bhaak 12y ago

Some of the "performance penalties" look like FUD to me. At least to the extent implied in the article. For example "use of an unitialized variable". Given that the compiler already has to analyze the program flow to determine if the variable could be use unitialized to output that warning, the optimization step to not initialize those variables that are written to before used doesn't seem so far away. This happens at compile time. But, there's also the fact that a modern OS gives you heap memory zeroed mostly for free. A call to calloc is highly optimized nowadays. It needs to be for security reasons, as a program must never receive memory with data left from a different process. Likewise the signed integer overflow check. Modern CPUs are so superscalar internally that an overflow check (which doesn't get flagged most of the time anyway) doesn't add anything to the execution speed. And finally we are able to use that without resorting to assembler. GCC 5 has added support for built-in functions for arithmetics with overflow checking.

haberman 12y ago
> But, there's also the fact that a modern OS gives you heap memory zeroed mostly for free. A call to calloc is highly optimized nowadays. It needs to be for security reasons, as a program must never receive memory with data left from a different process. Memory often is coming from previously-freed heap space within the process. There is no security need to zero such memory. Zeroing it wouldn't even help, because the process is free to read the memory before malloc returns it. And calloc, no matter how optimized it is, will still be more expensive than malloc which doesn't have to zero the memory. > Modern CPUs are so superscalar internally that an overflow check (which doesn't get flagged most of the time anyway) doesn't add anything to the execution speed. "Doesn't add anything" is not correct. These checks are not free: http://danluu.com/integer-overflow/ http://danluu.com/integer-overflow/ https://news.ycombinator.com/item?id=8765714#up_8765979 https://news.ycombinator.com/item?id=8765714#up_8765979
- barrkel 12y ago
  Zeroed memory should not be particularly expensive on modern machines, because we have a lot of parallelism available to zero it off working threads. There might be some constraint on memory bandwidth, but IMO memory latency is a bigger issue, usually preventing maximizing memory throughput, so there should be some spare. The danluu link doesn't really support your argument. Yes, the checks aren't free, but they aren't very costly either in practice, and much of the cost is down to C compilers not being optimized for generating range-checking code. I feel you're also possibly missing a longer term feedback loop issue. When software starts relying on things like overflow checks and zeroed memory, hardware manufacturers like Intel start building and optimizing hardware to make them faster. The "prefer performance over correctness" bias that's existed in the C world has been responsible for a lot of damage. We need to get out of it.

iopq 12y ago

Anyone know any other websites that talk about surprising things about C? I saw a really good one about syntax and semantics of C, but I lost it.

GregBuchholz 12y ago
You may like: http://www.gowrikumar.com/c/index.php http://www.gowrikumar.com/c/index.php ...and "Expert C Programming: Deep C Secrets": https://www.google.com/search?q=expert+c+programming+deep+c+secrets https://www.google.com/search?q=expert+c+programming+deep+c+...
alecco 12y ago
You might like SQLite's author take on C as a reliable language. http://www.sqlite.org/talks/wroclaw-20090310.pdf http://www.sqlite.org/talks/wroclaw-20090310.pdf If you forgive my plug, my bullet point summary plus copy-ready code for coverage testing examples: http://aleccolocco.blogspot.com.ar/2009/08/sqlite-lesson-in-low-defect-software.html http://aleccolocco.blogspot.com.ar/2009/08/sqlite-lesson-in-...

zvrba 12y ago

He forgot one: modifying & using a variable between sequence points (e.g., i = i++ + i++;). EDIT: not forgot, but only mentioned briefly at the end.

to3m 12y ago

Obligatory links: http://blog.metaobject.com/2014/04/cc-osmartass.html http://blog.metaobject.com/2014/04/cc-osmartass.html, http://robertoconcerto.blogspot.co.uk/2010/10/strict-aliasing.html http://robertoconcerto.blogspot.co.uk/2010/10/strict-aliasin... (I post these every time this subject crops up. TL;DR - stand up for yourself, and don't let your C compiler treat you like shit.)

userbinator 12y ago
See also the "maybe it's not the programmers who are being stupid, it's the language that is in need of change" proposal at http://blog.regehr.org/archives/1180 http://blog.regehr.org/archives/1180 with discussion https://news.ycombinator.com/item?id=8233484 https://news.ycombinator.com/item?id=8233484
- lmm 12y ago
  There are better alternatives to C if a friendly language is what you want. C has its niche and I don't see C changing, and I've yet to see a "dialect" be really successful. If you're willing to switch to "friendly C", why wouldn't you already be switching to Rust, or D, or even Haskell?
  - coldtea 12y ago
    Because you might need the low-level, reasoning about memory/cpu/etc capabilities of C -- which Haskell doesn't offer -- but don't want all the undefined behavior and extra unneeded complexity. As of Rust, it's not even 1.0-stable out yet. Even if it was, it would take 4-5 years to mature, get enough support, tooling etc to be something you'd use instead of C/C++ for everything.
    - lmm 12y ago
      > Because you might need the low-level, reasoning about memory/cpu/etc capabilities of C -- which Haskell doesn't offer -- but don't want all the undefined behavior and extra unneeded complexity. Surely if you need that low-level reasoning then you do need unions and strict-aliasing, and for the kernel not to be doing extra zeroing of your memory when you didn't ask for it. If you were willing to accept a factor of 2 overhead, you wouldn't be using C at all.

spott 12y ago

I'm actually curious: is there anything in C that would prevent some sort of annotation being made part of the standard that essentially meant "Do what I say"? For example: [[dwis]] { if ( signed_int + another_signed_int < signed_int ) { // do stuff if there is an overflow } } Or something similar, which wouldn't be optimized away?