22 ms·

Pointers Are More Abstract Than You Might Expect in C

269 points by ognyankulev 8y ago

gilgoomesh 8y ago
This scratches the surface of why I hope C slowly fades away as the default low-level language. C sounds simple when you look through K&R C. C lets you feel like you understand the stack, ALU and memory. A pointer is just an integer and I can manipulate it like an integer. But the reality is filled with a staggering number of weird special cases that exist because memory doesn't work like a simple flat address space; or the compiler needs to optimise field layouts, loops, functions, allocations, register assignments and local variables; or your CPU doesn't use the lower 4 bits or upper 24 bits when addressing. C has no way to shield common language constructs from these problems. So everything in the language is a little bit compromised. Program in C for long enough and you'll hit a lot of special special cases – usually in the worst way: runtime misbehavior. Type punning corruption, pointers with equal bit representations that are not equal, values that change mysteriously between lines of code, field offsets that are different at debug and release time. When using fundamental language constructs, we shouldn't need to worry about these ugly special cases – but C is built around these ideas. The need to specify memory layouts, direct memory access and other low level access should be gated by barriers that ensure the language's representation and the machine representation don't tread on each other's toes. Rust has a long way to go but is so much more robust at runtime. I think there's room for other languages to occupy a similar space but they're need to focus on no-std-lib no-runtime operation (not always the sexiest target).
- mpweiher 8y ago
  > the compiler needs to optimise ... wants to. There fixed that for you ;-) The weird special cases were largely(1) introduced recently by optimizer writers hijacking the standard in order to soften the semantics so that previously illegal optimizations would now be legal, by simply declaring the vast majority of existing C code as "undefined" and up for grabs. Which is why the Linux kernel, among others, has to set special flags in order to get back to a somewhat sane variant. (1) Except for those special architectures where the hardware is that weird, but in that case you typically know about it.
  - zodiac 8y ago
    What does "softening" the semantics mean? Aren't they pretty unambiguously defined by the standard?
    - mpweiher 8y ago
      Newer versions of the standard.
  - jcranmer 8y ago
    What changes are you thinking of? The only change with respect to undefined behavior I can think of off the top of my head is punning via unions, which C99 changed from undefined behavior to well-defined behavior.
    - mpweiher 8y ago
      According to http://www.complang.tuwien.ac.at/kps2015/proceedings/KPS_2015_submission_29.pdf http://www.complang.tuwien.ac.at/kps2015/proceedings/KPS_201... new undefined behaviors were added, and for significant bits. I think the bigger issue is that compiler writers have taken "undefined" to mean complete liberty to do whatever the hell they want, even fairly far away from the actual location of the undefined behavior. And it turns out that the C89 standard had a range of "permissible" actions on undefined behavior. To me "permissible" is restrictive, as in you are not allowed to do anything outside that range. Newer standards kept that section (3.4.3) the same, word for word, except for changing "permissible" to "possible". To me, "possible" is not restrictive but illustrative. Now you might think this is reading tea-leaves on my part, and you might be correct, but on the other hand that change of interpretation is exactly what happened. Compilers used to not do these things, and nowadays compilers do do these things, and compiler writers cite the standard as giving them permission. Also consider to what extent wording of standards is finessed and fought over, and that exactly this one word was changed, in a section that otherwise remained the same word for word.
  - xenadu02 8y ago
    > The weird special cases were largely(1) introduced recently by optimizer writers hijacking the standard in order to soften the semantics so that previously illegal optimizations would now be legal, by simply declaring the vast majority of existing C code as "undefined" and up for grabs. Citation needed. Signed overflow being undefined behavior was a direct consequence of different hardware representations of numbers. Inserting the instructions required to catch overflow and make it wrapping would be expensive, at least on some systems. The fact that arrays decay to pointers, C has no dynamic array support, and so on are decisions that go way back and more or less prohibit array bounds checking. After preprocessing, inlining, and various forms of optimization it can be very far from obvious that some random NULL check after the pointer has already been dereferenced is intentional and shouldn't be removed. Things like overlapping pointers have always had undefined behavior in the sense that inserting the instructions to check for the overlap all over the place would have a large performance impact, no one would use such a compiler, and thus there is no market for one. The standards committee largely documented existing practices rather than inventing new undefined behaviors. I keep seeing people like you claim we should just "fix" undefined behavior or the standards committee gleefully inserted undefined behaviors because they hate programmers. If you actually sat down and disassembled the C code you write then went through the exercise of inserting instructions to eliminate most types of undefined behavior I suspect it would be a very illuminating experience. I also doubt you'd be so certain of your position.
- ChrisSD 8y ago
  But as you know, CPU ISAs are designed for C programs and compilers are optimized for C programs. So everything, even new languages like Rust, have to buy into C's model to some extent. Truly getting out from under C's shadow is going to be very difficult. Maybe better languages are a first step on that path but they are only a small step.
  - pjmlp 8y ago
    Modern CPUs, but it used to be that they were designed for any high level language before C took over the show.
  - jstimpfle 8y ago
    That is said a lot, but what are actual desicions that are made to conform to the C model? What would be good ideas in CPU design that aren't made since they are not compatible with C? (I get it that CPUs have to support some common paradigms and use cases. For example, virtual memory / process isolation, maybe branch prediction things, or support for calling conventions. However, I don't think that is specific to C).
    - brandmeyer 8y ago
      Support for unaligned memory accesses in hardware is one. The exceeding majority of variable accesses are through pointers of the correct type, and are therefore to variables that are at least naturally aligned. But a very small handful of systems/procedures make unaligned accesses through type-punned pointers. Rather than just crash on these programs, processors will perform multiple memory accesses and stitch the results back together in hardware (worst case behavior: accesses that cross page boundaries). So there's extra hardware to perform this extra state tracking and detect the unaligned accesses in the first place. Hardware doesn't just get turned off for conforming programs. Its there consuming energy and giving off heat on all of them. Edit/addendum Here's another example: What should hardware do on integer overflow? There are several options, including signed-saturation, unsigned saturation, unsigned wraparound, and trap. The semantics of C are such that most hardware just picks 'unsigned wraparound' and makes programs eat the consequences. If you're very lucky then you also get 'an overflow bit, that gets clobbered on the next instruction'. One way to think about this problem might be to ask the question, "What hardware features would reduce the runtime cost of instrumenting programs with the various sanitizers that are becoming available?"
- nostalgeek 8y ago
  The following is purely my opinion. Rust might have gone too far the other way. Yes it strives to be a modern language, with a lot of functional programming features and "OOP" done right (aka no inheritance, simply method polymorphism). To me Rust is more a C++ replacement than a C replacement. A replacement for C should try to be as simple as possible while fixing C weak typing mess with strong typing for instance, including when it comes to pointers (for instance like in Ada where you can't just take the address of anything you want, you need to declare it as aliased to begin with). In fact Ada tries hard to make pointers redundant which is a great thing. This and range types and bound check arrays( Ada has a lot lot of great ideas, and "C style" Ada is a treat to use). So a C replacement should keep the procedural nature of C, with closures perhaps, but not go further in terms of paradigm. C aficionados often claim they love C because it's "simple"(it isn't) That's what they mean, they love C because you can't really do OO with it. On the other hand, Rust macros are exemplary of what a good macro system should be.
  - icedchai 8y ago
    I've been programming C on and off for 30 years. Rust feels too complex to be a C replacement. C++, yes... C's paradigm is really about working with memory. Anything that restricts memory access with bound or type checking is going to feel like "not C"...
    - bitwize 8y ago
      C doesn't have anything to say about whether bounds are checked or not. That's why so many things, including pointers to arbitrary locations in memory, are undefined: the standard was built to accommodate both embedded system implementations where scribbling all over RAM is desirable, and implementations that do things like bound checking.
  - pjmlp 8y ago
    I think it will only happen the way of Luddites in systems programming. Even if redo Modula-2 with curly brackets I am not sure it would pick up.
  - deleted 8y ago
- pjmlp 8y ago
  On top of that C just got here thanks to widespread of UNIX and POSIX, and we had so much better options. Yes they were languages with helmet, seat belts and airbag bolted on, but just like old technology cars on a motorway crash, the CVE database shows what happens when things go wrong.
- tomp 8y ago
  > memory doesn't work like a simple flat address space Can you expand on that? I thought that modern (non-segmented) memory does work like a simple flat address space, at least from the perspective of userspace programs. Isn't all the weirdness only a result of compiler optimisations?
  - thechao 8y ago
    Just to take one example: multisocket x86 is NUMA. There’s HW to paper over that fact, but the power/perf cost is quite high.
    - mastax 8y ago
      Also Epyc and (even more so) Threadripper have somewhat NUMA memory access in a single CPU.
- bradleyjg 8y ago
  These architectural bizarrities exist and have to be dealt with by someone. Of course it is better if that someone isn’t you—if some rust maintainer ports the language to the obscure hardware where the CPU ignores the lower four bits when addressing and presents nice, neat, and performance abstractions for you to use. But who is going to pay for that effort? If the hardware manufacturer cared enough to to invest a lot in developer tooling it probably wouldn’t have picked such a hostile interface to begin with.
- sureaboutthis 8y ago
  Some of your complaints could be said about assembly,too, but are you wanting to get rid of that, too? Some of your other complaints are about how processors and other hardware work, not C. In many ways, you are asking for C to be like other languages but then it wouldn't be C and all the advantages it does have. If you get rid of C, you get rid of most of the software your operating system runs on and that begs the question you need to ask yourself: if C is so bad, why is everyone using it?
  - bluejekyll 8y ago
    > if C is so bad, why is everyone using it? Don't confuse success for correctness, survivorship bias is something to consider here. As others will surely point out, when C was written, there were other languages available at the time that were as fast, but much safer. In many ways C won, because Unix won, to talk about them separately ignores how important both were to each others success. (Yes, other operating systems adopted C as well, but Unix is it's foundation)
  - pjmlp 8y ago
    Some mainframes like Unisys ClearPath don't have Assembly support at all, zero, nada, all necessary low level primitives are exposed as compiler intrisics.
- jcranmer 8y ago
  C sort of has this mindset built around it that you're writing portable assembly, except for messy bits like stack frame layout or register allocation. But that doesn't really hold true anymore, and arguably hasn't for decades. The first obvious issue is that C is specified by an abstract machine that doesn't really correspond to actual hardware. There's no concept of segmented memory, or multiple address spaces in C. Traps don't quite work the way you'd want in C [1]; there's no way to catch a trap that occurs in a specific region of code. Worse, the abstract machine leans on a definition of undefined behavior in such a way that it's painful for programmers but not useful enough to make automatic security permissible. (Unsigned overflow, which is going to be a more common cause of overflows that lead to security vulnerabilities, is well-defined). However, the more real problem with C is that it is outright missing hardware features that you'd like access to. Things like rotate instructions or wide multiply are easy enough to pattern match, so their lack isn't so devastating. But flag bits--particularly add with carry or checked overflow operations--aren't figured into at all. SIMD vectors have no conception. Returning multiple values via registers. Traps [1]. Coroutines. Multiple address spaces (especially important for GPUs). [1] Admittedly, the OS is also to blame here: POSIX signals are a really atrocious way to handle machine-generated faults.
  - nwmcsween 8y ago
    > flag bits That's sort of a processor specific implementation detail, IIRC you can access them via GNU extensions. > SIMD vectors Not having SIMD IMO is a feature as doing useful things will expose processor specific details, you can use SWAR to do some of SIMD.
    - jcranmer 8y ago
      Exposing processor-specific details is the desired feature, not a bug. The people who want to program C for portable assembly do so because they want to take advantage of things they know about the architecture: the same source code isn't really portable across different architectures without #ifdef anyways.
      - p1esk 8y ago
- MaxBarraclough 8y ago
  > I think there's room for other languages to occupy a similar space but they're need to focus on no-std-lib no-runtime operation (not always the sexiest target). You're describing the Zig language! It aims to be as fast as C, and unlike most languages that say this is a goal, it means it. There's no "almost as fast as C if you ignore the garbage-collector and the array-bounds-checking", it's actually as fast as C. Its author values the minimalism of C, but wants to create a far better language for doing that sort of work. [0] They're doing surprisingly well. They even managed to make a faster-than-C (faster even than hand-tuned assembly) SHA-2 [1] [0] https://andrewkelley.me/post/intro-to-zig.html https://andrewkelley.me/post/intro-to-zig.html , https://github.com/ziglang/zig/wiki/Why-Zig-When-There-is-Already-CPP,-D,-and-Rust%3F#first-class-support-for-no-standard-library https://github.com/ziglang/zig/wiki/Why-Zig-When-There-is-Al... [1] https://ziglang.org/download/0.2.0/release-notes.html https://ziglang.org/download/0.2.0/release-notes.html , it's also mentioned somewhere in this talk https://youtu.be/Z4oYSByyRak https://youtu.be/Z4oYSByyRak
  - kbenson 8y ago
    > faster even than hand-tuned assembly I think you and I are working off different definitions of what that means, as my definition doesn't really allow for a faster implementation (unless there's some really spooky stuff going on in the compiler). I suspect you mean faster than a popular hand tuned implementation.
    - MaxBarraclough 8y ago
      Obviously Zig's output (courtesy of its LLVM backend) can be expressed and distributed as assembly code, but it's not hand-tuned, it's compiler-generated. Whether the hand-tuned assembly code they used was the fastest out there, I'm not sure. I'd hope so, or they're being misleading. I might get time to re-watch the relevant part of the video - it's around the 21 minutes mark - https://youtu.be/Z4oYSByyRak?t=1292 https://youtu.be/Z4oYSByyRak?t=1292
      - isaachier 8y ago
        Andrew explains in the video that the speedup is mostly due to compile-time speed ups and potentially the use of the rorx instruction. It cannot be faster than the fastest assembly because the by definition is the fastest assembly.
- CyberDildonics 8y ago
  I feel like C++11 and Rust have shown a way forward and the C standard should focus on a design that takes into account ownership and protected arrays in a simple way, hopefully as backwards compatible as possible. Maybe the solution is for the major compilers to default to strong static analysis. Even though there are safe languages, the enormous integration and compatibility of C means that it isn't going anywhere soon, so I think even imperfect solutions can offer a lot if they are low hanging fruit.
  - pjmlp 8y ago
    It is a matter of mindset, multiple safe C variants have been attempted throughout the years. Even lint was already available in 1979, yet hardly anyone used static analysis tools on C codebases.
    - CyberDildonics 8y ago
      Which is why I mentioned defaulting to strong static analysis. Having it as a separate tool or an unknown flag is not enough.
- bitwize 8y ago
  > A pointer is just an integer and I can manipulate it like an integer. Except it's not. Not unless you cast it to uintptr_t. Doy. :) Understanding the underlying asm can give you context for what a pointer is, but you can't assume that a pointer is an integer, any more than you can assume 'int' is a machine word in length. There are reasons why C seems so abstruse and abstract. It has to exist on a huge variety of architectures and environments, from supercomputers to tiny embedded microcontrollers. Fun fact: The Lisp Machine had a C compiler! In Lisp Machine C, pointers were fat: they consisted of a machine pointer to an allocated chunk of memory and an offset within that chunk. And of course it was invalid to have a pointer outside an allocated object; the machine would throw an exception if you even tried. Hence why C pointers are only well-defined over valid references to objects and arrays, one past the end of an array, and NULL. And lest you say, "well no modern cpu works like that anymore", there are programs out there written for the Burroughs architecture that are still running. Are you going to cut off the maintainers of those programs because the architecture they're coding for doesn't resemble an x86? This is part of why I disdain the call for modernity in programming. "Modern" means "limited". "Modern" means "I have no intention of testing this on anything but my laptop, so I will make assumptions that suit my use case and fuck you if yours differs". C is a pain in the ass, but it becomes more approachable if you take into account its history. And it still fits in your head more easily than Rust.
  - hexane360 8y ago
    The obvious flipside of this is that those C programs written for Burroughs never had any chance of running on anything but Burroughs. The only difference is that C offloads the effort of portability onto the user, whereas other languages at least attempt to make it portable in the compiler.
    - pjmlp 8y ago
      What?! Since when did Burroughs had a C compiler? NEWP was and is more than enough, while being much safer.
      - bitwize 8y ago
- Koshkin 8y ago
  Road to Rust is paved with good intentions. (Same is true about C++.)
- andrewmcwatters 8y ago
  Is there a site or book out there that talks about these common pitfalls? I work in C89 so K&R C does seem pretty simple to me. I clearly haven't worked with it long enough to spot these things, or I have code in the wild that unknowingly is victim to these issues and I've yet to stumble on it.
- oppositelock 8y ago
  I've been writing C on and off since 1990, across embedded systems, game consoles, massive SGI supercomputers, and modern kernels. C is not fading away because of it's portability and low overhead. It has some nasty behaviors, as the original post shows, but you hit those once in a blue moon. The language is not unsound because of it, it just has some warts like any other language. I don't particularly enjoy writing C, as it's too low level for my liking and I have to write too much code to do basic things (slist_* makes me shudder), but the reason that I have used it is due to its tremendous portability and direct compilation to machine code. That still matters in some places. In places where memory is precious, or realtime performance precludes things like memory managers, C will continue to thrive, beside languages like Rust which are great for different goals. There's room for both. C, in my opinion, has stood the test of time on its simplicity, portability, and the ubiquity of the C standard library, tiny as it may be.
joosteto 8y ago
When I compile the example, i get: 0x7ffd0b57ebd0 0x7ffd0b57ebd8 0 OK, so gcc reordered a & b; I'll fix this by chaning the initialisation of p and q to: int *p = &a + 1; int *q = &b; But when I now run the example, I get: gcc -o c c.c && ./c 0x7ffe55eb0aa4 0x7ffe55eb0aa4 1 gcc -O -o c c.c && ./c 0x7ffcfeffd914 0x7ffcfeffd914 0 So, p==q only evaluates to 1 if optimisation is enabled. $ gcc --version gcc (Ubuntu 7.3.0-16ubuntu3) 7.3.0
- psyklic 8y ago
  It shows the exact options the author used if you click on GCC: Version: 8.1.1 Options: -std=c11 -O1 Target: x86_64-redhat-linux
mpweiher 8y ago
clang is a bit more sane: > cc pointereq.c > ./a.out 0x7ffee83163b8 0x7ffee83163b8 1 > cc -O pointereq.c > ./a.out 0x7ffeeeb8b3b8 0x7ffeeeb8b3c0 0 So without optimization, the pointers are the same and compare as equal. With optimization, the pointers compare as not equal. At first that seemed horrible, until I saw the pointers actually are not the same. Since I don't recall any guarantees about stack layout, that seems perfectly fine. > cat pointereq.c #include <stdio.h> int main(void) { int a, b; int *p = &a; int *q = &b + 1; printf("%p %p %d\n", (void *)p, (void *)q, p == q); return 0; }
andyjohnson0 8y ago
Just a datapoint: VS2017 on Win10 x64 gives me 00AFF89C 00AFF894 0, which is what I naively expect.
- JdeBP 8y ago
  Actually, you should not expect that, because you don't have a guarantee of ordering of the addresses of a and b on the stack. Now try with /O2 . (-:
  - andyjohnson0 8y ago
    Maybe I'm misunderstanding you, but it's precisely because I have no knowledge of the stack frame's layout that I didn't expect the pointers to be equal. With /O2 optimisation I get: 00B1F99C 00B1F9A4 0
    - JdeBP 8y ago
      That's where you are going wrong. You have no guarantee that the compiler places a and b next to each other, let alone in a defined order with a at a higher address than b such that one integer width beyond the end of b you find a. Expecting one thing is wrong. You are expecting the one thing that the comparison always returns false. You should not expect anything. The comparison result could be true or false. H:\>cl /O2 pointereq.c Microsoft (R) C/C++ Optimizing Compiler Version 19.14.26431 for x86 Copyright (C) Microsoft Corporation. All rights reserved. pointereq.c Microsoft (R) Incremental Linker Version 14.14.26431.0 Copyright (C) Microsoft Corporation. All rights reserved. /out:pointereq.exe pointereq.obj H:\>.\pointereq 00CFFA20 00CFFA20 1 H:\>
      - mpweiher 8y ago
        You are confusing "did not expect them to be equal", which is what the OP wrote, with "expected them to be not equal", which is what you are railing against.
nuriaion 8y ago
Basically you have no guarantee where the pointer q is pointing to. Some compiler/static code analyzer will yell at you with this code.
- jerrre 8y ago
  isn't it guaranteed that is points sizeof(int) bytes higher than the address of b? Whether something useful is behind that address is another question
  - ChrisSD 8y ago
    From reading the spec it seems that is undefined behaviour for anything other than `&b + 1`. That is one past the end of b is fine but not `&b + 2` etc.
    - zubyak 8y ago
      I wonder why someone decided that accessing "one past the end" should be fine
      - jwilk 8y ago
        It's explained in the addendum to the article.
        zubyak 8y ago
        thanks
      - mcguire 8y ago
        Short answer: so you can walk a pointer down the elements of an array without having to have weird code to deal with the end.
      - deleted 8y ago
        [deleted]
  - 8y ago
foxhill 8y ago
the comparison at the start is nonsense - there is no specification for the ordering or location of stack variables. by taking the address of these variables, you could see that they actually are the same value, and so intuitively you’d think they might be the same, but a different compiler might put them in different locations. or they may be elided entirely through optimisation. it’s far safer to fail the equality test in this case - this is what the model specifies. this is not even the first example of this counter-initiative behaviour. imagine two floating point values with exactly the same bit-representation. it is possible, without any trickery for them to fail an equality check - i.e, they are both NaN. this is what IEE754 demands of a compliant floating point implementation. and indeed, it’s a sane choice when you understand why it was made. similarly, it’s perfectly reasonable for this example to fail.
- jameshart 8y ago
  Surely the point of writing an if-block testing the equality of the pointers is precisely because the code isn’t assuming that they are the same - but maybe it has something it wants to do if it finds itself running somewhere where they are. Writing code which assumed those pointers were equivalent would be bad, but this code doesn’t do that. Instead, it looks like the compiler assumes that the pointers won’t be the same and refuses to let you entertain the possibility at all.
  - maxxxxx 8y ago
    What's the point of even trying to compare these two pointers?
  - cozzyd 8y ago
    I haven't tested it but I imagine casting to uint64_t will make it work as you want.
  - foxhill 8y ago
    > Instead, it looks like the compiler assumes that the pointers won’t be the same and refuses to let you entertain the possibility at all. and indeed it should - it’s sane to use the address of a stack variable for the purposes of writing something into it, but assuming that it forms some sort of array in memory is sure to cause pain if you rely on that assumption.
bluecalm 8y ago
I like the behavior of the compiler here. There is no guarantee that a and b are next to each other in memory. That's why the comparison fails, the alternative makes is runtime/compiler/optimization level dependent which would be a total mess. As usual with those C bashing articles you won't run into trouble if you don't try very hard to write contrived code. I mean, the moment you see: int *q = &b + 1; on your screen alarm bells should go off. Doing pointer arithmetic on something that is not an array is asking for trouble. If the standard should be amended in any way it should be undefined behavior right away you do pointer arithmetic on non-array objects.
- jstimpfle 8y ago
  That would require making a distinction between array-pointers and non-array pointers. There, another can of worms. Probably not worth the additional complexity.
- mpweiher 8y ago
  Agree on the alarm bells with the pointer arithmetic. Disagree that gcc is doing the right thing here. The clang behavior (different comment in this thread) is much more sane: if the pointers happen to be the same, they compare as equal. If the pointers happen to not be the same, they compare as not equal.
  - shakna 8y ago
    Except clang's behaviour changes depending on the optimisation level. If you use -O1, then you get a different result.
    - mpweiher 8y ago
      Yes, because the pointers are different at -O. As there are no guarantees as to the relative placement of auto/stack variables relative to each other, that is perfectly fine.
      - shakna 8y ago
Kenji 8y ago
tl;dr to save you guys time: Somebody writes C code with undefined behaviour and is surprised when the behaviour is indeed undefined and thus erratic. Completely pointless blogpost because the author does not understand C.
tzahola 8y ago
There's nothing surprising in the first example. Comparing the addresses of stack variables is undefined behaviour. The second one is more interesting: extern int _start[]; extern int _end[]; void foo(void) { for (int *i = _start; i != _end; ++i) { /* ... */ } } GCC optimized "i != _end" into "true". The kernel guys fixed this by turning "_start" and "_end" into "extern int*". I always thought [] was just syntactic sugar over a regular pointer, but seems like I was wrong.
- JdeBP 8y ago
  Actually, the headline article tells us that it optimized it to true. That affected some of my code from years ago, too; and another fix is to keep _start and instead of having another array called _end have an external size_t giving the number of elements of start and loop while i!=_start+count . In fairness to the compiler people there shouldn't really be anything surprising in the second example, either. There are x86 memory models (compact and large) where that loop never terminates in the general case, because the two arrays are not in the same data segment and no amount of incrementing a (far but not huge) pointer will get from one to the other. So it's not the case that this is a new pitfall. It was a pitfall decades ago. People just thought that it didn't exist with modern tiny (a.k.a. flat) memory models, because the old explanation wasn't applicable. It does, but with a different explanation.
  - tzahola 8y ago
    Yup, I've mistyped "true" as "false". If you set "_start" and "_end" to addresses of different objects, then you're invoking undefined behaviour and all bets are off. But this should be perfectly valid IMO: extern char* _start; extern char* _end; void foo() { for (char* c = start; c != end; c++) ... } int main(int, char**) { char buf[128]; _start = &buf[0]; _end = &buf[128]; foo(); } Right?
    - JdeBP 8y ago
      The thing there is indeed not the validity, but the introduction of extra memory references and variables. The code with the arrays compiles to immediate value loads with load-time fixups to the code; and if this is linking to an assembly language module the necessary assembly language is just some labels against the data. The code with the pointers compiles to loads from data memory, with (in the more usual paradigm where the start and end are bracketing some vector of initialized data in the program image) load-time fixups to the initial values of the variables; and the assembly language has to be different, some variables with these labels whose initial values are the addresses of some other labels against the actual data. Notice that the Linux Kernel developer in 2016 did not immediately appreciate the need to modify the concomitant assembly language and linker script stuff to match the change in the C language code. The approach that I mentioned keeps the immediate value for the start, and uses a load from initialized data memory for the count. Other approaches are possible, but they need more work in the build process. For example: By generating the C/C++ language declaration and definition from a script or suchlike, one could calculate and declare the number of elements in the array, making it possible to use sizeof.
tzs 8y ago
Note that if the two pointers are passed to a function, and the comparison is done in the function, the results are different: #include <stdio.h> void pcmp(int *p, int *q) { printf("%p %p %d\n", (void *)p, (void *)q, p == q); } int main(void) { int a, b; int *p = &a; int *q = &b + 1; printf("%p %p %d\n", (void *)p, (void *)q, p == q); pcmp(p, q); return 0; } That is giving me: 0x7ffebac1483c 0x7ffebac1483c 0 0x7ffebac1483c 0x7ffebac1483c 1 That's compiled with '-std=c11 -O1' as in the article. The result is the same of pcmp is moved into a separate file so that when compiling it the compiler has no knowledge of the origins of the two pointers. I don't like this at all. It bugs me that I can get different results comparing two pointers depending on where I happen to do the comparison.
- JoeAltmaier 8y ago
  Yes, I agree. It makes far more sense to make guarantees in a language, that two values are compared using a metric that is invariant. Especially in a language like C, where we expect it to be a very thin abstraction over the machine.
- deleted 8y ago
  [deleted]
- ash_gti 8y ago
  Did you use gcc or clang or...? When I ran this locally I get: 0x7ffeee2a4808 0x7ffeee2a4810 0 0x7ffeee2a4808 0x7ffeee2a4810 0 With -O0 I get 1/1 instead of 0/0 so this appears to be a compiler optimization.
- quelltext 8y ago
  Yup, what is described in the article looks like a bugged optimization to me were the runtime behavior differs from the static model used for the optimization. I am sure the actual runtime semantics is in fact to compare addresses but the optimization claims to know (falsely) enough about the pointers to determine they must be not equal. I don't understand the people in this comment section defending this. The cited spec gives no evidence that this should happen (I fail to see the undefined behavior here). It specifies when pointers are equal, and both pointers ended up pointing to the same object, so they should be equal. Just because the compiler doesn't know, shouldn't mean it can replace an equality check with false. The spec didn't say "also, they are not equal in case the conpiler thinks that they proooobably aren't pointing to the same object". I do agree that one should avoid this type of pointer arithmetic, but that doesn't excuse the confusing behavior of this primitive operstion. One should at least argue for the behavior to be consistent.
_RPM 8y ago
if you're adding 1 to a pointer, why would you expect it to be equal? I would not expect it to be equal.
- armitron 8y ago
  He's adding "1" to the pointer that points to the variable "b" which in this case is allocated immediately below "a" in the stack. Thus adding one will make the two pointers equal.
  - saagarjha 8y ago
    The layout of things on the stack is completely implementation-defined.
    - sannee 8y ago
      It has been a while since I read the standard, but this seems well into the realm of undefined behavior, not implementation defined.
      - saagarjha 8y ago
        From my understanding, how the stack is laid out is implementation-defined; attempting to actually observe this ordering in your program is undefined behavior.
crehn 8y ago
Two pointers compare equal if and only if both are null pointers, both are pointers to the same object (including a pointer to an object and a subobject at its beginning) or function, both are pointers to one past the last element of the same array object, or one is a pointer to one past the end of one array object and the other is a pointer to the start of a different array object that happens to immediately follow the first array object in the address space. Can someone explain to me the rationale behind this? Why not just "two pointers compare equal if they point to the same address"?
- Kenji 8y ago
  The rationale behind this "mess" is precisely that people cannot write code like in the first example of the blog post, effectively relying on undefined behaviour, and have this ugly code work most of the time.
- isaachier 8y ago
  My guess is type based alias analysis. The classic example is casting an int to a float like so may crash your program due to undefined behavior: float my_bad_function(void) { int x = 5; float* y = (float) &x; return y; }
- JdeBP 8y ago
  Because (a) pointers do not point to addresses, and (b) now you have to define the concept of addresses being equal in the language standard and haven't actually reached the goal. (-:
  - Sharlin 8y ago
    Indeed; pointers intentionally abstract away memory addresses and addressing, which, as the article states, can be much nastier business than the benign flat address space of i386 protected mode.
- thatcks 8y ago
  The C standard was (and is) written so that it allowed for machines that didn't have simple pointers; a highly relevant example for the time was x86 machines in real or protected mode using segment registers. Even when such environments have accessible linear address spaces under the hood, comparing the absolute address of two pointers requires some extra work (you have to reassemble the absolute address from the segment + offset pair). In the grand tradition of allowing the most efficient implementation possible, the C standard says that implementations don't have to do this work; they can simply compare pointer values directly, provided that they insure that pointers to the same object work (eg use the same segment + offset pair, which they normally naturally will). The standard was also written to allow for C interpreters, where you may not have an underlying linear memory model at all and all pointers are represented internally in some complex way (for example 'object ID + offset inside object'). Here you don't have any particular idea of 'an address' as a distinct thing and so you can't naturally compare two pointers to different objects in any meaningful way.
Sharlin 8y ago
> ...or one is a pointer to one past the end of one array object and the other is a pointer to the start of a different array object that happens to immediately follow the first array object in the address space. I was not aware of this special case. What's the rationale? Is there even a way in standard C to guarantee that two array objects are laid out in memory like that, with no padding?
- mcguire 8y ago
  Start with one large array and subdivide it into two segments.
- saagarjha 8y ago
  A two-dimensional array?
  - Sharlin 8y ago
    Yeah, good point.
oconnor663 8y ago
> If we step back from the standard and ask our self does it make sense to compare two pointers which are derived from two completely unrelated objects? The answer is probably always no. The one big counterexample I can think of is the difference between memcpy and memmove. The latter is supposed to be able to do arithmetic on memory regions, to see if they overlap. Is this article saying that the standard C implementation of memmove is relying on unspecified behavior?
- coldnose 8y ago
  Another example of arbitrary pointer comparison I'm familiar with is for resolving lock ordering problems. If I have a big set of locks that I need to take to complete a particular operation, a simple way to prevent deadlocking is to acquire those locks in the order of their pointers. E.g. if I have three locks I need to take, lock_t *a = (lock_t *)0x100; lock_t *b = (lock_t *)0x300; lock_t *c = (lock_t *)0x200; then I'll take them in order a, c, b;
  - bluecalm 8y ago
    If order of locks is important maybe it's worth considering putting them in an array.
    - oconnor663 8y ago
      Or maybe keeping a global atomic counter, and reading a unique (to the process) value out of it every time you create a mutex?
- slrz 8y ago
  Memmove is frequently cited as an example of a standard library function that can't be implemented using only standard C. It's incorrect, though: the way to do memmove using standard C is to use malloc to allocate a temporary buffer. But the whole question is not terribly relevant. When memmove is provided as part of the C implementation it can rely on non-portable platform behaviour just fine. There's no rule that you have to implement libc using only standard C facilities.
mabynogy 8y ago
Not related to the content but I find the colored links in the nav a simple and very good idea.
bsder 8y ago
Ummm, when I run this on gcc 7.3.0 on OS X I actually get: 0x7fff5dbd89fc 0x7fff5dbd89fc 1 Which kind of shoots the whole article in the foot ...
- zubyak 8y ago
  Did you compile it using '-std=c11 -O1' ?
  - bsder 8y ago
    No, but that just makes me assume that this is an optimization bug. Especially since -O4 actually coughs up different pointers: gcc-7 -std=c11 -O4 -o test test.c $ ./test 0x7fff5b5f5a08 0x7fff5b5f5a10 0 The following code works as expected even with your flags and the pointers are clearly not related to one another: #include <stdio.h> #include <stdint.h> int main(void) { uint32_t *p = ((uint32_t *)0x7fffffffffff3eac)+1; uint32_t *q = (uint32_t *)0x7fffffffffff3eb0; printf("%p %p %d\n", (void *)p, (void *)q, p == q); return 0; } So, I would either chalk this up to either 1) "compiler optimization bug" unless a gcc maintainer explicitly gave me a reason to believe to the contrary or 2) getting your underwear eaten by weasels because you optimized after invoking undefined behavior (attempting to probe stack layouts qualifies as undefined behavior in a big way). My assumption would be that the compiler had to do something screwball that it got wrong because you actually asked for a pointer to a stack object that it had optimized to always be in registers.

269 points by ognyankulev 8y ago

This scratches the surface of why I hope C slowly fades away as the default low-level language. C sounds simple when you look through K&R C. C lets you feel like you understand the stack, ALU and memory. A pointer is just an integer and I can manipulate it like an integer. But the reality is filled with a staggering number of weird special cases that exist because memory doesn't work like a simple flat address space; or the compiler needs to optimise field layouts, loops, functions, allocations, register assignments and local variables; or your CPU doesn't use the lower 4 bits or upper 24 bits when addressing. C has no way to shield common language constructs from these problems. So everything in the language is a little bit compromised. Program in C for long enough and you'll hit a lot of special special cases – usually in the worst way: runtime misbehavior. Type punning corruption, pointers with equal bit representations that are not equal, values that change mysteriously between lines of code, field offsets that are different at debug and release time. When using fundamental language constructs, we shouldn't need to worry about these ugly special cases – but C is built around these ideas. The need to specify memory layouts, direct memory access and other low level access should be gated by barriers that ensure the language's representation and the machine representation don't tread on each other's toes. Rust has a long way to go but is so much more robust at runtime. I think there's room for other languages to occupy a similar space but they're need to focus on no-std-lib no-runtime operation (not always the sexiest target).

mpweiher 8y ago
> the compiler needs to optimise ... wants to. There fixed that for you ;-) The weird special cases were largely(1) introduced recently by optimizer writers hijacking the standard in order to soften the semantics so that previously illegal optimizations would now be legal, by simply declaring the vast majority of existing C code as "undefined" and up for grabs. Which is why the Linux kernel, among others, has to set special flags in order to get back to a somewhat sane variant. (1) Except for those special architectures where the hardware is that weird, but in that case you typically know about it.
- zodiac 8y ago
  What does "softening" the semantics mean? Aren't they pretty unambiguously defined by the standard?
  - mpweiher 8y ago
    Newer versions of the standard.
- jcranmer 8y ago
  What changes are you thinking of? The only change with respect to undefined behavior I can think of off the top of my head is punning via unions, which C99 changed from undefined behavior to well-defined behavior.
  - mpweiher 8y ago
    According to http://www.complang.tuwien.ac.at/kps2015/proceedings/KPS_2015_submission_29.pdf http://www.complang.tuwien.ac.at/kps2015/proceedings/KPS_201... new undefined behaviors were added, and for significant bits. I think the bigger issue is that compiler writers have taken "undefined" to mean complete liberty to do whatever the hell they want, even fairly far away from the actual location of the undefined behavior. And it turns out that the C89 standard had a range of "permissible" actions on undefined behavior. To me "permissible" is restrictive, as in you are not allowed to do anything outside that range. Newer standards kept that section (3.4.3) the same, word for word, except for changing "permissible" to "possible". To me, "possible" is not restrictive but illustrative. Now you might think this is reading tea-leaves on my part, and you might be correct, but on the other hand that change of interpretation is exactly what happened. Compilers used to not do these things, and nowadays compilers do do these things, and compiler writers cite the standard as giving them permission. Also consider to what extent wording of standards is finessed and fought over, and that exactly this one word was changed, in a section that otherwise remained the same word for word.
- xenadu02 8y ago
  > The weird special cases were largely(1) introduced recently by optimizer writers hijacking the standard in order to soften the semantics so that previously illegal optimizations would now be legal, by simply declaring the vast majority of existing C code as "undefined" and up for grabs. Citation needed. Signed overflow being undefined behavior was a direct consequence of different hardware representations of numbers. Inserting the instructions required to catch overflow and make it wrapping would be expensive, at least on some systems. The fact that arrays decay to pointers, C has no dynamic array support, and so on are decisions that go way back and more or less prohibit array bounds checking. After preprocessing, inlining, and various forms of optimization it can be very far from obvious that some random NULL check after the pointer has already been dereferenced is intentional and shouldn't be removed. Things like overlapping pointers have always had undefined behavior in the sense that inserting the instructions to check for the overlap all over the place would have a large performance impact, no one would use such a compiler, and thus there is no market for one. The standards committee largely documented existing practices rather than inventing new undefined behaviors. I keep seeing people like you claim we should just "fix" undefined behavior or the standards committee gleefully inserted undefined behaviors because they hate programmers. If you actually sat down and disassembled the C code you write then went through the exercise of inserting instructions to eliminate most types of undefined behavior I suspect it would be a very illuminating experience. I also doubt you'd be so certain of your position.
ChrisSD 8y ago
But as you know, CPU ISAs are designed for C programs and compilers are optimized for C programs. So everything, even new languages like Rust, have to buy into C's model to some extent. Truly getting out from under C's shadow is going to be very difficult. Maybe better languages are a first step on that path but they are only a small step.
- pjmlp 8y ago
  Modern CPUs, but it used to be that they were designed for any high level language before C took over the show.
- jstimpfle 8y ago
  That is said a lot, but what are actual desicions that are made to conform to the C model? What would be good ideas in CPU design that aren't made since they are not compatible with C? (I get it that CPUs have to support some common paradigms and use cases. For example, virtual memory / process isolation, maybe branch prediction things, or support for calling conventions. However, I don't think that is specific to C).
  - brandmeyer 8y ago
    Support for unaligned memory accesses in hardware is one. The exceeding majority of variable accesses are through pointers of the correct type, and are therefore to variables that are at least naturally aligned. But a very small handful of systems/procedures make unaligned accesses through type-punned pointers. Rather than just crash on these programs, processors will perform multiple memory accesses and stitch the results back together in hardware (worst case behavior: accesses that cross page boundaries). So there's extra hardware to perform this extra state tracking and detect the unaligned accesses in the first place. Hardware doesn't just get turned off for conforming programs. Its there consuming energy and giving off heat on all of them. Edit/addendum Here's another example: What should hardware do on integer overflow? There are several options, including signed-saturation, unsigned saturation, unsigned wraparound, and trap. The semantics of C are such that most hardware just picks 'unsigned wraparound' and makes programs eat the consequences. If you're very lucky then you also get 'an overflow bit, that gets clobbered on the next instruction'. One way to think about this problem might be to ask the question, "What hardware features would reduce the runtime cost of instrumenting programs with the various sanitizers that are becoming available?"
nostalgeek 8y ago
The following is purely my opinion. Rust might have gone too far the other way. Yes it strives to be a modern language, with a lot of functional programming features and "OOP" done right (aka no inheritance, simply method polymorphism). To me Rust is more a C++ replacement than a C replacement. A replacement for C should try to be as simple as possible while fixing C weak typing mess with strong typing for instance, including when it comes to pointers (for instance like in Ada where you can't just take the address of anything you want, you need to declare it as aliased to begin with). In fact Ada tries hard to make pointers redundant which is a great thing. This and range types and bound check arrays( Ada has a lot lot of great ideas, and "C style" Ada is a treat to use). So a C replacement should keep the procedural nature of C, with closures perhaps, but not go further in terms of paradigm. C aficionados often claim they love C because it's "simple"(it isn't) That's what they mean, they love C because you can't really do OO with it. On the other hand, Rust macros are exemplary of what a good macro system should be.
- icedchai 8y ago
  I've been programming C on and off for 30 years. Rust feels too complex to be a C replacement. C++, yes... C's paradigm is really about working with memory. Anything that restricts memory access with bound or type checking is going to feel like "not C"...
  - bitwize 8y ago
    C doesn't have anything to say about whether bounds are checked or not. That's why so many things, including pointers to arbitrary locations in memory, are undefined: the standard was built to accommodate both embedded system implementations where scribbling all over RAM is desirable, and implementations that do things like bound checking.
- pjmlp 8y ago
  I think it will only happen the way of Luddites in systems programming. Even if redo Modula-2 with curly brackets I am not sure it would pick up.
- deleted 8y ago
pjmlp 8y ago
On top of that C just got here thanks to widespread of UNIX and POSIX, and we had so much better options. Yes they were languages with helmet, seat belts and airbag bolted on, but just like old technology cars on a motorway crash, the CVE database shows what happens when things go wrong.
tomp 8y ago
> memory doesn't work like a simple flat address space Can you expand on that? I thought that modern (non-segmented) memory does work like a simple flat address space, at least from the perspective of userspace programs. Isn't all the weirdness only a result of compiler optimisations?
- thechao 8y ago
  Just to take one example: multisocket x86 is NUMA. There’s HW to paper over that fact, but the power/perf cost is quite high.
  - mastax 8y ago
    Also Epyc and (even more so) Threadripper have somewhat NUMA memory access in a single CPU.
bradleyjg 8y ago
These architectural bizarrities exist and have to be dealt with by someone. Of course it is better if that someone isn’t you—if some rust maintainer ports the language to the obscure hardware where the CPU ignores the lower four bits when addressing and presents nice, neat, and performance abstractions for you to use. But who is going to pay for that effort? If the hardware manufacturer cared enough to to invest a lot in developer tooling it probably wouldn’t have picked such a hostile interface to begin with.
sureaboutthis 8y ago
Some of your complaints could be said about assembly,too, but are you wanting to get rid of that, too? Some of your other complaints are about how processors and other hardware work, not C. In many ways, you are asking for C to be like other languages but then it wouldn't be C and all the advantages it does have. If you get rid of C, you get rid of most of the software your operating system runs on and that begs the question you need to ask yourself: if C is so bad, why is everyone using it?
- bluejekyll 8y ago
  > if C is so bad, why is everyone using it? Don't confuse success for correctness, survivorship bias is something to consider here. As others will surely point out, when C was written, there were other languages available at the time that were as fast, but much safer. In many ways C won, because Unix won, to talk about them separately ignores how important both were to each others success. (Yes, other operating systems adopted C as well, but Unix is it's foundation)
- pjmlp 8y ago
  Some mainframes like Unisys ClearPath don't have Assembly support at all, zero, nada, all necessary low level primitives are exposed as compiler intrisics.
jcranmer 8y ago
C sort of has this mindset built around it that you're writing portable assembly, except for messy bits like stack frame layout or register allocation. But that doesn't really hold true anymore, and arguably hasn't for decades. The first obvious issue is that C is specified by an abstract machine that doesn't really correspond to actual hardware. There's no concept of segmented memory, or multiple address spaces in C. Traps don't quite work the way you'd want in C [1]; there's no way to catch a trap that occurs in a specific region of code. Worse, the abstract machine leans on a definition of undefined behavior in such a way that it's painful for programmers but not useful enough to make automatic security permissible. (Unsigned overflow, which is going to be a more common cause of overflows that lead to security vulnerabilities, is well-defined). However, the more real problem with C is that it is outright missing hardware features that you'd like access to. Things like rotate instructions or wide multiply are easy enough to pattern match, so their lack isn't so devastating. But flag bits--particularly add with carry or checked overflow operations--aren't figured into at all. SIMD vectors have no conception. Returning multiple values via registers. Traps [1]. Coroutines. Multiple address spaces (especially important for GPUs). [1] Admittedly, the OS is also to blame here: POSIX signals are a really atrocious way to handle machine-generated faults.
- nwmcsween 8y ago
  > flag bits That's sort of a processor specific implementation detail, IIRC you can access them via GNU extensions. > SIMD vectors Not having SIMD IMO is a feature as doing useful things will expose processor specific details, you can use SWAR to do some of SIMD.
  - jcranmer 8y ago
    Exposing processor-specific details is the desired feature, not a bug. The people who want to program C for portable assembly do so because they want to take advantage of things they know about the architecture: the same source code isn't really portable across different architectures without #ifdef anyways.
    - p1esk 8y ago
MaxBarraclough 8y ago
> I think there's room for other languages to occupy a similar space but they're need to focus on no-std-lib no-runtime operation (not always the sexiest target). You're describing the Zig language! It aims to be as fast as C, and unlike most languages that say this is a goal, it means it. There's no "almost as fast as C if you ignore the garbage-collector and the array-bounds-checking", it's actually as fast as C. Its author values the minimalism of C, but wants to create a far better language for doing that sort of work. [0] They're doing surprisingly well. They even managed to make a faster-than-C (faster even than hand-tuned assembly) SHA-2 [1] [0] https://andrewkelley.me/post/intro-to-zig.html https://andrewkelley.me/post/intro-to-zig.html , https://github.com/ziglang/zig/wiki/Why-Zig-When-There-is-Already-CPP,-D,-and-Rust%3F#first-class-support-for-no-standard-library https://github.com/ziglang/zig/wiki/Why-Zig-When-There-is-Al... [1] https://ziglang.org/download/0.2.0/release-notes.html https://ziglang.org/download/0.2.0/release-notes.html , it's also mentioned somewhere in this talk https://youtu.be/Z4oYSByyRak https://youtu.be/Z4oYSByyRak
- kbenson 8y ago
  > faster even than hand-tuned assembly I think you and I are working off different definitions of what that means, as my definition doesn't really allow for a faster implementation (unless there's some really spooky stuff going on in the compiler). I suspect you mean faster than a popular hand tuned implementation.
  - MaxBarraclough 8y ago
    Obviously Zig's output (courtesy of its LLVM backend) can be expressed and distributed as assembly code, but it's not hand-tuned, it's compiler-generated. Whether the hand-tuned assembly code they used was the fastest out there, I'm not sure. I'd hope so, or they're being misleading. I might get time to re-watch the relevant part of the video - it's around the 21 minutes mark - https://youtu.be/Z4oYSByyRak?t=1292 https://youtu.be/Z4oYSByyRak?t=1292
    - isaachier 8y ago
      Andrew explains in the video that the speedup is mostly due to compile-time speed ups and potentially the use of the rorx instruction. It cannot be faster than the fastest assembly because the by definition is the fastest assembly.
CyberDildonics 8y ago
I feel like C++11 and Rust have shown a way forward and the C standard should focus on a design that takes into account ownership and protected arrays in a simple way, hopefully as backwards compatible as possible. Maybe the solution is for the major compilers to default to strong static analysis. Even though there are safe languages, the enormous integration and compatibility of C means that it isn't going anywhere soon, so I think even imperfect solutions can offer a lot if they are low hanging fruit.
- pjmlp 8y ago
  It is a matter of mindset, multiple safe C variants have been attempted throughout the years. Even lint was already available in 1979, yet hardly anyone used static analysis tools on C codebases.
  - CyberDildonics 8y ago
    Which is why I mentioned defaulting to strong static analysis. Having it as a separate tool or an unknown flag is not enough.
bitwize 8y ago
> A pointer is just an integer and I can manipulate it like an integer. Except it's not. Not unless you cast it to uintptr_t. Doy. :) Understanding the underlying asm can give you context for what a pointer is, but you can't assume that a pointer is an integer, any more than you can assume 'int' is a machine word in length. There are reasons why C seems so abstruse and abstract. It has to exist on a huge variety of architectures and environments, from supercomputers to tiny embedded microcontrollers. Fun fact: The Lisp Machine had a C compiler! In Lisp Machine C, pointers were fat: they consisted of a machine pointer to an allocated chunk of memory and an offset within that chunk. And of course it was invalid to have a pointer outside an allocated object; the machine would throw an exception if you even tried. Hence why C pointers are only well-defined over valid references to objects and arrays, one past the end of an array, and NULL. And lest you say, "well no modern cpu works like that anymore", there are programs out there written for the Burroughs architecture that are still running. Are you going to cut off the maintainers of those programs because the architecture they're coding for doesn't resemble an x86? This is part of why I disdain the call for modernity in programming. "Modern" means "limited". "Modern" means "I have no intention of testing this on anything but my laptop, so I will make assumptions that suit my use case and fuck you if yours differs". C is a pain in the ass, but it becomes more approachable if you take into account its history. And it still fits in your head more easily than Rust.
- hexane360 8y ago
  The obvious flipside of this is that those C programs written for Burroughs never had any chance of running on anything but Burroughs. The only difference is that C offloads the effort of portability onto the user, whereas other languages at least attempt to make it portable in the compiler.
  - pjmlp 8y ago
    What?! Since when did Burroughs had a C compiler? NEWP was and is more than enough, while being much safer.
    - bitwize 8y ago
Koshkin 8y ago
Road to Rust is paved with good intentions. (Same is true about C++.)
andrewmcwatters 8y ago
Is there a site or book out there that talks about these common pitfalls? I work in C89 so K&R C does seem pretty simple to me. I clearly haven't worked with it long enough to spot these things, or I have code in the wild that unknowingly is victim to these issues and I've yet to stumble on it.
oppositelock 8y ago
I've been writing C on and off since 1990, across embedded systems, game consoles, massive SGI supercomputers, and modern kernels. C is not fading away because of it's portability and low overhead. It has some nasty behaviors, as the original post shows, but you hit those once in a blue moon. The language is not unsound because of it, it just has some warts like any other language. I don't particularly enjoy writing C, as it's too low level for my liking and I have to write too much code to do basic things (slist_* makes me shudder), but the reason that I have used it is due to its tremendous portability and direct compilation to machine code. That still matters in some places. In places where memory is precious, or realtime performance precludes things like memory managers, C will continue to thrive, beside languages like Rust which are great for different goals. There's room for both. C, in my opinion, has stood the test of time on its simplicity, portability, and the ubiquity of the C standard library, tiny as it may be.

joosteto 8y ago

When I compile the example, i get: 0x7ffd0b57ebd0 0x7ffd0b57ebd8 0 OK, so gcc reordered a & b; I'll fix this by chaning the initialisation of p and q to: int *p = &a + 1; int *q = &b; But when I now run the example, I get: gcc -o c c.c && ./c 0x7ffe55eb0aa4 0x7ffe55eb0aa4 1 gcc -O -o c c.c && ./c 0x7ffcfeffd914 0x7ffcfeffd914 0 So, p==q only evaluates to 1 if optimisation is enabled. $ gcc --version gcc (Ubuntu 7.3.0-16ubuntu3) 7.3.0

psyklic 8y ago
It shows the exact options the author used if you click on GCC: Version: 8.1.1 Options: -std=c11 -O1 Target: x86_64-redhat-linux

mpweiher 8y ago

clang is a bit more sane: > cc pointereq.c > ./a.out 0x7ffee83163b8 0x7ffee83163b8 1 > cc -O pointereq.c > ./a.out 0x7ffeeeb8b3b8 0x7ffeeeb8b3c0 0 So without optimization, the pointers are the same and compare as equal. With optimization, the pointers compare as not equal. At first that seemed horrible, until I saw the pointers actually are not the same. Since I don't recall any guarantees about stack layout, that seems perfectly fine. > cat pointereq.c #include <stdio.h> int main(void) { int a, b; int *p = &a; int *q = &b + 1; printf("%p %p %d\n", (void *)p, (void *)q, p == q); return 0; }

andyjohnson0 8y ago

Just a datapoint: VS2017 on Win10 x64 gives me 00AFF89C 00AFF894 0, which is what I naively expect.

JdeBP 8y ago
Actually, you should not expect that, because you don't have a guarantee of ordering of the addresses of a and b on the stack. Now try with /O2 . (-:
- andyjohnson0 8y ago
  Maybe I'm misunderstanding you, but it's precisely because I have no knowledge of the stack frame's layout that I didn't expect the pointers to be equal. With /O2 optimisation I get: 00B1F99C 00B1F9A4 0
  - JdeBP 8y ago
    That's where you are going wrong. You have no guarantee that the compiler places a and b next to each other, let alone in a defined order with a at a higher address than b such that one integer width beyond the end of b you find a. Expecting one thing is wrong. You are expecting the one thing that the comparison always returns false. You should not expect anything. The comparison result could be true or false. H:\>cl /O2 pointereq.c Microsoft (R) C/C++ Optimizing Compiler Version 19.14.26431 for x86 Copyright (C) Microsoft Corporation. All rights reserved. pointereq.c Microsoft (R) Incremental Linker Version 14.14.26431.0 Copyright (C) Microsoft Corporation. All rights reserved. /out:pointereq.exe pointereq.obj H:\>.\pointereq 00CFFA20 00CFFA20 1 H:\>
    - mpweiher 8y ago
      You are confusing "did not expect them to be equal", which is what the OP wrote, with "expected them to be not equal", which is what you are railing against.

nuriaion 8y ago

Basically you have no guarantee where the pointer q is pointing to. Some compiler/static code analyzer will yell at you with this code.

jerrre 8y ago
isn't it guaranteed that is points sizeof(int) bytes higher than the address of b? Whether something useful is behind that address is another question
- ChrisSD 8y ago
  From reading the spec it seems that is undefined behaviour for anything other than `&b + 1`. That is one past the end of b is fine but not `&b + 2` etc.
  - zubyak 8y ago
    I wonder why someone decided that accessing "one past the end" should be fine
    - jwilk 8y ago
      It's explained in the addendum to the article.
      - zubyak 8y ago
        thanks
    - mcguire 8y ago
      Short answer: so you can walk a pointer down the elements of an array without having to have weird code to deal with the end.
    - deleted 8y ago
      [deleted]
- 8y ago

foxhill 8y ago

the comparison at the start is nonsense - there is no specification for the ordering or location of stack variables. by taking the address of these variables, you could see that they actually are the same value, and so intuitively you’d think they might be the same, but a different compiler might put them in different locations. or they may be elided entirely through optimisation. it’s far safer to fail the equality test in this case - this is what the model specifies. this is not even the first example of this counter-initiative behaviour. imagine two floating point values with exactly the same bit-representation. it is possible, without any trickery for them to fail an equality check - i.e, they are both NaN. this is what IEE754 demands of a compliant floating point implementation. and indeed, it’s a sane choice when you understand why it was made. similarly, it’s perfectly reasonable for this example to fail.

jameshart 8y ago
Surely the point of writing an if-block testing the equality of the pointers is precisely because the code isn’t assuming that they are the same - but maybe it has something it wants to do if it finds itself running somewhere where they are. Writing code which assumed those pointers were equivalent would be bad, but this code doesn’t do that. Instead, it looks like the compiler assumes that the pointers won’t be the same and refuses to let you entertain the possibility at all.
- maxxxxx 8y ago
  What's the point of even trying to compare these two pointers?
- cozzyd 8y ago
  I haven't tested it but I imagine casting to uint64_t will make it work as you want.
- foxhill 8y ago
  > Instead, it looks like the compiler assumes that the pointers won’t be the same and refuses to let you entertain the possibility at all. and indeed it should - it’s sane to use the address of a stack variable for the purposes of writing something into it, but assuming that it forms some sort of array in memory is sure to cause pain if you rely on that assumption.

bluecalm 8y ago

I like the behavior of the compiler here. There is no guarantee that a and b are next to each other in memory. That's why the comparison fails, the alternative makes is runtime/compiler/optimization level dependent which would be a total mess. As usual with those C bashing articles you won't run into trouble if you don't try very hard to write contrived code. I mean, the moment you see: int *q = &b + 1; on your screen alarm bells should go off. Doing pointer arithmetic on something that is not an array is asking for trouble. If the standard should be amended in any way it should be undefined behavior right away you do pointer arithmetic on non-array objects.

jstimpfle 8y ago
That would require making a distinction between array-pointers and non-array pointers. There, another can of worms. Probably not worth the additional complexity.
mpweiher 8y ago
Agree on the alarm bells with the pointer arithmetic. Disagree that gcc is doing the right thing here. The clang behavior (different comment in this thread) is much more sane: if the pointers happen to be the same, they compare as equal. If the pointers happen to not be the same, they compare as not equal.
- shakna 8y ago
  Except clang's behaviour changes depending on the optimisation level. If you use -O1, then you get a different result.
  - mpweiher 8y ago
    Yes, because the pointers are different at -O. As there are no guarantees as to the relative placement of auto/stack variables relative to each other, that is perfectly fine.
    - shakna 8y ago

Kenji 8y ago

tl;dr to save you guys time: Somebody writes C code with undefined behaviour and is surprised when the behaviour is indeed undefined and thus erratic. Completely pointless blogpost because the author does not understand C.

tzahola 8y ago

There's nothing surprising in the first example. Comparing the addresses of stack variables is undefined behaviour. The second one is more interesting: extern int _start[]; extern int _end[]; void foo(void) { for (int *i = _start; i != _end; ++i) { /* ... */ } } GCC optimized "i != _end" into "true". The kernel guys fixed this by turning "_start" and "_end" into "extern int*". I always thought [] was just syntactic sugar over a regular pointer, but seems like I was wrong.

JdeBP 8y ago
Actually, the headline article tells us that it optimized it to true. That affected some of my code from years ago, too; and another fix is to keep _start and instead of having another array called _end have an external size_t giving the number of elements of start and loop while i!=_start+count . In fairness to the compiler people there shouldn't really be anything surprising in the second example, either. There are x86 memory models (compact and large) where that loop never terminates in the general case, because the two arrays are not in the same data segment and no amount of incrementing a (far but not huge) pointer will get from one to the other. So it's not the case that this is a new pitfall. It was a pitfall decades ago. People just thought that it didn't exist with modern tiny (a.k.a. flat) memory models, because the old explanation wasn't applicable. It does, but with a different explanation.
- tzahola 8y ago
  Yup, I've mistyped "true" as "false". If you set "_start" and "_end" to addresses of different objects, then you're invoking undefined behaviour and all bets are off. But this should be perfectly valid IMO: extern char* _start; extern char* _end; void foo() { for (char* c = start; c != end; c++) ... } int main(int, char**) { char buf[128]; _start = &buf[0]; _end = &buf[128]; foo(); } Right?
  - JdeBP 8y ago
    The thing there is indeed not the validity, but the introduction of extra memory references and variables. The code with the arrays compiles to immediate value loads with load-time fixups to the code; and if this is linking to an assembly language module the necessary assembly language is just some labels against the data. The code with the pointers compiles to loads from data memory, with (in the more usual paradigm where the start and end are bracketing some vector of initialized data in the program image) load-time fixups to the initial values of the variables; and the assembly language has to be different, some variables with these labels whose initial values are the addresses of some other labels against the actual data. Notice that the Linux Kernel developer in 2016 did not immediately appreciate the need to modify the concomitant assembly language and linker script stuff to match the change in the C language code. The approach that I mentioned keeps the immediate value for the start, and uses a load from initialized data memory for the count. Other approaches are possible, but they need more work in the build process. For example: By generating the C/C++ language declaration and definition from a script or suchlike, one could calculate and declare the number of elements in the array, making it possible to use sizeof.

tzs 8y ago

Note that if the two pointers are passed to a function, and the comparison is done in the function, the results are different: #include <stdio.h> void pcmp(int *p, int *q) { printf("%p %p %d\n", (void *)p, (void *)q, p == q); } int main(void) { int a, b; int *p = &a; int *q = &b + 1; printf("%p %p %d\n", (void *)p, (void *)q, p == q); pcmp(p, q); return 0; } That is giving me: 0x7ffebac1483c 0x7ffebac1483c 0 0x7ffebac1483c 0x7ffebac1483c 1 That's compiled with '-std=c11 -O1' as in the article. The result is the same of pcmp is moved into a separate file so that when compiling it the compiler has no knowledge of the origins of the two pointers. I don't like this at all. It bugs me that I can get different results comparing two pointers depending on where I happen to do the comparison.

JoeAltmaier 8y ago
Yes, I agree. It makes far more sense to make guarantees in a language, that two values are compared using a metric that is invariant. Especially in a language like C, where we expect it to be a very thin abstraction over the machine.
deleted 8y ago
[deleted]
ash_gti 8y ago
Did you use gcc or clang or...? When I ran this locally I get: 0x7ffeee2a4808 0x7ffeee2a4810 0 0x7ffeee2a4808 0x7ffeee2a4810 0 With -O0 I get 1/1 instead of 0/0 so this appears to be a compiler optimization.
quelltext 8y ago
Yup, what is described in the article looks like a bugged optimization to me were the runtime behavior differs from the static model used for the optimization. I am sure the actual runtime semantics is in fact to compare addresses but the optimization claims to know (falsely) enough about the pointers to determine they must be not equal. I don't understand the people in this comment section defending this. The cited spec gives no evidence that this should happen (I fail to see the undefined behavior here). It specifies when pointers are equal, and both pointers ended up pointing to the same object, so they should be equal. Just because the compiler doesn't know, shouldn't mean it can replace an equality check with false. The spec didn't say "also, they are not equal in case the conpiler thinks that they proooobably aren't pointing to the same object". I do agree that one should avoid this type of pointer arithmetic, but that doesn't excuse the confusing behavior of this primitive operstion. One should at least argue for the behavior to be consistent.

_RPM 8y ago

if you're adding 1 to a pointer, why would you expect it to be equal? I would not expect it to be equal.

armitron 8y ago
He's adding "1" to the pointer that points to the variable "b" which in this case is allocated immediately below "a" in the stack. Thus adding one will make the two pointers equal.
- saagarjha 8y ago
  The layout of things on the stack is completely implementation-defined.
  - sannee 8y ago
    It has been a while since I read the standard, but this seems well into the realm of undefined behavior, not implementation defined.
    - saagarjha 8y ago
      From my understanding, how the stack is laid out is implementation-defined; attempting to actually observe this ordering in your program is undefined behavior.

crehn 8y ago

Two pointers compare equal if and only if both are null pointers, both are pointers to the same object (including a pointer to an object and a subobject at its beginning) or function, both are pointers to one past the last element of the same array object, or one is a pointer to one past the end of one array object and the other is a pointer to the start of a different array object that happens to immediately follow the first array object in the address space. Can someone explain to me the rationale behind this? Why not just "two pointers compare equal if they point to the same address"?

Kenji 8y ago
The rationale behind this "mess" is precisely that people cannot write code like in the first example of the blog post, effectively relying on undefined behaviour, and have this ugly code work most of the time.
isaachier 8y ago
My guess is type based alias analysis. The classic example is casting an int to a float like so may crash your program due to undefined behavior: float my_bad_function(void) { int x = 5; float* y = (float) &x; return y; }
JdeBP 8y ago
Because (a) pointers do not point to addresses, and (b) now you have to define the concept of addresses being equal in the language standard and haven't actually reached the goal. (-:
- Sharlin 8y ago
  Indeed; pointers intentionally abstract away memory addresses and addressing, which, as the article states, can be much nastier business than the benign flat address space of i386 protected mode.
thatcks 8y ago
The C standard was (and is) written so that it allowed for machines that didn't have simple pointers; a highly relevant example for the time was x86 machines in real or protected mode using segment registers. Even when such environments have accessible linear address spaces under the hood, comparing the absolute address of two pointers requires some extra work (you have to reassemble the absolute address from the segment + offset pair). In the grand tradition of allowing the most efficient implementation possible, the C standard says that implementations don't have to do this work; they can simply compare pointer values directly, provided that they insure that pointers to the same object work (eg use the same segment + offset pair, which they normally naturally will). The standard was also written to allow for C interpreters, where you may not have an underlying linear memory model at all and all pointers are represented internally in some complex way (for example 'object ID + offset inside object'). Here you don't have any particular idea of 'an address' as a distinct thing and so you can't naturally compare two pointers to different objects in any meaningful way.

Sharlin 8y ago

> ...or one is a pointer to one past the end of one array object and the other is a pointer to the start of a different array object that happens to immediately follow the first array object in the address space. I was not aware of this special case. What's the rationale? Is there even a way in standard C to guarantee that two array objects are laid out in memory like that, with no padding?

mcguire 8y ago
Start with one large array and subdivide it into two segments.
saagarjha 8y ago
A two-dimensional array?
- Sharlin 8y ago
  Yeah, good point.

oconnor663 8y ago

> If we step back from the standard and ask our self does it make sense to compare two pointers which are derived from two completely unrelated objects? The answer is probably always no. The one big counterexample I can think of is the difference between memcpy and memmove. The latter is supposed to be able to do arithmetic on memory regions, to see if they overlap. Is this article saying that the standard C implementation of memmove is relying on unspecified behavior?

coldnose 8y ago
Another example of arbitrary pointer comparison I'm familiar with is for resolving lock ordering problems. If I have a big set of locks that I need to take to complete a particular operation, a simple way to prevent deadlocking is to acquire those locks in the order of their pointers. E.g. if I have three locks I need to take, lock_t *a = (lock_t *)0x100; lock_t *b = (lock_t *)0x300; lock_t *c = (lock_t *)0x200; then I'll take them in order a, c, b;
- bluecalm 8y ago
  If order of locks is important maybe it's worth considering putting them in an array.
  - oconnor663 8y ago
    Or maybe keeping a global atomic counter, and reading a unique (to the process) value out of it every time you create a mutex?
slrz 8y ago
Memmove is frequently cited as an example of a standard library function that can't be implemented using only standard C. It's incorrect, though: the way to do memmove using standard C is to use malloc to allocate a temporary buffer. But the whole question is not terribly relevant. When memmove is provided as part of the C implementation it can rely on non-portable platform behaviour just fine. There's no rule that you have to implement libc using only standard C facilities.

mabynogy 8y ago

Not related to the content but I find the colored links in the nav a simple and very good idea.

bsder 8y ago

Ummm, when I run this on gcc 7.3.0 on OS X I actually get: 0x7fff5dbd89fc 0x7fff5dbd89fc 1 Which kind of shoots the whole article in the foot ...

zubyak 8y ago
Did you compile it using '-std=c11 -O1' ?
- bsder 8y ago
  No, but that just makes me assume that this is an optimization bug. Especially since -O4 actually coughs up different pointers: gcc-7 -std=c11 -O4 -o test test.c $ ./test 0x7fff5b5f5a08 0x7fff5b5f5a10 0 The following code works as expected even with your flags and the pointers are clearly not related to one another: #include <stdio.h> #include <stdint.h> int main(void) { uint32_t *p = ((uint32_t *)0x7fffffffffff3eac)+1; uint32_t *q = (uint32_t *)0x7fffffffffff3eb0; printf("%p %p %d\n", (void *)p, (void *)q, p == q); return 0; } So, I would either chalk this up to either 1) "compiler optimization bug" unless a gcc maintainer explicitly gave me a reason to believe to the contrary or 2) getting your underwear eaten by weasels because you optimized after invoking undefined behavior (attempting to probe stack layouts qualifies as undefined behavior in a big way). My assumption would be that the compiler had to do something screwball that it got wrong because you actually asked for a pointer to a stack object that it had optimized to always be in registers.