13 ms·

What’s the difference between an integer and a pointer?

170 points by luu 8y ago

Upvoter33 8y ago
Regehr and colleagues have been doing excellent work on undefined behavior for some time, which has been fantastic. I do personally wish the C standard was changed a bit to have less undefined behavior, leaning towards getting programmer intent correct and not worrying (as much) about getting every possible optimization into the generated code.
- saagarjha 8y ago
  > I do personally wish the C standard was changed a bit to have less undefined behavior Agreed. There are some particularly egregious ones where the behavior should at the very least be implementation defined.
algesten 8y ago
I can't replicate this example on my mac. Regardless of `diff` or 16 (which is the diff for me), I get the same output for clang and gcc: 7 5 7
- micro-ram 8y ago
  Agreed. I get the intended result of 7 5 7 on macOS 10.13.6 LLVM. Mac:tmp $ clang -O3 mem1d.c ; ./a.out diff = -96 x = 0x7f9f4fc00300, *x = 7 y = 0x7f9f4fc00360, *y = 5 p = 0x7f9f4fc00300, *p = 7 Mac:tmp $ diff mem1d.c mem2d.c 13c13 < int *p = (int *)(yi + diff); --- > int *p = (int *)(yi - 96); Mac:tmp $ clang -O3 mem2d.c ; ./a.out diff = -96 x = 0x7fc2b5c00300, *x = 7 y = 0x7fc2b5c00360, *y = 5 p = 0x7fc2b5c00300, *p = 7 Mac:tmp$ clang --version Apple LLVM version 10.0.0 (clang-1000.11.45.2) Target: x86_64-apple-darwin17.7.0 Thread model: posix
  - daveFNbuck 8y ago
    You only pasted a clang run here. It's gcc that's supposed to give 3 5 7.
- daveFNbuck 8y ago
  I get a 16 diff too, but I'm seeing 3 5 7 on gcc on High Sierra with a 2013 Macbook Pro. I'm using gcc 8.2.0 installed via Homebrew. Did you run gcc-8 with the -O3 flag?
cesarb 8y ago
The Linux kernel uses a macro called RELOC_HIDE to stop gcc from seeing through the conversion from pointer to integer in these cases: https://elixir.bootlin.com/linux/v4.18/source/include/linux/compiler-gcc.h#L50 https://elixir.bootlin.com/linux/v4.18/source/include/linux/... Edit: a good explanation of that macro is at http://lists.linuxcoding.com/kernel/2006-q3/msg17979.html http://lists.linuxcoding.com/kernel/2006-q3/msg17979.html : "The reason for it is that gcc assumes that if you add something on to the address of a symbol, the resulting address is still inside the bounds of the symbol, and do optimizations based on that. The RELOC_HIDE macro is designed to prevent gcc knowing that the resulting pointer is obtained by adding an offset to the address of a symbol. As far as gcc knows, the resulting pointer could point to anything."
- chrisseaton 8y ago
  > gcc assumes that if you add something on to the address of a symbol, the resulting address is still inside the bounds of the symbol How can one C object span two symbols in the first place for this assumption to be invalid?
  - 21 8y ago
    If I understand correctly what this is about, one example is buffer overflows, where you write to the first symbol but end up in the second: char symbolA[10]; char symbolB[10]; symbolA[15] = 0; // actually writes to symbolB
    - chrisseaton 8y ago
      This isn't one C object spanning two symbols - it's two C objects, and reading one object from a pointer to another object.
  - Sharlin 8y ago
    Hm? The point is that compilers are allowed to assume that a pointer to x can never be formed by offseting a pointer to y (except if they are subobjects in the same array). But of course you’re not prevented from trying, either on purpose or accidentally, it just results in undefined behavior.
aknoob 8y ago
Conceptually, Pointer is a higher and a more well defined, level of abstraction than a raw integer.In much the same way as an Iterator is a higher and more well defined level of abstraction than a raw pointer. The fact that an Iterator, a pointer and an integer all tend to be different ways of looking at same integer value is more or less an implementation detail.
OnlyRepliesToBS 8y ago
the same difference between a raven and a writing desk
simias 8y ago
I sort of see the point of the article but I think talking about differences between pointers and integers obscures the real issue here: the problem as far as I can tell is just that offsetting a pointer more than one byte after the end of the object it points to is UB. TFA argues that converting to integers first sidesteps the issue but I'm not convinced (although I'm not a C standard lawyer). Surely if you convert a pointer to an integer, add an offset that's greater than the size of the original object and then convert that to a pointer you trigger UB? How else could this be defined? You can't really expect the compiler to figure out what you're doing when you're casting "random" integers as pointers. Also the article mentions Rust in its introduction but in Rust anything surrounding pointers (including pointer arithmetics) is unsafe so I don't think there's any conflict whatsoever here.
- cjcole 8y ago
  Related discussion at internals.rust-lang.org: https://internals.rust-lang.org/t/pointers-are-complicated-or-whats-in-a-byte/8045 https://internals.rust-lang.org/t/pointers-are-complicated-o... "This summer, I will again work (amongst other things) on a “memory model” for Rust/MIR. However, before I can talk about the ideas I have for this year, I have to finally take the time and dispel the myth that “pointers are simple: they are just integers”. Both parts of this statement are false, at least in languages with unsafe features like Rust or C: Pointers are neither simple nor (just) integers. I also want to define a piece of the memory model that has to be fixed before we can even talk about some of the more complex parts: Just what is the data that is stored in memory? It is organized in bytes, the minimal addressable unit and the smallest piece that can be accessed (at least on most platforms), but what are the possible values of a byte? Again, it turns out “it’s just an 8-bit integer” does not actually work as the answer." Ralf Jung is a coauthor of the "What’s the difference between an integer and a pointer?" paper.
  - simias 8y ago
    Okay I can agree with that but then there's no need to look for contrived examples, the C standard itself spells out a bunch of UB that arise from using pointers as "just integers". It's probably worth repeating, especially for people new to these languages, but it's not exactly breaking news. I guess I just expected something else when I started reading this article.
csours 8y ago
A pointer is an integer with a shiv
captainmuon 8y ago
I miss the good old times where a pointer was just a number, and if you did `*addr` you plainly read from that memory address. I'm not sure if the standard was ever that way, but compilers sure used to behave like that. Maybe we can one day get a `--std=simple-world` flag that forgoes a few optimizations and makes UB code do the "obvious" thing.
- api 8y ago
  It never was guaranteed by any standard, and in fact numerous old machines had weird pointer formats: segmented addressing, strange widths (36 bits), and tagged pointers come to mind. Its more like we had this window of time when pointers were not weird and got too used to it.
  - captainmuon 8y ago
    Sure, but back then the standard was not relevant. You had your x86 computer, your compiler (MSVC, DJGPP, CodeWarrior, Borland...) and maybe a "how to write C/C++" book. You did have weird things like segmented addressing, but for most purposes, pointers were just memory addresses, and you could use casts to reinterpret data as you wished. In fact, using bit-cast-magic and doing pointer arithmetics were the preferred way of doing things. It's strange that we put so much effort into learning all this arcane knowledge and now we're supposed to unlearn it...
    - wruza 8y ago
      I can’t tell for sure, since I was too young back then and didn’t write something complex enough, but I suspect that compilers were not evil genies back then. It was probably later gcc who first used UB as an excuse to evil optimizations. I also don’t understand a reason behind dragging all that legacy into current standards. Does a real application of it make at least 0.1% of all usage? You cannot even buy a chip that implements segments, tagged pointers, etc etc. at least they could make a special mode where you can span across symbol whatever it means or see a pointer as a cpu sees it. All this can be solved with simple ptr_untag(p), ptr_span(expr) and similar constructs, but instead we have resort to outsmarting the specific compiler logic or introducing ourself with complicated type systems. That went insane. Personally I just want my bytes in linear address space and a way to tell which bytes point to which and which do not. I liked asm and then C, they were two of my first three languages, but what C became today is just a horrible mess.
bluto 8y ago
One is a number and the other is a dog often used when hunting
dahart 8y ago
> First, it is a big mistake to try to understand pointers in a programming language as if they follow the same rules as pointer-sized integer values. I’m a bit confused by this statement. I don’t know the C & C++ standards, but isn’t pointer arithmetic defined and allowed within the bounds of the heap allocation of the base pointer? Like, on an array for example, aren’t pointers required to “follow the same rules as pointer-sized integer values”, as long as you’re not indexing outside your array allocation? Wouldn’t the issue here be more clearly stated as “don’t use pointer arithmetic to index outside the bounds of the base pointer’s allocation” rather than “never use pointer artithmetic”?
- camgunz 8y ago
  Yeah it's fine inside an array, but as soon as you move a pointer from one allocation to another, boom. Dunno what it means for stack pointers though, probably the same except now you've made it so they can't be registers.
utopcell 8y ago
This pointer manipulation is fine. GCC simply optimizes the parameters for printf(): https://godbolt.org/z/AHY7Nd https://godbolt.org/z/AHY7Nd *x will be 7 in the end.
deleted 8y ago
[deleted]
paulddraper 8y ago
> For example, they assume that a pointer derived from one heap allocation cannot alias a pointer derived from a different heap allocation. Doesn't this depends on which allocations have been freed?
tmalsburg2 8y ago
One question that is not answered by the post: if the gcc-compiled program does not overwrite *x, what does it overwrite instead?

170 points by luu 8y ago

Regehr and colleagues have been doing excellent work on undefined behavior for some time, which has been fantastic. I do personally wish the C standard was changed a bit to have less undefined behavior, leaning towards getting programmer intent correct and not worrying (as much) about getting every possible optimization into the generated code.

saagarjha 8y ago
> I do personally wish the C standard was changed a bit to have less undefined behavior Agreed. There are some particularly egregious ones where the behavior should at the very least be implementation defined.

algesten 8y ago

I can't replicate this example on my mac. Regardless of `diff` or 16 (which is the diff for me), I get the same output for clang and gcc: 7 5 7

micro-ram 8y ago
Agreed. I get the intended result of 7 5 7 on macOS 10.13.6 LLVM. Mac:tmp $ clang -O3 mem1d.c ; ./a.out diff = -96 x = 0x7f9f4fc00300, *x = 7 y = 0x7f9f4fc00360, *y = 5 p = 0x7f9f4fc00300, *p = 7 Mac:tmp $ diff mem1d.c mem2d.c 13c13 < int *p = (int *)(yi + diff); --- > int *p = (int *)(yi - 96); Mac:tmp $ clang -O3 mem2d.c ; ./a.out diff = -96 x = 0x7fc2b5c00300, *x = 7 y = 0x7fc2b5c00360, *y = 5 p = 0x7fc2b5c00300, *p = 7 Mac:tmp$ clang --version Apple LLVM version 10.0.0 (clang-1000.11.45.2) Target: x86_64-apple-darwin17.7.0 Thread model: posix
- daveFNbuck 8y ago
  You only pasted a clang run here. It's gcc that's supposed to give 3 5 7.
daveFNbuck 8y ago
I get a 16 diff too, but I'm seeing 3 5 7 on gcc on High Sierra with a 2013 Macbook Pro. I'm using gcc 8.2.0 installed via Homebrew. Did you run gcc-8 with the -O3 flag?

cesarb 8y ago

The Linux kernel uses a macro called RELOC_HIDE to stop gcc from seeing through the conversion from pointer to integer in these cases: https://elixir.bootlin.com/linux/v4.18/source/include/linux/compiler-gcc.h#L50 https://elixir.bootlin.com/linux/v4.18/source/include/linux/... Edit: a good explanation of that macro is at http://lists.linuxcoding.com/kernel/2006-q3/msg17979.html http://lists.linuxcoding.com/kernel/2006-q3/msg17979.html : "The reason for it is that gcc assumes that if you add something on to the address of a symbol, the resulting address is still inside the bounds of the symbol, and do optimizations based on that. The RELOC_HIDE macro is designed to prevent gcc knowing that the resulting pointer is obtained by adding an offset to the address of a symbol. As far as gcc knows, the resulting pointer could point to anything."

chrisseaton 8y ago
> gcc assumes that if you add something on to the address of a symbol, the resulting address is still inside the bounds of the symbol How can one C object span two symbols in the first place for this assumption to be invalid?
- 21 8y ago
  If I understand correctly what this is about, one example is buffer overflows, where you write to the first symbol but end up in the second: char symbolA[10]; char symbolB[10]; symbolA[15] = 0; // actually writes to symbolB
  - chrisseaton 8y ago
    This isn't one C object spanning two symbols - it's two C objects, and reading one object from a pointer to another object.
- Sharlin 8y ago
  Hm? The point is that compilers are allowed to assume that a pointer to x can never be formed by offseting a pointer to y (except if they are subobjects in the same array). But of course you’re not prevented from trying, either on purpose or accidentally, it just results in undefined behavior.

aknoob 8y ago

Conceptually, Pointer is a higher and a more well defined, level of abstraction than a raw integer.In much the same way as an Iterator is a higher and more well defined level of abstraction than a raw pointer. The fact that an Iterator, a pointer and an integer all tend to be different ways of looking at same integer value is more or less an implementation detail.

OnlyRepliesToBS 8y ago

the same difference between a raven and a writing desk

simias 8y ago

I sort of see the point of the article but I think talking about differences between pointers and integers obscures the real issue here: the problem as far as I can tell is just that offsetting a pointer more than one byte after the end of the object it points to is UB. TFA argues that converting to integers first sidesteps the issue but I'm not convinced (although I'm not a C standard lawyer). Surely if you convert a pointer to an integer, add an offset that's greater than the size of the original object and then convert that to a pointer you trigger UB? How else could this be defined? You can't really expect the compiler to figure out what you're doing when you're casting "random" integers as pointers. Also the article mentions Rust in its introduction but in Rust anything surrounding pointers (including pointer arithmetics) is unsafe so I don't think there's any conflict whatsoever here.

cjcole 8y ago
Related discussion at internals.rust-lang.org: https://internals.rust-lang.org/t/pointers-are-complicated-or-whats-in-a-byte/8045 https://internals.rust-lang.org/t/pointers-are-complicated-o... "This summer, I will again work (amongst other things) on a “memory model” for Rust/MIR. However, before I can talk about the ideas I have for this year, I have to finally take the time and dispel the myth that “pointers are simple: they are just integers”. Both parts of this statement are false, at least in languages with unsafe features like Rust or C: Pointers are neither simple nor (just) integers. I also want to define a piece of the memory model that has to be fixed before we can even talk about some of the more complex parts: Just what is the data that is stored in memory? It is organized in bytes, the minimal addressable unit and the smallest piece that can be accessed (at least on most platforms), but what are the possible values of a byte? Again, it turns out “it’s just an 8-bit integer” does not actually work as the answer." Ralf Jung is a coauthor of the "What’s the difference between an integer and a pointer?" paper.
- simias 8y ago
  Okay I can agree with that but then there's no need to look for contrived examples, the C standard itself spells out a bunch of UB that arise from using pointers as "just integers". It's probably worth repeating, especially for people new to these languages, but it's not exactly breaking news. I guess I just expected something else when I started reading this article.

csours 8y ago

A pointer is an integer with a shiv

captainmuon 8y ago

I miss the good old times where a pointer was just a number, and if you did `*addr` you plainly read from that memory address. I'm not sure if the standard was ever that way, but compilers sure used to behave like that. Maybe we can one day get a `--std=simple-world` flag that forgoes a few optimizations and makes UB code do the "obvious" thing.

api 8y ago
It never was guaranteed by any standard, and in fact numerous old machines had weird pointer formats: segmented addressing, strange widths (36 bits), and tagged pointers come to mind. Its more like we had this window of time when pointers were not weird and got too used to it.
- captainmuon 8y ago
  Sure, but back then the standard was not relevant. You had your x86 computer, your compiler (MSVC, DJGPP, CodeWarrior, Borland...) and maybe a "how to write C/C++" book. You did have weird things like segmented addressing, but for most purposes, pointers were just memory addresses, and you could use casts to reinterpret data as you wished. In fact, using bit-cast-magic and doing pointer arithmetics were the preferred way of doing things. It's strange that we put so much effort into learning all this arcane knowledge and now we're supposed to unlearn it...
  - wruza 8y ago
    I can’t tell for sure, since I was too young back then and didn’t write something complex enough, but I suspect that compilers were not evil genies back then. It was probably later gcc who first used UB as an excuse to evil optimizations. I also don’t understand a reason behind dragging all that legacy into current standards. Does a real application of it make at least 0.1% of all usage? You cannot even buy a chip that implements segments, tagged pointers, etc etc. at least they could make a special mode where you can span across symbol whatever it means or see a pointer as a cpu sees it. All this can be solved with simple ptr_untag(p), ptr_span(expr) and similar constructs, but instead we have resort to outsmarting the specific compiler logic or introducing ourself with complicated type systems. That went insane. Personally I just want my bytes in linear address space and a way to tell which bytes point to which and which do not. I liked asm and then C, they were two of my first three languages, but what C became today is just a horrible mess.

bluto 8y ago

One is a number and the other is a dog often used when hunting

dahart 8y ago

> First, it is a big mistake to try to understand pointers in a programming language as if they follow the same rules as pointer-sized integer values. I’m a bit confused by this statement. I don’t know the C & C++ standards, but isn’t pointer arithmetic defined and allowed within the bounds of the heap allocation of the base pointer? Like, on an array for example, aren’t pointers required to “follow the same rules as pointer-sized integer values”, as long as you’re not indexing outside your array allocation? Wouldn’t the issue here be more clearly stated as “don’t use pointer arithmetic to index outside the bounds of the base pointer’s allocation” rather than “never use pointer artithmetic”?

camgunz 8y ago
Yeah it's fine inside an array, but as soon as you move a pointer from one allocation to another, boom. Dunno what it means for stack pointers though, probably the same except now you've made it so they can't be registers.

utopcell 8y ago

This pointer manipulation is fine. GCC simply optimizes the parameters for printf(): https://godbolt.org/z/AHY7Nd https://godbolt.org/z/AHY7Nd *x will be 7 in the end.

deleted 8y ago

[deleted]

paulddraper 8y ago

> For example, they assume that a pointer derived from one heap allocation cannot alias a pointer derived from a different heap allocation. Doesn't this depends on which allocations have been freed?

tmalsburg2 8y ago

One question that is not answered by the post: if the gcc-compiled program does not overwrite *x, what does it overwrite instead?