5 ms·

I don't think that pointer provenance rears its ugly head here. You are generally free to convert a pointer to an integer and then back, which is what's happeni

I don't think that pointer provenance rears its ugly head here. You are generally free to convert a pointer to an integer and then back, which is what's happening here, and you get pointer provenance problems because different pointers may be supposed to point to different objects. Basically, casting to an integer and back is supposed to be "safe", but doing math and getting pointers to different objects is not. Note that there are some implementation-dependent factors here which I'm not getting into. > And reading the int half of the union to check if the pointer is valid should always be sound, but I'm not 100% sure on that. After various alias problems a while back (Linux kernel folks had some words), everyone got together and agreed that you can do type punning through a union... you just get the byte representation of one type, reinterpreted as the byte representation of another type. This was codified a few C standards ago and it's fairly explicit in the spec now. I'll also just mention that you don't need an architecture than enforces alignment--you just need an allocator that returns aligned pointers.

MaulingMonkey 4y ago
> I don't think that pointer provenance rears its ugly head here. You are generally free to convert a pointer to an integer and then back It's worth noting that Rust is experimenting with making this not so free for one to do: https://doc.rust-lang.org/std/ptr/index.html#pointer-usize-pointer-roundtrips-and-exposed-provenance https://doc.rust-lang.org/std/ptr/index.html#pointer-usize-p...
wahern 4y ago
> I don't think that pointer provenance rears its ugly head here. You are generally free to convert a pointer to an integer and then back, which is what's happening here C [optionally] supports converting pointers to intptr_t (or uintptr_t), not to any integer type (even with nominally sufficient width), and certainly not to a floating point type, which is how NaN boxing works. In CHERI intptr_t requires special treatment by the compiler. Plus, the nominal width of both pointers and intptr_t double in size--128-bits on 64-bit architectures. Most environments don't even have a 128-bit floating point type, even when the hardware might support it. (According to Wikipedia, Fortran is an exception, but it's still uncommon in C and other popular language environments.)
- Dylan16807 4y ago
  > C [optionally] supports converting pointers to intptr_t (or uintptr_t) Yes it's optional, but it's "generally" supported. > (even with nominally sufficient width) C99 says: "Any pointer type may be converted to an integer type. Except as previously specified, the result is implementation-defined. If the result cannot be represented in the integer type, the behavior is undefined. The result need not be in the range of values of any integer type." I'm not entirely sure how I'm supposed to parse that, but it sounds like if the result is in range of an integer then it's supposed to work? I guess you could technically say all results have a zillion added to the actual value? Rules lawyering gets annoying pretty fast. > and certainly not to a floating point type, which is how NaN boxing works NaN boxing works by converting to an integer type first. What it does with the bits afterwards is not important to C, as long as it reconstructs the same integer before converting back.
  - lokedhs 4y ago
    The part you quoted seems pretty clear. It's implementation defined. That's why it works done on most modern architectures, but you can't rely on it in portable code, as there are certainly architectures where it doesn't.
    - kazinator 4y ago
      implementation-defined is not "optional". The implementation must provide the conversion and document how it's done; it cannot say that, "oh, it's not supported and the program will refuse to compile, or behave erratically". However some stupid definition of behavior could be given, like converting any pointer to integer results in a reliable 0. C has pointer arithmetic; it must support subtraction of pointers that point into the same array-like object. So even on that implementation, you might perhaps be able to obtain a large-ish array and treat that as a heap in which you can convert pointers to integers by subtracting the base.
- klodolph 4y ago
  > not to any integer type (even with nominally sufficient width), I’m going to have to say that you’re definitely wrong about that, because when I read the C standard, it seems pretty clear to me. From N2176 (C17 draft) §6.3.2.3: > An integer may be converted to any pointer type. Except as previously specified, the result is implementation-defined, might not be correctly aligned, might not point to an entity of the referenced type, and might be a trap representation. > Any pointer type may be converted to an integer type. Except as previously specified, the result is implementation-defined. If the result cannot be represented in the integer type, the behavior is undefined. The result need not be in the range of values of any integer type. As I said, there are some implementation-dependent factors here, but it should be clear that casting a pointer to and from an integer is OK, depending on the implementation, and depending on what integer type—and it’s not just limited to intptr types. The C standard guarantees that you can always cast to and from intptr_t / uintptr_t (if it exists), and other casts are implementation-defined. Relying on implementation-defined behavior is ok. There’s nothing here that requires casting to float. You’re passing around a union which has both a float and a uint64_t in it.
LoganDark 4y ago
> You are generally free to convert a pointer to an integer and then back (I'm going to ignore your use of the term "generally" to acknowledge that there are exceptions, because I want to elaborate on a particular exception that highlights an error in reasoning that I believe most programmers make when they manipulate memory addresses and then cast them straight to pointers.) On CHERI, pure integer addressing can be turned off. In that case, pointers cannot necessarily be derived from integers alone. This is because CHERI implements pointer capabilities in hardware. You have to use special capability-preserving instructions to handle them, or else they permanently become invalid. You can still manipulate memory addresses, but you generally cannot use them directly without them being part of a valid pointer. So it's valid to take a memory address, but invalid to try to use it directly. I personally consider it unsound to try to transmute an integer into a pointer. It doesn't make sense, because [pointers are not necessarily memory addresses](https://www.ralfj.de/blog/2018/07/24/pointers-and-bytes.html https://www.ralfj.de/blog/2018/07/24/pointers-and-bytes.html). The Rust programming language is also considering taking this stance -- they already have a lint against treating pointers as integers, and with it an option to forbid it, though I'm unsure what the status on that is (it will probably remain optional forever). This doesn't necessarily make it wrong or incorrect if the language allows for it, but I would advise not doing so, unless it's a domain-specific optimization (i.e. there is a fallback for architectures where it does not apply).

MaulingMonkey 4y ago

> I don't think that pointer provenance rears its ugly head here. You are generally free to convert a pointer to an integer and then back It's worth noting that Rust is experimenting with making this not so free for one to do: https://doc.rust-lang.org/std/ptr/index.html#pointer-usize-pointer-roundtrips-and-exposed-provenance https://doc.rust-lang.org/std/ptr/index.html#pointer-usize-p...

wahern 4y ago

> I don't think that pointer provenance rears its ugly head here. You are generally free to convert a pointer to an integer and then back, which is what's happening here C [optionally] supports converting pointers to intptr_t (or uintptr_t), not to any integer type (even with nominally sufficient width), and certainly not to a floating point type, which is how NaN boxing works. In CHERI intptr_t requires special treatment by the compiler. Plus, the nominal width of both pointers and intptr_t double in size--128-bits on 64-bit architectures. Most environments don't even have a 128-bit floating point type, even when the hardware might support it. (According to Wikipedia, Fortran is an exception, but it's still uncommon in C and other popular language environments.)

Dylan16807 4y ago
> C [optionally] supports converting pointers to intptr_t (or uintptr_t) Yes it's optional, but it's "generally" supported. > (even with nominally sufficient width) C99 says: "Any pointer type may be converted to an integer type. Except as previously specified, the result is implementation-defined. If the result cannot be represented in the integer type, the behavior is undefined. The result need not be in the range of values of any integer type." I'm not entirely sure how I'm supposed to parse that, but it sounds like if the result is in range of an integer then it's supposed to work? I guess you could technically say all results have a zillion added to the actual value? Rules lawyering gets annoying pretty fast. > and certainly not to a floating point type, which is how NaN boxing works NaN boxing works by converting to an integer type first. What it does with the bits afterwards is not important to C, as long as it reconstructs the same integer before converting back.
- lokedhs 4y ago
  The part you quoted seems pretty clear. It's implementation defined. That's why it works done on most modern architectures, but you can't rely on it in portable code, as there are certainly architectures where it doesn't.
  - kazinator 4y ago
    implementation-defined is not "optional". The implementation must provide the conversion and document how it's done; it cannot say that, "oh, it's not supported and the program will refuse to compile, or behave erratically". However some stupid definition of behavior could be given, like converting any pointer to integer results in a reliable 0. C has pointer arithmetic; it must support subtraction of pointers that point into the same array-like object. So even on that implementation, you might perhaps be able to obtain a large-ish array and treat that as a heap in which you can convert pointers to integers by subtracting the base.
klodolph 4y ago
> not to any integer type (even with nominally sufficient width), I’m going to have to say that you’re definitely wrong about that, because when I read the C standard, it seems pretty clear to me. From N2176 (C17 draft) §6.3.2.3: > An integer may be converted to any pointer type. Except as previously specified, the result is implementation-defined, might not be correctly aligned, might not point to an entity of the referenced type, and might be a trap representation. > Any pointer type may be converted to an integer type. Except as previously specified, the result is implementation-defined. If the result cannot be represented in the integer type, the behavior is undefined. The result need not be in the range of values of any integer type. As I said, there are some implementation-dependent factors here, but it should be clear that casting a pointer to and from an integer is OK, depending on the implementation, and depending on what integer type—and it’s not just limited to intptr types. The C standard guarantees that you can always cast to and from intptr_t / uintptr_t (if it exists), and other casts are implementation-defined. Relying on implementation-defined behavior is ok. There’s nothing here that requires casting to float. You’re passing around a union which has both a float and a uint64_t in it.

LoganDark 4y ago

> You are generally free to convert a pointer to an integer and then back (I'm going to ignore your use of the term "generally" to acknowledge that there are exceptions, because I want to elaborate on a particular exception that highlights an error in reasoning that I believe most programmers make when they manipulate memory addresses and then cast them straight to pointers.) On CHERI, pure integer addressing can be turned off. In that case, pointers cannot necessarily be derived from integers alone. This is because CHERI implements pointer capabilities in hardware. You have to use special capability-preserving instructions to handle them, or else they permanently become invalid. You can still manipulate memory addresses, but you generally cannot use them directly without them being part of a valid pointer. So it's valid to take a memory address, but invalid to try to use it directly. I personally consider it unsound to try to transmute an integer into a pointer. It doesn't make sense, because [pointers are not necessarily memory addresses](https://www.ralfj.de/blog/2018/07/24/pointers-and-bytes.html https://www.ralfj.de/blog/2018/07/24/pointers-and-bytes.html). The Rust programming language is also considering taking this stance -- they already have a lint against treating pointers as integers, and with it an option to forbid it, though I'm unsure what the status on that is (it will probably remain optional forever). This doesn't necessarily make it wrong or incorrect if the language allows for it, but I would advise not doing so, unless it's a domain-specific optimization (i.e. there is a fallback for architectures where it does not apply).