3 ms·

> not to any integer type (even with nominally sufficient width), I’m going to have to say that you’re definitely wrong about that, because when I read the C s

> not to any integer type (even with nominally sufficient width), I’m going to have to say that you’re definitely wrong about that, because when I read the C standard, it seems pretty clear to me. From N2176 (C17 draft) §6.3.2.3: > An integer may be converted to any pointer type. Except as previously specified, the result is implementation-defined, might not be correctly aligned, might not point to an entity of the referenced type, and might be a trap representation. > Any pointer type may be converted to an integer type. Except as previously specified, the result is implementation-defined. If the result cannot be represented in the integer type, the behavior is undefined. The result need not be in the range of values of any integer type. As I said, there are some implementation-dependent factors here, but it should be clear that casting a pointer to and from an integer is OK, depending on the implementation, and depending on what integer type—and it’s not just limited to intptr types. The C standard guarantees that you can always cast to and from intptr_t / uintptr_t (if it exists), and other casts are implementation-defined. Relying on implementation-defined behavior is ok. There’s nothing here that requires casting to float. You’re passing around a union which has both a float and a uint64_t in it.

wahern 4y ago
This issue isn't round-tripping integers through object pointers, but round-tripping object pointers through integers.[1] C only guarantees the latter to occur through intptr_t, if at all. And you can't use plain pointers to round-trip integers because they have no observable structure by which you could inspect them for these user space tagging tricks. So you're stuck with intptr_t. Also, none of your points pertain to NaN boxing in particular. NaN boxing literally requires floating point types, and nothing in the standard permits floating point types to be used this way. NaN boxing fundamentally relies upon undefined behavior. All the modern hysteria regarding undefined behavior notwithstanding, however, clearly people can manage to implement workable solutions even in the face of what the standard makes undefined. But such de facto allowances are the first to be discarded as situations change, otherwise they would have been codified as part of a standard. As the article describes, most of what can be achieved from an optimization point with NaN boxing can be achieved by smuggling floats through pointers or integers, rather than vice-versa, but strictly speaking NaN boxing is especially problematic. [1] The standard says "Any pointer type may be converted to an integer type. Except as previously specified, the result is implementation-defined. If the result cannot be represented in the integer type, the behavior is undefined." But in CHERI 1 bit of capability data (the most critical bit!) is stored outside addressable memory and by design cannot be represented as part of an integer, except intptr_t, which as I described requires special compiler treatment. Moreover, this special treatment might not be free, obviating the performance advantages of type punning tricks. I forget the details, but IIRC in some cases a call into the kernel is actually required to preserve C's existing conversion semantics. And intptr_t semantics in CHERI more generallly likely present barriers to other kinds of compiler optimizations, even ignoring the fact that our memory saving type punning hack doesn't actually save memory anymore on account of both pointers and intptr_t being double the width of every other type in the union.
- Dylan16807 4y ago
  > nothing in the standard permits floating point types to be used this way. NaN boxing fundamentally relies upon undefined behavior. What do you mean? memcpy from bytes to a NaN should work fine, and there's also a "nan" function that can take a number to insert into the NaN payload. There are certainly implementation-defined aspects, but all of floating point is implementation defined. As long as you can get a number back out, you can make it into an integer and cast back to a pointer on 98% of platforms.
  - MaulingMonkey 4y ago
    > memcpy from bytes to a NaN should work fine Signaling NaNs are explicitly undefined in C11 F.2.1.: "This specification does not define the behavior of signaling NaNs." - and in practice may be "quieted" by conversion to Quiet NaNs, changing their bit patterns, or may trigger actual signals. Fast math optimization flags will also break the hell out of your code by assuming NaNs are impossible. I want to say there are more circumstances where optimizers and compiler generated code can butcher your NaN payloads, but I'd be working off recollected hearsay and I can't find a source, so don't quote me on that. NaN boxing is common enough that, if you take the right precautions, a modern compiler should probably support it, maybe. NaN boxing is uncommon enough that, if your codebase needs to be sufficiently portable, you need an opt out for when it breaks. Let's review duktape's scars: https://github.com/svaarala/duktape/blob/123d9426d5e5b36d5dab39b01bb3b9ef34be806d/src/duk_features.h.in#L1271-L1292 https://github.com/svaarala/duktape/blob/123d9426d5e5b36d5da... https://github.com/svaarala/duktape/blob/5252b7a50611a3cb8bfcd6856b6a4a899bc622ed/config/architectures/architecture_x86.h.in#L8-L13 https://github.com/svaarala/duktape/blob/5252b7a50611a3cb8bf... https://github.com/svaarala/duktape/blob/224a0b89ca08a36e37eea3dce4d254e4d0c188ac/misc/clang_aliasing.c https://github.com/svaarala/duktape/blob/224a0b89ca08a36e37e... (for context, a "packed" duk_tval involves NaN boxing) Note that "the right precautions" involve unions and proper integer types to avoid optimizer-invoked rewrites of the value and debugging when things go wrong, not simply YOLOing bytes into a double via memcpy. Note that debugging when it all goes terribly wrong can be quite painful. I've personally had the misfortune of being forced to debug duktape being built with fast math optimizatoins enabled on one "rare" platform + build configuration that wasn't caught by duktape's #if defined(__FAST_MATH__) checks linked above (wasn't Clang nor GCC, so go figure it didn't make the same #define)

wahern 4y ago

This issue isn't round-tripping integers through object pointers, but round-tripping object pointers through integers.[1] C only guarantees the latter to occur through intptr_t, if at all. And you can't use plain pointers to round-trip integers because they have no observable structure by which you could inspect them for these user space tagging tricks. So you're stuck with intptr_t. Also, none of your points pertain to NaN boxing in particular. NaN boxing literally requires floating point types, and nothing in the standard permits floating point types to be used this way. NaN boxing fundamentally relies upon undefined behavior. All the modern hysteria regarding undefined behavior notwithstanding, however, clearly people can manage to implement workable solutions even in the face of what the standard makes undefined. But such de facto allowances are the first to be discarded as situations change, otherwise they would have been codified as part of a standard. As the article describes, most of what can be achieved from an optimization point with NaN boxing can be achieved by smuggling floats through pointers or integers, rather than vice-versa, but strictly speaking NaN boxing is especially problematic. [1] The standard says "Any pointer type may be converted to an integer type. Except as previously specified, the result is implementation-defined. If the result cannot be represented in the integer type, the behavior is undefined." But in CHERI 1 bit of capability data (the most critical bit!) is stored outside addressable memory and by design cannot be represented as part of an integer, except intptr_t, which as I described requires special compiler treatment. Moreover, this special treatment might not be free, obviating the performance advantages of type punning tricks. I forget the details, but IIRC in some cases a call into the kernel is actually required to preserve C's existing conversion semantics. And intptr_t semantics in CHERI more generallly likely present barriers to other kinds of compiler optimizations, even ignoring the fact that our memory saving type punning hack doesn't actually save memory anymore on account of both pointers and intptr_t being double the width of every other type in the union.

Dylan16807 4y ago
> nothing in the standard permits floating point types to be used this way. NaN boxing fundamentally relies upon undefined behavior. What do you mean? memcpy from bytes to a NaN should work fine, and there's also a "nan" function that can take a number to insert into the NaN payload. There are certainly implementation-defined aspects, but all of floating point is implementation defined. As long as you can get a number back out, you can make it into an integer and cast back to a pointer on 98% of platforms.
- MaulingMonkey 4y ago
  > memcpy from bytes to a NaN should work fine Signaling NaNs are explicitly undefined in C11 F.2.1.: "This specification does not define the behavior of signaling NaNs." - and in practice may be "quieted" by conversion to Quiet NaNs, changing their bit patterns, or may trigger actual signals. Fast math optimization flags will also break the hell out of your code by assuming NaNs are impossible. I want to say there are more circumstances where optimizers and compiler generated code can butcher your NaN payloads, but I'd be working off recollected hearsay and I can't find a source, so don't quote me on that. NaN boxing is common enough that, if you take the right precautions, a modern compiler should probably support it, maybe. NaN boxing is uncommon enough that, if your codebase needs to be sufficiently portable, you need an opt out for when it breaks. Let's review duktape's scars: https://github.com/svaarala/duktape/blob/123d9426d5e5b36d5dab39b01bb3b9ef34be806d/src/duk_features.h.in#L1271-L1292 https://github.com/svaarala/duktape/blob/123d9426d5e5b36d5da... https://github.com/svaarala/duktape/blob/5252b7a50611a3cb8bfcd6856b6a4a899bc622ed/config/architectures/architecture_x86.h.in#L8-L13 https://github.com/svaarala/duktape/blob/5252b7a50611a3cb8bf... https://github.com/svaarala/duktape/blob/224a0b89ca08a36e37eea3dce4d254e4d0c188ac/misc/clang_aliasing.c https://github.com/svaarala/duktape/blob/224a0b89ca08a36e37e... (for context, a "packed" duk_tval involves NaN boxing) Note that "the right precautions" involve unions and proper integer types to avoid optimizer-invoked rewrites of the value and debugging when things go wrong, not simply YOLOing bytes into a double via memcpy. Note that debugging when it all goes terribly wrong can be quite painful. I've personally had the misfortune of being forced to debug duktape being built with fast math optimizatoins enabled on one "rare" platform + build configuration that wasn't caught by duktape's #if defined(__FAST_MATH__) checks linked above (wasn't Clang nor GCC, so go figure it didn't make the same #define)