4 ms·

I found this article hard to read. I was trying to understand what the current state of affairs is, and how these new abstractions improve the matter. E.g. the

I found this article hard to read. I was trying to understand what the current state of affairs is, and how these new abstractions improve the matter. E.g. the consistency properties, the weakenings, and when they help. Instead it read more like an exasperated rant, that links to rust toolchain improvements. It does have an account of how to use the new features, which was good. But I couldn't glean when and how that would help me as a user

kmeisthax 5y ago
There's a prior post that this one is building off of, but basically: 1. Pointers are not addresses; they are pairs of (allocation, address). This is known as pointers having provenance. 2. Treating a pointer as an int-sized address is merely a hardware optimization; and there is already hardware that deliberately does not do this (e.g. CHERI and maybe ARM PAC). 3. The compiler needs to know the allocation part of the pointer (the one that gets lost at runtime) in order to determine if a pointer write will change a local variable. If this association is lost then you get miscompiles. 4. Converting an integer back into a pointer (e.g. usize as ptr) does not establish pointer provenance, will break CHERI, and will miscompile on other architectures. 5. Rust made the mistake of allowing #4 in unsafe code. This is entirely unsound. 6. The proposed strict-provenance APIs allows doing something like #4, but sound, by letting the user stitch an address onto a pointer with a compatible allocation. This re-establishes the chain of provenance and avoids the miscompile.
- Gankra 5y ago
  Rust is no more unsound than C/C++/Swift here. It basically works but that's a really frustrating answer for anyone who cares about validation/sanitizers/optimization/docs. It's just better if most code doesn't poke the dragon (llvm) and has trivially correct semantics instead of "yeah this has to work but um, don't ask me how or why".
- zozbot234 5y ago
  Looks like the root mistake here is conflating `usize` (the integer type giving the maximum size of a data object, and the type of offsets into an allocation) and `uptr` (the size of a pointer's integer representation). In a segmented memory architecture, or one using 'object-oriented memory', the two need not be the same. You need pretty much the same sort of tweaks to be able to effectively target systems like the old 8086/80286, the 65816 (of Apple II GS and SNES fame) or the Intel iAPX 432.
  - kmeisthax 5y ago
    Even on segmented memory, `uptr` doesn't carry provenance. You still either need `with_addr` to tell the compiler that this integer aliases an existing allocation, or `with_exposed_addr` to tell the compiler that you want your use-after-free bug to also have a CVE number.
  - xenadu02 5y ago
    It is more complicated than that. The issue is really that sometimes you legitimately need to access or manipulate the bits of a pointer, but once you convert a pointer to any integer type then convert it back to a pointer that allows such manipulation the compiler only has two choices: 1. Assume the new pointer could be pointing at _anything_ and so disable all optimizations that might possibly be affected. So even trivial things like deducing that "x == y" can be eliminated because x is never modified is right out the window... after all that random integer could somehow have ended up pointing at the memory for x. 2. Assume the new pointer doesn't point at anything the compiler knows about, causing it to make optimizations that break the program. The "x == y" that it eliminated because it hoisted x out of the loop and nothing modified x... turns out your magic pointer from nowhere was pointing at x and now your modifications are never read. Or maybe it decided to put x in a register so it has no memory address. Programmers get really angry when compilers do this. Provenance is the compiler tracking that you converted ptr -> x, fiddled the bits, then converted x -> ptr2 and being more conservative around optimizations. The Rust change means by default you would only convert a pointer to an integer type right at the point you need to twiddle the bits and only for as long as you need to do so. The compiler then has a better understanding of what you are doing and doesn't need to assume pointers could be pointing at anything. You have to remember the compiler is doing various kinds of inlining, code transformations, etc. Normal things we expect compilers to do. A bunch of small optimizations executed, sometimes iteratively. You can't point to any specific optimization and say "that's the one that breaks things" so you can't just magic this problem away. Programmers hate compilers that make really slow code (#1), then get really angry when the compiler breaks their program (#2), then get upset about the fix slowing down their code (#1) and the cycle repeats. Often programmers assume compiler writers are just psychopaths who refuse to fix such "obvious" problems. They often pronounce exactly how things should work, demonstrating they're at "step 1" of baby's first memory model. If fixing this problem with pointers were easy we'd have done it a long time ago.

kmeisthax 5y ago

There's a prior post that this one is building off of, but basically: 1. Pointers are not addresses; they are pairs of (allocation, address). This is known as pointers having provenance. 2. Treating a pointer as an int-sized address is merely a hardware optimization; and there is already hardware that deliberately does not do this (e.g. CHERI and maybe ARM PAC). 3. The compiler needs to know the allocation part of the pointer (the one that gets lost at runtime) in order to determine if a pointer write will change a local variable. If this association is lost then you get miscompiles. 4. Converting an integer back into a pointer (e.g. usize as ptr) does not establish pointer provenance, will break CHERI, and will miscompile on other architectures. 5. Rust made the mistake of allowing #4 in unsafe code. This is entirely unsound. 6. The proposed strict-provenance APIs allows doing something like #4, but sound, by letting the user stitch an address onto a pointer with a compatible allocation. This re-establishes the chain of provenance and avoids the miscompile.

Gankra 5y ago
Rust is no more unsound than C/C++/Swift here. It basically works but that's a really frustrating answer for anyone who cares about validation/sanitizers/optimization/docs. It's just better if most code doesn't poke the dragon (llvm) and has trivially correct semantics instead of "yeah this has to work but um, don't ask me how or why".
zozbot234 5y ago
Looks like the root mistake here is conflating `usize` (the integer type giving the maximum size of a data object, and the type of offsets into an allocation) and `uptr` (the size of a pointer's integer representation). In a segmented memory architecture, or one using 'object-oriented memory', the two need not be the same. You need pretty much the same sort of tweaks to be able to effectively target systems like the old 8086/80286, the 65816 (of Apple II GS and SNES fame) or the Intel iAPX 432.
- kmeisthax 5y ago
  Even on segmented memory, `uptr` doesn't carry provenance. You still either need `with_addr` to tell the compiler that this integer aliases an existing allocation, or `with_exposed_addr` to tell the compiler that you want your use-after-free bug to also have a CVE number.
- xenadu02 5y ago
  It is more complicated than that. The issue is really that sometimes you legitimately need to access or manipulate the bits of a pointer, but once you convert a pointer to any integer type then convert it back to a pointer that allows such manipulation the compiler only has two choices: 1. Assume the new pointer could be pointing at _anything_ and so disable all optimizations that might possibly be affected. So even trivial things like deducing that "x == y" can be eliminated because x is never modified is right out the window... after all that random integer could somehow have ended up pointing at the memory for x. 2. Assume the new pointer doesn't point at anything the compiler knows about, causing it to make optimizations that break the program. The "x == y" that it eliminated because it hoisted x out of the loop and nothing modified x... turns out your magic pointer from nowhere was pointing at x and now your modifications are never read. Or maybe it decided to put x in a register so it has no memory address. Programmers get really angry when compilers do this. Provenance is the compiler tracking that you converted ptr -> x, fiddled the bits, then converted x -> ptr2 and being more conservative around optimizations. The Rust change means by default you would only convert a pointer to an integer type right at the point you need to twiddle the bits and only for as long as you need to do so. The compiler then has a better understanding of what you are doing and doesn't need to assume pointers could be pointing at anything. You have to remember the compiler is doing various kinds of inlining, code transformations, etc. Normal things we expect compilers to do. A bunch of small optimizations executed, sometimes iteratively. You can't point to any specific optimization and say "that's the one that breaks things" so you can't just magic this problem away. Programmers hate compilers that make really slow code (#1), then get really angry when the compiler breaks their program (#2), then get upset about the fix slowing down their code (#1) and the cycle repeats. Often programmers assume compiler writers are just psychopaths who refuse to fix such "obvious" problems. They often pronounce exactly how things should work, demonstrating they're at "step 1" of baby's first memory model. If fixing this problem with pointers were easy we'd have done it a long time ago.