4 ms·

Pointer provenance is a nice example. A block of memory cannot be read as an array of simd types sometimes and scalar types otherwise. It can't contain atomic v

Pointer provenance is a nice example. A block of memory cannot be read as an array of simd types sometimes and scalar types otherwise. It can't contain atomic values which are operated on using non-atomic operations during program startup before you spawn any threads. There were proposals to let one mmap existing structures but I don't know if any landed. Usually done with reinterpret cast and hoping that rule violation doesn't break you. Pointer provenance does make most application code faster but other times it opens a performance gap that you have to step outside of C++ to close. Compiler extensions, switching off the analysis, changing language.

angiosperm 3y ago
Use of mmap itself is undefined in the language. Posix provides a definition that programs rely on, instead. Implementers are allowed to define literally anything the union of all standards leaves undefined.
- JonChesterfield 3y ago
  Mmap itself is alright. You've got a void* from somewhere, that's OK. You can placement new into it to make objects. What isn't allowed is casting it to a hashtable type and then using it as such. Because there is no hashtable instance anywhere, and specifically not there, so you've violated the pointer aliasing rules. The obvious fix is to guarantee that placement new doesn't change the bytes, perhaps only for trivially copyable types or similar constraint. I didn't see the proposals in that direction land but also didn't see them fail, so maybe the newer standard permits it.
  - LegionMammal978 3y ago
    As I understand it, that's precisely what std::start_lifetime_as<T>() does: it effectively performs a placement new to create a T object, except that it retains the existing bytes at the address. It only works with implicit-lifetime types (i.e., scalars, or classes with a trivial constructor), though, so it probably wouldn't work with your hash table example, except perhaps for an inline hash table.
    - JonChesterfield 3y ago
      Superb! Looking through https://en.cppreference.com/w/cpp/memory/start_lifetime_as https://en.cppreference.com/w/cpp/memory/start_lifetime_as, this appears to be the right thing. It also has volatile overloads (which it looks like placement new still does not). This doesn't appear to be implemented in libc++ yet but that seems fixable, it'll go down the same object construction logic placement new does. Thank you for the reference, that'll fix some ugly edge cases in one of my libraries.
  - klankerzz 3y ago
    Can't you also just say that mmap() is a magical function that you don't know what it does? For all the compiler knows, mmap() could just be a: static Hash_Table h; return (void *)&h; And make that the rule for all externally defined functions.
    - gpderetta 3y ago
      Indeed. That's the way to reason about correctness of opaque functions.
  - angiosperm 3y ago
    To call mmap, you are calling a function that is not in the collection of translation units that makes the program. Libraries are beyond the Standard. Include files not listed in the Standard, likewise. So you rely on Posix, there. For objects got by casting void* to a known type, you rely on the compiler being unable to prove that the objects didn't exist already, somewhere in the program. Pray the compiler doesn't get smart enough to notice no constructor for that type is linked, meaning you that you couldn't have made that object.
agalunar 3y ago
> A block of memory cannot be read as an array of simd types sometimes and scalar types otherwise. As far as I can tell, it is currently the case that, using raw pointers, this is not actually undefined behavior (but I never entirely trust my conclusions on these matters). "&mut T and &T follow LLVM’s scoped noalias model" [1][referring to 2 and 3] but I am fairly sure this does not currently apply to raw pointers, and "provenance is implicitly shared with all pointers transitively derived from the original pointer through operations like offset, borrowing, and pointer casts." [4] [1] https://doc.rust-lang.org/reference/behavior-considered-undefined.html https://doc.rust-lang.org/reference/behavior-considered-unde... [2] https://llvm.org/docs/LangRef.html#pointeraliasing https://llvm.org/docs/LangRef.html#pointeraliasing [3] "noalias" under https://llvm.org/docs/LangRef.html#parameter-attributes https://llvm.org/docs/LangRef.html#parameter-attributes [4] https://doc.rust-lang.org/core/ptr/index.html https://doc.rust-lang.org/core/ptr/index.html Also excellent are https://faultlore.com/blah/fix-rust-pointers https://faultlore.com/blah/fix-rust-pointers https://www.ralfj.de/blog/2018/07/24/pointers-and-bytes.html https://www.ralfj.de/blog/2018/07/24/pointers-and-bytes.html https://www.ralfj.de/blog/2020/12/14/provenance.html https://www.ralfj.de/blog/2020/12/14/provenance.html https://www.ralfj.de/blog/2022/04/11/provenance-exposed.html https://www.ralfj.de/blog/2022/04/11/provenance-exposed.html It seems likely you'd already be familiar with these; I'm just putting them out there for anyone interested.
- JonChesterfield 3y ago
  LLVM can represent various aliasing relationships, modulo some risk of C++ inspired bugs in some passes. They might all be stamped out now. I remember a bug report about one that was open for many years. I'm happy to hear rust can (probably) represent the same relationships LLVM can. C++ cannot, at least as of about two years ago when I last looked through the corresponding papers. All it can do is different types do not alias, where atomic_int and int are different types.
  - tialaramex 3y ago
    No, LLVM definitely still has big problems. https://github.com/llvm/llvm-project/issues/45725 https://github.com/llvm/llvm-project/issues/45725 is an example, the symptom in Rust is that you can write what is in effect a pointer comparison in which LLVM ends up claiming that two things are different, although they are also identical...

angiosperm 3y ago

Use of mmap itself is undefined in the language. Posix provides a definition that programs rely on, instead. Implementers are allowed to define literally anything the union of all standards leaves undefined.

JonChesterfield 3y ago
Mmap itself is alright. You've got a void* from somewhere, that's OK. You can placement new into it to make objects. What isn't allowed is casting it to a hashtable type and then using it as such. Because there is no hashtable instance anywhere, and specifically not there, so you've violated the pointer aliasing rules. The obvious fix is to guarantee that placement new doesn't change the bytes, perhaps only for trivially copyable types or similar constraint. I didn't see the proposals in that direction land but also didn't see them fail, so maybe the newer standard permits it.
- LegionMammal978 3y ago
  As I understand it, that's precisely what std::start_lifetime_as<T>() does: it effectively performs a placement new to create a T object, except that it retains the existing bytes at the address. It only works with implicit-lifetime types (i.e., scalars, or classes with a trivial constructor), though, so it probably wouldn't work with your hash table example, except perhaps for an inline hash table.
  - JonChesterfield 3y ago
    Superb! Looking through https://en.cppreference.com/w/cpp/memory/start_lifetime_as https://en.cppreference.com/w/cpp/memory/start_lifetime_as, this appears to be the right thing. It also has volatile overloads (which it looks like placement new still does not). This doesn't appear to be implemented in libc++ yet but that seems fixable, it'll go down the same object construction logic placement new does. Thank you for the reference, that'll fix some ugly edge cases in one of my libraries.
- klankerzz 3y ago
  Can't you also just say that mmap() is a magical function that you don't know what it does? For all the compiler knows, mmap() could just be a: static Hash_Table h; return (void *)&h; And make that the rule for all externally defined functions.
  - gpderetta 3y ago
    Indeed. That's the way to reason about correctness of opaque functions.
- angiosperm 3y ago
  To call mmap, you are calling a function that is not in the collection of translation units that makes the program. Libraries are beyond the Standard. Include files not listed in the Standard, likewise. So you rely on Posix, there. For objects got by casting void* to a known type, you rely on the compiler being unable to prove that the objects didn't exist already, somewhere in the program. Pray the compiler doesn't get smart enough to notice no constructor for that type is linked, meaning you that you couldn't have made that object.

agalunar 3y ago

> A block of memory cannot be read as an array of simd types sometimes and scalar types otherwise. As far as I can tell, it is currently the case that, using raw pointers, this is not actually undefined behavior (but I never entirely trust my conclusions on these matters). "&mut T and &T follow LLVM’s scoped noalias model" [1][referring to 2 and 3] but I am fairly sure this does not currently apply to raw pointers, and "provenance is implicitly shared with all pointers transitively derived from the original pointer through operations like offset, borrowing, and pointer casts." [4] [1] https://doc.rust-lang.org/reference/behavior-considered-undefined.html https://doc.rust-lang.org/reference/behavior-considered-unde... [2] https://llvm.org/docs/LangRef.html#pointeraliasing https://llvm.org/docs/LangRef.html#pointeraliasing [3] "noalias" under https://llvm.org/docs/LangRef.html#parameter-attributes https://llvm.org/docs/LangRef.html#parameter-attributes [4] https://doc.rust-lang.org/core/ptr/index.html https://doc.rust-lang.org/core/ptr/index.html Also excellent are https://faultlore.com/blah/fix-rust-pointers https://faultlore.com/blah/fix-rust-pointers https://www.ralfj.de/blog/2018/07/24/pointers-and-bytes.html https://www.ralfj.de/blog/2018/07/24/pointers-and-bytes.html https://www.ralfj.de/blog/2020/12/14/provenance.html https://www.ralfj.de/blog/2020/12/14/provenance.html https://www.ralfj.de/blog/2022/04/11/provenance-exposed.html https://www.ralfj.de/blog/2022/04/11/provenance-exposed.html It seems likely you'd already be familiar with these; I'm just putting them out there for anyone interested.

JonChesterfield 3y ago
LLVM can represent various aliasing relationships, modulo some risk of C++ inspired bugs in some passes. They might all be stamped out now. I remember a bug report about one that was open for many years. I'm happy to hear rust can (probably) represent the same relationships LLVM can. C++ cannot, at least as of about two years ago when I last looked through the corresponding papers. All it can do is different types do not alias, where atomic_int and int are different types.
- tialaramex 3y ago
  No, LLVM definitely still has big problems. https://github.com/llvm/llvm-project/issues/45725 https://github.com/llvm/llvm-project/issues/45725 is an example, the symptom in Rust is that you can write what is in effect a pointer comparison in which LLVM ends up claiming that two things are different, although they are also identical...