3 ms·

I think we should agree that `intmax_t` was a mistake and not focus on it so much in articles like this. It's enough that the C ABI is so hard to pin down, and

I think we should agree that `intmax_t` was a mistake and not focus on it so much in articles like this. It's enough that the C ABI is so hard to pin down, and that C headers are hard to parse. What we really need to interop w/ C but w/o C is a C compiler based tool that outputs DWARF or DWARF-like debug output that actually lays out everything -- type sizes and layouts down to the individual bits that bitfields map to, enum values, constant-like macros, etc. Function-like macros you'll never really be able to use from other languages, so, oh well. Such debug metadata would have to be encoded in a stable/committed encoding, expressed in a stable schema. With that sort of debug info a language like Rust could take care of all C interop natively w/o C, especially if those debug files were included with the OS so that Rust wouldn't need to invoke the tool that generates them. Though, obviously some `-D...` C compiler/pre-processor arguments can radically change the contents of the parsed headers, so some rationalization would be needed to cut down the number of "ABIs" to a manageable set (like GNU, BSD, POSIX).

nerdponx 4y ago
> C compiler based tool that outputs DWARF or DWARF-like debug output that actually lays out everything That sounds like C2FFI: https://github.com/rpav/c2ffi https://github.com/rpav/c2ffi The only language ecosystem where I've seen it used is Common Lisp, in the CL-Autowrap library (https://github.com/rpav/cl-autowrap https://github.com/rpav/cl-autowrap). But C2FFI emits plain JSON, so I don't see any reason why you couldn't build e.g. a Python auto-binding library on top of it. It depends on LLVM to generate the spec file, but end users don't need to have that.
- trynewideas 4y ago
  It also kinda sorta sounds like this tool the author wrote: https://github.com/Gankra/abi-cafe https://github.com/Gankra/abi-cafe > This tool helps automate testing that two languages/compilers agree on ABIs for the purposes of FFI. This is still in early development so lots of stuff is stubbed out. ... > By running this natively on whatever platform you care about, this will tell you what FFI interfaces do and don't currently work. Ideally all you need to do is cargo run, but we're dealing with native toolchains so, expect toolchain bugs!
- skissane 4y ago
  The idea of plain JSON as a format for expressing ABIs is great. I wish it was in LLVM/GCC out-of-the-box, rather than a separate open source project (whose maintainer only seems to have rather limited time to spend on it). Ideally there would be a standard format so LLVM, GCC, MSVC, etc, could all produce it (obviously with some extension points since each has some unique features the other doesn’t). From memory, the actual JSON produced by c2ffi follows rather closely clang’s internal data structures and so might not be the right design for something intended to be generic.
  - cryptonector 4y ago
    It doesn't have to be JSON. It just has to be in a formal syntax and semantics so we can interchange.
    - skissane 4y ago
      Sure it doesn’t have to be JSON. It could be YAML, XML, ASN.1, S-expressions, whatever. But I think JSON is a clear winner (nowadays) on the basis of being simple, easy to parse (compared to C/C++/etc which have very complex syntax), parsing libraries being extremely widely available (almost every language, even the most obscure, has a JSON parser available; languages which lack JSON support in their standard library, such as C, are now the exceptional minority.) Of course it is more voluminous and slower than a binary format. One could always define a binary encoding which straightforwardly maps to the JSON one (or just reuse an existing one such as UBJSON, BSON, CBOR, etc)
      - cryptonector 4y ago
        Verbosity doesn't really matter in this context. Schema does. I agree that JSON is probably the best choice from among the choices you listed. (Also, ASN.1 has JSON encoding rules, FYI. ASN.1 is not just the awful DER.)

nerdponx 4y ago

> C compiler based tool that outputs DWARF or DWARF-like debug output that actually lays out everything That sounds like C2FFI: https://github.com/rpav/c2ffi https://github.com/rpav/c2ffi The only language ecosystem where I've seen it used is Common Lisp, in the CL-Autowrap library (https://github.com/rpav/cl-autowrap https://github.com/rpav/cl-autowrap). But C2FFI emits plain JSON, so I don't see any reason why you couldn't build e.g. a Python auto-binding library on top of it. It depends on LLVM to generate the spec file, but end users don't need to have that.

trynewideas 4y ago
It also kinda sorta sounds like this tool the author wrote: https://github.com/Gankra/abi-cafe https://github.com/Gankra/abi-cafe > This tool helps automate testing that two languages/compilers agree on ABIs for the purposes of FFI. This is still in early development so lots of stuff is stubbed out. ... > By running this natively on whatever platform you care about, this will tell you what FFI interfaces do and don't currently work. Ideally all you need to do is cargo run, but we're dealing with native toolchains so, expect toolchain bugs!
skissane 4y ago
The idea of plain JSON as a format for expressing ABIs is great. I wish it was in LLVM/GCC out-of-the-box, rather than a separate open source project (whose maintainer only seems to have rather limited time to spend on it). Ideally there would be a standard format so LLVM, GCC, MSVC, etc, could all produce it (obviously with some extension points since each has some unique features the other doesn’t). From memory, the actual JSON produced by c2ffi follows rather closely clang’s internal data structures and so might not be the right design for something intended to be generic.
- cryptonector 4y ago
  It doesn't have to be JSON. It just has to be in a formal syntax and semantics so we can interchange.
  - skissane 4y ago
    Sure it doesn’t have to be JSON. It could be YAML, XML, ASN.1, S-expressions, whatever. But I think JSON is a clear winner (nowadays) on the basis of being simple, easy to parse (compared to C/C++/etc which have very complex syntax), parsing libraries being extremely widely available (almost every language, even the most obscure, has a JSON parser available; languages which lack JSON support in their standard library, such as C, are now the exceptional minority.) Of course it is more voluminous and slower than a binary format. One could always define a binary encoding which straightforwardly maps to the JSON one (or just reuse an existing one such as UBJSON, BSON, CBOR, etc)
    - cryptonector 4y ago
      Verbosity doesn't really matter in this context. Schema does. I agree that JSON is probably the best choice from among the choices you listed. (Also, ASN.1 has JSON encoding rules, FYI. ASN.1 is not just the awful DER.)