8 ms·

Pair Your Compilers at the ABI Café

130 points by nrabulinski 2y ago

fowl2 2y ago
I mean there's tons of IDLs, so many no one can agree on which one! oblig: https://xkcd.com/927/ https://xkcd.com/927/
gumby 2y ago
It's Interop for compilers!
not2b 2y ago
There is a specified common C++ ABI that gcc, clang, Intel's proprietary compiler, and others use. It was originally developed for the Itanium processor but is now used by gcc and clang for everything. See https://itanium-cxx-abi.github.io/cxx-abi/abi.html https://itanium-cxx-abi.github.io/cxx-abi/abi.html Unfortunately this ABI didn't specify how __int128 (and other nonstandard types) are to be passed.
- jcranmer 2y ago
  The Itanium ABI effectively specifies how to lower the C++ ABI to an assumed C ABI, and the C ABI is given by what is known as the "psABI" (processor-specific ABI). The (not-most-recent) x86-64 ABI is here: https://raw.githubusercontent.com/wiki/hjl-tools/x86-psABI/x86-64-psABI-1.0.pdf https://raw.githubusercontent.com/wiki/hjl-tools/x86-psABI/x..., and it does actually explain how to pass __int128.
- kevingadd 2y ago
  WebAssembly also has a de-facto standard ABI: https://github.com/WebAssembly/tool-conventions/blob/main/BasicCABI.md#function-signatures https://github.com/WebAssembly/tool-conventions/blob/main/Ba...
jcranmer 2y ago
I can definitely feel the pain of trying to work out ABI mismatch concerns. It doesn't help that it often isn't clear from the output what some of the underlying assumptions are--expected stack alignment, for example, or structs being broken up into registers or not. It would be nice if compilers could output some sort of metadata that basically says "ah yes, here's a struct, it requires this alignment, fields are at these offsets and have these sizes, and are present in these cases" (the latter option being able to support discriminated unions) or "function call, parameters are here, here, and here." You'd think this is what DWARF itself provides, but if you play around with DWARF for a bit, you discover that it actually lacks a lot of the low-level ABI details you want to uncover; instead, it's more of a format that's meant to be generic enough to convey to the debugger the AST of the source program along with some hint of how to map the binary code to that AST--you can't really write a language-agnostic DWARF-based debugger.
- PartiallyTyped 2y ago
  What you are asking for sounds quite a bit like what rustc does. Rustc (outside `extern "c"`) offers no guarantees on the ordering of the fields, however, it guarantees that every instance of struct A will have the same ordering during that particular compilation. This allows rustc to compile external crates (as long as no monomorphization is needed) in a consistent manner across all crates that depend on that.
  - menaerus 2y ago
    Most of the ABI issues arise when you start to mix and match shared libaries produced by different compilers, or even the libraries produced by the different versions of the same compiler. Rust has none of that, nor does support dynamic linking, so I fail to understand what is it that rustc can offer in that solution space. There is none.
    - bluGill 2y ago
      Rust works around the issue by not allowing all the useful things that get you there. There are other useful things like sharing pointers across threads that rust will not let you do - for both better and worse. (better in that you avoid a lot of problems for something you rarely need - worse for those few cases where you actually need to do those and cannot)
      - Georgelemental 2y ago
        You can share pointers across threads in Rust, it's just `unsafe`.
- orivej 2y ago
  Some Common Lisp FFIs have opted to coax this information out of the compiler. https://github.com/rpav/c2ffi https://github.com/rpav/c2ffi is a C++ tool that links to libclang-cpp and literally outputs JSON with sizes and alignments. (It is then used by https://github.com/rpav/cl-autowrap https://github.com/rpav/cl-autowrap to autogenerate a Lisp wrapper.) The older CFFI Groveller [1] works by generating C code which is compiled by the system C compiler (e.g. GCC or Clang) and, when executed, prints Lisp code that contains resolved values of constants, sizes, alignments, etc. [1] https://cffi.common-lisp.dev/manual/html_node/The-Groveller.html#The-Groveller https://cffi.common-lisp.dev/manual/html_node/The-Groveller....
w10-1 2y ago
Also function pointers, errors & exception-handling, async/channels/thread-local's, go stacks, swift @objc, @cdecl and cpp inter-op, FFI dialects... It's not really pain anymore; it's a kind of hilarity
zX41ZdbW 2y ago
I think the right way to avoid this problem is to avoid using ABI at runtime or build time. At runtime, it means - don't use shared libraries. At build time, it means - build every library from the source, don't use pre-built artifacts. This sounds controversial... But it allows you to change compiler or compiler options at any time, and you don't have to bother. It also enables cross-compilation, reproducible builds, and portable binaries. You no longer have to ask developers to set up a complex build environment on a specific Linux distribution because it works everywhere. I use this approach for ClickHouse.
- comex 2y ago
  Even then, you still need ABI consistency between compilers if you want to link together codebases written in different languages (e.g. C and Rust). In practice this almost always 'just works' because most cross-language calls simply don't use the kinds of complicated types discussed in the blog post. They tend to stick to simple integer and pointer types, where ABI consistency is usually a given. Though you can still get into trouble when passing function pointers, especially when combined with some modern control-flow integrity systems.
  - tester756 2y ago
    >Even then, you still need ABI consistency between compilers if you want to link together codebases written in different languages (e.g. C and Rust). Let's talk over http, queue or other IPC-ish way
    - 0xdeafbeef 2y ago
      You can't pass values in registers using this model
    - dwattttt 2y ago
      You _still_ need a consistent way to talk about values; IPC systems tackle the same problems under the name marshalling & de/serialisation. They just tend to take much more conservative options to deal with exactly this kind of problem (you don't have to care about integer endian-ness if integers are expressed as strings).
gigel82 2y ago
I struggled with this many times and at the end of the day threw down the towel and just wrapped everything in plain C exports. That's the only way I know to get ABI compatibility across different compilers/toolsets/versions. COM-like constructs come as a close second. It's an unfortunate state.
tialaramex 2y ago
If I understand correctly there's also an ABI problem for synchronization rules. Within a compiler, it actually doesn't matter whether we think about a problem as A mustn't happen after B, or as B mustn't happen before A. But expressing this across an ABI we have to be careful that we don't have buck passing. Suppose Language #1 thinks of it the first way, and Language #2 the second way, now if Language #1 is responsible for B while Language #2 is responsible for A, each may believe the other will have taken care of ordering and no synchronization is actually implemented. Overall, "ABI" turns out to mean something like "Every assumption you've made which can be detected by other software, including assumptions you didn't realise you had". Discovering all your assumptions is hard, accepting that other people assumed different and they aren't just wrong is also surprisingly hard.
- JonChesterfield 2y ago
  Yes, if you're brave enough to have fence position as part of the calling convention. I think you're safe if ordering is expressed within a given function, or by sequence of calls to functions with the same ideas of fences.

130 points by nrabulinski 2y ago

fowl2 2y ago

I mean there's tons of IDLs, so many no one can agree on which one! oblig: https://xkcd.com/927/ https://xkcd.com/927/

gumby 2y ago

It's Interop for compilers!

not2b 2y ago

There is a specified common C++ ABI that gcc, clang, Intel's proprietary compiler, and others use. It was originally developed for the Itanium processor but is now used by gcc and clang for everything. See https://itanium-cxx-abi.github.io/cxx-abi/abi.html https://itanium-cxx-abi.github.io/cxx-abi/abi.html Unfortunately this ABI didn't specify how __int128 (and other nonstandard types) are to be passed.

jcranmer 2y ago
The Itanium ABI effectively specifies how to lower the C++ ABI to an assumed C ABI, and the C ABI is given by what is known as the "psABI" (processor-specific ABI). The (not-most-recent) x86-64 ABI is here: https://raw.githubusercontent.com/wiki/hjl-tools/x86-psABI/x86-64-psABI-1.0.pdf https://raw.githubusercontent.com/wiki/hjl-tools/x86-psABI/x..., and it does actually explain how to pass __int128.
kevingadd 2y ago
WebAssembly also has a de-facto standard ABI: https://github.com/WebAssembly/tool-conventions/blob/main/BasicCABI.md#function-signatures https://github.com/WebAssembly/tool-conventions/blob/main/Ba...

jcranmer 2y ago

I can definitely feel the pain of trying to work out ABI mismatch concerns. It doesn't help that it often isn't clear from the output what some of the underlying assumptions are--expected stack alignment, for example, or structs being broken up into registers or not. It would be nice if compilers could output some sort of metadata that basically says "ah yes, here's a struct, it requires this alignment, fields are at these offsets and have these sizes, and are present in these cases" (the latter option being able to support discriminated unions) or "function call, parameters are here, here, and here." You'd think this is what DWARF itself provides, but if you play around with DWARF for a bit, you discover that it actually lacks a lot of the low-level ABI details you want to uncover; instead, it's more of a format that's meant to be generic enough to convey to the debugger the AST of the source program along with some hint of how to map the binary code to that AST--you can't really write a language-agnostic DWARF-based debugger.

PartiallyTyped 2y ago
What you are asking for sounds quite a bit like what rustc does. Rustc (outside `extern "c"`) offers no guarantees on the ordering of the fields, however, it guarantees that every instance of struct A will have the same ordering during that particular compilation. This allows rustc to compile external crates (as long as no monomorphization is needed) in a consistent manner across all crates that depend on that.
- menaerus 2y ago
  Most of the ABI issues arise when you start to mix and match shared libaries produced by different compilers, or even the libraries produced by the different versions of the same compiler. Rust has none of that, nor does support dynamic linking, so I fail to understand what is it that rustc can offer in that solution space. There is none.
  - bluGill 2y ago
    Rust works around the issue by not allowing all the useful things that get you there. There are other useful things like sharing pointers across threads that rust will not let you do - for both better and worse. (better in that you avoid a lot of problems for something you rarely need - worse for those few cases where you actually need to do those and cannot)
    - Georgelemental 2y ago
      You can share pointers across threads in Rust, it's just `unsafe`.
orivej 2y ago
Some Common Lisp FFIs have opted to coax this information out of the compiler. https://github.com/rpav/c2ffi https://github.com/rpav/c2ffi is a C++ tool that links to libclang-cpp and literally outputs JSON with sizes and alignments. (It is then used by https://github.com/rpav/cl-autowrap https://github.com/rpav/cl-autowrap to autogenerate a Lisp wrapper.) The older CFFI Groveller [1] works by generating C code which is compiled by the system C compiler (e.g. GCC or Clang) and, when executed, prints Lisp code that contains resolved values of constants, sizes, alignments, etc. [1] https://cffi.common-lisp.dev/manual/html_node/The-Groveller.html#The-Groveller https://cffi.common-lisp.dev/manual/html_node/The-Groveller....

w10-1 2y ago

Also function pointers, errors & exception-handling, async/channels/thread-local's, go stacks, swift @objc, @cdecl and cpp inter-op, FFI dialects... It's not really pain anymore; it's a kind of hilarity

zX41ZdbW 2y ago

I think the right way to avoid this problem is to avoid using ABI at runtime or build time. At runtime, it means - don't use shared libraries. At build time, it means - build every library from the source, don't use pre-built artifacts. This sounds controversial... But it allows you to change compiler or compiler options at any time, and you don't have to bother. It also enables cross-compilation, reproducible builds, and portable binaries. You no longer have to ask developers to set up a complex build environment on a specific Linux distribution because it works everywhere. I use this approach for ClickHouse.

comex 2y ago
Even then, you still need ABI consistency between compilers if you want to link together codebases written in different languages (e.g. C and Rust). In practice this almost always 'just works' because most cross-language calls simply don't use the kinds of complicated types discussed in the blog post. They tend to stick to simple integer and pointer types, where ABI consistency is usually a given. Though you can still get into trouble when passing function pointers, especially when combined with some modern control-flow integrity systems.
- tester756 2y ago
  >Even then, you still need ABI consistency between compilers if you want to link together codebases written in different languages (e.g. C and Rust). Let's talk over http, queue or other IPC-ish way
  - 0xdeafbeef 2y ago
    You can't pass values in registers using this model
  - dwattttt 2y ago
    You _still_ need a consistent way to talk about values; IPC systems tackle the same problems under the name marshalling & de/serialisation. They just tend to take much more conservative options to deal with exactly this kind of problem (you don't have to care about integer endian-ness if integers are expressed as strings).

gigel82 2y ago

I struggled with this many times and at the end of the day threw down the towel and just wrapped everything in plain C exports. That's the only way I know to get ABI compatibility across different compilers/toolsets/versions. COM-like constructs come as a close second. It's an unfortunate state.

tialaramex 2y ago

If I understand correctly there's also an ABI problem for synchronization rules. Within a compiler, it actually doesn't matter whether we think about a problem as A mustn't happen after B, or as B mustn't happen before A. But expressing this across an ABI we have to be careful that we don't have buck passing. Suppose Language #1 thinks of it the first way, and Language #2 the second way, now if Language #1 is responsible for B while Language #2 is responsible for A, each may believe the other will have taken care of ordering and no synchronization is actually implemented. Overall, "ABI" turns out to mean something like "Every assumption you've made which can be detected by other software, including assumptions you didn't realise you had". Discovering all your assumptions is hard, accepting that other people assumed different and they aren't just wrong is also surprisingly hard.

JonChesterfield 2y ago
Yes, if you're brave enough to have fence position as part of the calling convention. I think you're safe if ordering is expressed within a given function, or by sequence of calls to functions with the same ideas of fences.