9 ms·

Experimental Rust Feature: Safer Interoperable ABI

210 points by mvelbaum 4y ago

the_mitsuhiko 4y ago
This is an incredible exciting development. One thing I would like to point out is that the existence and progress of this PR is showing that there has been a change in approach in Rust over the last year or so which wasn't there previously. A lot of long standing issues were basically blocked on them just being too big to solve due to the surface area. Now the Rust project is more willing to slice problems down into smaller sets and giving those a try. This ABI will be limited, but despite those limits it has a lot of utility.
pabs3 4y ago
Would this allow Rust libraries to be dynamically linked like C libraries are often?
- masklinn 4y ago
  > The interoperable ABI does not aim to support the full richness of Rust's type system, or that of other languages. It aims to support common cases more safely and simply. So not much (or any) more than it currently is: it's not a stable Rust ABI, it's instead a more expressive ABI compared to C.
  - kibwen 4y ago
    That said, even though it won't be "a stable Rust ABI", it will (if the experiment succeeds) be a stable ABI that is more Rust-like than the current C ABI.
- JoshTriplett 4y ago
  Yes, to some degree. You can dynamically link Rust today using the C ABI, but then you lose all the richness and safety of Rust types. The interoperable ABI will allow more of Rust's type system to work across the boundary.
- flohofwoe 4y ago
  As far as I understand the proposal, it's "just" standardizing how some high level Rust types would be automatically translated into a "C ABI" compatible representation, sort of like a builtin bindings generator. But I wonder why DLL linkage wouldn't already be possible without this proposal. C++ doesn't have a standardized ABI either, but it's possible to load shared libraries with C++ interfaces just fine - as long as they've been created with the same compiler and compiler version. Since there's only a single Rust compiler vendor I'd imagine that dynamic linkage should already be much less trouble than in C++ land?
  - lmm 4y ago
    Rust moves much faster than C++, at least at the moment. So the practical benefit of sharing libraries created with the same compiler version would be much smaller.
  - leni536 4y ago
    C++ absolutely has standardised ABIs, like the Itanium C++ ABI implemented by clang and gcc. The situation is not any better or worse than for C.
    - pjmlp 4y ago
      And on Windows, COM and WinRT, although they are language agnostic.
  - Rusky 4y ago
    DLL linkage with the same compiler has been possible since the beginning- for instance rustc itself is built this way. This is about going beyond that so people can interop with other languages, link with DLLs without rebuilding them every 6 weeks, etc
- matheusmoreira 4y ago
  If it's a simple enough binary interface, sure. Simplicity is really important. C++ has standardized ABIs but you just don't see people dynamically loading a C++ shared object, obtaining a pointer to a C++ object and calling its member functions. They always do it through the C ABI if they do it at all. Some would rather rewrite stuff in C than put up with that. C++ ABIs are so obnoxiously complex the only things that dare to touch it are C++ compilers and even they have been known to break things. I just get this incredibly hopeless feeling whenever I try to contemplate how I'd handle a C++ exception from Python code. The only reasonable answer I can think of is I wouldn't, I'd just disable C++ exceptions instead and hope for the best. People use C because it's just simple symbols and unchanging calling conventions. Look up symbols to get function pointers, put parameters in specific registers, call the function and off you go. Nobody will interface with Rust if the ABI is too complex. Given the language's many high level abstractions, I think it's likely we'll see a repeat of the C++ situation above.
  - AshamedCaptain 4y ago
    > C++ has standardized ABIs but you just don't see people dynamically loading a C++ shared object, obtaining a pointer to a C++ object and calling its member functions. They always do it through the C ABI if they do it at all. That does not match my experience at all. From the top of my head, Qt (QtPlugin) does C++ ABI plugins, and they do that almost transparently in a myriad of platforms. > I just get this incredibly hopeless feeling whenever I try to contemplate how I'd handle a C++ exception from Python code. At one point I could even catch C++ exceptions from Java, and viceversa (see gcj).
    - matheusmoreira 4y ago
      > From the top of my head, Qt (QtPlugin) does C++ ABI plugins This? https://wiki.qt.io/Plugins https://wiki.qt.io/Plugins > provides a bunch of macros that helps us create the C-function that generates the plugin object Looks like it uses the C ABI. > At one point I could even catch C++ exceptions from Java, and viceversa (see gcj). I'm interested in the details of that. Perhaps the GCJ compiler generated some sort of bridge between the languages?
ianlevesque 4y ago
So, COM?
- speps 4y ago
  No COM is a ref-counted land of objects with virtual tables of functions. This proposal seems to define how to convert C ABI to/from Rust types in a standard way. This will likely improve Rust+C use cases over time.
shmerl 4y ago
This looks very cool. Can it help Rust / C++ interop?
- the_mitsuhiko 4y ago
  This primarily helps you write dynamically loadable modules without being restricted to the unsafe C ABI that Rust already provides.
  - shmerl 4y ago
    So it mostly helps making libraries in Rust for other languages to use?
    - the_mitsuhiko 4y ago
      It primarily helps making dynamically loadable Rust libraries that Rust can load. Everything else is downstream from there. So for instance if you have a web server written in Rust and you want to dynamically load a plugin also written in Rust.
    - cormacrelf 4y ago
      Yes. Note that they are intending to build it on top of the C ABI that already exists. So anything you can do with the new ABI, you can already do today. It just currently takes a lot of work. Every &[u8] slice you want to pass has to be rewritten as two arguments for pointer and length. That’s the easy stuff; error handling is as painful as it always is in C. You have to sit down and essentially bang out a C header file as Rust extern “C” functions, using error codes, fancy ways to represent error conditions as invalid return values (eg negative numbers) and all those tricks. Even if your target language is eg Swift and Swift can absolutely be taught to understand slice types, result types, nullable types etc, since it has all of these things natively. And then you have to write an actual C header file or use cbindgen, and then do the whole thing in reverse to climb back up to the high level types in the target language. As I said: a lot of work. The general idea is for someone to eventually teach Swift about the slice type etc. When Swift defines the same mappings from its high level types (Optional etc) to C ABI representation as Rust does. And when that work is done you don’t need to drop down to banging out C headers and writing the repetitive conversion code on either end. All that would remain is keeping the “extern” declarations/imports in sync with the exports, which will be much easier once higher level types can be used easily in FFIs. Making an IDL to describe them is out of scope for now but clearly a possibility.
raydiatian 4y ago
Relevant/related topic & HN discussion which can help contextualize the value add I think maybe “C isnt a programming language, it’s a protocol” https://faultlore.com/blah/c-isnt-a-language/ https://faultlore.com/blah/c-isnt-a-language/ https://news.ycombinator.com/item?id=33509223 https://news.ycombinator.com/item?id=33509223
- matheusmoreira 4y ago
  These articles about ABIs are really good, thank you. I'd like to comment on the following: > Um sorry what? This is Bappyscript, not C. Where’s the Bappyscript interface for Linux? Right here! https://man7.org/linux/man-pages/man2/syscall.2.html#NOTES https://man7.org/linux/man-pages/man2/syscall.2.html#NOTES On Linux, you don't need C at all since you can interface with the kernel directly. The ABI is stable and simple enough that a new programming language could have a system_call keyword that makes the compiler emit the system call code. With this one piece of functionality, it's possible to do literally anything on Linux. No need for C libraries and their legacy at all. The hardest part will be describing the Linux user space API data structures in the new language so that they can be passed to and from the kernel.
  - striking 4y ago
    Well, alright, but then you still need to parse C to be able to divine the shape of the data structures, as they're defined in C (or you have to make some assumptions).
    - matheusmoreira 4y ago
      Yes. Fortunately, these definitions are significantly less complex than libc stuff. They all use typedefs prefixed with __kernel that are defined in asm and asm-generic headers. The Linux kernel headers with all relevant structure definitions are here: https://github.com/torvalds/linux/tree/master/include/uapi https://github.com/torvalds/linux/tree/master/include/uapi
    - mhh__ 4y ago
kevincox 4y ago
This is interesting but seems very basic. It looks like it is just lowering types that were previously too complex into C ABI. Notably it doesn't solve anything about versioning. I honestly quite like the Swift approach. It isn't zero overhead but it defaults to really good comparability. Basically it adds a vtable for everything including member offsets and object sizes. But the really nice thing is that you can tell the compiler when you don't need compatibility (code and types in the same library, or a library that you statically link) and the overhead just disappears. Of course you lose a few language features (you can't know the size of a type) but then they have tags that you can apply to trade compatibility/flexibility to get those features back.
- the_mitsuhiko 4y ago
  > Notably it doesn't solve anything about versioning. It does promise backwards compatibility. How is that not sufficient?
  - kevincox 4y ago
    That just pushes the problem onto the user. There is lots of well-documented problems with C versioning. Just saying "don't change the size, order, alignment or layout of anything ever" isn't a great option, it can be done (Microsoft has done a pretty good job) but it requires infinite foresight and is very expensive. Swift allows libraries to naturally evolve while maintaining comparability. Most things that you would expect to work (add a new field) just work. Good luck adding a new field to `jmp_buf`. It's effectively impossible to do in an ABI compatible way. The fact that C's ABI is stable/backwards compatible doesn't help. Basically a backwards compatible ABI is the absolute minimum requirement for making backwards compatible libraries. Having more options removes heaps of complexity for library authors.
    - the_mitsuhiko 4y ago
      That’s a fair ask but I’m quite okay with that not being solved. Once the building blocks are stable you can build on top of that.
mlindner 4y ago
This is huge news!! I've been waiting for something like this from the Rust team for years! I hope libraries can be built for other languages that would allow cleaner interfacing directly with Rust that doesn't rely on the boat anchor of the legacy C ABI that all languages are tied to! This is the most important "feature" of Rust for it's long term success. Edit: It looks like this is still using C ABI as it's base, which increases the burden for other languages to implement it. What a let down...
- generichuman 4y ago
  > It looks like this is still using C ABI as it's base, which increases the burden for other languages to implement it. What a let down... How so? It seems to me that actually decreases the burden. For example C# people may not implement the Rust ABI if they don't want to, but since they (and basically every language) already support C ABI, I can implement a library over the existing C ABI so that I can access Rust types.
  - kibwen 4y ago
    Correct, the entire goal of defining it in terms of the C ABI is that C is the lingua franca that every language already speaks, so anything built on top of that has a dramatically lowered barrier to entry. I assume that the grandparent poster was thinking about making it possible for hypothetical future languages to someday be able to forego supporting the C ABI entirely, which is certainly a laudable goal (there's plenty of historical baggage there), but what the Rust developers care more about here is simply being able to do FFI that's safer than what currently exists.
  - mlindner 4y ago
    Because if you're implementing it properly with the advanced features of a language such as Rust, you need to similarly create an entirely different ABI that goes down to the C ABI, increasing the complexity of the implementation because the C ABI has so many edge cases. Basing it on the C ABI makes it only easier for "simple" languages like C to communicate with Rust.
afranchuk 4y ago
This is great! For those unaware, the abi_stable crate makes stable ABIs (even with complex features like trait objects) pretty easy and, importantly, verifiable. It is primarily useful for rust-to-rust abi stability (for instance when creating a plugin system).
amluto 4y ago
I think it would be amazing if there was also a way to export a machine readable description of the layout of an interoperable object. Then other languages could parse it instead of needing to parse Rust code and know all the ABI rules.
- duped 4y ago
  This is typically called an IDL and TLA mentions inspiration from several examples under "prior art." At the top is the Web Assembly Interface proposal which has the wit [0] format, [0] https://github.com/WebAssembly/component-model/blob/main/design/mvp/WIT.md https://github.com/WebAssembly/component-model/blob/main/des...
  - mrfox321 4y ago
    To expand acronyms / jargon IDL := Interface Description Language TLA := linked article
    - duped 4y ago
      TLA - the linked article
brundolf 4y ago
Would this simplify things like wasm-bindgen too?

210 points by mvelbaum 4y ago

This is an incredible exciting development. One thing I would like to point out is that the existence and progress of this PR is showing that there has been a change in approach in Rust over the last year or so which wasn't there previously. A lot of long standing issues were basically blocked on them just being too big to solve due to the surface area. Now the Rust project is more willing to slice problems down into smaller sets and giving those a try. This ABI will be limited, but despite those limits it has a lot of utility.

pabs3 4y ago

Would this allow Rust libraries to be dynamically linked like C libraries are often?

masklinn 4y ago
> The interoperable ABI does not aim to support the full richness of Rust's type system, or that of other languages. It aims to support common cases more safely and simply. So not much (or any) more than it currently is: it's not a stable Rust ABI, it's instead a more expressive ABI compared to C.
- kibwen 4y ago
  That said, even though it won't be "a stable Rust ABI", it will (if the experiment succeeds) be a stable ABI that is more Rust-like than the current C ABI.
JoshTriplett 4y ago
Yes, to some degree. You can dynamically link Rust today using the C ABI, but then you lose all the richness and safety of Rust types. The interoperable ABI will allow more of Rust's type system to work across the boundary.
flohofwoe 4y ago
As far as I understand the proposal, it's "just" standardizing how some high level Rust types would be automatically translated into a "C ABI" compatible representation, sort of like a builtin bindings generator. But I wonder why DLL linkage wouldn't already be possible without this proposal. C++ doesn't have a standardized ABI either, but it's possible to load shared libraries with C++ interfaces just fine - as long as they've been created with the same compiler and compiler version. Since there's only a single Rust compiler vendor I'd imagine that dynamic linkage should already be much less trouble than in C++ land?
- lmm 4y ago
  Rust moves much faster than C++, at least at the moment. So the practical benefit of sharing libraries created with the same compiler version would be much smaller.
- leni536 4y ago
  C++ absolutely has standardised ABIs, like the Itanium C++ ABI implemented by clang and gcc. The situation is not any better or worse than for C.
  - pjmlp 4y ago
    And on Windows, COM and WinRT, although they are language agnostic.
- Rusky 4y ago
  DLL linkage with the same compiler has been possible since the beginning- for instance rustc itself is built this way. This is about going beyond that so people can interop with other languages, link with DLLs without rebuilding them every 6 weeks, etc
matheusmoreira 4y ago
If it's a simple enough binary interface, sure. Simplicity is really important. C++ has standardized ABIs but you just don't see people dynamically loading a C++ shared object, obtaining a pointer to a C++ object and calling its member functions. They always do it through the C ABI if they do it at all. Some would rather rewrite stuff in C than put up with that. C++ ABIs are so obnoxiously complex the only things that dare to touch it are C++ compilers and even they have been known to break things. I just get this incredibly hopeless feeling whenever I try to contemplate how I'd handle a C++ exception from Python code. The only reasonable answer I can think of is I wouldn't, I'd just disable C++ exceptions instead and hope for the best. People use C because it's just simple symbols and unchanging calling conventions. Look up symbols to get function pointers, put parameters in specific registers, call the function and off you go. Nobody will interface with Rust if the ABI is too complex. Given the language's many high level abstractions, I think it's likely we'll see a repeat of the C++ situation above.
- AshamedCaptain 4y ago
  > C++ has standardized ABIs but you just don't see people dynamically loading a C++ shared object, obtaining a pointer to a C++ object and calling its member functions. They always do it through the C ABI if they do it at all. That does not match my experience at all. From the top of my head, Qt (QtPlugin) does C++ ABI plugins, and they do that almost transparently in a myriad of platforms. > I just get this incredibly hopeless feeling whenever I try to contemplate how I'd handle a C++ exception from Python code. At one point I could even catch C++ exceptions from Java, and viceversa (see gcj).
  - matheusmoreira 4y ago
    > From the top of my head, Qt (QtPlugin) does C++ ABI plugins This? https://wiki.qt.io/Plugins https://wiki.qt.io/Plugins > provides a bunch of macros that helps us create the C-function that generates the plugin object Looks like it uses the C ABI. > At one point I could even catch C++ exceptions from Java, and viceversa (see gcj). I'm interested in the details of that. Perhaps the GCJ compiler generated some sort of bridge between the languages?

ianlevesque 4y ago

So, COM?

speps 4y ago
No COM is a ref-counted land of objects with virtual tables of functions. This proposal seems to define how to convert C ABI to/from Rust types in a standard way. This will likely improve Rust+C use cases over time.

shmerl 4y ago

This looks very cool. Can it help Rust / C++ interop?

the_mitsuhiko 4y ago
This primarily helps you write dynamically loadable modules without being restricted to the unsafe C ABI that Rust already provides.
- shmerl 4y ago
  So it mostly helps making libraries in Rust for other languages to use?
  - the_mitsuhiko 4y ago
    It primarily helps making dynamically loadable Rust libraries that Rust can load. Everything else is downstream from there. So for instance if you have a web server written in Rust and you want to dynamically load a plugin also written in Rust.
  - cormacrelf 4y ago
    Yes. Note that they are intending to build it on top of the C ABI that already exists. So anything you can do with the new ABI, you can already do today. It just currently takes a lot of work. Every &[u8] slice you want to pass has to be rewritten as two arguments for pointer and length. That’s the easy stuff; error handling is as painful as it always is in C. You have to sit down and essentially bang out a C header file as Rust extern “C” functions, using error codes, fancy ways to represent error conditions as invalid return values (eg negative numbers) and all those tricks. Even if your target language is eg Swift and Swift can absolutely be taught to understand slice types, result types, nullable types etc, since it has all of these things natively. And then you have to write an actual C header file or use cbindgen, and then do the whole thing in reverse to climb back up to the high level types in the target language. As I said: a lot of work. The general idea is for someone to eventually teach Swift about the slice type etc. When Swift defines the same mappings from its high level types (Optional etc) to C ABI representation as Rust does. And when that work is done you don’t need to drop down to banging out C headers and writing the repetitive conversion code on either end. All that would remain is keeping the “extern” declarations/imports in sync with the exports, which will be much easier once higher level types can be used easily in FFIs. Making an IDL to describe them is out of scope for now but clearly a possibility.

raydiatian 4y ago

Relevant/related topic & HN discussion which can help contextualize the value add I think maybe “C isnt a programming language, it’s a protocol” https://faultlore.com/blah/c-isnt-a-language/ https://faultlore.com/blah/c-isnt-a-language/ https://news.ycombinator.com/item?id=33509223 https://news.ycombinator.com/item?id=33509223

matheusmoreira 4y ago
These articles about ABIs are really good, thank you. I'd like to comment on the following: > Um sorry what? This is Bappyscript, not C. Where’s the Bappyscript interface for Linux? Right here! https://man7.org/linux/man-pages/man2/syscall.2.html#NOTES https://man7.org/linux/man-pages/man2/syscall.2.html#NOTES On Linux, you don't need C at all since you can interface with the kernel directly. The ABI is stable and simple enough that a new programming language could have a system_call keyword that makes the compiler emit the system call code. With this one piece of functionality, it's possible to do literally anything on Linux. No need for C libraries and their legacy at all. The hardest part will be describing the Linux user space API data structures in the new language so that they can be passed to and from the kernel.
- striking 4y ago
  Well, alright, but then you still need to parse C to be able to divine the shape of the data structures, as they're defined in C (or you have to make some assumptions).
  - matheusmoreira 4y ago
    Yes. Fortunately, these definitions are significantly less complex than libc stuff. They all use typedefs prefixed with __kernel that are defined in asm and asm-generic headers. The Linux kernel headers with all relevant structure definitions are here: https://github.com/torvalds/linux/tree/master/include/uapi https://github.com/torvalds/linux/tree/master/include/uapi
  - mhh__ 4y ago

kevincox 4y ago

This is interesting but seems very basic. It looks like it is just lowering types that were previously too complex into C ABI. Notably it doesn't solve anything about versioning. I honestly quite like the Swift approach. It isn't zero overhead but it defaults to really good comparability. Basically it adds a vtable for everything including member offsets and object sizes. But the really nice thing is that you can tell the compiler when you don't need compatibility (code and types in the same library, or a library that you statically link) and the overhead just disappears. Of course you lose a few language features (you can't know the size of a type) but then they have tags that you can apply to trade compatibility/flexibility to get those features back.

the_mitsuhiko 4y ago
> Notably it doesn't solve anything about versioning. It does promise backwards compatibility. How is that not sufficient?
- kevincox 4y ago
  That just pushes the problem onto the user. There is lots of well-documented problems with C versioning. Just saying "don't change the size, order, alignment or layout of anything ever" isn't a great option, it can be done (Microsoft has done a pretty good job) but it requires infinite foresight and is very expensive. Swift allows libraries to naturally evolve while maintaining comparability. Most things that you would expect to work (add a new field) just work. Good luck adding a new field to `jmp_buf`. It's effectively impossible to do in an ABI compatible way. The fact that C's ABI is stable/backwards compatible doesn't help. Basically a backwards compatible ABI is the absolute minimum requirement for making backwards compatible libraries. Having more options removes heaps of complexity for library authors.
  - the_mitsuhiko 4y ago
    That’s a fair ask but I’m quite okay with that not being solved. Once the building blocks are stable you can build on top of that.

mlindner 4y ago

This is huge news!! I've been waiting for something like this from the Rust team for years! I hope libraries can be built for other languages that would allow cleaner interfacing directly with Rust that doesn't rely on the boat anchor of the legacy C ABI that all languages are tied to! This is the most important "feature" of Rust for it's long term success. Edit: It looks like this is still using C ABI as it's base, which increases the burden for other languages to implement it. What a let down...

generichuman 4y ago
> It looks like this is still using C ABI as it's base, which increases the burden for other languages to implement it. What a let down... How so? It seems to me that actually decreases the burden. For example C# people may not implement the Rust ABI if they don't want to, but since they (and basically every language) already support C ABI, I can implement a library over the existing C ABI so that I can access Rust types.
- kibwen 4y ago
  Correct, the entire goal of defining it in terms of the C ABI is that C is the lingua franca that every language already speaks, so anything built on top of that has a dramatically lowered barrier to entry. I assume that the grandparent poster was thinking about making it possible for hypothetical future languages to someday be able to forego supporting the C ABI entirely, which is certainly a laudable goal (there's plenty of historical baggage there), but what the Rust developers care more about here is simply being able to do FFI that's safer than what currently exists.
- mlindner 4y ago
  Because if you're implementing it properly with the advanced features of a language such as Rust, you need to similarly create an entirely different ABI that goes down to the C ABI, increasing the complexity of the implementation because the C ABI has so many edge cases. Basing it on the C ABI makes it only easier for "simple" languages like C to communicate with Rust.

afranchuk 4y ago

This is great! For those unaware, the abi_stable crate makes stable ABIs (even with complex features like trait objects) pretty easy and, importantly, verifiable. It is primarily useful for rust-to-rust abi stability (for instance when creating a plugin system).

amluto 4y ago

I think it would be amazing if there was also a way to export a machine readable description of the layout of an interoperable object. Then other languages could parse it instead of needing to parse Rust code and know all the ABI rules.

duped 4y ago
This is typically called an IDL and TLA mentions inspiration from several examples under "prior art." At the top is the Web Assembly Interface proposal which has the wit [0] format, [0] https://github.com/WebAssembly/component-model/blob/main/design/mvp/WIT.md https://github.com/WebAssembly/component-model/blob/main/des...
- mrfox321 4y ago
  To expand acronyms / jargon IDL := Interface Description Language TLA := linked article
  - duped 4y ago
    TLA - the linked article

brundolf 4y ago

Would this simplify things like wasm-bindgen too?