3 ms·

Talking about foundational software but no mention of the biggest missing part of rust IMO: an ABI. If you want to write an OS in rust and provide rich service

Talking about foundational software but no mention of the biggest missing part of rust IMO: an ABI. If you want to write an OS in rust and provide rich services for applications to use, you need to offer libraries they can call without needing to recompile when the OS is upgraded. Windows mostly does this with COM, Apple historically had ObjC’s dynamic dispatch (and now Swift’s ABI), Android does this with a JVM and bytecode… Rust can only really offer extern "C", and that really limits how useful dynamic libraries can be. Doing an ABI without a VM-like layer (JVM, .NET) is really difficult though, and requires you to commit to certain implementation details without ever changing them, so I can understand why it’s not a priority. To my knowledge the only success stories are Swift (which faced the problem head-on) and COM (which has a language-independent IDL and other pain points.) ObjC has such extreme late binding that it almost feels like cheating. If rust had a full-featured ABI it would be nearly the perfect language. (It would improve compile times as well, because we’d move to a world where dependencies could just be binaries, which would cut compile times massively.)

loglog 1y ago
Not your main point, but how would monomorphization of generics work with binary dependencies?
- ninkendo 1y ago
  It wouldn’t, there would have to be dynamic dispatch. Swift provides some prior art for “monomorphization in the same binary, dynamic dispatch across linker boundaries” as a general approach, and it works pretty well.
jenadine 1y ago
Not sure if this is really needed though. I mean, it would be more convenient to be able to pass more "things" around interfaces (slices, trait objects, ...), but you can still do everything you need to with an explicit extern "C" abi. (And I've seen proposal to extend extern ABI to more things)
- boredatoms 1y ago
  Something slightly more than extern C would be nice, such as definitive way to use Option/Result/Enum across the C boundary
  - eptcyka 1y ago
    Would you want to define that boundary in a way that still maintains compatibility with C? Or would all other languages be expected to link in a Rust library to then do the "extern C" conversion from a rust native type to something that is FFI stable? Swift can do this because Apple will tell you to use Swift or GTFO. Windows has COM. Rust is an orphan without a Primary Platform™ pushing for it, even if the rust language team came up with a technical solution, be it an IDL or a stable ABI with bend-over-backwards run-time linking magic, it would only be useful for other Rust binaries or it would be just as limited as the lingua franca of ABIs - C.
  - ronsor 1y ago
    The way C3[0] handles this is fairly interesting. An function returning an optional in C3, such as: fn int? run_operation_by_id(int id); Would be equivalent to the following in C: /* ABI type definitions omitted for brevity */ c3fault_t run_operation_by_id(int *return_value, int id); [0] https://c3-lang.org https://c3-lang.org
    - deleted 1y ago
      [deleted]
robertknight 1y ago
I've seen talks on this topic at Rust conferences which seemed strongly influenced by Swift's approach, so that will probably be the direction this ends up going in.
- IshKebab 1y ago
  The WASI Component Model also has a richer ABI. I wonder if that could be copied to native platforms somehow.
ameliaquining 1y ago
I don't think Rust ever wants to do ABI stability that isn't opt-in, because Rust is about zero-cost abstractions and a stable ABI is not zero-cost. See this explanation of how C++'s de facto stable ABI significantly reduces performance: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p2028r0.pdf https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p20... [PDF] For the use case of letting Rust programs dlopen plugins that are also written in Rust, the current solutions are stabby and abi_stable. They're not perfect but they seem to work fairly well in practice. For more general use cases like cross-language interop, the hope is to get somewhere on crABI: https://github.com/joshtriplett/rfcs/blob/crabi-v1/text/3470-crabi-v1.md https://github.com/joshtriplett/rfcs/blob/crabi-v1/text/3470... This is intended to be a superior successor to C's ABI, useful for all the same use cases (at least if the code you want to talk to is new enough to support it). Note that this isn't something the Rust maintainers can do unilaterally; there's a need to get buy-in from maintainers of other languages, Linux distros, etc. I'm not sure that the "everything is a shared library" approach is workable in a language where heap allocation is generally explicit and opt-in. While Swift has some support for stack allocation, most types are heap-allocated, and I think that reduces the extent to which the rest of the language has to be warped to accommodate the stable ABI.
- ninkendo 1y ago
  > While Swift has some support for stack allocation, most types are heap-allocated, and I think that reduces the extent to which the rest of the language has to be warped to accommodate the stable ABI. Stack vs heap in swift is simple: structs are stack, classes are heap. This is what makes Swift’s effort so impressive: it works with stack data just as well as heap. Stack allocated types being used across library boundaries has a ton of plumbing to make it work, there’s the concept of a value witness table (like a vtable but for size/stride information) that can allow looking up sizes at runtime. Stack allocation makes heavy use of alloca(), which most languages shy away from (since it means stack frames aren’t of a static size.) It was difficult but swift managed to do it. For rust to do something similar it would have to fix a lot of cases where object safety is broken, and yeah, this may require changing the language, particularly to support dynamic/runtime size information. (It would also make the dyn keyword effectively obsolete, because to make generics work across library boundaries, any type has to be able to “become” dynamic if the linker needs it to be.) It would definitely require changes to the language, but Swift provides some inspiring prior art here. This article goes into the details: https://faultlore.com/blah/swift-abi/ https://faultlore.com/blah/swift-abi/
yoshuaw 1y ago
We are more or less trying to solve this exact problem with Wasm Components [1]. If core WebAssembly is a portable instruction format, WebAssembly Components are a portable executable/linkable format. Using them is unfortunately not yet quite as easy as using repr(wasm)/extern "wasm". But with wit-bindgen [2] and the wasm32-wasip2 target [3], it's not that hard either. [1]: https://youtu.be/tAACYA1Mwv4 https://youtu.be/tAACYA1Mwv4 [2]: https://github.com/bytecodealliance/wit-bindgen https://github.com/bytecodealliance/wit-bindgen [3]: https://doc.rust-lang.org/nightly/rustc/platform-support/wasm32-wasip2.html https://doc.rust-lang.org/nightly/rustc/platform-support/was...
- drogus 1y ago
  I love WASM and WASI, but it's not nearly the same, unfortunately. Performance takes a hit, you can't use async in a straightforward way, launching hundreds of thousands of tasks is problematic etc. WASM is great for allowing to extend your app, but I don't see it as a replacement for an ABI anytime soon
nh2 1y ago
I don't really understand the point for writing an "OS in rust and provide rich services for applications to use". You can use any serialisation method you want (C structs, Protobuf, JSON, ...). An ABI is for passing data in the programming language and trusting it blindly, e.g. what functions use when calling each other. Any sane OS still has to validate the data that crosses its boundaries. For example, if you make a Linux system call, it doesn't blindly use a pointer you pass it, but instead actually checks if the pointer belongs to the calling process's address space. It seems pointless to pick Rust's internal function-calling data serialisation for the OS boundary; that wouldn't make it easier for any userspace programs running in that OS than explicitly defining a serialisation format. Why would an OS want to limit itself to Rust's internal representation, or why would Rust want to freeze its internal representation, making improvements impossible on both sides? This seems to bring only drawbacks. The only benefit I can see so far for a stable ABI is dynamic linking (and its use in "plugins" for programs where the code is equally trusted, e.g. writing some dlopen()able `.so` files in Rust, as plugins for Rust-only programs), where you can really "blindly call the symbol". But even there it's a bit questionable, as any mistake or mismatch in ABI can result in immediately memory corruption, which is something that Rust users rather dislike.
anon-3988 1y ago
We already have? Its called C.

loglog 1y ago

Not your main point, but how would monomorphization of generics work with binary dependencies?

ninkendo 1y ago
It wouldn’t, there would have to be dynamic dispatch. Swift provides some prior art for “monomorphization in the same binary, dynamic dispatch across linker boundaries” as a general approach, and it works pretty well.

jenadine 1y ago

Not sure if this is really needed though. I mean, it would be more convenient to be able to pass more "things" around interfaces (slices, trait objects, ...), but you can still do everything you need to with an explicit extern "C" abi. (And I've seen proposal to extend extern ABI to more things)

boredatoms 1y ago
Something slightly more than extern C would be nice, such as definitive way to use Option/Result/Enum across the C boundary
- eptcyka 1y ago
  Would you want to define that boundary in a way that still maintains compatibility with C? Or would all other languages be expected to link in a Rust library to then do the "extern C" conversion from a rust native type to something that is FFI stable? Swift can do this because Apple will tell you to use Swift or GTFO. Windows has COM. Rust is an orphan without a Primary Platform™ pushing for it, even if the rust language team came up with a technical solution, be it an IDL or a stable ABI with bend-over-backwards run-time linking magic, it would only be useful for other Rust binaries or it would be just as limited as the lingua franca of ABIs - C.
- ronsor 1y ago
  The way C3[0] handles this is fairly interesting. An function returning an optional in C3, such as: fn int? run_operation_by_id(int id); Would be equivalent to the following in C: /* ABI type definitions omitted for brevity */ c3fault_t run_operation_by_id(int *return_value, int id); [0] https://c3-lang.org https://c3-lang.org
  - deleted 1y ago
    [deleted]

robertknight 1y ago

I've seen talks on this topic at Rust conferences which seemed strongly influenced by Swift's approach, so that will probably be the direction this ends up going in.

IshKebab 1y ago
The WASI Component Model also has a richer ABI. I wonder if that could be copied to native platforms somehow.

ameliaquining 1y ago

I don't think Rust ever wants to do ABI stability that isn't opt-in, because Rust is about zero-cost abstractions and a stable ABI is not zero-cost. See this explanation of how C++'s de facto stable ABI significantly reduces performance: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p2028r0.pdf https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p20... [PDF] For the use case of letting Rust programs dlopen plugins that are also written in Rust, the current solutions are stabby and abi_stable. They're not perfect but they seem to work fairly well in practice. For more general use cases like cross-language interop, the hope is to get somewhere on crABI: https://github.com/joshtriplett/rfcs/blob/crabi-v1/text/3470-crabi-v1.md https://github.com/joshtriplett/rfcs/blob/crabi-v1/text/3470... This is intended to be a superior successor to C's ABI, useful for all the same use cases (at least if the code you want to talk to is new enough to support it). Note that this isn't something the Rust maintainers can do unilaterally; there's a need to get buy-in from maintainers of other languages, Linux distros, etc. I'm not sure that the "everything is a shared library" approach is workable in a language where heap allocation is generally explicit and opt-in. While Swift has some support for stack allocation, most types are heap-allocated, and I think that reduces the extent to which the rest of the language has to be warped to accommodate the stable ABI.

ninkendo 1y ago
> While Swift has some support for stack allocation, most types are heap-allocated, and I think that reduces the extent to which the rest of the language has to be warped to accommodate the stable ABI. Stack vs heap in swift is simple: structs are stack, classes are heap. This is what makes Swift’s effort so impressive: it works with stack data just as well as heap. Stack allocated types being used across library boundaries has a ton of plumbing to make it work, there’s the concept of a value witness table (like a vtable but for size/stride information) that can allow looking up sizes at runtime. Stack allocation makes heavy use of alloca(), which most languages shy away from (since it means stack frames aren’t of a static size.) It was difficult but swift managed to do it. For rust to do something similar it would have to fix a lot of cases where object safety is broken, and yeah, this may require changing the language, particularly to support dynamic/runtime size information. (It would also make the dyn keyword effectively obsolete, because to make generics work across library boundaries, any type has to be able to “become” dynamic if the linker needs it to be.) It would definitely require changes to the language, but Swift provides some inspiring prior art here. This article goes into the details: https://faultlore.com/blah/swift-abi/ https://faultlore.com/blah/swift-abi/

yoshuaw 1y ago

We are more or less trying to solve this exact problem with Wasm Components [1]. If core WebAssembly is a portable instruction format, WebAssembly Components are a portable executable/linkable format. Using them is unfortunately not yet quite as easy as using repr(wasm)/extern "wasm". But with wit-bindgen [2] and the wasm32-wasip2 target [3], it's not that hard either. [1]: https://youtu.be/tAACYA1Mwv4 https://youtu.be/tAACYA1Mwv4 [2]: https://github.com/bytecodealliance/wit-bindgen https://github.com/bytecodealliance/wit-bindgen [3]: https://doc.rust-lang.org/nightly/rustc/platform-support/wasm32-wasip2.html https://doc.rust-lang.org/nightly/rustc/platform-support/was...

drogus 1y ago
I love WASM and WASI, but it's not nearly the same, unfortunately. Performance takes a hit, you can't use async in a straightforward way, launching hundreds of thousands of tasks is problematic etc. WASM is great for allowing to extend your app, but I don't see it as a replacement for an ABI anytime soon

nh2 1y ago

I don't really understand the point for writing an "OS in rust and provide rich services for applications to use". You can use any serialisation method you want (C structs, Protobuf, JSON, ...). An ABI is for passing data in the programming language and trusting it blindly, e.g. what functions use when calling each other. Any sane OS still has to validate the data that crosses its boundaries. For example, if you make a Linux system call, it doesn't blindly use a pointer you pass it, but instead actually checks if the pointer belongs to the calling process's address space. It seems pointless to pick Rust's internal function-calling data serialisation for the OS boundary; that wouldn't make it easier for any userspace programs running in that OS than explicitly defining a serialisation format. Why would an OS want to limit itself to Rust's internal representation, or why would Rust want to freeze its internal representation, making improvements impossible on both sides? This seems to bring only drawbacks. The only benefit I can see so far for a stable ABI is dynamic linking (and its use in "plugins" for programs where the code is equally trusted, e.g. writing some dlopen()able `.so` files in Rust, as plugins for Rust-only programs), where you can really "blindly call the symbol". But even there it's a bit questionable, as any mistake or mismatch in ABI can result in immediately memory corruption, which is something that Rust users rather dislike.

anon-3988 1y ago

We already have? Its called C.