8 ms·

Here's the chasm I want to see Rust cross: Dynamic linking with a safe ABI, where if you change and recompile one library then the outcome has to obey some def

Here's the chasm I want to see Rust cross: Dynamic linking with a safe ABI, where if you change and recompile one library then the outcome has to obey some definition of safety, and ABI stability is about as good as C or Objective-C or Swift. Until that happens, it'll be hard to adopt Rust in a lot of C/C++ strongholds where C's ABI and dynamic linking are the thing that enables the software to get huge.

eptcyka 8mo ago
The C ABI can already be used, it comes with all the existing safety guarantees that C will provide. Isn’t this as good as C?
- ahartmetz 8mo ago
  Not really. The foreign ABI requires a foreign API, which adds friction that you don't have with C exporting a C API / ABI. I've never tried, but I would guess that it adds a lot of friction.
- pizlonator 8mo ago
  It is as good as C. It's also as bad as C. I'm saying that the chasm to cross is a safe ABI.
  - sagarm 8mo ago
    There is no existing safe ABI, so this cannot be an adoption barrier.
    - pizlonator 8mo ago
      Lots of reasons why it is. I'll give you two. 1) It can't be that replacing 20 C/C++ shared objects with 20 Rust shared objects results in 20 copies of the Rust standard library and other dependencies that those Rust libraries pull in. But, today, that is what happens. For some situations, this is too much of a memory usage regression to be tolerable. 2) If you really have 20 libraries calling into one another using C ABI, then you end up with manual memory management and manual buffer offset management everywhere even if you rewrite the innards in Rust. So long as Rust doesn't have a safe ABI, the upside of a Rust rewrite might be too low in terms of safety/security gained to be worth doing
      - zozbot234 8mo ago
        Many Rust core/standard library functions are trivial and inlining them is not really a concern. For those that do involve significant amount of code, C ABI-compatible code could be exported from some .so dynamic object, with only a small safe wrapper being statically linked.
- ozgrakkurt 8mo ago
  I found c ABI a bit too difficult in rust compared to c or zig. Mainly because of destructors. I am guessing c++ would be difficult in a similar way. Also unsafe rust has always on strict-aliasing, which makes writing code difficult unless you do it in certain ways. Having glue libraries like pyo3 makes it good in rust. But that introduces bloat and other issues. This has been the biggest issue I had with rust, it is too hard to write something so you use a dependency. And before you know it, you are bloating out of control
elteto 8mo ago
> Until that happens, it'll be hard to adopt Rust in a lot of C/C++ strongholds where C's ABI and dynamic linking are the thing that enables the software to get huge. Wait, Rust can already communicate using the C ABI. In fact, it offers exactly the same capabilities as C++ in this regard (dynamic linking).
- pizlonator 8mo ago
  That's an unsafe ABI.
  - elteto 8mo ago
    As unsafe as C or C++. In fact, safer, because only the ABI surface is unsafe, the rust code behind it can be as safe or unsafe as you want it to be. I was addressing this portion of your comment: "C's ABI and dynamic linking are the thing that enables the software to get huge". If the C ABI is what enables software to get huge then Rust is already there. There is a second claim in your comment about a "safe ABI", but that is something that neither C or C++ offers right now.
    - pizlonator 8mo ago
      Here's the problem. If you told me that you rebuilt the Linux userland with Rust but you used C ABI at all of the boundaries, then I would be pretty convinced that you did not create a meaningful improvement to security because of how many dynamic linking boundaries there are. So many of the libraries involved are small, and big or small they expose ABIs that involve pointers to buffers and manual memory management. > There is a second claim in your comment about a "safe ABI", but that is something that neither C or C++ offers right now. Of course C and C++ are no safer in this regard. (Well, with Fil-C they are safer, but like whatever.) But that misses the point, which is that: - It would be a big deal if Rust did have a safe dynamic linking ABI. Someone should do it. That's the main point I'm making. I don't think deflecting by saying "but C is no safer" is super interesting. - So long as this problem isn't fixed, the upside of using Rust to replace a lot of the load bearing stuff in an OS is much lower than it should be to justify the effort. This point is debatable for sure, but your arguments don't address it.
ahartmetz 8mo ago
Dynamic linking is also great for compile time of debug builds. If a large library or application is split up into smaller shared libraries, ones unaffected by changes don't need to be touched at all. Runtime dynamic linking has a small overhead, but it's several orders of magnitude faster than compile-time linking, so not a problem in debug builds.
- PaulDavisThe1st 8mo ago
  for developer turnaround time, it is huge. we explicitly do not statically link Ardour because as developers we are in the edit-compile-debug cycle all day every day, and speeding up the link step (which dynamic linking does dramatically, especially with parallel linkers like lld) is a gigantic improvement to our quality of life and productivity.
  - elteto 8mo ago
    A common pattern is dynamic linking for development and static linking for production-ready releases.
    - PaulDavisThe1st 8mo ago
      We considered doing both, but it turned out that the GUI toolkit we use was really, really not designed to be statically linked, so we stopped trying.
    - ahartmetz 8mo ago
      Yes, that's a good way to do it.
jjmarr 8mo ago
C++ ABI stability is the main reason improvements to the language get rejected. You cannot change anything that would affect the class layout of something in the STL. For templated functions where the implementation is in the header, ODR means you can't add optimizations later on. Maybe this was OK in the 90s when companies deleted the source code and laid off the programmers once the software was done, but it's not a feature Rust should ever support or guarantee. The "stable ABI" is C functions and nothing else for a very good reason.
- pizlonator 8mo ago
  I think if Rust wants to evolve even more aggressively than C++ evolves, then that is a chasm that needs to be crossed. In lots of domains, having a language that doesn't change very much, or that only changes very carefully with backcompat being taken super seriously, is more important than the memory safety guarantees Rust offers.
  - jjmarr 8mo ago
    In my view, this is a good thing. As a C++ developer, I regularly deal with people that think creating a compiled object file and throwing away the source code is acceptable, or decide to hide source code for "security" while distributing object files. This makes my life hell. Rust preventing this makes my life so much better.
    - pizlonator 8mo ago
      > As a C++ developer, I regularly deal with people that think creating a compiled object file and throwing away the source code is acceptable, or decide to hide source code for "security" while distributing object files. This makes my life hell. I mean yeah that's bad. > Rust preventing this makes my life so much better. I'm talking about a different issue, which is: how do you create software that's in the billions of lines of code in scale. That's the scale of desktop OSes. Probably also the scale of some other things too. At that scale, you can't just give everyone the source and tell them to do a world compile. Stable ABIs fix that. Also, you can't coordinate between all of the people involved other than via stable ABIs. So stable ABIs save both individual build time and reduce cognitive load. This is true even and especially if everyone has access to everyone else's source code
pjmlp 8mo ago
Indeed, Victor Ciura from Microsoft DevDiv has several talks on how this is currently an adoption problem at Microsoft. They have been working around it with DLLs, and COM/WinRT, but still the tooling isn't ideal.
- rhdunn 8mo ago
  COM is interesting as it implements interfaces using the C++ vtable layout, which can be done in C. Dynamic COM (DCOM) is used to provide interoperability with Visual Basic. You can also access .NET/C# objects/interfaces via COM. It has an interface to allow you to get the type metadata but that isn't necessary. This makes it possible to e.g. get the C#/.NET exception stack trace from a C/C++ application.
  - tonyedgecombe 8mo ago
    >Dynamic COM (DCOM) is used to provide interoperability with Visual Basic. DCOM is Distributed COM not Dynamic COM[1]. COM does have an interface for dynamic dispatch called IDispatch[2] which is used for scripting languages like VBScript or JScript. It isn't required for Visual Basic though. VB is compiled and supports early binding to interfaces. [1] https://en.wikipedia.org/wiki/Distributed_Component_Object_Model https://en.wikipedia.org/wiki/Distributed_Component_Object_M... [2] https://en.wikipedia.org/wiki/IDispatch https://en.wikipedia.org/wiki/IDispatch
    - rhdunn 8mo ago
      Ah yes, that's what I was thinking of. It's been a while since I've worked with COM.
  - deleted 8mo ago
    [deleted]
  - deleted 8mo ago
    [deleted]
cryptonector 8mo ago
Eh, some people can work on moving to Rust, while others work on adding dynamic linking to Rust. Or maybe we can some how get used to living with static linking. (I don't think so, but many seem to think so in spite of my advice to the contrary!) Another possibility is to use IPC as the dynamic linking boundary of sorts, but this will consume lots more memory, and as is stated elsewhere in this thread, memory ain't cheap no more.
choeger 8mo ago
That would be great, but Rust relies on compile-time monomorphization for efficiency (very much like C++, if you consider templates polymorphic functions/classes). This means that any Rust ABI would have to cater for link-time specialization. I think this should be doable, but it would require a solution that's better than just to move the code generation into the linker. Instead, one would need to carefully consider the usage of the "shape" of all parameters of a function.
- hyperman1 8mo ago
  I wonder if we look at it from a too narrow perspective. We use the C ABI because it's the only game in town. We should be aiming for a safe cross language ABI. I'd love to make Rust, C, PHP, Swift, Java and Python easily talk to each other inside 1 process. It should extend the C ABI with things like strings, arrays, objects with a way to destruct them, and provide some safety guarantees. As an example, the windows world has COM, which is at the core pretty reasonable for its design constraints, even if gnarly sometimes.
  - zozbot234 8mo ago
    You'll find that all of these languages ultimately build FFI on top of C ABI conventions, though Swift's own internally stable ABI uses a lot of alloca() to place dynamically sized objects on the stack, in a way that's somewhat unidiomatic (the Rust folks are trying to back out of their alloca() equivalent). You can even interface to COM from pure C.
  - pjmlp 8mo ago
    That is why platforms like Common Language Runtime exist, not only COM. CLR was going to be the COM Runtime+, and idea was reborn again as Windows team with their anti-.NET bias decided to redo Longhorn in C++, with WinRT. "Turning to the past to power Windows’ future: An in-depth look at WinRT" https://arstechnica.com/features/2012/10/windows-8-and-winrt-everything-old-is-new-again/ https://arstechnica.com/features/2012/10/windows-8-and-winrt... It is also how Android IPC and Apple's XPC kind of get into the picture. The elephant in the room is that FOSS OSes hardly embrace such solutions.
lelanthran 8mo ago
> Here's the chasm I want to see Rust cross: That's not important. What I want to see is the Rewrite-it-in-Rust movement move towards GPL. GPL is pro-user. MIT is pro-business. In their zeal to convert, they are happily replacing pro-user software with pro-business software. Their primary goal is to convert, not to safeguard. If they shifted their goal from spreading Rust to protecting users, I'd be a lot happier about the community.
- bayindirh 8mo ago
  > In their zeal to convert, they are happily replacing pro-user software with pro-business software. This is one of the two main reasons I'm not using Rust. Second reason is being addressed by gccrs team, so I have no big gripes there, since they are progressing well.
  - tcfhgj 8mo ago
    I think this makes you implicitly are a part of this trend, because even less pro user software exists in Rust because of your decision
    - bayindirh 8mo ago
      Seriously, that's a good point. I'll seriously consider my position when gccrs becomes a bit more mature. Thanks for your reply.
  - kibwen 8mo ago
    By this same metric, do you refuse to use C because the vast majority of OSS C codebases are permissively licensed? Surely you see that this makes no sense, yes? Neither Rust-the-language nor Rust-the-ecosystem are any more hostile to GPL than any other language and ecosystem.
    - lelanthran 8mo ago
      > By this same metric, do you refuse to use C because the vast majority of OSS C codebases are permissively licensed? It's not comparable - the Rewrite-it-in-Rust community is aiming to replace the existing pro-user products, with new pro-business products. The last significant online C community was the one that gave us the pro-user products in the first place. > Surely you see that this makes no sense, yes? Neither Rust-the-language nor Rust-the-ecosystem are any more hostile to GPL than any other language and ecosystem. I don't care whether or not they are hostile, that is not relevant. What is relevant to the complaints you are reading is that their primary goal is the spread of Rust, not the interests of the users. It is totally reasonable to be against a community who are working very hard to replace pro-user software with pro-business software.

eptcyka 8mo ago

The C ABI can already be used, it comes with all the existing safety guarantees that C will provide. Isn’t this as good as C?

ahartmetz 8mo ago
Not really. The foreign ABI requires a foreign API, which adds friction that you don't have with C exporting a C API / ABI. I've never tried, but I would guess that it adds a lot of friction.
pizlonator 8mo ago
It is as good as C. It's also as bad as C. I'm saying that the chasm to cross is a safe ABI.
- sagarm 8mo ago
  There is no existing safe ABI, so this cannot be an adoption barrier.
  - pizlonator 8mo ago
    Lots of reasons why it is. I'll give you two. 1) It can't be that replacing 20 C/C++ shared objects with 20 Rust shared objects results in 20 copies of the Rust standard library and other dependencies that those Rust libraries pull in. But, today, that is what happens. For some situations, this is too much of a memory usage regression to be tolerable. 2) If you really have 20 libraries calling into one another using C ABI, then you end up with manual memory management and manual buffer offset management everywhere even if you rewrite the innards in Rust. So long as Rust doesn't have a safe ABI, the upside of a Rust rewrite might be too low in terms of safety/security gained to be worth doing
    - zozbot234 8mo ago
      Many Rust core/standard library functions are trivial and inlining them is not really a concern. For those that do involve significant amount of code, C ABI-compatible code could be exported from some .so dynamic object, with only a small safe wrapper being statically linked.
ozgrakkurt 8mo ago
I found c ABI a bit too difficult in rust compared to c or zig. Mainly because of destructors. I am guessing c++ would be difficult in a similar way. Also unsafe rust has always on strict-aliasing, which makes writing code difficult unless you do it in certain ways. Having glue libraries like pyo3 makes it good in rust. But that introduces bloat and other issues. This has been the biggest issue I had with rust, it is too hard to write something so you use a dependency. And before you know it, you are bloating out of control

elteto 8mo ago

> Until that happens, it'll be hard to adopt Rust in a lot of C/C++ strongholds where C's ABI and dynamic linking are the thing that enables the software to get huge. Wait, Rust can already communicate using the C ABI. In fact, it offers exactly the same capabilities as C++ in this regard (dynamic linking).

pizlonator 8mo ago
That's an unsafe ABI.
- elteto 8mo ago
  As unsafe as C or C++. In fact, safer, because only the ABI surface is unsafe, the rust code behind it can be as safe or unsafe as you want it to be. I was addressing this portion of your comment: "C's ABI and dynamic linking are the thing that enables the software to get huge". If the C ABI is what enables software to get huge then Rust is already there. There is a second claim in your comment about a "safe ABI", but that is something that neither C or C++ offers right now.
  - pizlonator 8mo ago
    Here's the problem. If you told me that you rebuilt the Linux userland with Rust but you used C ABI at all of the boundaries, then I would be pretty convinced that you did not create a meaningful improvement to security because of how many dynamic linking boundaries there are. So many of the libraries involved are small, and big or small they expose ABIs that involve pointers to buffers and manual memory management. > There is a second claim in your comment about a "safe ABI", but that is something that neither C or C++ offers right now. Of course C and C++ are no safer in this regard. (Well, with Fil-C they are safer, but like whatever.) But that misses the point, which is that: - It would be a big deal if Rust did have a safe dynamic linking ABI. Someone should do it. That's the main point I'm making. I don't think deflecting by saying "but C is no safer" is super interesting. - So long as this problem isn't fixed, the upside of using Rust to replace a lot of the load bearing stuff in an OS is much lower than it should be to justify the effort. This point is debatable for sure, but your arguments don't address it.

ahartmetz 8mo ago

Dynamic linking is also great for compile time of debug builds. If a large library or application is split up into smaller shared libraries, ones unaffected by changes don't need to be touched at all. Runtime dynamic linking has a small overhead, but it's several orders of magnitude faster than compile-time linking, so not a problem in debug builds.

PaulDavisThe1st 8mo ago
for developer turnaround time, it is huge. we explicitly do not statically link Ardour because as developers we are in the edit-compile-debug cycle all day every day, and speeding up the link step (which dynamic linking does dramatically, especially with parallel linkers like lld) is a gigantic improvement to our quality of life and productivity.
- elteto 8mo ago
  A common pattern is dynamic linking for development and static linking for production-ready releases.
  - PaulDavisThe1st 8mo ago
    We considered doing both, but it turned out that the GUI toolkit we use was really, really not designed to be statically linked, so we stopped trying.
  - ahartmetz 8mo ago
    Yes, that's a good way to do it.

jjmarr 8mo ago

C++ ABI stability is the main reason improvements to the language get rejected. You cannot change anything that would affect the class layout of something in the STL. For templated functions where the implementation is in the header, ODR means you can't add optimizations later on. Maybe this was OK in the 90s when companies deleted the source code and laid off the programmers once the software was done, but it's not a feature Rust should ever support or guarantee. The "stable ABI" is C functions and nothing else for a very good reason.

pizlonator 8mo ago
I think if Rust wants to evolve even more aggressively than C++ evolves, then that is a chasm that needs to be crossed. In lots of domains, having a language that doesn't change very much, or that only changes very carefully with backcompat being taken super seriously, is more important than the memory safety guarantees Rust offers.
- jjmarr 8mo ago
  In my view, this is a good thing. As a C++ developer, I regularly deal with people that think creating a compiled object file and throwing away the source code is acceptable, or decide to hide source code for "security" while distributing object files. This makes my life hell. Rust preventing this makes my life so much better.
  - pizlonator 8mo ago
    > As a C++ developer, I regularly deal with people that think creating a compiled object file and throwing away the source code is acceptable, or decide to hide source code for "security" while distributing object files. This makes my life hell. I mean yeah that's bad. > Rust preventing this makes my life so much better. I'm talking about a different issue, which is: how do you create software that's in the billions of lines of code in scale. That's the scale of desktop OSes. Probably also the scale of some other things too. At that scale, you can't just give everyone the source and tell them to do a world compile. Stable ABIs fix that. Also, you can't coordinate between all of the people involved other than via stable ABIs. So stable ABIs save both individual build time and reduce cognitive load. This is true even and especially if everyone has access to everyone else's source code

pjmlp 8mo ago

Indeed, Victor Ciura from Microsoft DevDiv has several talks on how this is currently an adoption problem at Microsoft. They have been working around it with DLLs, and COM/WinRT, but still the tooling isn't ideal.

rhdunn 8mo ago
COM is interesting as it implements interfaces using the C++ vtable layout, which can be done in C. Dynamic COM (DCOM) is used to provide interoperability with Visual Basic. You can also access .NET/C# objects/interfaces via COM. It has an interface to allow you to get the type metadata but that isn't necessary. This makes it possible to e.g. get the C#/.NET exception stack trace from a C/C++ application.
- tonyedgecombe 8mo ago
  >Dynamic COM (DCOM) is used to provide interoperability with Visual Basic. DCOM is Distributed COM not Dynamic COM[1]. COM does have an interface for dynamic dispatch called IDispatch[2] which is used for scripting languages like VBScript or JScript. It isn't required for Visual Basic though. VB is compiled and supports early binding to interfaces. [1] https://en.wikipedia.org/wiki/Distributed_Component_Object_Model https://en.wikipedia.org/wiki/Distributed_Component_Object_M... [2] https://en.wikipedia.org/wiki/IDispatch https://en.wikipedia.org/wiki/IDispatch
  - rhdunn 8mo ago
    Ah yes, that's what I was thinking of. It's been a while since I've worked with COM.
- deleted 8mo ago
  [deleted]
- deleted 8mo ago
  [deleted]

cryptonector 8mo ago

Eh, some people can work on moving to Rust, while others work on adding dynamic linking to Rust. Or maybe we can some how get used to living with static linking. (I don't think so, but many seem to think so in spite of my advice to the contrary!) Another possibility is to use IPC as the dynamic linking boundary of sorts, but this will consume lots more memory, and as is stated elsewhere in this thread, memory ain't cheap no more.

choeger 8mo ago

That would be great, but Rust relies on compile-time monomorphization for efficiency (very much like C++, if you consider templates polymorphic functions/classes). This means that any Rust ABI would have to cater for link-time specialization. I think this should be doable, but it would require a solution that's better than just to move the code generation into the linker. Instead, one would need to carefully consider the usage of the "shape" of all parameters of a function.

hyperman1 8mo ago
I wonder if we look at it from a too narrow perspective. We use the C ABI because it's the only game in town. We should be aiming for a safe cross language ABI. I'd love to make Rust, C, PHP, Swift, Java and Python easily talk to each other inside 1 process. It should extend the C ABI with things like strings, arrays, objects with a way to destruct them, and provide some safety guarantees. As an example, the windows world has COM, which is at the core pretty reasonable for its design constraints, even if gnarly sometimes.
- zozbot234 8mo ago
  You'll find that all of these languages ultimately build FFI on top of C ABI conventions, though Swift's own internally stable ABI uses a lot of alloca() to place dynamically sized objects on the stack, in a way that's somewhat unidiomatic (the Rust folks are trying to back out of their alloca() equivalent). You can even interface to COM from pure C.
- pjmlp 8mo ago
  That is why platforms like Common Language Runtime exist, not only COM. CLR was going to be the COM Runtime+, and idea was reborn again as Windows team with their anti-.NET bias decided to redo Longhorn in C++, with WinRT. "Turning to the past to power Windows’ future: An in-depth look at WinRT" https://arstechnica.com/features/2012/10/windows-8-and-winrt-everything-old-is-new-again/ https://arstechnica.com/features/2012/10/windows-8-and-winrt... It is also how Android IPC and Apple's XPC kind of get into the picture. The elephant in the room is that FOSS OSes hardly embrace such solutions.

lelanthran 8mo ago

> Here's the chasm I want to see Rust cross: That's not important. What I want to see is the Rewrite-it-in-Rust movement move towards GPL. GPL is pro-user. MIT is pro-business. In their zeal to convert, they are happily replacing pro-user software with pro-business software. Their primary goal is to convert, not to safeguard. If they shifted their goal from spreading Rust to protecting users, I'd be a lot happier about the community.

bayindirh 8mo ago
> In their zeal to convert, they are happily replacing pro-user software with pro-business software. This is one of the two main reasons I'm not using Rust. Second reason is being addressed by gccrs team, so I have no big gripes there, since they are progressing well.
- tcfhgj 8mo ago
  I think this makes you implicitly are a part of this trend, because even less pro user software exists in Rust because of your decision
  - bayindirh 8mo ago
    Seriously, that's a good point. I'll seriously consider my position when gccrs becomes a bit more mature. Thanks for your reply.
- kibwen 8mo ago
  By this same metric, do you refuse to use C because the vast majority of OSS C codebases are permissively licensed? Surely you see that this makes no sense, yes? Neither Rust-the-language nor Rust-the-ecosystem are any more hostile to GPL than any other language and ecosystem.
  - lelanthran 8mo ago
    > By this same metric, do you refuse to use C because the vast majority of OSS C codebases are permissively licensed? It's not comparable - the Rewrite-it-in-Rust community is aiming to replace the existing pro-user products, with new pro-business products. The last significant online C community was the one that gave us the pro-user products in the first place. > Surely you see that this makes no sense, yes? Neither Rust-the-language nor Rust-the-ecosystem are any more hostile to GPL than any other language and ecosystem. I don't care whether or not they are hostile, that is not relevant. What is relevant to the complaints you are reading is that their primary goal is the spread of Rust, not the interests of the users. It is totally reasonable to be against a community who are working very hard to replace pro-user software with pro-business software.