6 ms·

My theory is that in order to replace C/C++ in the systems language space we are going to need several different types of languages. Rust is pretty nice, but t

by Verdex 6y ago

My theory is that in order to replace C/C++ in the systems language space we are going to need several different types of languages. Rust is pretty nice, but the options for allocation and it's ownership type system make it a mismatch for certain types of applications (probably games and OS's but we'll see how that plays out). I think regions are neat. And I'm looking for an opportunity both for playing around with it myself and for the programming language community at large getting more exposure to them. More ideas exposed to more people leads to better ideas of how to fit everything together and how to leverage what features to solve some problems better.

aldanor 6y ago
Games often use arena-based allocation systems, slotmaps etc to manage myriads of instances. I don't see how any of that is a problem in Rust?
- steveklabnik 6y ago
  It is not. The usual arguments against Rust for games are more cultural. Rust makes a tradeoff of being a bit harder up front, for less time debugging later. Many people say that games specifically would prefer the opposite tradeoff. We'll see as things continue to develop here.
  - vvanders 6y ago
    Honestly most of the iterative parts of gamedev I worked on was either already up in the tooling/editor or in a scripting language. I think Rust for core engine and Lua for game logic would be a pretty solid combo.
- megameter 6y ago
  It's mostly the ergonomics of Rust being fine-grained that present a problem. While it's gotten better with revision, fundamentally, you start in an assume-nothing mode in Rust and then boilerplate appears to say "actually, this one is mutable". This incurs friction on nearly every line of state manipulation and when you're doing game-style allocation, you mostly don't care. You have a tremendous amount of mutable state; some of it you blow away every frame, the rest you may want to lock sometimes. What Vale does that is attractive is to make simplifying assumptions that address this problem domain closer to the level of granularity that it deserves.
  - danbolt 6y ago
    I think a pure ECS has its state manipulation well-encapsulated and organizable. Part of me thinks that sort of design would mesh pretty well with Rust’s ownership system.
    - steveklabnik 6y ago
      Rust folks think this as well; there has been a lot of activity in this area, and even spread some of the ideas behind ECSes to areas outside of games. https://kyren.github.io/2018/09/14/rustconf-talk.html https://kyren.github.io/2018/09/14/rustconf-talk.html
ampdepolymerase 6y ago
A modern alternative to C's FFI needs to be developed, perhaps a standard like gRPC but as a compiler extension. C's FFI contains so little information due to its weak type system that it is wholly unsuitable for higher level languages. Yet we use it because it is the lowest common denominator. The C FFI has cost hundreds of hours of lost productivity due to having to re-annotate all the types in the non-C caller. This is before we even take into account the mess that is function pointers and types generated using the pre-processor. And no, tools like bindgen and c2rust are stop gaps, not a real solution. A lot of low level code cannot be effectively called safely in a higher level language at all without hundreds of hours of manual work wrapping and annotating each individual function. We need a new standardized, zero cost, cross language interop format that functions on the ABI/memory level. Rust's Cxx is a good start but it doesn't scale m:n with regards to new languages. Every new language has to implement a custom wrapper for calling another high level language if they do not want to lose 80% of the benefits of the respective language's type system when passing through the primitive C FFI. For example, calling Rust from Julia, calling Julia from V lang. The lack of a common interop format is hurting the developer ecosystem as more proliferation of programming languages simply leads to greater fragmentation. The web solved this a long time ago with JSON APIs, generated OpenAPI clients (and now Graphql), and to a lesser extent, gRPC. We need to do better when it comes to systems programming.
- simias 6y ago
  >We need a new standardized, zero cost, cross language interop format functions on the ABI/memory level I don't think that's doable unless the underlying language already has the same ownership and lifetime guarantees as Rust does, otherwise these bindings have to protect against memory issues by effectively taking ownership of the data to garantee its lifetime, which is not zero cost. I simply don't think that you can auto-generate zero-cost safe C or C++ bindings automatically.
  - estebank 6y ago
    > I simply don't think that you can auto-generate zero-cost safe C or C++ bindings automatically. As stated, that's likely an accurate statement, but if you separate the requirements for the bindings the problem looks way more tractable: - auto-generate: I'm pretty positive that this point is only hard given the following, and I feel you would agree with that assessment, so I'll just focus on the other ones. - zero-cost: doing zero-cost ffi is the best option, but a lot of advanced constructs that people want to use cannot be well represented in that way because they rely on the internal consistency of the host language. That means that dynamically loading a library written in Rust might make it so that you can only interact with ADTs, trait objects and whitelisted provided traits on those ADTs, potentially only through trait objects. That's not zero-cost, but it is super powerful. ObjectiveC has lived its entire existence going through vtables and that has even enabled runtime introspection that was super useful. - safe: safety can be accomplished easily enough if you don't constrain yourself to zero-cost or if you put constraints on what is supported for zero-cost ffi. You could say that method calls on trait objects is safe, but type parameters are out of scope because ffi and monomorphization are not compatible. You could restrict yourself to either no borrows crossing ffi, or no mutable data (only internal mutability through trait methods) allowed. - C or C++: I would focus on other languages, personally. Having easy interop with Java, C#, Python and JavaScript would open opportunities that do not have anything to do with systems programming per-se, but that would make it more likely that the next numpy isn't written in pure C. - automatically: the need for extra annotations in either end of the interop simplifies part of the problem and might even help with documentation. Adding some light ownership information to APIs in non-Rust languages would be great for non-Rust users of those APIs.
WalterBright 6y ago
I'll be presenting a video on D's prototype ownership/borrowing system tomorrow at 14:00 UTC. https://dconf.org/2020/online/index.html#schedule https://dconf.org/2020/online/index.html#schedule https://dlang.org/blog/2020/11/20/dconf-online-2020-how-to-participate/ https://dlang.org/blog/2020/11/20/dconf-online-2020-how-to-p... It's free to participate!

by Verdex 6y ago

aldanor 6y ago

Games often use arena-based allocation systems, slotmaps etc to manage myriads of instances. I don't see how any of that is a problem in Rust?

steveklabnik 6y ago
It is not. The usual arguments against Rust for games are more cultural. Rust makes a tradeoff of being a bit harder up front, for less time debugging later. Many people say that games specifically would prefer the opposite tradeoff. We'll see as things continue to develop here.
- vvanders 6y ago
  Honestly most of the iterative parts of gamedev I worked on was either already up in the tooling/editor or in a scripting language. I think Rust for core engine and Lua for game logic would be a pretty solid combo.
megameter 6y ago
It's mostly the ergonomics of Rust being fine-grained that present a problem. While it's gotten better with revision, fundamentally, you start in an assume-nothing mode in Rust and then boilerplate appears to say "actually, this one is mutable". This incurs friction on nearly every line of state manipulation and when you're doing game-style allocation, you mostly don't care. You have a tremendous amount of mutable state; some of it you blow away every frame, the rest you may want to lock sometimes. What Vale does that is attractive is to make simplifying assumptions that address this problem domain closer to the level of granularity that it deserves.
- danbolt 6y ago
  I think a pure ECS has its state manipulation well-encapsulated and organizable. Part of me thinks that sort of design would mesh pretty well with Rust’s ownership system.
  - steveklabnik 6y ago
    Rust folks think this as well; there has been a lot of activity in this area, and even spread some of the ideas behind ECSes to areas outside of games. https://kyren.github.io/2018/09/14/rustconf-talk.html https://kyren.github.io/2018/09/14/rustconf-talk.html

ampdepolymerase 6y ago

A modern alternative to C's FFI needs to be developed, perhaps a standard like gRPC but as a compiler extension. C's FFI contains so little information due to its weak type system that it is wholly unsuitable for higher level languages. Yet we use it because it is the lowest common denominator. The C FFI has cost hundreds of hours of lost productivity due to having to re-annotate all the types in the non-C caller. This is before we even take into account the mess that is function pointers and types generated using the pre-processor. And no, tools like bindgen and c2rust are stop gaps, not a real solution. A lot of low level code cannot be effectively called safely in a higher level language at all without hundreds of hours of manual work wrapping and annotating each individual function. We need a new standardized, zero cost, cross language interop format that functions on the ABI/memory level. Rust's Cxx is a good start but it doesn't scale m:n with regards to new languages. Every new language has to implement a custom wrapper for calling another high level language if they do not want to lose 80% of the benefits of the respective language's type system when passing through the primitive C FFI. For example, calling Rust from Julia, calling Julia from V lang. The lack of a common interop format is hurting the developer ecosystem as more proliferation of programming languages simply leads to greater fragmentation. The web solved this a long time ago with JSON APIs, generated OpenAPI clients (and now Graphql), and to a lesser extent, gRPC. We need to do better when it comes to systems programming.

simias 6y ago
>We need a new standardized, zero cost, cross language interop format functions on the ABI/memory level I don't think that's doable unless the underlying language already has the same ownership and lifetime guarantees as Rust does, otherwise these bindings have to protect against memory issues by effectively taking ownership of the data to garantee its lifetime, which is not zero cost. I simply don't think that you can auto-generate zero-cost safe C or C++ bindings automatically.
- estebank 6y ago
  > I simply don't think that you can auto-generate zero-cost safe C or C++ bindings automatically. As stated, that's likely an accurate statement, but if you separate the requirements for the bindings the problem looks way more tractable: - auto-generate: I'm pretty positive that this point is only hard given the following, and I feel you would agree with that assessment, so I'll just focus on the other ones. - zero-cost: doing zero-cost ffi is the best option, but a lot of advanced constructs that people want to use cannot be well represented in that way because they rely on the internal consistency of the host language. That means that dynamically loading a library written in Rust might make it so that you can only interact with ADTs, trait objects and whitelisted provided traits on those ADTs, potentially only through trait objects. That's not zero-cost, but it is super powerful. ObjectiveC has lived its entire existence going through vtables and that has even enabled runtime introspection that was super useful. - safe: safety can be accomplished easily enough if you don't constrain yourself to zero-cost or if you put constraints on what is supported for zero-cost ffi. You could say that method calls on trait objects is safe, but type parameters are out of scope because ffi and monomorphization are not compatible. You could restrict yourself to either no borrows crossing ffi, or no mutable data (only internal mutability through trait methods) allowed. - C or C++: I would focus on other languages, personally. Having easy interop with Java, C#, Python and JavaScript would open opportunities that do not have anything to do with systems programming per-se, but that would make it more likely that the next numpy isn't written in pure C. - automatically: the need for extra annotations in either end of the interop simplifies part of the problem and might even help with documentation. Adding some light ownership information to APIs in non-Rust languages would be great for non-Rust users of those APIs.

WalterBright 6y ago

I'll be presenting a video on D's prototype ownership/borrowing system tomorrow at 14:00 UTC. https://dconf.org/2020/online/index.html#schedule https://dconf.org/2020/online/index.html#schedule https://dlang.org/blog/2020/11/20/dconf-online-2020-how-to-participate/ https://dlang.org/blog/2020/11/20/dconf-online-2020-how-to-p... It's free to participate!