7 ms·

Untapped potential in Rust's type system

201 points by lukastyrychtr 5y ago

eptcyka 5y ago
An interesting, yet somewhat unrelated discussion could be had about cases where one might want the type ID to not change when the type changes structurally. There are many structures in many ABI interfaces that are structurally different between releases, yet still binary compatible. This particular characteristic has always been defined adhoc, and I do wonder what a more formal system might mean. This almost feels like the issue of identity in distributed systems.
- lasagnaphil 5y ago
  Code generation probably works the best here, although it makes the build system more complex. You would have the typeid mappings as an actually existing file that you can commit into your git repo. Or if your language supports metaprogramming (something like Nim, Zig, or Jai), you could write compile-time code that maps each of your annotated types into deterministically-determined ids. (An example of this in Nim: https://gist.github.com/PhilipWitte/dd6c670fca3baf573490 https://gist.github.com/PhilipWitte/dd6c670fca3baf573490)
- zozbot234 5y ago
  > There are many structures in many ABI interfaces that are structurally different between releases, yet still binary compatible. These should be defined in the API as simple newtype wrappers over some basic binary-level type (generally either unsigned binary word or u8 array) with conversions to and from safer higher-level types defined in Rust code, leaving it to the compiler to optimize these to no-ops whenever possible. This ensures that "binary compatible" types also keep the same structural identity, and conversely, that binary incompatible types are automatically detected as well.
willeh 5y ago
Interesting article, but I think the key to writing idiomatic Rust is not to stretch what the type system can do but rather be happy at what It can express and avoid unnecessary abstraction. The compile-time guarantees that we have to prove in Rust, also serve to give a hint for when not to abstract.
- nine_k 5y ago
  I think pushing the boundaries of what is idiomatic can sometimes be valuable. Look how much idiomatic C++ has changed since 1995. Or what is idiomatic in JavaScript now.
  - fnord123 5y ago
    "sometimes" is doing a lot of heavy lifting here.
- klyrs 5y ago
  This article is about curiosity, not best practices.
- marshray 5y ago
  Sometimes life hands you lemons, and sometimes it hands you Box<dyn Fruit>.
dshpala 5y ago
I've always wondered if it's possible to use Rust' ownership chains to attribute memory usage? I.e. calculate total bytes retained by an object?
- ashtonkem 5y ago
  Shouldn't that be possible with any memory management system? I believe Java has a tree behind the scenes in memory for management.
  - squiggleblaz 5y ago
    The Java model ties one object to another. The rust model ties a consumed memory to a function. A debug log output that shows how much memory this function call required, in the same way it shows how much time it uses?
- simias 5y ago
  I don't think ownership really lets you do anything more than what "sizeof" tells you in C. In particular: - Some types have dynamic memory allocation (Vec, HashMaps etc...), so those would have to be computed at runtime and can change at any moment. - Some types have shared ownership (Rc/Arc), so it's unclear how you would measure memory usage then. - Some types, especially in foreign interfaces, will effectively just hold a pointer to some black box data, you'd need a special API to figure out how much memory it hides. For instance what's the memory usage of a database handle or a JPEG compression library context? - When you care about memory usage things like fragmentation are usually very important, and the amount of memory used by a given object can be misleading. If you have a string that takes up 12 bytes but it's the only object left in the middle of a 4KiB page, then just counting "12 bytes" for this object is misleading because you have a huge fragmentation overhead. The only advantage of the borrow checker is that safe Rust forces you to make ownership relationships explicit but not all Rust code is safe and there are many escape hatches that muddy the water (like Rc/Arc mentioned above, but also threads and a few other things).
  - squiggleblaz 5y ago
richardwhiuk 5y ago
Note, you have to be quite careful with typeid - you can end up with types not matching because you have two different versions of a crate. Normally this would be a compile time error, but using typeid can turn this into a runtime error. https://docs.rs/assert-type-eq/0.1.0/assert_type_eq/ https://docs.rs/assert-type-eq/0.1.0/assert_type_eq/ allows you to assert that two types are the same, forcing a compile time error for this scenario.
- simiones 5y ago
  Off-topic mostly, but is this sort of "single line" function/macro typical for Rust crates? This seems like something I would expect from a Node library, not a serious programming language community that cares about robust software.
  - roblabla 5y ago
    Macros can be quite hard to understand, so having a simple crate wrapping it with extensive documentation and tests can be quite beneficial. And yes, this also applies to node. The attack is entirely unnecessary.
    - simiones 5y ago
      This crate has no tests, and no documentation beyond a usage example; it would do much better as a copy/paste than an entire dependency, in any ecosystem. It would be great to have as part of some extended "utils" package if it can't be included in the standard library (something like C++'s boost or Java's Apache Commons).
  - SpaceNugget 5y ago
    That isn't a single line macro. Did you mean a single macro crate? Most crates in rust have more than a single macro/function, but some do. Regardless, it's not at all related to how "robust" or not the language ecosystem is.
    - deleted 5y ago
weltensturm 5y ago
Heh, this sounds just like the global event system I made for my D window manager: https://github.com/weltensturm/flatman/blob/master/common/event.d#L221 https://github.com/weltensturm/flatman/blob/master/common/ev... (which has some extra jingles with method annotations and event filters for convenience)
demurgos 5y ago
I am currently working on a configuration manager (to get rid of Ansible for my needs). To support running code as different users, I spawn multiple processes. I want to be able to call functions across processes without limitations, and be forward-compatible with a system where the processes could actually be on different machines. I basically had similar goals and implemented it with a similar design as the one described in the last part. I had the same interrogations about what should should be captured by a universal type id. In my own code, I decide to punt the discussion and require the user code to pick a globally unique name. I planned initially on using Typetag [0], a library achieving the same result as `UniversalId` from the article. The reason why I couldn't use `Typetag` in the end is the lack of support for generics. Generics are a tough problem to deal with when deriving an id because it is not clear how they should influence it. In my case I have a common pattern where structs are defined as: Message<S: AsRef<str>> { message: S, } This allows me define methods on both borrowed (`Message<&str>`) and owned (`Message<String>`) versions without issues. When remaining inside the process, the borrowed version is passed around and when sent across the process boundaries I can deserialize it to an owned version. This pattern prevents me from using Typetag and I am still not sure how it should be solved. A related problem is also how the registry is built on the receiving end. In my project the registry is built manually, similarly to the example in the article. There are also crates such as Inventory [1] and Linkme [2]. Which allow to mark types at their definition point and then collect them in order to register them. [0]: https://github.com/dtolnay/typetag https://github.com/dtolnay/typetag [1]: https://github.com/dtolnay/inventory https://github.com/dtolnay/inventory [2]: https://github.com/dtolnay/linkme https://github.com/dtolnay/linkme
- lpghatguy 5y ago
  I've written a lot of structs that have "maybe borrowed" contents. I usually reach for std::borrow::Cow for this. It's not exactly the same: we need to branch at runtime instead of having multiple generated codepaths. struct Message<'a> { message: Cow<'a, str>, }
  - demurgos 5y ago
    I agree that in this case I can move to branch at runtime (the runtime cost is irrelevant for my use case), I still feel that it illustrates how it complicates deriving a universal type id. The more general issue is that removing generic requires the lib to pick the implementation and remove choice from the consumer. Sometimes it's not that important, sometimes it matters more. Another example I have in my project is that I have a trait describing a `User` (with e.g. `get_name`) it may be implemented as a `LinuxUser` or `WindowsUser`, each providing extra fields. How do you generate a universal id for a struct generic over a `User`?

201 points by lukastyrychtr 5y ago

An interesting, yet somewhat unrelated discussion could be had about cases where one might want the type ID to not change when the type changes structurally. There are many structures in many ABI interfaces that are structurally different between releases, yet still binary compatible. This particular characteristic has always been defined adhoc, and I do wonder what a more formal system might mean. This almost feels like the issue of identity in distributed systems.

lasagnaphil 5y ago
Code generation probably works the best here, although it makes the build system more complex. You would have the typeid mappings as an actually existing file that you can commit into your git repo. Or if your language supports metaprogramming (something like Nim, Zig, or Jai), you could write compile-time code that maps each of your annotated types into deterministically-determined ids. (An example of this in Nim: https://gist.github.com/PhilipWitte/dd6c670fca3baf573490 https://gist.github.com/PhilipWitte/dd6c670fca3baf573490)
zozbot234 5y ago
> There are many structures in many ABI interfaces that are structurally different between releases, yet still binary compatible. These should be defined in the API as simple newtype wrappers over some basic binary-level type (generally either unsigned binary word or u8 array) with conversions to and from safer higher-level types defined in Rust code, leaving it to the compiler to optimize these to no-ops whenever possible. This ensures that "binary compatible" types also keep the same structural identity, and conversely, that binary incompatible types are automatically detected as well.

willeh 5y ago

Interesting article, but I think the key to writing idiomatic Rust is not to stretch what the type system can do but rather be happy at what It can express and avoid unnecessary abstraction. The compile-time guarantees that we have to prove in Rust, also serve to give a hint for when not to abstract.

nine_k 5y ago
I think pushing the boundaries of what is idiomatic can sometimes be valuable. Look how much idiomatic C++ has changed since 1995. Or what is idiomatic in JavaScript now.
- fnord123 5y ago
  "sometimes" is doing a lot of heavy lifting here.
klyrs 5y ago
This article is about curiosity, not best practices.
marshray 5y ago
Sometimes life hands you lemons, and sometimes it hands you Box<dyn Fruit>.

dshpala 5y ago

I've always wondered if it's possible to use Rust' ownership chains to attribute memory usage? I.e. calculate total bytes retained by an object?

ashtonkem 5y ago
Shouldn't that be possible with any memory management system? I believe Java has a tree behind the scenes in memory for management.
- squiggleblaz 5y ago
  The Java model ties one object to another. The rust model ties a consumed memory to a function. A debug log output that shows how much memory this function call required, in the same way it shows how much time it uses?
simias 5y ago
I don't think ownership really lets you do anything more than what "sizeof" tells you in C. In particular: - Some types have dynamic memory allocation (Vec, HashMaps etc...), so those would have to be computed at runtime and can change at any moment. - Some types have shared ownership (Rc/Arc), so it's unclear how you would measure memory usage then. - Some types, especially in foreign interfaces, will effectively just hold a pointer to some black box data, you'd need a special API to figure out how much memory it hides. For instance what's the memory usage of a database handle or a JPEG compression library context? - When you care about memory usage things like fragmentation are usually very important, and the amount of memory used by a given object can be misleading. If you have a string that takes up 12 bytes but it's the only object left in the middle of a 4KiB page, then just counting "12 bytes" for this object is misleading because you have a huge fragmentation overhead. The only advantage of the borrow checker is that safe Rust forces you to make ownership relationships explicit but not all Rust code is safe and there are many escape hatches that muddy the water (like Rc/Arc mentioned above, but also threads and a few other things).
- squiggleblaz 5y ago

richardwhiuk 5y ago

Note, you have to be quite careful with typeid - you can end up with types not matching because you have two different versions of a crate. Normally this would be a compile time error, but using typeid can turn this into a runtime error. https://docs.rs/assert-type-eq/0.1.0/assert_type_eq/ https://docs.rs/assert-type-eq/0.1.0/assert_type_eq/ allows you to assert that two types are the same, forcing a compile time error for this scenario.

simiones 5y ago
Off-topic mostly, but is this sort of "single line" function/macro typical for Rust crates? This seems like something I would expect from a Node library, not a serious programming language community that cares about robust software.
- roblabla 5y ago
  Macros can be quite hard to understand, so having a simple crate wrapping it with extensive documentation and tests can be quite beneficial. And yes, this also applies to node. The attack is entirely unnecessary.
  - simiones 5y ago
    This crate has no tests, and no documentation beyond a usage example; it would do much better as a copy/paste than an entire dependency, in any ecosystem. It would be great to have as part of some extended "utils" package if it can't be included in the standard library (something like C++'s boost or Java's Apache Commons).
- SpaceNugget 5y ago
  That isn't a single line macro. Did you mean a single macro crate? Most crates in rust have more than a single macro/function, but some do. Regardless, it's not at all related to how "robust" or not the language ecosystem is.
  - deleted 5y ago

weltensturm 5y ago

Heh, this sounds just like the global event system I made for my D window manager: https://github.com/weltensturm/flatman/blob/master/common/event.d#L221 https://github.com/weltensturm/flatman/blob/master/common/ev... (which has some extra jingles with method annotations and event filters for convenience)

demurgos 5y ago

I am currently working on a configuration manager (to get rid of Ansible for my needs). To support running code as different users, I spawn multiple processes. I want to be able to call functions across processes without limitations, and be forward-compatible with a system where the processes could actually be on different machines. I basically had similar goals and implemented it with a similar design as the one described in the last part. I had the same interrogations about what should should be captured by a universal type id. In my own code, I decide to punt the discussion and require the user code to pick a globally unique name. I planned initially on using Typetag [0], a library achieving the same result as `UniversalId` from the article. The reason why I couldn't use `Typetag` in the end is the lack of support for generics. Generics are a tough problem to deal with when deriving an id because it is not clear how they should influence it. In my case I have a common pattern where structs are defined as: Message<S: AsRef<str>> { message: S, } This allows me define methods on both borrowed (`Message<&str>`) and owned (`Message<String>`) versions without issues. When remaining inside the process, the borrowed version is passed around and when sent across the process boundaries I can deserialize it to an owned version. This pattern prevents me from using Typetag and I am still not sure how it should be solved. A related problem is also how the registry is built on the receiving end. In my project the registry is built manually, similarly to the example in the article. There are also crates such as Inventory [1] and Linkme [2]. Which allow to mark types at their definition point and then collect them in order to register them. [0]: https://github.com/dtolnay/typetag https://github.com/dtolnay/typetag [1]: https://github.com/dtolnay/inventory https://github.com/dtolnay/inventory [2]: https://github.com/dtolnay/linkme https://github.com/dtolnay/linkme

lpghatguy 5y ago
I've written a lot of structs that have "maybe borrowed" contents. I usually reach for std::borrow::Cow for this. It's not exactly the same: we need to branch at runtime instead of having multiple generated codepaths. struct Message<'a> { message: Cow<'a, str>, }
- demurgos 5y ago
  I agree that in this case I can move to branch at runtime (the runtime cost is irrelevant for my use case), I still feel that it illustrates how it complicates deriving a universal type id. The more general issue is that removing generic requires the lib to pick the implementation and remove choice from the consumer. Sometimes it's not that important, sometimes it matters more. Another example I have in my project is that I have a trait describing a `User` (with e.g. `get_name`) it may be implemented as a `LinuxUser` or `WindowsUser`, each providing extra fields. How do you generate a universal id for a struct generic over a `User`?