6 ms·

The no-foreign-traits-on-foreign-types rule and the terrible ergonomics behind the newtype "pattern" combine to make one of the worst development experiences I'

by nmilo 3y ago

The no-foreign-traits-on-foreign-types rule and the terrible ergonomics behind the newtype "pattern" combine to make one of the worst development experiences I've ever seen in a language. It basically means you can only use one large library, ever. (With caveats.) For example, if I'm writing a game, I include a math library and pull in a Vec3 type, then add Vec3 as a member to all my objects, and then later I want to pull in a serialization library that defines, for ex, a Serializable trait, I can't, unless: 1. I write my own Vec3 class, not fun. 2. I write my own serialization library, could be fun depending on who you are but surely a waste of time. 3. I make a newtype, implement Serializable on it, find-and-replace all Vec3s with my own SerializableVec3, fix about 500 errors one by one by adding .0 everywhere, implement wrappers for all the Vec3 methods I want to use, rinse and repeat for every other foreign type I want to serialize. 4. Hope the math library and the serialization library know each other and the math library implements all the traits you need. In practice this is what happens, but this makes a weird situation where serde is the de facto standard/monopoly because it is literally impossible to use anything else. Which may be fine for now, serde is the standard precisely because it's so extensible, but if you ever want to use another library with traits, well you're SOL and it's back to step 3. How they managed to ship a language with dev ergonomics this bad is beyond me. Especially when the solution is so simple. Allow foreign traits on foreign types for executable projects only. Put it behind a compiler flag for all I care. -fallow-foreign-traits-I-dont-care-if-this-breaks-things. Just let me implement my damn traits, the current state is ridiculous.

fiedzia 3y ago
3. should be one click in your IDE.
staunton 3y ago
I strongly disagree that these are big problems in practice. There are macros you can use to make newtypes more ergonomic to use and reduce boilerplate, etc. The advantage it gives is forced adherence to many implicit assumptions of library authors, which means that it's a lot easier (read: preventing subtle bugs) to compose different libraries. Meanwhile, allowing it might split and completely break the ecosystem.
- pdimitar 3y ago
  Can you give examples of those macros? Very interested in using them.
justinpombrio 3y ago
You're missing: 5. Write a function `serialize_vec3(vec: Vec3) -> String`, and call it in the places you want to serialize a Vec3. The body of the function can convert it to a new type that #derives Serializable, or it can implement the Serializable trait directly.
dragonwriter 3y ago
> I make a newtype, implement Serializable on it, find-and-replace all Vec3s with my own SerializableVec3, fix about 500 errors one by one by adding .0 everywhere, implement wrappers for all the Vec3 methods I want to use, rinse and repeat for every other foreign type I want to serialize. You could also implement Deref (and, if appropriate, DerefMut) on the newtype, though there seems to be controversy over this pattern; for the specific case of a newtype that exists for the specific purpose of enabling foreign trait implementation, it seems to be a fairly straightforward and effective way of dealing with things.
- rascul 3y ago
  That pattern is documented here https://rust-unofficial.github.io/patterns/anti_patterns/deref.html https://rust-unofficial.github.io/patterns/anti_patterns/der...
  - themulticaster 3y ago
    The anti-pattern described as Deref Polymorphism is not the same as using Deref with the newtype pattern, in order to allow using the wrapped type transparently. In the latter case, the Target type of the Deref trait is always going to be perfectly clear. In case of the antipattern described on the linked page, it is not perfectly clear what the Target type is going to be. In short, a Deref impl for some type T signals that T represents some level of indirection, and following/dereferencing that indirection can always be done in an unsurprising and trivial manner.
stouset 3y ago
I won’t deny that the newtype pattern leaves a lot to be desired, but I think I’m practice this isn’t nearly as big a deal as you make it out to be. All that aside, > Especially when the solution is so simple. This is completely unwarranted and presupposes incompetency or apathy on the Rust dev team, either of which couldn’t be farther from the truth. Designing programming languages is hard and full of tradeoffs. While we can of course disagree about those tradeoffs, “simple” changes like these are never actually simple in practice and involve complex sets of tradeoffs that invariably have been discussed to obscene lengths. Even “simple” workarounds often commit to implementation guarantees that language designers are hesitant to make until they’re certain it won’t box them into a corner in the future. Put bluntly, anyone who asserts that a programming language design change is “simple” is only highlighting their own ignorance on the subject.
- nmilo 3y ago
  I didn't mean any offence when I called the change simple, though I do now see how it can be interpreted that way. When I call it simple it's because I am ignorant about programming language design changes, because, well, I don't really care. I'm not a PL designer. The change is simple to me because it: (a) seems arbitrary and easily changed, (b) solves a class of problem that I've never run into in other languages nor can ever really envision happening in a realistic scenario, and (c) if there are complex sets of tradeoffs involved in enabling this feature then they haven't been communicated well to the end-user at all. Especially with how bad the current state of affairs is, my calling the change simple should be a sign to the Rust team that if this really is the best possible decision for the language, they should put a clear statement in the compiler saying why. Because as it stands, from my end-user perspective, I still haven't seen a great reason why this shouldn't be implemented.
- ithkuil 3y ago
  Another example: https://crates.io/crates/deepsize https://crates.io/crates/deepsize You can implement that for your types. Deepsize crate implements the trait for a few popular libraries. You can derive the trait in your structures and it will work for the few structures that happen to contain only your structures and the few hand picked structures supported by the deepsize crate. And then? What about the 99% of the other cases? Adding a trait impl is acceptable. Wrapping all the other types not much
IshKebab 3y ago
It's definitely annoying. But to say it's > one of the worst development experiences I've ever seen in a language is pure hyperbole. All languages have issues at least as big as this, and most have far far bigger issues. Name a language and I'll tell you a much worse issue.
- nathanwh 3y ago
  > Name a language and I'll tell you a much worse issue. Not sure if the offer was only open to OP but I'll bite. How about Java?
  - monocasa 3y ago
    Boxed Integers will compare with == for values -128 to 127 but not other valid ints. Integer.valueOf(5) == Integer.valueOf(5) true Integer.valueOf(200) == Integer.valueOf(200) false
    - chaosite 3y ago
      That's a quirk of the language, sure, but it's barely an issue, and definitely not a worse issue. Don't get many wrong, Java has many issues, but this barely qualifies. You don't compare non-primitives (which boxed integers are) with == in Java, you use the equals method.
      - monocasa 3y ago
        You're not supposed to, but I've certainly found code out there that does use == to compare boxed integers because the test cases they used happen to work. And in fact when I googled to find the exact range where it switches from cached integers to truly creating new objects, the top link was at best highly misleading. https://www.tutorialspoint.com/check-two-numbers-for-equality-in-java https://www.tutorialspoint.com/check-two-numbers-for-equalit...
      - earthling8118 3y ago
        You definitely shouldn't compare them that way. But it still allows you to. That becomes a fairly big issue where you have the behavior working when you test it out and suddenly it doesn't work when the values become larger. Sure, an experienced Java developer will know this. How about a developer that is new to the language? Not as likely. I've personally lost over 6 hours to that one years and years ago.
ryukoposting 3y ago
The effort involved in (3) can be substantially reduced with some AsRef/AsMutRef/Deref/etc implementations on the wrapper type. If I had to add 500 instances of ".0" to my code, I'd ask myself if I'm taking the right approach to the problem I'm solving. .0 basically means "I don't care about the abstraction the newtype provides, I need the thing inside of it." It will be necessary from time to time, sure, but 500 times? Maybe instead, I can make something like struct Wrapper<'a>(&'a Vec3<_>); Then, I only put a Vec3 inside the Wrapper when I actually need to go serialize something. "Allowing foreign traits on foreign types for executable projects only" relies on the assumption that every Rust file is either part of a library, or part of an executable. Never both. This assumption is already false because hybrid crates exist, but it's particularly faulty when you consider that the dichotomy of executables and libraries can be extremely blurry in some domains, like in the case of loadable kernel modules or embedded firmware. A systems language cannot make validity choices based on assumptions about underlying ABI/binary formats without kneecapping the language's usefulness. The monopoly in serialization was inevitable regardless of their design choices about trait coherence. It's far more sane for everyone to agree on a single implementation rather than have 5 different feature flags so everyone can choose their favorite serialization lib. Imagine if you imported a library and discovered that it includes a bunch of functions that return johns_cool_library::Vec instead of std::Vec. Do we complain about std having a monopoly on vectors and strings? No.
- MrBuddyCasino 3y ago
  The majority of Rust code isn’t kernel extensions or embedded firmware, and its not like those use cases suffer any downsides by allowing the majority some more leeway. Therefore this strikes me as perfect being the enemy of good, but I guess that is Rust‘s culture.
rdedev 3y ago
Would phantom types help in this case ? I know that rust supports it
duped 3y ago
Most languages don't even have the anatomy of features that lead to this problem in the first place. The real solution is to open a PR on your math library and add the derives yourself, or fork it. Which is coincidentally what you would have to do in almost any other language, since you can't usually derive an interface for a class when you don't own the interface and the class.
jcrites 3y ago
I agree with you. I'd like to see Rust change to allow defining traits on foreign types, somehow. I've discussed this a number of times with folks on the Rust community Discord. There are unfortunately obstacles to making this happen. One obstacle is that if you implement a trait for a type, then that implementation applies to it everywhere, including in all other library code using the type. By implementing a trait you change its behavior in surprising and likely conflicting ways. Additionally, if you were to implement `Serialize` for some struct, then that would also directly conflict with the struct owner trying to do the same thing. My proposed solution to this problem is to conceptualized trait implementations as something that can be `use`d. The basic idea is that, if I implement `Serialize` for `Foo` (and `Foo` comes from another namespace) then the type-trait implementation remains private to mine (or perhaps, private to my namespace). That's not how Rust works today, but I'd be curious whether it could work. I realize this will create a new set of challenges. It will mean `Foo` comes with one set of behavior everywhere else, and another set of behavior in my namespaces where the trait is implemented for it. I don't know enough about Rust to foresee what kind of problems this would cause, but it seems tempting to explore. Usually the kinds of traits that you'd want to implement are not ones that will cause problems for other code.
- ithkuil 3y ago
  What if the author of the trait explicitly marks it to allow that?
- glandium 3y ago
  Sometimes I just wish that it were possible to do something akin to "pub(crate) impl Trait for Type", that would a) only make the trait implemented for the type in the current crate b) possibly override any other impl of the trait for said type.
- erutuon 3y ago
  I can see the convenience of sort of importing implementations of traits into scope. One difficulty is that traits are used for core functionality, like dereferencing and comparison operators. It would be inconvenient to have to import the implementations to do these basic things and confusing if some commonly used types could have their basic behavior dramatically changed. Maybe that could be solved by choosing some traits that are more central and preventing them from being switched out by importation, so you couldn't for instance change what `==` does to a pair of `f64`s.
deleted 3y ago
[deleted]
- deleted 3y ago
  [deleted]
armchairhacker 3y ago
This isn’t an issue with the “newtype” pattern, it’s an issue with orphan instances. Newtypes have other uses (e.g. contracts, type-safety for integers and other data which has specific meaning). This is just using the newtype pattern to get around the fundamental issue, which is that you can’t define a third-party instance for a third-party type. Personally, I’d like Rust to have “selectable” implementations, which are named and must be explicitly imported, and explicitly “applied” if there are multiple in scope. Selectable implementations always override regular implementations, so you can also replace the library’s provided implementation if it’s buggy or not what you wanted. I don’t even think there’s an RFC for this though… Aside: newtype isn’t even Rust-specific. It goes to show how popular Rust is and how much Rustaceons love type safety, that when you search “newtype pattern”, the first results are all Rust. The keyword “newtype” comes from Haskell, which also has the orphan rule and associated issues, but at least lets you disable it with a GHC rule. And a zero-cost wrapper is something you can do in Swift, Kotlin, C++, and even C, and the general newtype pattern (although not necessarily zero-cost) is something you can do in practically any typed language, even untyped ones like JavaScript if you consider runtime exceptions ok (use a struct with the custom type name as the field name).
dureuill 3y ago
I think I'd implement Serialize on a different type and implement conversions from the foreign type to my serializable type. That would mean that the locations where I want to serialize the foreign type would have to be marked with `.into()`, but that would be the sole maintenance overhead.

by nmilo 3y ago

fiedzia 3y ago

3. should be one click in your IDE.

staunton 3y ago

I strongly disagree that these are big problems in practice. There are macros you can use to make newtypes more ergonomic to use and reduce boilerplate, etc. The advantage it gives is forced adherence to many implicit assumptions of library authors, which means that it's a lot easier (read: preventing subtle bugs) to compose different libraries. Meanwhile, allowing it might split and completely break the ecosystem.

pdimitar 3y ago
Can you give examples of those macros? Very interested in using them.

justinpombrio 3y ago

You're missing: 5. Write a function `serialize_vec3(vec: Vec3) -> String`, and call it in the places you want to serialize a Vec3. The body of the function can convert it to a new type that #derives Serializable, or it can implement the Serializable trait directly.

dragonwriter 3y ago

> I make a newtype, implement Serializable on it, find-and-replace all Vec3s with my own SerializableVec3, fix about 500 errors one by one by adding .0 everywhere, implement wrappers for all the Vec3 methods I want to use, rinse and repeat for every other foreign type I want to serialize. You could also implement Deref (and, if appropriate, DerefMut) on the newtype, though there seems to be controversy over this pattern; for the specific case of a newtype that exists for the specific purpose of enabling foreign trait implementation, it seems to be a fairly straightforward and effective way of dealing with things.

rascul 3y ago
That pattern is documented here https://rust-unofficial.github.io/patterns/anti_patterns/deref.html https://rust-unofficial.github.io/patterns/anti_patterns/der...
- themulticaster 3y ago
  The anti-pattern described as Deref Polymorphism is not the same as using Deref with the newtype pattern, in order to allow using the wrapped type transparently. In the latter case, the Target type of the Deref trait is always going to be perfectly clear. In case of the antipattern described on the linked page, it is not perfectly clear what the Target type is going to be. In short, a Deref impl for some type T signals that T represents some level of indirection, and following/dereferencing that indirection can always be done in an unsurprising and trivial manner.

stouset 3y ago

I won’t deny that the newtype pattern leaves a lot to be desired, but I think I’m practice this isn’t nearly as big a deal as you make it out to be. All that aside, > Especially when the solution is so simple. This is completely unwarranted and presupposes incompetency or apathy on the Rust dev team, either of which couldn’t be farther from the truth. Designing programming languages is hard and full of tradeoffs. While we can of course disagree about those tradeoffs, “simple” changes like these are never actually simple in practice and involve complex sets of tradeoffs that invariably have been discussed to obscene lengths. Even “simple” workarounds often commit to implementation guarantees that language designers are hesitant to make until they’re certain it won’t box them into a corner in the future. Put bluntly, anyone who asserts that a programming language design change is “simple” is only highlighting their own ignorance on the subject.

nmilo 3y ago
I didn't mean any offence when I called the change simple, though I do now see how it can be interpreted that way. When I call it simple it's because I am ignorant about programming language design changes, because, well, I don't really care. I'm not a PL designer. The change is simple to me because it: (a) seems arbitrary and easily changed, (b) solves a class of problem that I've never run into in other languages nor can ever really envision happening in a realistic scenario, and (c) if there are complex sets of tradeoffs involved in enabling this feature then they haven't been communicated well to the end-user at all. Especially with how bad the current state of affairs is, my calling the change simple should be a sign to the Rust team that if this really is the best possible decision for the language, they should put a clear statement in the compiler saying why. Because as it stands, from my end-user perspective, I still haven't seen a great reason why this shouldn't be implemented.
ithkuil 3y ago
Another example: https://crates.io/crates/deepsize https://crates.io/crates/deepsize You can implement that for your types. Deepsize crate implements the trait for a few popular libraries. You can derive the trait in your structures and it will work for the few structures that happen to contain only your structures and the few hand picked structures supported by the deepsize crate. And then? What about the 99% of the other cases? Adding a trait impl is acceptable. Wrapping all the other types not much

IshKebab 3y ago

It's definitely annoying. But to say it's > one of the worst development experiences I've ever seen in a language is pure hyperbole. All languages have issues at least as big as this, and most have far far bigger issues. Name a language and I'll tell you a much worse issue.

nathanwh 3y ago
> Name a language and I'll tell you a much worse issue. Not sure if the offer was only open to OP but I'll bite. How about Java?
- monocasa 3y ago
  Boxed Integers will compare with == for values -128 to 127 but not other valid ints. Integer.valueOf(5) == Integer.valueOf(5) true Integer.valueOf(200) == Integer.valueOf(200) false
  - chaosite 3y ago
    That's a quirk of the language, sure, but it's barely an issue, and definitely not a worse issue. Don't get many wrong, Java has many issues, but this barely qualifies. You don't compare non-primitives (which boxed integers are) with == in Java, you use the equals method.
    - monocasa 3y ago
      You're not supposed to, but I've certainly found code out there that does use == to compare boxed integers because the test cases they used happen to work. And in fact when I googled to find the exact range where it switches from cached integers to truly creating new objects, the top link was at best highly misleading. https://www.tutorialspoint.com/check-two-numbers-for-equality-in-java https://www.tutorialspoint.com/check-two-numbers-for-equalit...
    - earthling8118 3y ago
      You definitely shouldn't compare them that way. But it still allows you to. That becomes a fairly big issue where you have the behavior working when you test it out and suddenly it doesn't work when the values become larger. Sure, an experienced Java developer will know this. How about a developer that is new to the language? Not as likely. I've personally lost over 6 hours to that one years and years ago.

ryukoposting 3y ago

The effort involved in (3) can be substantially reduced with some AsRef/AsMutRef/Deref/etc implementations on the wrapper type. If I had to add 500 instances of ".0" to my code, I'd ask myself if I'm taking the right approach to the problem I'm solving. .0 basically means "I don't care about the abstraction the newtype provides, I need the thing inside of it." It will be necessary from time to time, sure, but 500 times? Maybe instead, I can make something like struct Wrapper<'a>(&'a Vec3<_>); Then, I only put a Vec3 inside the Wrapper when I actually need to go serialize something. "Allowing foreign traits on foreign types for executable projects only" relies on the assumption that every Rust file is either part of a library, or part of an executable. Never both. This assumption is already false because hybrid crates exist, but it's particularly faulty when you consider that the dichotomy of executables and libraries can be extremely blurry in some domains, like in the case of loadable kernel modules or embedded firmware. A systems language cannot make validity choices based on assumptions about underlying ABI/binary formats without kneecapping the language's usefulness. The monopoly in serialization was inevitable regardless of their design choices about trait coherence. It's far more sane for everyone to agree on a single implementation rather than have 5 different feature flags so everyone can choose their favorite serialization lib. Imagine if you imported a library and discovered that it includes a bunch of functions that return johns_cool_library::Vec instead of std::Vec. Do we complain about std having a monopoly on vectors and strings? No.

MrBuddyCasino 3y ago
The majority of Rust code isn’t kernel extensions or embedded firmware, and its not like those use cases suffer any downsides by allowing the majority some more leeway. Therefore this strikes me as perfect being the enemy of good, but I guess that is Rust‘s culture.

rdedev 3y ago

Would phantom types help in this case ? I know that rust supports it

duped 3y ago

Most languages don't even have the anatomy of features that lead to this problem in the first place. The real solution is to open a PR on your math library and add the derives yourself, or fork it. Which is coincidentally what you would have to do in almost any other language, since you can't usually derive an interface for a class when you don't own the interface and the class.

jcrites 3y ago

I agree with you. I'd like to see Rust change to allow defining traits on foreign types, somehow. I've discussed this a number of times with folks on the Rust community Discord. There are unfortunately obstacles to making this happen. One obstacle is that if you implement a trait for a type, then that implementation applies to it everywhere, including in all other library code using the type. By implementing a trait you change its behavior in surprising and likely conflicting ways. Additionally, if you were to implement `Serialize` for some struct, then that would also directly conflict with the struct owner trying to do the same thing. My proposed solution to this problem is to conceptualized trait implementations as something that can be `use`d. The basic idea is that, if I implement `Serialize` for `Foo` (and `Foo` comes from another namespace) then the type-trait implementation remains private to mine (or perhaps, private to my namespace). That's not how Rust works today, but I'd be curious whether it could work. I realize this will create a new set of challenges. It will mean `Foo` comes with one set of behavior everywhere else, and another set of behavior in my namespaces where the trait is implemented for it. I don't know enough about Rust to foresee what kind of problems this would cause, but it seems tempting to explore. Usually the kinds of traits that you'd want to implement are not ones that will cause problems for other code.

ithkuil 3y ago
What if the author of the trait explicitly marks it to allow that?
glandium 3y ago
Sometimes I just wish that it were possible to do something akin to "pub(crate) impl Trait for Type", that would a) only make the trait implemented for the type in the current crate b) possibly override any other impl of the trait for said type.
erutuon 3y ago
I can see the convenience of sort of importing implementations of traits into scope. One difficulty is that traits are used for core functionality, like dereferencing and comparison operators. It would be inconvenient to have to import the implementations to do these basic things and confusing if some commonly used types could have their basic behavior dramatically changed. Maybe that could be solved by choosing some traits that are more central and preventing them from being switched out by importation, so you couldn't for instance change what `==` does to a pair of `f64`s.

deleted 3y ago

[deleted]

deleted 3y ago
[deleted]

armchairhacker 3y ago

This isn’t an issue with the “newtype” pattern, it’s an issue with orphan instances. Newtypes have other uses (e.g. contracts, type-safety for integers and other data which has specific meaning). This is just using the newtype pattern to get around the fundamental issue, which is that you can’t define a third-party instance for a third-party type. Personally, I’d like Rust to have “selectable” implementations, which are named and must be explicitly imported, and explicitly “applied” if there are multiple in scope. Selectable implementations always override regular implementations, so you can also replace the library’s provided implementation if it’s buggy or not what you wanted. I don’t even think there’s an RFC for this though… Aside: newtype isn’t even Rust-specific. It goes to show how popular Rust is and how much Rustaceons love type safety, that when you search “newtype pattern”, the first results are all Rust. The keyword “newtype” comes from Haskell, which also has the orphan rule and associated issues, but at least lets you disable it with a GHC rule. And a zero-cost wrapper is something you can do in Swift, Kotlin, C++, and even C, and the general newtype pattern (although not necessarily zero-cost) is something you can do in practically any typed language, even untyped ones like JavaScript if you consider runtime exceptions ok (use a struct with the custom type name as the field name).

dureuill 3y ago

I think I'd implement Serialize on a different type and implement conversions from the foreign type to my serializable type. That would mean that the locations where I want to serialize the foreign type would have to be marked with `.into()`, but that would be the sole maintenance overhead.