6 ms·

Nah, not like it is in Rust. For the vast majority of projects, serde makes parsing / serializing something they don't even have to think about, without having

Nah, not like it is in Rust. For the vast majority of projects, serde makes parsing / serializing something they don't even have to think about, without having to sacrifice type safety to get there. Until you experience that in a large project with a lot of moving parts and developers, it's hard to appreciate just how many developer cycles you were wasting before on stuff that can be totally automated. It's format-agnostic, it's really fast (not the absolute fastest, but fast enough for most situations), it's insanely customizable for even the weirdest use cases, it's absolutely everywhere, it incorporates proper error handling, and adding it to your own types is trivial. Is there any actual reason something like serde couldn't exist in other languages? None that I know of, but it doesn't, at least not when you consider just how pervasive serde is in the ecosystem (almost every library that stores things you want to serialize will have it). I really hope every language can standardize on something similar; it just makes development so much nicer and faster when you don't have to worry about this stuff.

fiddlerwoaroof 5y ago
These sorts of things are really convenient, but the idea of tightly binding your type definitions to serialization formats always makes be a little nervous: I haven’t used Rust enough in practice to have an opinion here, but I’m wary of a solution in this space that makes it too easy to ignore the forward/backwards compatibility of your external format.
- Jweb_Guru 5y ago
  Yeah, for sure that is something to be cautious of. For cases where I expect backwards compatibility to be a concern, I will usually make the top-level type of the message or format an enum with each variant being a version (initially with just a single version, V1 or V0). That way, if we need to make backwards-incompatible format changes, we can create a new variant, and potentially even automatically migrate between formats (I've done this a few times). Then all we have to do is call a function to migrate to the latest format, and we can use whatever that is for all the actual program code (generally it will be a type alias for the version structure of the same name). Of course, if this gets too complex or you start to have non-treelike relationships between values, you still want to fall back on a more traditional database schema, but this kind of versioning can get you a long way. That said, it's incredibly refreshing to be able to focus on these semantic and migration concerns without having to worry about how to deal with parsing them later. serde doesn't solve enforcing backwards compatibility for you, but it sure makes doing the right thing a lot easier. You never have to worry about some code accidentally forgetting to check the version field, people carelessly mixing and matching messages with different versions, or fields that "should" never be set but are there for backwards compatibility. That's worth a lot to me!
throwaway894345 5y ago
> For the vast majority of projects, serde makes parsing / serializing something they don't even have to think about, without having to sacrifice type safety to get there. This has not been remotely my experience. Deserializing has a lot of implications for ownership and you get into complicated deserializer trait specifications pretty quickly. In particular, there doesn’t seem to be a way to deserialize a type that has a reference because you don’t seem to be able to tell serde who should own that memory. Maybe I’m wrong, but I’ve walked through this multiple times with experienced Rust users and no one could get it working. Serde is impressive, I’m sure, but far from “don’t even have to think about it”.
- roca 5y ago
  We almost always deserialize to owned data, partly because our deserialization source is almost always compressed so the data has to be copied. There are a few cases where we want to serialize with a reference and deserialize to owned, but that's no big deal using custom "with" functions. We have hit one serde footgun, which is that skip-serializing-if corrupts your data with formats like bincode :-(. https://github.com/serde-rs/serde/issues/1732 https://github.com/serde-rs/serde/issues/1732 really wish that could be fixed.
deleted 5y ago
[deleted]
UK-Al05 5y ago
I rarely have to think about deserialisation in c# or java either...
- Jweb_Guru 5y ago
  I wrote a lot of Java back in the day. I elaborated on why I disagree in the other comment. I do not know if the situation in C# is similar.
  - Yacoby 5y ago
    For context, the example from the Serde wiki written in Java using Jackson public record Point(int x, int y) {} // [...] var objectMapper = new ObjectMapper(); var jsonString = objectMapper.writeValueAsString(new Point(2,1)); It isn't obvious to me what advantage Serde offers that you couldn't get with Jackson or other similar libraries in Java (although I get that Java and Rust are different languages, and that there may not be something this ergonomic in the likes of C++)
    - Jweb_Guru 5y ago
      For that example there's no advantage. But it's really, really easy to find examples where there are (the simplest example is, anything with pointers with @NonNull annotations, or anything which is performance-sensitive and contains other objects).
Const-me 5y ago
> Is there any actual reason something like serde couldn't exist in other languages? C# standard library has DataContractSerializer for decades. That thing reads/write text XML by default, but with minor tweaks can do binary XML or JSON. I like their binary XML format the most. Very fast because doesn’t waste time printing or parsing numbers or Base64 bytes. Also, pre-shared XML dictionary, and session-accumulating dynamic dictionary, makes the serialized representation very compact, sometimes an order of magnitude better than text/xml.
brundolf 5y ago
Macros are definitely key. I wonder if ownership helps too? Circular references are always weird when serializing
pjmlp 5y ago
That has been the same in Java and .NET world for 20 years now, just plug the desired serializer from the standard library and be done with it.
- Jweb_Guru 5y ago
  It absolutely does not provide the same guarantees, and I've covered why in several posts now. Do you believe that people who disagree with you have never used any language other than Rust before? Having to deal with data structure invariants getting violated due to reflection is a huge headache and makes the parsers way less useful than they would be otherwise.
  - pjmlp 5y ago
    Yes, because those same people seem to be unaware reflection is not the only way to use such APIs, revealing a superfical knowledge of how those ecosystems have been working for the last 25 years.
    - spoiler 5y ago
      How is it implemented in Java or C# if not via runtime reflection? I've always assumed that's the way its done...
      - andrewaylett 5y ago
        I'd always recommend folk use constructor-based deserialisation, rather than getting the deserialiser to create empty objects and poke at fields. That's still (at least naively) using reflection, but it's a lot safer. In practice, using reflection for every object being deserialised is far too slow and Jackson at least generates code at runtime rather than repeatedly reflecting. That's magic, but it's nice magic because it's not breaking the language: one can see that it's possible to implement it in pure Java.
      - pjmlp 5y ago
        You can use stuff like VS T4 templates or compiler plugins via annotation processors. So at compile time, just like serde works, you create your type safe serializer.

fiddlerwoaroof 5y ago

These sorts of things are really convenient, but the idea of tightly binding your type definitions to serialization formats always makes be a little nervous: I haven’t used Rust enough in practice to have an opinion here, but I’m wary of a solution in this space that makes it too easy to ignore the forward/backwards compatibility of your external format.

Jweb_Guru 5y ago
Yeah, for sure that is something to be cautious of. For cases where I expect backwards compatibility to be a concern, I will usually make the top-level type of the message or format an enum with each variant being a version (initially with just a single version, V1 or V0). That way, if we need to make backwards-incompatible format changes, we can create a new variant, and potentially even automatically migrate between formats (I've done this a few times). Then all we have to do is call a function to migrate to the latest format, and we can use whatever that is for all the actual program code (generally it will be a type alias for the version structure of the same name). Of course, if this gets too complex or you start to have non-treelike relationships between values, you still want to fall back on a more traditional database schema, but this kind of versioning can get you a long way. That said, it's incredibly refreshing to be able to focus on these semantic and migration concerns without having to worry about how to deal with parsing them later. serde doesn't solve enforcing backwards compatibility for you, but it sure makes doing the right thing a lot easier. You never have to worry about some code accidentally forgetting to check the version field, people carelessly mixing and matching messages with different versions, or fields that "should" never be set but are there for backwards compatibility. That's worth a lot to me!

throwaway894345 5y ago

> For the vast majority of projects, serde makes parsing / serializing something they don't even have to think about, without having to sacrifice type safety to get there. This has not been remotely my experience. Deserializing has a lot of implications for ownership and you get into complicated deserializer trait specifications pretty quickly. In particular, there doesn’t seem to be a way to deserialize a type that has a reference because you don’t seem to be able to tell serde who should own that memory. Maybe I’m wrong, but I’ve walked through this multiple times with experienced Rust users and no one could get it working. Serde is impressive, I’m sure, but far from “don’t even have to think about it”.

roca 5y ago
We almost always deserialize to owned data, partly because our deserialization source is almost always compressed so the data has to be copied. There are a few cases where we want to serialize with a reference and deserialize to owned, but that's no big deal using custom "with" functions. We have hit one serde footgun, which is that skip-serializing-if corrupts your data with formats like bincode :-(. https://github.com/serde-rs/serde/issues/1732 https://github.com/serde-rs/serde/issues/1732 really wish that could be fixed.

deleted 5y ago

[deleted]

UK-Al05 5y ago

I rarely have to think about deserialisation in c# or java either...

Jweb_Guru 5y ago
I wrote a lot of Java back in the day. I elaborated on why I disagree in the other comment. I do not know if the situation in C# is similar.
- Yacoby 5y ago
  For context, the example from the Serde wiki written in Java using Jackson public record Point(int x, int y) {} // [...] var objectMapper = new ObjectMapper(); var jsonString = objectMapper.writeValueAsString(new Point(2,1)); It isn't obvious to me what advantage Serde offers that you couldn't get with Jackson or other similar libraries in Java (although I get that Java and Rust are different languages, and that there may not be something this ergonomic in the likes of C++)
  - Jweb_Guru 5y ago
    For that example there's no advantage. But it's really, really easy to find examples where there are (the simplest example is, anything with pointers with @NonNull annotations, or anything which is performance-sensitive and contains other objects).

Const-me 5y ago

> Is there any actual reason something like serde couldn't exist in other languages? C# standard library has DataContractSerializer for decades. That thing reads/write text XML by default, but with minor tweaks can do binary XML or JSON. I like their binary XML format the most. Very fast because doesn’t waste time printing or parsing numbers or Base64 bytes. Also, pre-shared XML dictionary, and session-accumulating dynamic dictionary, makes the serialized representation very compact, sometimes an order of magnitude better than text/xml.

brundolf 5y ago

Macros are definitely key. I wonder if ownership helps too? Circular references are always weird when serializing

pjmlp 5y ago

That has been the same in Java and .NET world for 20 years now, just plug the desired serializer from the standard library and be done with it.

Jweb_Guru 5y ago
It absolutely does not provide the same guarantees, and I've covered why in several posts now. Do you believe that people who disagree with you have never used any language other than Rust before? Having to deal with data structure invariants getting violated due to reflection is a huge headache and makes the parsers way less useful than they would be otherwise.
- pjmlp 5y ago
  Yes, because those same people seem to be unaware reflection is not the only way to use such APIs, revealing a superfical knowledge of how those ecosystems have been working for the last 25 years.
  - spoiler 5y ago
    How is it implemented in Java or C# if not via runtime reflection? I've always assumed that's the way its done...
    - andrewaylett 5y ago
      I'd always recommend folk use constructor-based deserialisation, rather than getting the deserialiser to create empty objects and poke at fields. That's still (at least naively) using reflection, but it's a lot safer. In practice, using reflection for every object being deserialised is far too slow and Jackson at least generates code at runtime rather than repeatedly reflecting. That's magic, but it's nice magic because it's not breaking the language: one can see that it's possible to implement it in pure Java.
    - pjmlp 5y ago
      You can use stuff like VS T4 templates or compiler plugins via annotation processors. So at compile time, just like serde works, you create your type safe serializer.