8 ms·

I think in most ecosystems it's easier to use a prebuilt serializer/deserializer than your own format.

dexwiz 5y ago
Yeah I don’t know why you would try to write your own textbased format. GSON and Jackson have been around for over a decade in Java, and equivalents in other languages. Unless you are in some perf critical application, a library is almost always better for this problem.
roca 5y ago
Fair enough. I have not found that to be true in C/C++, which is my main point of comparison. For very simple cases it's generally easier to write a few lines to a text file, maybe with some whitespace separators in the lines, than to use an "real" format. For JS and Python, JSON.stringify/parse and json.loads/dumps are a step up, but you still end up with an untyped mess with no schema validation, which makes them only halfway solutions to me. I'm a static typing guy at heart, sue me.
- eptcyka 5y ago
  Yea, but with serde, you don't get an untyped mess. You either get the type you expected or you get an error.
  - vips7L 5y ago
    Same is true in any typed language.
    - spoiler 5y ago
      I don't think that's true. Or at least we have different ideas about what's being discussed... Let's take JSON or YAML for example: Rust's closest siblings C and C++ are typed, and you get a pretty much untyped messes when working with serialisation or deserialisation. You have to manually inspect every node or write manual ”NodeType" to struct conversation. They allow unwrapping to primitives at best (eg via template specialisation). In Haskell it's a bit more awkward, but you can kinda pull something similar off in terms of ergonomics. I've not worked with JSON in Haskell that much that I need to look for it, but I don't remember there being something as ergonomic as serde. Typescript doesn't support this type of validation either out of the box (there might be tools that add validation). In Elixir there's the Poison library, or it could be done with Kernel.struct/2 (not 100% sure this will work though). Of the "mainstream" languages Go is pretty much the only one that has as good ergonomics as Rust in that it supports it pretty much natively (via struct tags) --- This list excludes codegen tools which can generate (de)serialisers from an external schema (a la capnp, grpc, jsonschema, etc).
      - UK-Al05 5y ago
        Most typed languages it's like this var account = JsonConvert.DeserializeObject<Account>(string/stream) That's it. No inspection or looping through trees. You can often get it to create objects via constructor for validation. It's been like this for 15 years...
        deleted 5y ago
        [deleted]
      - vips7L 5y ago
        Maybe we are having different ideas about what is being discussed, but in Java and C# (arguably the most mainstream languages) you have the same experience supported natively via annotations and attributes.
        spoiler 5y ago
        Ah, I stand corrected then. I've worked with the Microsoft tech stack very very little, so I've never bothered to learn C#. I've completely forgot about Java; I haven't used it in ages. Now that you mentioned Java, I realised Kotlin has it too.
      - rudedogg 5y ago
        > Of the "mainstream" languages Go is pretty much the only one that has as good ergonomics as Rust in that it supports it pretty much natively (via struct tags) Swift does this too, using the Codable Protocol. The compiler generates the protocol conformance code for you, if you say a type is `Codable`, and the properties the type contains are all Codable types (Int, Float, String, URL, etc. conform to it), you don't have to do anything. https://developer.apple.com/documentation/foundation/archives_and_serialization/encoding_and_decoding_custom_types https://developer.apple.com/documentation/foundation/archive...
        spoiler 5y ago
        Ah, I don't use Apple devices, so I've never been able to play with Swift(1). So I didn't know Swift had it too; thank you for pointing it out! (1): which is a shame, I remember when it came out that it seemed like a nice language
pornel 5y ago
The trick with Serde is that it decodes straight into a native Rust struct. You get most of validation for free (it also nicely takes advantage of Rust enums with data), and struct access is maximally fast. A generic JSON decoder would give you a dynamic structure that can contain anything, and then you'd have to pick it apart.
- UK-Al05 5y ago
  Most deserializers in typed languages allow you to deserialize straight into structs or typed objects... Serde just doesn't use reflection to do it.
  - Taywee 5y ago
    C++ isn't super easy about it, though. The best C++ json library (in my opinion), nlohmann's json, still requires you to define to_json and from_json for your types to use them. It's not too bad as a one-off, but when you have dozens of types, it gets really tedious compared to Serde, and managing std::variant with it is way more annoying than Serde with Rust enums.
  - Jweb_Guru 5y ago
    APIs like GSON and Jackson have a lot more footguns, because (1) they attempt to (de)serialize types whether or not they were ever intended to be (de)serialized, (2) they are missing information about important constraints due to losing information at runtime (the excuse usually used here is that "they don't do validation" when in reality they are violating expectations of package modularity and encapsulation in a pretty blunt way). For example, they will happily ignore non-null constraints on your fields. In practice, this is a major annoyance and ends up making you very paranoid about (de)serializing any structure you don't have full control over. Performance of a lot of the reflection-based parsing APIs also leaves something to be desired, which means that projects with more stringent performance requirements often have to resort to stuff like Avro. This still happens with serde, but much more rarely--besides having more compile-time information at its disposal and needing to allocate less, it also provides relatively straightforward hooks to achieve things like zero-copy deserialization for strings where the whole buffer is available at once.
Jweb_Guru 5y ago
Nah, not like it is in Rust. For the vast majority of projects, serde makes parsing / serializing something they don't even have to think about, without having to sacrifice type safety to get there. Until you experience that in a large project with a lot of moving parts and developers, it's hard to appreciate just how many developer cycles you were wasting before on stuff that can be totally automated. It's format-agnostic, it's really fast (not the absolute fastest, but fast enough for most situations), it's insanely customizable for even the weirdest use cases, it's absolutely everywhere, it incorporates proper error handling, and adding it to your own types is trivial. Is there any actual reason something like serde couldn't exist in other languages? None that I know of, but it doesn't, at least not when you consider just how pervasive serde is in the ecosystem (almost every library that stores things you want to serialize will have it). I really hope every language can standardize on something similar; it just makes development so much nicer and faster when you don't have to worry about this stuff.
- fiddlerwoaroof 5y ago
  These sorts of things are really convenient, but the idea of tightly binding your type definitions to serialization formats always makes be a little nervous: I haven’t used Rust enough in practice to have an opinion here, but I’m wary of a solution in this space that makes it too easy to ignore the forward/backwards compatibility of your external format.
  - Jweb_Guru 5y ago
    Yeah, for sure that is something to be cautious of. For cases where I expect backwards compatibility to be a concern, I will usually make the top-level type of the message or format an enum with each variant being a version (initially with just a single version, V1 or V0). That way, if we need to make backwards-incompatible format changes, we can create a new variant, and potentially even automatically migrate between formats (I've done this a few times). Then all we have to do is call a function to migrate to the latest format, and we can use whatever that is for all the actual program code (generally it will be a type alias for the version structure of the same name). Of course, if this gets too complex or you start to have non-treelike relationships between values, you still want to fall back on a more traditional database schema, but this kind of versioning can get you a long way. That said, it's incredibly refreshing to be able to focus on these semantic and migration concerns without having to worry about how to deal with parsing them later. serde doesn't solve enforcing backwards compatibility for you, but it sure makes doing the right thing a lot easier. You never have to worry about some code accidentally forgetting to check the version field, people carelessly mixing and matching messages with different versions, or fields that "should" never be set but are there for backwards compatibility. That's worth a lot to me!
thedracle 5y ago
At least in languages with similar native performance as Rust (C/C++), in my personal experience, serialization has often been difficult and manual, and most libraries have fallen short of my expectations, or had serious performance, security, or toolchain issues, to the point I usually hand-serialize myself.

Yeah I don’t know why you would try to write your own textbased format. GSON and Jackson have been around for over a decade in Java, and equivalents in other languages. Unless you are in some perf critical application, a library is almost always better for this problem.

roca 5y ago

Fair enough. I have not found that to be true in C/C++, which is my main point of comparison. For very simple cases it's generally easier to write a few lines to a text file, maybe with some whitespace separators in the lines, than to use an "real" format. For JS and Python, JSON.stringify/parse and json.loads/dumps are a step up, but you still end up with an untyped mess with no schema validation, which makes them only halfway solutions to me. I'm a static typing guy at heart, sue me.

eptcyka 5y ago
Yea, but with serde, you don't get an untyped mess. You either get the type you expected or you get an error.
- vips7L 5y ago
  Same is true in any typed language.
  - spoiler 5y ago
    I don't think that's true. Or at least we have different ideas about what's being discussed... Let's take JSON or YAML for example: Rust's closest siblings C and C++ are typed, and you get a pretty much untyped messes when working with serialisation or deserialisation. You have to manually inspect every node or write manual ”NodeType" to struct conversation. They allow unwrapping to primitives at best (eg via template specialisation). In Haskell it's a bit more awkward, but you can kinda pull something similar off in terms of ergonomics. I've not worked with JSON in Haskell that much that I need to look for it, but I don't remember there being something as ergonomic as serde. Typescript doesn't support this type of validation either out of the box (there might be tools that add validation). In Elixir there's the Poison library, or it could be done with Kernel.struct/2 (not 100% sure this will work though). Of the "mainstream" languages Go is pretty much the only one that has as good ergonomics as Rust in that it supports it pretty much natively (via struct tags) --- This list excludes codegen tools which can generate (de)serialisers from an external schema (a la capnp, grpc, jsonschema, etc).
    - UK-Al05 5y ago
      Most typed languages it's like this var account = JsonConvert.DeserializeObject<Account>(string/stream) That's it. No inspection or looping through trees. You can often get it to create objects via constructor for validation. It's been like this for 15 years...
      - deleted 5y ago
        [deleted]
    - vips7L 5y ago
      Maybe we are having different ideas about what is being discussed, but in Java and C# (arguably the most mainstream languages) you have the same experience supported natively via annotations and attributes.
      - spoiler 5y ago
        Ah, I stand corrected then. I've worked with the Microsoft tech stack very very little, so I've never bothered to learn C#. I've completely forgot about Java; I haven't used it in ages. Now that you mentioned Java, I realised Kotlin has it too.
    - rudedogg 5y ago
      > Of the "mainstream" languages Go is pretty much the only one that has as good ergonomics as Rust in that it supports it pretty much natively (via struct tags) Swift does this too, using the Codable Protocol. The compiler generates the protocol conformance code for you, if you say a type is `Codable`, and the properties the type contains are all Codable types (Int, Float, String, URL, etc. conform to it), you don't have to do anything. https://developer.apple.com/documentation/foundation/archives_and_serialization/encoding_and_decoding_custom_types https://developer.apple.com/documentation/foundation/archive...
      - spoiler 5y ago
        Ah, I don't use Apple devices, so I've never been able to play with Swift(1). So I didn't know Swift had it too; thank you for pointing it out! (1): which is a shame, I remember when it came out that it seemed like a nice language

pornel 5y ago

The trick with Serde is that it decodes straight into a native Rust struct. You get most of validation for free (it also nicely takes advantage of Rust enums with data), and struct access is maximally fast. A generic JSON decoder would give you a dynamic structure that can contain anything, and then you'd have to pick it apart.

UK-Al05 5y ago
Most deserializers in typed languages allow you to deserialize straight into structs or typed objects... Serde just doesn't use reflection to do it.
- Taywee 5y ago
  C++ isn't super easy about it, though. The best C++ json library (in my opinion), nlohmann's json, still requires you to define to_json and from_json for your types to use them. It's not too bad as a one-off, but when you have dozens of types, it gets really tedious compared to Serde, and managing std::variant with it is way more annoying than Serde with Rust enums.
- Jweb_Guru 5y ago
  APIs like GSON and Jackson have a lot more footguns, because (1) they attempt to (de)serialize types whether or not they were ever intended to be (de)serialized, (2) they are missing information about important constraints due to losing information at runtime (the excuse usually used here is that "they don't do validation" when in reality they are violating expectations of package modularity and encapsulation in a pretty blunt way). For example, they will happily ignore non-null constraints on your fields. In practice, this is a major annoyance and ends up making you very paranoid about (de)serializing any structure you don't have full control over. Performance of a lot of the reflection-based parsing APIs also leaves something to be desired, which means that projects with more stringent performance requirements often have to resort to stuff like Avro. This still happens with serde, but much more rarely--besides having more compile-time information at its disposal and needing to allocate less, it also provides relatively straightforward hooks to achieve things like zero-copy deserialization for strings where the whole buffer is available at once.

Jweb_Guru 5y ago

Nah, not like it is in Rust. For the vast majority of projects, serde makes parsing / serializing something they don't even have to think about, without having to sacrifice type safety to get there. Until you experience that in a large project with a lot of moving parts and developers, it's hard to appreciate just how many developer cycles you were wasting before on stuff that can be totally automated. It's format-agnostic, it's really fast (not the absolute fastest, but fast enough for most situations), it's insanely customizable for even the weirdest use cases, it's absolutely everywhere, it incorporates proper error handling, and adding it to your own types is trivial. Is there any actual reason something like serde couldn't exist in other languages? None that I know of, but it doesn't, at least not when you consider just how pervasive serde is in the ecosystem (almost every library that stores things you want to serialize will have it). I really hope every language can standardize on something similar; it just makes development so much nicer and faster when you don't have to worry about this stuff.

fiddlerwoaroof 5y ago
These sorts of things are really convenient, but the idea of tightly binding your type definitions to serialization formats always makes be a little nervous: I haven’t used Rust enough in practice to have an opinion here, but I’m wary of a solution in this space that makes it too easy to ignore the forward/backwards compatibility of your external format.
- Jweb_Guru 5y ago
  Yeah, for sure that is something to be cautious of. For cases where I expect backwards compatibility to be a concern, I will usually make the top-level type of the message or format an enum with each variant being a version (initially with just a single version, V1 or V0). That way, if we need to make backwards-incompatible format changes, we can create a new variant, and potentially even automatically migrate between formats (I've done this a few times). Then all we have to do is call a function to migrate to the latest format, and we can use whatever that is for all the actual program code (generally it will be a type alias for the version structure of the same name). Of course, if this gets too complex or you start to have non-treelike relationships between values, you still want to fall back on a more traditional database schema, but this kind of versioning can get you a long way. That said, it's incredibly refreshing to be able to focus on these semantic and migration concerns without having to worry about how to deal with parsing them later. serde doesn't solve enforcing backwards compatibility for you, but it sure makes doing the right thing a lot easier. You never have to worry about some code accidentally forgetting to check the version field, people carelessly mixing and matching messages with different versions, or fields that "should" never be set but are there for backwards compatibility. That's worth a lot to me!

thedracle 5y ago

At least in languages with similar native performance as Rust (C/C++), in my personal experience, serialization has often been difficult and manual, and most libraries have fallen short of my expectations, or had serious performance, security, or toolchain issues, to the point I usually hand-serialize myself.