15 ms·

The Serde Rust Framework

360 points by ur-whale 5y ago

deleted 5y ago
[deleted]
roca 5y ago
serde is one of the best things about Rust in practice. It is more convenient to use serde to serialize/deserialize to some standard format like JSON or YAML than it is to write your own half-baked format and serialization/deserialization code --- even for the simplest tasks --- so you just stop doing the latter. This is a fundamental shift, and very good for your software. "Convenience" here actually covers a lot of ground. The APIs are convenient. Boilerplate code is minimal (#[derive(Serialize)]). serde's attributes give you lots of control over the serialization/deserialization, while still being convenient to use. cargo makes importing serde into your project effortless. All of these are necessary to achieve that shift away from custom formats. Before Rust I used to write a lot of one-off half-baked formats. Pernosco uses Rust and serde and it uses no half-baked formats. Everything's JSON, or YAML if it needs to be more human-editable, or bincode if it needs to be fast and compact and not human-readable or extensible.
- UK-Al05 5y ago
  I think in most ecosystems it's easier to use a prebuilt serializer/deserializer than your own format.
  - dexwiz 5y ago
    Yeah I don’t know why you would try to write your own textbased format. GSON and Jackson have been around for over a decade in Java, and equivalents in other languages. Unless you are in some perf critical application, a library is almost always better for this problem.
  - roca 5y ago
    Fair enough. I have not found that to be true in C/C++, which is my main point of comparison. For very simple cases it's generally easier to write a few lines to a text file, maybe with some whitespace separators in the lines, than to use an "real" format. For JS and Python, JSON.stringify/parse and json.loads/dumps are a step up, but you still end up with an untyped mess with no schema validation, which makes them only halfway solutions to me. I'm a static typing guy at heart, sue me.
    - eptcyka 5y ago
      Yea, but with serde, you don't get an untyped mess. You either get the type you expected or you get an error.
      - vips7L 5y ago
        Same is true in any typed language.
        spoiler 5y ago
        I don't think that's true. Or at least we have different ideas about what's being discussed... Let's take JSON or YAML for example: Rust's closest siblings C and C++ are typed, and you get a pretty much untyped messes when working with serialisation or deserialisation. You have to manually inspect every node or write manual ”NodeType" to struct conversation. They allow unwrapping to primitives at best (eg via template specialisation). In Haskell it's a bit more awkward, but you can kinda pull something similar off in terms of ergonomics. I've not worked with JSON in Haskell that much that I need to look for it, but I don't remember there being something as ergonomic as serde. Typescript doesn't support this type of validation either out of the box (there might be tools that add validation). In Elixir there's the Poison library, or it could be done with Kernel.struct/2 (not 100% sure this will work though). Of the "mainstream" languages Go is pretty much the only one that has as good ergonomics as Rust in that it supports it pretty much natively (via struct tags) --- This list excludes codegen tools which can generate (de)serialisers from an external schema (a la capnp, grpc, jsonschema, etc).
        UK-Al05 5y ago
        Most typed languages it's like this var account = JsonConvert.DeserializeObject<Account>(string/stream) That's it. No inspection or looping through trees. You can often get it to create objects via constructor for validation. It's been like this for 15 years...
        deleted 5y ago
  - pornel 5y ago
    The trick with Serde is that it decodes straight into a native Rust struct. You get most of validation for free (it also nicely takes advantage of Rust enums with data), and struct access is maximally fast. A generic JSON decoder would give you a dynamic structure that can contain anything, and then you'd have to pick it apart.
    - UK-Al05 5y ago
      Most deserializers in typed languages allow you to deserialize straight into structs or typed objects... Serde just doesn't use reflection to do it.
      - Taywee 5y ago
        C++ isn't super easy about it, though. The best C++ json library (in my opinion), nlohmann's json, still requires you to define to_json and from_json for your types to use them. It's not too bad as a one-off, but when you have dozens of types, it gets really tedious compared to Serde, and managing std::variant with it is way more annoying than Serde with Rust enums.
      - Jweb_Guru 5y ago
        APIs like GSON and Jackson have a lot more footguns, because (1) they attempt to (de)serialize types whether or not they were ever intended to be (de)serialized, (2) they are missing information about important constraints due to losing information at runtime (the excuse usually used here is that "they don't do validation" when in reality they are violating expectations of package modularity and encapsulation in a pretty blunt way). For example, they will happily ignore non-null constraints on your fields. In practice, this is a major annoyance and ends up making you very paranoid about (de)serializing any structure you don't have full control over. Performance of a lot of the reflection-based parsing APIs also leaves something to be desired, which means that projects with more stringent performance requirements often have to resort to stuff like Avro. This still happens with serde, but much more rarely--besides having more compile-time information at its disposal and needing to allocate less, it also provides relatively straightforward hooks to achieve things like zero-copy deserialization for strings where the whole buffer is available at once.
  - Jweb_Guru 5y ago
    Nah, not like it is in Rust. For the vast majority of projects, serde makes parsing / serializing something they don't even have to think about, without having to sacrifice type safety to get there. Until you experience that in a large project with a lot of moving parts and developers, it's hard to appreciate just how many developer cycles you were wasting before on stuff that can be totally automated. It's format-agnostic, it's really fast (not the absolute fastest, but fast enough for most situations), it's insanely customizable for even the weirdest use cases, it's absolutely everywhere, it incorporates proper error handling, and adding it to your own types is trivial. Is there any actual reason something like serde couldn't exist in other languages? None that I know of, but it doesn't, at least not when you consider just how pervasive serde is in the ecosystem (almost every library that stores things you want to serialize will have it). I really hope every language can standardize on something similar; it just makes development so much nicer and faster when you don't have to worry about this stuff.
    - fiddlerwoaroof 5y ago
      These sorts of things are really convenient, but the idea of tightly binding your type definitions to serialization formats always makes be a little nervous: I haven’t used Rust enough in practice to have an opinion here, but I’m wary of a solution in this space that makes it too easy to ignore the forward/backwards compatibility of your external format.
      - Jweb_Guru 5y ago
        Yeah, for sure that is something to be cautious of. For cases where I expect backwards compatibility to be a concern, I will usually make the top-level type of the message or format an enum with each variant being a version (initially with just a single version, V1 or V0). That way, if we need to make backwards-incompatible format changes, we can create a new variant, and potentially even automatically migrate between formats (I've done this a few times). Then all we have to do is call a function to migrate to the latest format, and we can use whatever that is for all the actual program code (generally it will be a type alias for the version structure of the same name). Of course, if this gets too complex or you start to have non-treelike relationships between values, you still want to fall back on a more traditional database schema, but this kind of versioning can get you a long way. That said, it's incredibly refreshing to be able to focus on these semantic and migration concerns without having to worry about how to deal with parsing them later. serde doesn't solve enforcing backwards compatibility for you, but it sure makes doing the right thing a lot easier. You never have to worry about some code accidentally forgetting to check the version field, people carelessly mixing and matching messages with different versions, or fields that "should" never be set but are there for backwards compatibility. That's worth a lot to me!
  - thedracle 5y ago
    At least in languages with similar native performance as Rust (C/C++), in my personal experience, serialization has often been difficult and manual, and most libraries have fallen short of my expectations, or had serious performance, security, or toolchain issues, to the point I usually hand-serialize myself.
- danobi 5y ago
  Completely agree. FWIW, I think procedural macros make a huge difference for ergonomics. For example, a "similar" library in C++ (libcereal) is still a PITA to use and debug.
- vvanders 5y ago
  It's also blazing-fast. I've done some benchmarking on JSON -> in-memory borrow structs and it was pretty darn impressive while being fairly ergonomic to use compared to a streaming parser.
- the_duke 5y ago
  The only downside is compile time bloat. Serde generates heaps and heaps of generic code. This all gets optimized away to be very efficient, but only once it reaches LLVM. Ever tried working on a crate with hundreds or thousands of de/serializable types? Compile times shoot through the roof really quickly, and serde is often the culprit. The maintainer of serde also created `miniserde` [1] to tackle this problem, which uses dynamic dispatch instead of generics and can have 4x compile time improvements. Due to Rusts lack of orphan instances you really depend on a pervasive standard for serialization, though, so the ecosystem is pretty locked in to serde. [1] https://github.com/dtolnay/miniserde https://github.com/dtolnay/miniserde
  - munro 5y ago
    I've been using Rust for just a little bit, building applications/services/scrapers/databases/forking libraries, but I haven't noticed these issues--the biggest compile time issue I had was including RocksDB. I'm trying to imagine what you're building. :} 99% of the time I'm just doing incremental debug builds, and haven't really minded the compile times when doing release builds. Overall it everything has been fast, to the point where I want to start scripting with Rust.
  - unanswered 5y ago
    Studies have shown that more than 87% of complaints about rust compile time are because people are enabling serde everywhere and for everything.
    - johnfn 5y ago
      These concerns are valid. Data serialization/deserialization is an extremely standard thing to need to be able to do. It's not the user's fault if they use the obvious tool for an obvious job and suddenly have bloated compile times.
      - vlovich123 5y ago
        Seems like there’s opportunity for tighter integration with the compiler where serde moves into the compiler or there are acceleration primitives rather than doing the AST macros thing. This sounds similar to the problems of c++ template bloat.
- curun1r 5y ago
  The other benefit is that almost all of the Rust ecosystem supports serde, often by way of a feature. So using serde can mean being able to embed a type from a crate in your serializable type or being able to pass your own types to crate functions/methods that serialize/deserialize. It all ends up being interoperable.
- thedracle 5y ago
  The extendibility of it is impressive, as well as the list of formats supported by the community. The ability to output JSON for debugging next to tight-concise binary formats, is really appealing. I've dealt with a number of C++ based serialization libraries, and they always have serious downsides to the point I often end up hand serializing structures with one-off half-baked formats just like you, which is error prone and laborious.
paulgb 5y ago
To me, Serde is one of the killer features of Rust. Once you realize how much of programming is (for better or worse) shuttling data between different representations, it's hard for me to program in a language where I don't have a tool like Serde. A big part of its flexibility is how modular it is. Most common Rust libraries support serde serialization (at least as an optional feature), so if you use crates that do, you can plug any backend in and serialize those data structures to it. It doesn't even have to be string-based; I've been using Serde to store arbitrary data structures as objects on Google Cloud Firestore.
AaronO 5y ago
Whilst not classical de/serialization I wrote serde_v8 (https://github.com/denoland/serde_v8 https://github.com/denoland/serde_v8), an expressive and ~maximally efficient bijection between v8 & rust. It has powered Deno's op-layer since 1.9 (https://deno.com/blog/v1.9#faster-calls-into-rust-with-serde_v8 https://deno.com/blog/v1.9#faster-calls-into-rust-with-serde...) and has enabled significant improvements in opcall overhead (close to 100x) whilst also simplifying said op-layer.
- apendleton 5y ago
  Honestly using serde for language interop is one of my favorite things about serde, whether it's "classical de/serialization" or not. I've recently had the very-pleasant experience of writing some code that needs to pass geospatial data back and forth between Python and Rust, and found that the geojson crate, even though it's nominally for JSON, actually works with other serde-compatible things, including (something I found kind of miraculous) Python objects, using the pythonize crate, which can walk them with serde visitors. So as long as I can get my data into a roughly-geojson-shaped thing on the python side, I can consume it on the Rust side, without having to ever actually produce json.
- josephg 5y ago
  Thats fantastic. I maintain the nodejs bindings for foundationdb. Foundationdb refuses to publish a wire protocol, so the bindings are implemented as native code wrapped by n_api. The code is a rat's nest of calls to methods like `napi_get_value_string_utf8` to parse javascript objects into C. (Eg [1]). As well as being difficult to read and write, I'm sure there's weird bugs lurking somewhere in all that boilerplate code. I've made my error checking a bit easier using macros, but that might have only made things worse. I'd much prefer all that code to just be in rust. serde-v8 looks way easier to use than all the goopy serialization nonsense I'm doing now. (Though I'd want a serde_napi variant instead of going straight to v8). [1] https://github.com/josephg/node-foundationdb/blob/c1165539e5d4aea4765d8f6f6444a684156a1a21/src/transaction.cpp https://github.com/josephg/node-foundationdb/blob/c1165539e5...
- the_mitsuhiko 5y ago
  While what I am using it for is not for performance, but I'm also using Serde as a bridge for my template engine and it's such a nice experience. I just wish it was possible to pass auxiliary information between serializers without having to resort to thread locals.
xet7 5y ago
Is there any Rust web framework that does not have problem with Slow-Loris ? https://github.com/SergioBenitez/Rocket/issues/1405 https://github.com/SergioBenitez/Rocket/issues/1405
- eptcyka 5y ago
  I think this should be fixed in hyper.
- ttymck 5y ago
  This is interesting, and news to me. Your comment suggests something about Rust imposes this problem on all web frameworks, is that true? If so could you share the root cause?
  - xet7 5y ago
    I'm just newbie in Rust. Does some web framework not use hyper? I'm trying to find web framework that does not have problem with Slow-Loris.
    - Ericson2314 5y ago
      I don't think anyone will want to avoid hyper, tbh, that's a lot of work to hand-roll instead. so you better advertise the issue there instead, and if they refuse to do anything about it then make a big stink downstream.
    - fwip 5y ago
      This doesn't solve your problem out of the box, but I think most reverse-proxies have configurations that can protect the services behind them from these sorts of attacks.
- pornel 5y ago
  Because of the way Rust's async works, you can put a timeout on anything (e.g. you don't need your http parser support timeouts, you can kill it at any await point). So it's only a matter of choosing timeout policy for your app. One person's DoS attack is another person's long polling API. Anyway, it has nothing to do with Serde, which doesn't have a network component. For Serde you'd typically buffer the input first, or use external framing (like line-oriented JSON).
the_gipsy 5y ago
I just came from rust to go and… what a disappointment. Missing fields default to some kind of default “zero value” - for any type, even full blown structs. You can’t tell the difference of a missing field or the field having the default value. So if a field has a validation of “must be greater than zero”, you can’t really give a proper error message. If user puts in “0” or omits the field, you always get a value of “0”.
- deleted 5y ago
  [deleted]
- rtkaratekid 5y ago
  I just went from two years of professional Rust development to Go, and I’m right there with you. Thankfully we have some terrible C++ that looks to be a good candidate for a rewrite.
- apendleton 5y ago
  If you want a field to be able to be not-set, you just need to specify that by making it an `Option<_>` type. So like, if the field is a number, make it an `Option<u64>` -- that way you can distinguish between not-set (`None`) and set to zero (`Some(0)`). Edit: whoops, misread which language it was you were frustrated with. Nevermind!
  - tylerhou 5y ago
    I know you’re giving helpful advice, but the author is talking (complaining) about Golang, which has no option typing (unless you want to use a pointer).
  - the_duke 5y ago
    You misunderstood, parent is complaining about Go, not about Rust. In Go you would have to use a pointer to represent Option<T>, which makes code really awkward. Go serialization really has quite a few issues apart from default initialization. Configuration by somewhat weird struct tag strings which are only evaluated/validated at runtime, de/serialization is all done via reflection (unless you want to use code generation), ...
pornel 5y ago
Because Serde is a common API for pretty much all serialisation formats supported in Rust, it's very easy to try multiple formats to choose the best size/speed you need. JSON too big? You can make it CBOR or Msgpack with a couple of lines. Want faster? Call bincode instead.
- roca 5y ago
  Yeah, the format independence is amazing. In Pernosco we use bincode for IPC payloads and when we want to log IPCs we just serialize the payloads to JSON with zero developer effort.
  - mwcampbell 5y ago
    Have you considered using a zero-copy serialization format like capnproto, Flat Buffers, or rkyv for binary IPC payloads? Or did you decide that getting that last increment of performance wasn't worth the trouble of using something less easy than serde?
    - roca 5y ago
      Yeah our IPC marshalling just isn't that relevant for performance AFAIK.
yazaddaruvala 5y ago
Frankly Serde’s continued success has made it basically necessary for most Rust development. When the Bazzar organically builds and loves a mini-Cathedral it’s probably time to upgrade it to standard. The increased complie times are the only real issue with Serde. Especially if upgrading this crate into std would allow for a way to reduce the compile time “penalty” of very common formats like JSON, etc.
elktea 5y ago
Serde is great. I wish it did XML as well but that's a trickier proposition
- bruuuh 5y ago
  I have found the quick-xml crate to work well.
- rascul 5y ago
  Maybe this would work https://crates.io/crates/serde-xml-rs https://crates.io/crates/serde-xml-rs
  - deleted 5y ago
    [deleted]
- Kinrany 5y ago
  Perhaps XML serializers could be derived from schema. That way you always know which tag name, attribute or child node is meaningful.
praveen9920 5y ago
I’m surprised to see that they don’t support protobuf yet which seem to be gaining. But otherwise seems like a good framework
- felipellrocha 5y ago
  I believe that is implemented by a separate crate that simply implements ‘(De)Serialiazable’
- wongarsu 5y ago
  Yes, that would be really nice. Protobuf doesn't really fit the model of serde though because of how closely coupled it is to its schema files. Still, there's two good implementations with the prost [0] and the protobuf [1] crates to choose from. 0: https://crates.io/crates/prost https://crates.io/crates/prost 1: https://crates.io/crates/protobuf https://crates.io/crates/protobuf
lmm 5y ago
Serde solves a lot of problems but using a dedicated macro for it feels like a real missed opportunity compared to using a generic macro for the part that needs a macro (treating structures generically), i.e. what frunk does, and then building serialization as just one of many possible use cases on top of that. Compare how this is done in the Scala ecosystem, where circe-generic is just one of many libraries that use shapeless to represent datatypes as generic records (and then serialization proceeds from there), and the same representation code is also used by e.g. doobie (a sort of lightweight ORM) to transform records to and from database rows. (Or, better still, building generic representation of structs as records into the language/compiler, where the whole ecosystem can depend on it and we don't have to bloat compile times generating it with a macro).
- lihaoyi 5y ago
  On the other hand, Scala's uPickle works almost identically as how Serde does, originally for performance, but the final visitor-based architecture ended up being almost identical. KotlinX serialization works basically the same way too; it definitely seems like a case of convergent evolution
  - lmm 5y ago
    Well "walk the object graph with a visitor" goes back at least to early (reflection-based) Java XML serializers; a generated-code version of the same thing is not a huge leap. I find it a frustratingly imperative way of approaching the problem: you can't think about what happens without thinking about control flow, because the visit can't be defunctionalized as a value. You may not want to physically reify the generic representation of the value - there would be a performance cost to that - but having it available to think about helps keep your code (and understanding) clear, in the same way that when doing a series of transformations of a large collection you probably want to be able to conceptualise "the collection in between step 1 and step 2" even if you don't want to actually instantiate that in memory at runtime.
waynesonfire 5y ago
But but apache arrow!
nicopappl 5y ago
Two things make serde great: (1) It's format agnostic (2) everyone in the community uses it. My experience with other language ecosystem (I'm thinking Haskell or Scala) is that serialization/deserialization libraries are limited to one format, with incompatible APIs. Even if the libraries in those ecosystem provide the same kind of guarantees, it still doesn't compare to serde. Typically any framework or library (cats, play, scalaz, akka) will be either inflexible with format supports or require glue code (provided by the user of the library or a 3rd party) to even work with your pet codec library, which end up also limited to a specific storage format. With serde, the assumption is just that things work, and it's fantastic. Example: I have my web server (written in rust) communicating messages to the frontend (in elm) with a websocket. For fun, I decided to test druid's wasm output when it came out (druid is a UI framework), see how feasible it would be to replace the frontend. And I could just pick up the type definitions from my rust server, add a websocket dependency that works on wasm and _it just worked_! I could then switch from json to msgpack and it worked the same. The strength of rust is the community. There is a strong sense of collaboration and under the rich diversity of frameworks and high level libraries, there is a culture of modular and shared code and contributions that makes every thing that more flexible and reliable.
- JoelMcCracken 5y ago
  Ah, cool. If you implement ToJSON from aeson for a type in Haskell, you also can do yaml serialization without any extra work (besides bringing in the YAML library, but the point is that the yaml library reuses ToJSON), but I don't know about alllll of those others that I see here: https://serde.rs/#data-formats https://serde.rs/#data-formats) Pretty cool!
dathinab 5y ago
I know it's nit picking but I would say it's a library not a framework. There are varying definitions of framework but what tend to be common is that they have a strong effect of "locking you into their ecosystem", which isn't really the case for serde.
jlrubin 5y ago
one thing that has kinda irked me about serde is that there's relatively limited support for bounding memory usage for types like a Vec<T> at deserialize time without a lot of manual work. It is doable (e.g. see SafePSBT deserializer here https://github.com/sapio-lang/sapio/blob/master/ctv_emulators/src/msgs.rs https://github.com/sapio-lang/sapio/blob/master/ctv_emulator...), just not particularly ergonomic.
- jlrubin 5y ago
  addendum: if you're like 'when could you want this?', imagine you are deserializing from a Reader which has an internal circular buffer or something from the network (bounded) and you want to limit a peers total resident memory (or something) on your server. Very difficult to do this with Serde!
cztomsik 5y ago
I'm glad serde exists but I don't like HOW it works: I think there should be a push for rust-lang to provide introspection. If you want to use serde, every crate needs to derive serde traits as well. This is not only bad architecture but it also prevents other libraries to get popular. For example, there are many "mini-serde" variants and I personally think, they are awesome (and surprisingly fast), yet you cannot practically use them because literally no crate supports anything else than serde. https://github.com/dtolnay/miniserde https://github.com/dtolnay/miniserde https://github.com/not-fl3/nanoserde https://github.com/not-fl3/nanoserde https://github.com/makepad/makepad/tree/master/render/microserde https://github.com/makepad/makepad/tree/master/render/micros...
- jcelerier 5y ago
  I wonder if the c++ approach of boost.pfr would be portable to rust ? It allows reflection on aggregates without needing to annotate anything: https://github.com/boostorg/pfr https://github.com/boostorg/pfr
  - cztomsik 5y ago
    I think currently, the only closest thing is bevy-reflect. Which is nice, but IMHO it should be in the language because nobody supports it ATM and to get there, it will take at least the same time it took for serde to get supported. https://crates.io/crates/bevy_reflect https://crates.io/crates/bevy_reflect

360 points by ur-whale 5y ago

deleted 5y ago

[deleted]

roca 5y ago

serde is one of the best things about Rust in practice. It is more convenient to use serde to serialize/deserialize to some standard format like JSON or YAML than it is to write your own half-baked format and serialization/deserialization code --- even for the simplest tasks --- so you just stop doing the latter. This is a fundamental shift, and very good for your software. "Convenience" here actually covers a lot of ground. The APIs are convenient. Boilerplate code is minimal (#[derive(Serialize)]). serde's attributes give you lots of control over the serialization/deserialization, while still being convenient to use. cargo makes importing serde into your project effortless. All of these are necessary to achieve that shift away from custom formats. Before Rust I used to write a lot of one-off half-baked formats. Pernosco uses Rust and serde and it uses no half-baked formats. Everything's JSON, or YAML if it needs to be more human-editable, or bincode if it needs to be fast and compact and not human-readable or extensible.

UK-Al05 5y ago
I think in most ecosystems it's easier to use a prebuilt serializer/deserializer than your own format.
- dexwiz 5y ago
  Yeah I don’t know why you would try to write your own textbased format. GSON and Jackson have been around for over a decade in Java, and equivalents in other languages. Unless you are in some perf critical application, a library is almost always better for this problem.
- roca 5y ago
  Fair enough. I have not found that to be true in C/C++, which is my main point of comparison. For very simple cases it's generally easier to write a few lines to a text file, maybe with some whitespace separators in the lines, than to use an "real" format. For JS and Python, JSON.stringify/parse and json.loads/dumps are a step up, but you still end up with an untyped mess with no schema validation, which makes them only halfway solutions to me. I'm a static typing guy at heart, sue me.
  - eptcyka 5y ago
    Yea, but with serde, you don't get an untyped mess. You either get the type you expected or you get an error.
    - vips7L 5y ago
      Same is true in any typed language.
      - spoiler 5y ago
        I don't think that's true. Or at least we have different ideas about what's being discussed... Let's take JSON or YAML for example: Rust's closest siblings C and C++ are typed, and you get a pretty much untyped messes when working with serialisation or deserialisation. You have to manually inspect every node or write manual ”NodeType" to struct conversation. They allow unwrapping to primitives at best (eg via template specialisation). In Haskell it's a bit more awkward, but you can kinda pull something similar off in terms of ergonomics. I've not worked with JSON in Haskell that much that I need to look for it, but I don't remember there being something as ergonomic as serde. Typescript doesn't support this type of validation either out of the box (there might be tools that add validation). In Elixir there's the Poison library, or it could be done with Kernel.struct/2 (not 100% sure this will work though). Of the "mainstream" languages Go is pretty much the only one that has as good ergonomics as Rust in that it supports it pretty much natively (via struct tags) --- This list excludes codegen tools which can generate (de)serialisers from an external schema (a la capnp, grpc, jsonschema, etc).
        UK-Al05 5y ago
        Most typed languages it's like this var account = JsonConvert.DeserializeObject<Account>(string/stream) That's it. No inspection or looping through trees. You can often get it to create objects via constructor for validation. It's been like this for 15 years...
        deleted 5y ago
- pornel 5y ago
  The trick with Serde is that it decodes straight into a native Rust struct. You get most of validation for free (it also nicely takes advantage of Rust enums with data), and struct access is maximally fast. A generic JSON decoder would give you a dynamic structure that can contain anything, and then you'd have to pick it apart.
  - UK-Al05 5y ago
    Most deserializers in typed languages allow you to deserialize straight into structs or typed objects... Serde just doesn't use reflection to do it.
    - Taywee 5y ago
      C++ isn't super easy about it, though. The best C++ json library (in my opinion), nlohmann's json, still requires you to define to_json and from_json for your types to use them. It's not too bad as a one-off, but when you have dozens of types, it gets really tedious compared to Serde, and managing std::variant with it is way more annoying than Serde with Rust enums.
    - Jweb_Guru 5y ago
      APIs like GSON and Jackson have a lot more footguns, because (1) they attempt to (de)serialize types whether or not they were ever intended to be (de)serialized, (2) they are missing information about important constraints due to losing information at runtime (the excuse usually used here is that "they don't do validation" when in reality they are violating expectations of package modularity and encapsulation in a pretty blunt way). For example, they will happily ignore non-null constraints on your fields. In practice, this is a major annoyance and ends up making you very paranoid about (de)serializing any structure you don't have full control over. Performance of a lot of the reflection-based parsing APIs also leaves something to be desired, which means that projects with more stringent performance requirements often have to resort to stuff like Avro. This still happens with serde, but much more rarely--besides having more compile-time information at its disposal and needing to allocate less, it also provides relatively straightforward hooks to achieve things like zero-copy deserialization for strings where the whole buffer is available at once.
- Jweb_Guru 5y ago
  Nah, not like it is in Rust. For the vast majority of projects, serde makes parsing / serializing something they don't even have to think about, without having to sacrifice type safety to get there. Until you experience that in a large project with a lot of moving parts and developers, it's hard to appreciate just how many developer cycles you were wasting before on stuff that can be totally automated. It's format-agnostic, it's really fast (not the absolute fastest, but fast enough for most situations), it's insanely customizable for even the weirdest use cases, it's absolutely everywhere, it incorporates proper error handling, and adding it to your own types is trivial. Is there any actual reason something like serde couldn't exist in other languages? None that I know of, but it doesn't, at least not when you consider just how pervasive serde is in the ecosystem (almost every library that stores things you want to serialize will have it). I really hope every language can standardize on something similar; it just makes development so much nicer and faster when you don't have to worry about this stuff.
  - fiddlerwoaroof 5y ago
    These sorts of things are really convenient, but the idea of tightly binding your type definitions to serialization formats always makes be a little nervous: I haven’t used Rust enough in practice to have an opinion here, but I’m wary of a solution in this space that makes it too easy to ignore the forward/backwards compatibility of your external format.
    - Jweb_Guru 5y ago
      Yeah, for sure that is something to be cautious of. For cases where I expect backwards compatibility to be a concern, I will usually make the top-level type of the message or format an enum with each variant being a version (initially with just a single version, V1 or V0). That way, if we need to make backwards-incompatible format changes, we can create a new variant, and potentially even automatically migrate between formats (I've done this a few times). Then all we have to do is call a function to migrate to the latest format, and we can use whatever that is for all the actual program code (generally it will be a type alias for the version structure of the same name). Of course, if this gets too complex or you start to have non-treelike relationships between values, you still want to fall back on a more traditional database schema, but this kind of versioning can get you a long way. That said, it's incredibly refreshing to be able to focus on these semantic and migration concerns without having to worry about how to deal with parsing them later. serde doesn't solve enforcing backwards compatibility for you, but it sure makes doing the right thing a lot easier. You never have to worry about some code accidentally forgetting to check the version field, people carelessly mixing and matching messages with different versions, or fields that "should" never be set but are there for backwards compatibility. That's worth a lot to me!
- thedracle 5y ago
  At least in languages with similar native performance as Rust (C/C++), in my personal experience, serialization has often been difficult and manual, and most libraries have fallen short of my expectations, or had serious performance, security, or toolchain issues, to the point I usually hand-serialize myself.
danobi 5y ago
Completely agree. FWIW, I think procedural macros make a huge difference for ergonomics. For example, a "similar" library in C++ (libcereal) is still a PITA to use and debug.
vvanders 5y ago
It's also blazing-fast. I've done some benchmarking on JSON -> in-memory borrow structs and it was pretty darn impressive while being fairly ergonomic to use compared to a streaming parser.
the_duke 5y ago
The only downside is compile time bloat. Serde generates heaps and heaps of generic code. This all gets optimized away to be very efficient, but only once it reaches LLVM. Ever tried working on a crate with hundreds or thousands of de/serializable types? Compile times shoot through the roof really quickly, and serde is often the culprit. The maintainer of serde also created `miniserde` [1] to tackle this problem, which uses dynamic dispatch instead of generics and can have 4x compile time improvements. Due to Rusts lack of orphan instances you really depend on a pervasive standard for serialization, though, so the ecosystem is pretty locked in to serde. [1] https://github.com/dtolnay/miniserde https://github.com/dtolnay/miniserde
- munro 5y ago
  I've been using Rust for just a little bit, building applications/services/scrapers/databases/forking libraries, but I haven't noticed these issues--the biggest compile time issue I had was including RocksDB. I'm trying to imagine what you're building. :} 99% of the time I'm just doing incremental debug builds, and haven't really minded the compile times when doing release builds. Overall it everything has been fast, to the point where I want to start scripting with Rust.
- unanswered 5y ago
  Studies have shown that more than 87% of complaints about rust compile time are because people are enabling serde everywhere and for everything.
  - johnfn 5y ago
    These concerns are valid. Data serialization/deserialization is an extremely standard thing to need to be able to do. It's not the user's fault if they use the obvious tool for an obvious job and suddenly have bloated compile times.
    - vlovich123 5y ago
      Seems like there’s opportunity for tighter integration with the compiler where serde moves into the compiler or there are acceleration primitives rather than doing the AST macros thing. This sounds similar to the problems of c++ template bloat.
curun1r 5y ago
The other benefit is that almost all of the Rust ecosystem supports serde, often by way of a feature. So using serde can mean being able to embed a type from a crate in your serializable type or being able to pass your own types to crate functions/methods that serialize/deserialize. It all ends up being interoperable.
thedracle 5y ago
The extendibility of it is impressive, as well as the list of formats supported by the community. The ability to output JSON for debugging next to tight-concise binary formats, is really appealing. I've dealt with a number of C++ based serialization libraries, and they always have serious downsides to the point I often end up hand serializing structures with one-off half-baked formats just like you, which is error prone and laborious.

paulgb 5y ago

To me, Serde is one of the killer features of Rust. Once you realize how much of programming is (for better or worse) shuttling data between different representations, it's hard for me to program in a language where I don't have a tool like Serde. A big part of its flexibility is how modular it is. Most common Rust libraries support serde serialization (at least as an optional feature), so if you use crates that do, you can plug any backend in and serialize those data structures to it. It doesn't even have to be string-based; I've been using Serde to store arbitrary data structures as objects on Google Cloud Firestore.

AaronO 5y ago

Whilst not classical de/serialization I wrote serde_v8 (https://github.com/denoland/serde_v8 https://github.com/denoland/serde_v8), an expressive and ~maximally efficient bijection between v8 & rust. It has powered Deno's op-layer since 1.9 (https://deno.com/blog/v1.9#faster-calls-into-rust-with-serde_v8 https://deno.com/blog/v1.9#faster-calls-into-rust-with-serde...) and has enabled significant improvements in opcall overhead (close to 100x) whilst also simplifying said op-layer.

apendleton 5y ago
Honestly using serde for language interop is one of my favorite things about serde, whether it's "classical de/serialization" or not. I've recently had the very-pleasant experience of writing some code that needs to pass geospatial data back and forth between Python and Rust, and found that the geojson crate, even though it's nominally for JSON, actually works with other serde-compatible things, including (something I found kind of miraculous) Python objects, using the pythonize crate, which can walk them with serde visitors. So as long as I can get my data into a roughly-geojson-shaped thing on the python side, I can consume it on the Rust side, without having to ever actually produce json.
josephg 5y ago
Thats fantastic. I maintain the nodejs bindings for foundationdb. Foundationdb refuses to publish a wire protocol, so the bindings are implemented as native code wrapped by n_api. The code is a rat's nest of calls to methods like `napi_get_value_string_utf8` to parse javascript objects into C. (Eg [1]). As well as being difficult to read and write, I'm sure there's weird bugs lurking somewhere in all that boilerplate code. I've made my error checking a bit easier using macros, but that might have only made things worse. I'd much prefer all that code to just be in rust. serde-v8 looks way easier to use than all the goopy serialization nonsense I'm doing now. (Though I'd want a serde_napi variant instead of going straight to v8). [1] https://github.com/josephg/node-foundationdb/blob/c1165539e5d4aea4765d8f6f6444a684156a1a21/src/transaction.cpp https://github.com/josephg/node-foundationdb/blob/c1165539e5...
the_mitsuhiko 5y ago
While what I am using it for is not for performance, but I'm also using Serde as a bridge for my template engine and it's such a nice experience. I just wish it was possible to pass auxiliary information between serializers without having to resort to thread locals.

xet7 5y ago

Is there any Rust web framework that does not have problem with Slow-Loris ? https://github.com/SergioBenitez/Rocket/issues/1405 https://github.com/SergioBenitez/Rocket/issues/1405

eptcyka 5y ago
I think this should be fixed in hyper.
ttymck 5y ago
This is interesting, and news to me. Your comment suggests something about Rust imposes this problem on all web frameworks, is that true? If so could you share the root cause?
- xet7 5y ago
  I'm just newbie in Rust. Does some web framework not use hyper? I'm trying to find web framework that does not have problem with Slow-Loris.
  - Ericson2314 5y ago
    I don't think anyone will want to avoid hyper, tbh, that's a lot of work to hand-roll instead. so you better advertise the issue there instead, and if they refuse to do anything about it then make a big stink downstream.
  - fwip 5y ago
    This doesn't solve your problem out of the box, but I think most reverse-proxies have configurations that can protect the services behind them from these sorts of attacks.
pornel 5y ago
Because of the way Rust's async works, you can put a timeout on anything (e.g. you don't need your http parser support timeouts, you can kill it at any await point). So it's only a matter of choosing timeout policy for your app. One person's DoS attack is another person's long polling API. Anyway, it has nothing to do with Serde, which doesn't have a network component. For Serde you'd typically buffer the input first, or use external framing (like line-oriented JSON).

the_gipsy 5y ago

I just came from rust to go and… what a disappointment. Missing fields default to some kind of default “zero value” - for any type, even full blown structs. You can’t tell the difference of a missing field or the field having the default value. So if a field has a validation of “must be greater than zero”, you can’t really give a proper error message. If user puts in “0” or omits the field, you always get a value of “0”.

deleted 5y ago
[deleted]
rtkaratekid 5y ago
I just went from two years of professional Rust development to Go, and I’m right there with you. Thankfully we have some terrible C++ that looks to be a good candidate for a rewrite.
apendleton 5y ago
If you want a field to be able to be not-set, you just need to specify that by making it an `Option<_>` type. So like, if the field is a number, make it an `Option<u64>` -- that way you can distinguish between not-set (`None`) and set to zero (`Some(0)`). Edit: whoops, misread which language it was you were frustrated with. Nevermind!
- tylerhou 5y ago
  I know you’re giving helpful advice, but the author is talking (complaining) about Golang, which has no option typing (unless you want to use a pointer).
- the_duke 5y ago
  You misunderstood, parent is complaining about Go, not about Rust. In Go you would have to use a pointer to represent Option<T>, which makes code really awkward. Go serialization really has quite a few issues apart from default initialization. Configuration by somewhat weird struct tag strings which are only evaluated/validated at runtime, de/serialization is all done via reflection (unless you want to use code generation), ...

pornel 5y ago

Because Serde is a common API for pretty much all serialisation formats supported in Rust, it's very easy to try multiple formats to choose the best size/speed you need. JSON too big? You can make it CBOR or Msgpack with a couple of lines. Want faster? Call bincode instead.

roca 5y ago
Yeah, the format independence is amazing. In Pernosco we use bincode for IPC payloads and when we want to log IPCs we just serialize the payloads to JSON with zero developer effort.
- mwcampbell 5y ago
  Have you considered using a zero-copy serialization format like capnproto, Flat Buffers, or rkyv for binary IPC payloads? Or did you decide that getting that last increment of performance wasn't worth the trouble of using something less easy than serde?
  - roca 5y ago
    Yeah our IPC marshalling just isn't that relevant for performance AFAIK.

yazaddaruvala 5y ago

Frankly Serde’s continued success has made it basically necessary for most Rust development. When the Bazzar organically builds and loves a mini-Cathedral it’s probably time to upgrade it to standard. The increased complie times are the only real issue with Serde. Especially if upgrading this crate into std would allow for a way to reduce the compile time “penalty” of very common formats like JSON, etc.

elktea 5y ago

Serde is great. I wish it did XML as well but that's a trickier proposition

bruuuh 5y ago
I have found the quick-xml crate to work well.
rascul 5y ago
Maybe this would work https://crates.io/crates/serde-xml-rs https://crates.io/crates/serde-xml-rs
- deleted 5y ago
  [deleted]
Kinrany 5y ago
Perhaps XML serializers could be derived from schema. That way you always know which tag name, attribute or child node is meaningful.

praveen9920 5y ago

I’m surprised to see that they don’t support protobuf yet which seem to be gaining. But otherwise seems like a good framework

felipellrocha 5y ago
I believe that is implemented by a separate crate that simply implements ‘(De)Serialiazable’
wongarsu 5y ago
Yes, that would be really nice. Protobuf doesn't really fit the model of serde though because of how closely coupled it is to its schema files. Still, there's two good implementations with the prost [0] and the protobuf [1] crates to choose from. 0: https://crates.io/crates/prost https://crates.io/crates/prost 1: https://crates.io/crates/protobuf https://crates.io/crates/protobuf

lmm 5y ago

Serde solves a lot of problems but using a dedicated macro for it feels like a real missed opportunity compared to using a generic macro for the part that needs a macro (treating structures generically), i.e. what frunk does, and then building serialization as just one of many possible use cases on top of that. Compare how this is done in the Scala ecosystem, where circe-generic is just one of many libraries that use shapeless to represent datatypes as generic records (and then serialization proceeds from there), and the same representation code is also used by e.g. doobie (a sort of lightweight ORM) to transform records to and from database rows. (Or, better still, building generic representation of structs as records into the language/compiler, where the whole ecosystem can depend on it and we don't have to bloat compile times generating it with a macro).

lihaoyi 5y ago
On the other hand, Scala's uPickle works almost identically as how Serde does, originally for performance, but the final visitor-based architecture ended up being almost identical. KotlinX serialization works basically the same way too; it definitely seems like a case of convergent evolution
- lmm 5y ago
  Well "walk the object graph with a visitor" goes back at least to early (reflection-based) Java XML serializers; a generated-code version of the same thing is not a huge leap. I find it a frustratingly imperative way of approaching the problem: you can't think about what happens without thinking about control flow, because the visit can't be defunctionalized as a value. You may not want to physically reify the generic representation of the value - there would be a performance cost to that - but having it available to think about helps keep your code (and understanding) clear, in the same way that when doing a series of transformations of a large collection you probably want to be able to conceptualise "the collection in between step 1 and step 2" even if you don't want to actually instantiate that in memory at runtime.

waynesonfire 5y ago

But but apache arrow!

nicopappl 5y ago

Two things make serde great: (1) It's format agnostic (2) everyone in the community uses it. My experience with other language ecosystem (I'm thinking Haskell or Scala) is that serialization/deserialization libraries are limited to one format, with incompatible APIs. Even if the libraries in those ecosystem provide the same kind of guarantees, it still doesn't compare to serde. Typically any framework or library (cats, play, scalaz, akka) will be either inflexible with format supports or require glue code (provided by the user of the library or a 3rd party) to even work with your pet codec library, which end up also limited to a specific storage format. With serde, the assumption is just that things work, and it's fantastic. Example: I have my web server (written in rust) communicating messages to the frontend (in elm) with a websocket. For fun, I decided to test druid's wasm output when it came out (druid is a UI framework), see how feasible it would be to replace the frontend. And I could just pick up the type definitions from my rust server, add a websocket dependency that works on wasm and _it just worked_! I could then switch from json to msgpack and it worked the same. The strength of rust is the community. There is a strong sense of collaboration and under the rich diversity of frameworks and high level libraries, there is a culture of modular and shared code and contributions that makes every thing that more flexible and reliable.

JoelMcCracken 5y ago
Ah, cool. If you implement ToJSON from aeson for a type in Haskell, you also can do yaml serialization without any extra work (besides bringing in the YAML library, but the point is that the yaml library reuses ToJSON), but I don't know about alllll of those others that I see here: https://serde.rs/#data-formats https://serde.rs/#data-formats) Pretty cool!

dathinab 5y ago

I know it's nit picking but I would say it's a library not a framework. There are varying definitions of framework but what tend to be common is that they have a strong effect of "locking you into their ecosystem", which isn't really the case for serde.

jlrubin 5y ago

one thing that has kinda irked me about serde is that there's relatively limited support for bounding memory usage for types like a Vec<T> at deserialize time without a lot of manual work. It is doable (e.g. see SafePSBT deserializer here https://github.com/sapio-lang/sapio/blob/master/ctv_emulators/src/msgs.rs https://github.com/sapio-lang/sapio/blob/master/ctv_emulator...), just not particularly ergonomic.

jlrubin 5y ago
addendum: if you're like 'when could you want this?', imagine you are deserializing from a Reader which has an internal circular buffer or something from the network (bounded) and you want to limit a peers total resident memory (or something) on your server. Very difficult to do this with Serde!

cztomsik 5y ago

I'm glad serde exists but I don't like HOW it works: I think there should be a push for rust-lang to provide introspection. If you want to use serde, every crate needs to derive serde traits as well. This is not only bad architecture but it also prevents other libraries to get popular. For example, there are many "mini-serde" variants and I personally think, they are awesome (and surprisingly fast), yet you cannot practically use them because literally no crate supports anything else than serde. https://github.com/dtolnay/miniserde https://github.com/dtolnay/miniserde https://github.com/not-fl3/nanoserde https://github.com/not-fl3/nanoserde https://github.com/makepad/makepad/tree/master/render/microserde https://github.com/makepad/makepad/tree/master/render/micros...

jcelerier 5y ago
I wonder if the c++ approach of boost.pfr would be portable to rust ? It allows reflection on aggregates without needing to annotate anything: https://github.com/boostorg/pfr https://github.com/boostorg/pfr
- cztomsik 5y ago
  I think currently, the only closest thing is bevy-reflect. Which is nice, but IMHO it should be in the language because nobody supports it ATM and to get there, it will take at least the same time it took for serde to get supported. https://crates.io/crates/bevy_reflect https://crates.io/crates/bevy_reflect