10 ms·
serde is one of the best things about Rust in practice. It is more convenient to use serde to serialize/deserialize to some standard format like JSON or YAML t
by roca 5y ago
serde is one of the best things about Rust in practice.
It is more convenient to use serde to serialize/deserialize to some standard format like JSON or YAML than it is to write your own half-baked format and serialization/deserialization code --- even for the simplest tasks --- so you just stop doing the latter. This is a fundamental shift, and very good for your software.
"Convenience" here actually covers a lot of ground. The APIs are convenient. Boilerplate code is minimal (#[derive(Serialize)]). serde's attributes give you lots of control over the serialization/deserialization, while still being convenient to use. cargo makes importing serde into your project effortless. All of these are necessary to achieve that shift away from custom formats.
Before Rust I used to write a lot of one-off half-baked formats. Pernosco uses Rust and serde and it uses no half-baked formats. Everything's JSON, or YAML if it needs to be more human-editable, or bincode if it needs to be fast and compact and not human-readable or extensible.
- UK-Al05 5y agoI think in most ecosystems it's easier to use a prebuilt serializer/deserializer than your own format.
- dexwiz 5y agoYeah I don’t know why you would try to write your own textbased format. GSON and Jackson have been around for over a decade in Java, and equivalents in other languages. Unless you are in some perf critical application, a library is almost always better for this problem.
- roca 5y agoFair enough. I have not found that to be true in C/C++, which is my main point of comparison. For very simple cases it's generally easier to write a few lines to a text file, maybe with some whitespace separators in the lines, than to use an "real" format. For JS and Python, JSON.stringify/parse and json.loads/dumps are a step up, but you still end up with an untyped mess with no schema validation, which makes them only halfway solutions to me. I'm a static typing guy at heart, sue me.
- eptcyka 5y agoYea, but with serde, you don't get an untyped mess. You either get the type you expected or you get an error.
- vips7L 5y agoSame is true in any typed language.
- spoiler 5y agoI don't think that's true. Or at least we have different ideas about what's being discussed... Let's take JSON or YAML for example: Rust's closest siblings C and C++ are typed, and you get a pretty much untyped messes when working with serialisation or deserialisation. You have to manually inspect every node or write manual ”NodeType" to struct conversation. They allow unwrapping to primitives at best (eg via template specialisation). In Haskell it's a bit more awkward, but you can kinda pull something similar off in terms of ergonomics. I've not worked with JSON in Haskell that much that I need to look for it, but I don't remember there being something as ergonomic as serde. Typescript doesn't support this type of validation either out of the box (there might be tools that add validation). In Elixir there's the Poison library, or it could be done with Kernel.struct/2 (not 100% sure this will work though). Of the "mainstream" languages Go is pretty much the only one that has as good ergonomics as Rust in that it supports it pretty much natively (via struct tags) --- This list excludes codegen tools which can generate (de)serialisers from an external schema (a la capnp, grpc, jsonschema, etc).
- UK-Al05 5y agoMost typed languages it's like this var account = JsonConvert.DeserializeObject<Account>(string/stream) That's it. No inspection or looping through trees. You can often get it to create objects via constructor for validation. It's been like this for 15 years...
- deleted 5y ago
- pornel 5y agoThe trick with Serde is that it decodes straight into a native Rust struct. You get most of validation for free (it also nicely takes advantage of Rust enums with data), and struct access is maximally fast. A generic JSON decoder would give you a dynamic structure that can contain anything, and then you'd have to pick it apart.
- UK-Al05 5y agoMost deserializers in typed languages allow you to deserialize straight into structs or typed objects... Serde just doesn't use reflection to do it.
- Taywee 5y agoC++ isn't super easy about it, though. The best C++ json library (in my opinion), nlohmann's json, still requires you to define to_json and from_json for your types to use them. It's not too bad as a one-off, but when you have dozens of types, it gets really tedious compared to Serde, and managing std::variant with it is way more annoying than Serde with Rust enums.
- Jweb_Guru 5y agoAPIs like GSON and Jackson have a lot more footguns, because (1) they attempt to (de)serialize types whether or not they were ever intended to be (de)serialized, (2) they are missing information about important constraints due to losing information at runtime (the excuse usually used here is that "they don't do validation" when in reality they are violating expectations of package modularity and encapsulation in a pretty blunt way). For example, they will happily ignore non-null constraints on your fields. In practice, this is a major annoyance and ends up making you very paranoid about (de)serializing any structure you don't have full control over. Performance of a lot of the reflection-based parsing APIs also leaves something to be desired, which means that projects with more stringent performance requirements often have to resort to stuff like Avro. This still happens with serde, but much more rarely--besides having more compile-time information at its disposal and needing to allocate less, it also provides relatively straightforward hooks to achieve things like zero-copy deserialization for strings where the whole buffer is available at once.
- Jweb_Guru 5y agoNah, not like it is in Rust. For the vast majority of projects, serde makes parsing / serializing something they don't even have to think about, without having to sacrifice type safety to get there. Until you experience that in a large project with a lot of moving parts and developers, it's hard to appreciate just how many developer cycles you were wasting before on stuff that can be totally automated. It's format-agnostic, it's really fast (not the absolute fastest, but fast enough for most situations), it's insanely customizable for even the weirdest use cases, it's absolutely everywhere, it incorporates proper error handling, and adding it to your own types is trivial. Is there any actual reason something like serde couldn't exist in other languages? None that I know of, but it doesn't, at least not when you consider just how pervasive serde is in the ecosystem (almost every library that stores things you want to serialize will have it). I really hope every language can standardize on something similar; it just makes development so much nicer and faster when you don't have to worry about this stuff.
- fiddlerwoaroof 5y agoThese sorts of things are really convenient, but the idea of tightly binding your type definitions to serialization formats always makes be a little nervous: I haven’t used Rust enough in practice to have an opinion here, but I’m wary of a solution in this space that makes it too easy to ignore the forward/backwards compatibility of your external format.
- Jweb_Guru 5y agoYeah, for sure that is something to be cautious of. For cases where I expect backwards compatibility to be a concern, I will usually make the top-level type of the message or format an enum with each variant being a version (initially with just a single version, V1 or V0). That way, if we need to make backwards-incompatible format changes, we can create a new variant, and potentially even automatically migrate between formats (I've done this a few times). Then all we have to do is call a function to migrate to the latest format, and we can use whatever that is for all the actual program code (generally it will be a type alias for the version structure of the same name). Of course, if this gets too complex or you start to have non-treelike relationships between values, you still want to fall back on a more traditional database schema, but this kind of versioning can get you a long way. That said, it's incredibly refreshing to be able to focus on these semantic and migration concerns without having to worry about how to deal with parsing them later. serde doesn't solve enforcing backwards compatibility for you, but it sure makes doing the right thing a lot easier. You never have to worry about some code accidentally forgetting to check the version field, people carelessly mixing and matching messages with different versions, or fields that "should" never be set but are there for backwards compatibility. That's worth a lot to me!
- thedracle 5y agoAt least in languages with similar native performance as Rust (C/C++), in my personal experience, serialization has often been difficult and manual, and most libraries have fallen short of my expectations, or had serious performance, security, or toolchain issues, to the point I usually hand-serialize myself.
- danobi 5y agoCompletely agree. FWIW, I think procedural macros make a huge difference for ergonomics. For example, a "similar" library in C++ (libcereal) is still a PITA to use and debug.
- vvanders 5y agoIt's also blazing-fast. I've done some benchmarking on JSON -> in-memory borrow structs and it was pretty darn impressive while being fairly ergonomic to use compared to a streaming parser.
- the_duke 5y agoThe only downside is compile time bloat. Serde generates heaps and heaps of generic code. This all gets optimized away to be very efficient, but only once it reaches LLVM. Ever tried working on a crate with hundreds or thousands of de/serializable types? Compile times shoot through the roof really quickly, and serde is often the culprit. The maintainer of serde also created `miniserde` [1] to tackle this problem, which uses dynamic dispatch instead of generics and can have 4x compile time improvements. Due to Rusts lack of orphan instances you really depend on a pervasive standard for serialization, though, so the ecosystem is pretty locked in to serde. [1] https://github.com/dtolnay/miniserde https://github.com/dtolnay/miniserde
- munro 5y agoI've been using Rust for just a little bit, building applications/services/scrapers/databases/forking libraries, but I haven't noticed these issues--the biggest compile time issue I had was including RocksDB. I'm trying to imagine what you're building. :} 99% of the time I'm just doing incremental debug builds, and haven't really minded the compile times when doing release builds. Overall it everything has been fast, to the point where I want to start scripting with Rust.
- unanswered 5y agoStudies have shown that more than 87% of complaints about rust compile time are because people are enabling serde everywhere and for everything.
- johnfn 5y agoThese concerns are valid. Data serialization/deserialization is an extremely standard thing to need to be able to do. It's not the user's fault if they use the obvious tool for an obvious job and suddenly have bloated compile times.
- vlovich123 5y agoSeems like there’s opportunity for tighter integration with the compiler where serde moves into the compiler or there are acceleration primitives rather than doing the AST macros thing. This sounds similar to the problems of c++ template bloat.
- curun1r 5y agoThe other benefit is that almost all of the Rust ecosystem supports serde, often by way of a feature. So using serde can mean being able to embed a type from a crate in your serializable type or being able to pass your own types to crate functions/methods that serialize/deserialize. It all ends up being interoperable.
- thedracle 5y agoThe extendibility of it is impressive, as well as the list of formats supported by the community. The ability to output JSON for debugging next to tight-concise binary formats, is really appealing. I've dealt with a number of C++ based serialization libraries, and they always have serious downsides to the point I often end up hand serializing structures with one-off half-baked formats just like you, which is error prone and laborious.