2 ms·
> What about serialization makes it so intractable? Could some kind of language support make it better? I think the main problem with serialization is that it
by jcranmer 6y ago
> What about serialization makes it so intractable? Could some kind of language support make it better?
I think the main problem with serialization is that it is used to mean several different things which at first glance could all be handled by one system but experience instead shows are just too different to be handled in that manner.
One kind of serialization is handling something that is mostly "bit blitting." I've got a struct in memory, it needs to be copied to disk to be read back into memory. Maybe you need to do some normalization for things like endianness or charset conversion.
A different kind of serialization is when you're trying to read/write a standard-specified binary layout. This sounds like it's the same thing as before, but the difference here is that the structs are not readily blittable. You might have cases like x86's definitions, where fields are splayed out in different locations (zmm0 is spread out across three locations in the XSAVE struct). Or you might have the ... interesting features of ASN.1's DER format. Handling these kinds of cases requires at the same time as a more blittable struct requires adding lots of customization points.
A third situation is where you're reading in something like JSON or XML: now you have symbolic field names that can occur in any order, and your serialization code needs to account for unexpected orders. Customization points really kick in here, because now you also usually want to have default values and optional values and maybe conditionally required (this field is required only if some other field has a particular value).
On top of these, you also start to conflate orthogonal issues of validation and conversion. Are the structs in memory meant to be close to on-disk representations and clients responsible for handling data representation issues (consider the way the object crate handles endianness in ELF files)? Where, and when, does validation of parsing data happen? How much of this is automatable from the serialization framework itself (e.g., the value must have values in such-and-such range)? What about supporting multiple versions of save data--where is the ground truth about how to upconvert (or maybe even downconvert!) old versions to the current library version?
This is a situation where you can build something pretty tight for a single use case, but feature scope quickly comes in when it comes to extending it for multiple use cases, to the point that it's too unwieldy if you're not using any advanced features and the advanced features are too advanced to use.
As for what language support makes it better, the obvious answer is reflection. Having written my own auto-derive-serialization crate, what I mostly need is a) given this struct, give me a list of fields, their names, types, and attributes, b) let me do specialization on types, and c) let me turn that into writing some pretty unconstrained code. Rust's current macro system mostly falls down on the type system, although the need to include a hefty parsing quote to reparse everything to get the field lists is pretty poor (not to mention actually handling the corner cases in Rust's type system is challenging, although serialization code is likely to screw up here anyways). There's probably room to specify a MVP to make implementing serialization easier that wouldn't need to go through syn.