4 ms·

> What about serialization makes it so intractable? Could some kind of language support make it better? I've thought about serialization a lot over the years,

> What about serialization makes it so intractable? Could some kind of language support make it better? I've thought about serialization a lot over the years, and I think a big part of the problem is that we try to map our programming language data types directly to serialized types (e.g. mapping JSON objects to Rust structs). That can be useful, but I think often the serialized representation is sufficiently different from the desired data type that it's not good enough. So we end up with one set of data types for serialization/deserialization, and a bunch of code that maps between those and the desired representation. I wonder if a system similar to Scheme's `syntax-rules` (or Rust's `macro_rules!`) for mapping the serialized data into the program's data types (and vice-versa) would be a nice mechanism.

zackbrown 6y ago
> I think a big part of the problem is that we try to map our programming language data types directly to serialized types Much of the time, in-program data is relational with a possibly-cyclic graph describing relationships, ownership, etc. Systems that work with RDBMSes do a fine job of de/serializing data for relational stores, and ORMs are a powerful and delightful way to do this. The problem with JSON and XML for these kinds of data is these formats are inherently non-relational. They are simple trees of data, not arbitrary graphs. In practice, e.g. in every full-stack web codebase I've ever seen, JSON blobs are hacked together e.g. by doing a makeshift front-end "join by ID" when needed — or adjusting a back-end method to perform the join and return — wait for it — a flattened JSON blob. It seems like we need a "JSON" or "XML" where relationships are a first-class citizen. To my knowledge, the closest we've come to this is when the data broker and format are tightly coupled, e.g. with SOAP or GraphQL. Can this be achieved strictly declaratively, in a format that's better than a SQL data dump? Maybe! > I wonder if a system similar to Scheme's `syntax-rules` (or Rust's `macro_rules!`) for mapping the serialized data into the program's data types (and vice-versa) would be a nice mechanism. Does `serde`[1] look like it's close to your imagined mark? I haven't tried it personally, but it looks akin to Java's `Serializable` or C#'s `ISerializable`— all reasonably elegant ways of handling the messy binding logic of serialization over arbitrary formats. [1] https://serde.rs/ https://serde.rs/
- wtetzner 6y ago
  > and ORMs are a powerful and delightful way to do this I think we'll end up disagreeing there. However, I think this type of model could maybe work better for "static" serialization/deserialization of file formats. But even so, I think the biggest part that's missing is a convenient way to describe the mapping between the structure of the serialized data and the internal representation. ORMs typically give you a single method of doing that mapping, which can work when you have control over the serialized representation. But it doesn't work very well when you're serializing to an existing file format, for example. > Does `serde`[1] look like it's close to your imagined mark? No, serde is exactly what I was talking about in terms of the serialized data matching your internal data structures. I think it's a necessary piece, but I think the part that's missing is a nice way to map that data to the representation your program want's to work on.

zackbrown 6y ago

> I think a big part of the problem is that we try to map our programming language data types directly to serialized types Much of the time, in-program data is relational with a possibly-cyclic graph describing relationships, ownership, etc. Systems that work with RDBMSes do a fine job of de/serializing data for relational stores, and ORMs are a powerful and delightful way to do this. The problem with JSON and XML for these kinds of data is these formats are inherently non-relational. They are simple trees of data, not arbitrary graphs. In practice, e.g. in every full-stack web codebase I've ever seen, JSON blobs are hacked together e.g. by doing a makeshift front-end "join by ID" when needed — or adjusting a back-end method to perform the join and return — wait for it — a flattened JSON blob. It seems like we need a "JSON" or "XML" where relationships are a first-class citizen. To my knowledge, the closest we've come to this is when the data broker and format are tightly coupled, e.g. with SOAP or GraphQL. Can this be achieved strictly declaratively, in a format that's better than a SQL data dump? Maybe! > I wonder if a system similar to Scheme's `syntax-rules` (or Rust's `macro_rules!`) for mapping the serialized data into the program's data types (and vice-versa) would be a nice mechanism. Does `serde`[1] look like it's close to your imagined mark? I haven't tried it personally, but it looks akin to Java's `Serializable` or C#'s `ISerializable`— all reasonably elegant ways of handling the messy binding logic of serialization over arbitrary formats. [1] https://serde.rs/ https://serde.rs/

wtetzner 6y ago
> and ORMs are a powerful and delightful way to do this I think we'll end up disagreeing there. However, I think this type of model could maybe work better for "static" serialization/deserialization of file formats. But even so, I think the biggest part that's missing is a convenient way to describe the mapping between the structure of the serialized data and the internal representation. ORMs typically give you a single method of doing that mapping, which can work when you have control over the serialized representation. But it doesn't work very well when you're serializing to an existing file format, for example. > Does `serde`[1] look like it's close to your imagined mark? No, serde is exactly what I was talking about in terms of the serialized data matching your internal data structures. I think it's a necessary piece, but I think the part that's missing is a nice way to map that data to the representation your program want's to work on.