12 ms·

Understanding Serde

121 points by JoshMcguigan 7y ago

bestouff 7y ago
Serde is one of Rust's killer feature which makes you appreciate the philosophy behind the language. With just a few annotations you have a robust and complete parser for whatever format (JSON, TOML, etc.) automatically built for serializing and deserializing your data structures. And it's fast, and optionnally zero-copy. I've never seen this in other languages.
- pjmlp 7y ago
  Common Lisp?
  - fnord123 7y ago
    Have you got a link to the Common Lisp library that lets you annotate data structures so you can use the annotations to parse or dump data by 3rd party libraries for various formats?
    - pjmlp 7y ago
      No, but given that Common Lisp macro capabilities trump what Rust is capable of, that would be easily achievable. Or now that I think of it, Template Haskell.
- jimenezrick 7y ago
  Haskell? Data generics isn't really nothing new. We had this technology for years.
  - bjz_ 7y ago
    At the same level of performance though?
    - jimenezrick 7y ago
      If you are asking if it's fast? yes it's fast serialization. Not sure which numbers you have at hand, but for me fast means it isn't the bottleneck of my application.
    - ghostwriter 7y ago
      it doesn't have to be to make the point, as Serde isn't on the same level of verbosity as Aeson.
  - earthboundkid 7y ago
    Haskell has been around for more than a decade and had huge uptake by language nerds, but the only significant OSS written in it is Pandoc. In contrast, Rust already has Servo and a number of CLI tools, ripgrep being the most prominent.
    - leshow 7y ago
      I'm not sure what you consider 'significant', but to my mind, there are many great haskell open source projects: xmonad, shellcheck, stack, haxl, to name a few. Not to mention how many languages are written in it: Elm, Purescript, Idris, GHC itself.
      - earthboundkid 7y ago
        I almost listed xmonad. I haven’t heard of the other ones. Doesn’t change my point.
    - jimenezrick 7y ago
      Leaving popularity contests aside, i worked in a company using Haskell and i did used a lot these techniques, if that counts \o/ i guess today it only matter how many stars you repo has in github...
  - nicoburns 7y ago
    True. Not widely available though. The great thing about Rust is that has a whole of the advantages of Haskell, and C++, and to a lesser extent even languages like Python.
- rkangel 7y ago
  It is best not to confuse deserialisation and parsing. Parsing is a separate task with separate issues. For JSON deserialisation, Serde is a very convenient API to use an existing JSON parser that someone else has (hand) written. If you needed a new syntax, you'd have to write a new parser. I often implement communications protocols as part of my work. In most cases, Serde doesn't help me much there - I have to parse the protocol myself (e.g. using Nom).
  - legulere 7y ago
    What is the difference between parsing and deserializing? With both you have a string representation of a data structure that you read and bring into the native form. Do you call it parsing if the grammar is more complicated?
    - earthboundkid 7y ago
      Serde only helps with moving the values from the parser into a struct. It doesn't help with figuring out the data format.
    - p0llard 7y ago
      Deserialisation (in this context) is the act of decoding data from some "string representation" to a representation suitable for use, typically a native format. Typically, although this depends on the specific encoding, a parser is required as one of the first stages of a deserialiser; normally a parser refers to something which applies the rules of something as least as complex as a CFG, as opposed to a lexer/scanner which typically just applies regular expressions; a combination of the two allows the serialised format to be interpreted. You can show quite easily that JSON (and indeed any CFL) cannot be "parsed" using a lexer alone.
    - specialist 7y ago
      Good question, thanks. I had the vague notion that parsing was for human readable content and deserialization was for computer-to-computer content. Now I tentatively make the distinction that compiling transforms streams into code (stream -> lex -> parse -> magic -> code) whereas deserialization is for data. Making parsing just one of the transformation steps for both compilation and deserialization.
- skocznymroczny 7y ago
  D also has serialization libraries based on introspection capabilities. Orange is an example of such library https://code.dlang.org/packages/orange https://code.dlang.org/packages/orange . Here's a blog series (in progress) showing how to write a serializer/deserializer from scratch in D: https://bradley.chatha.dev/Home/Blog?post=JsonSerialiser1 https://bradley.chatha.dev/Home/Blog?post=JsonSerialiser1
  - JoshMcguigan 7y ago
    Thanks for sharing. I'm not familiar with D, but it looks like compile time type introspection is something which is built into the language, which removes the need for something like the serde derive macro. Is that correct?
    - skocznymroczny 7y ago
      Yes. D metaprogramming is mostly based around template and it gives you compile-time traits which you can use to query structs/classes for their fields, determining if something is an array or a pointer etc. This way, you can write a serialize(foo) function which will work for any type, without needing any derives or marking the type as Serializable like in Java. The closest equivalent to derive in D would be a mixin. You could declare a mixin which autogenerates the toJSON() fromJSON() methods based on the fields of a struct.
- chris_overseas 7y ago
  Kotlin serialization[0] is similar to this, implemented via a compiler plugin that allows optimal serializers/deserializers to be generated at compile time. [0] https://github.com/Kotlin/kotlinx.serialization https://github.com/Kotlin/kotlinx.serialization
- jcl 7y ago
  I've read, though, that one downside of Serde is that it tends to generate large binaries -- the cost of performance, I guess. https://raphlinus.github.io/rust/2019/08/21/rust-bloat.html#use-serialization-sparingly https://raphlinus.github.io/rust/2019/08/21/rust-bloat.html#...
kolektiv 7y ago
Serde is very strong indeed (although not the easiest to understand, this would have been useful when I implemented a Serde library for [de]serialising with the AWS DynamoDB AttributeValue format). That said, I've always had a slight problem with this approach to serialisation in general, which is that it makes it very hard to have multiple serialisation approaches for a single type (e.g. I want one casing rule for one target, and a different one for another). This seems effectively impossible with an approach based on annotations/macros. Would love to know how others have dealt with that.
- JoshMcguigan 7y ago
  > I want one casing rule for one target, and a different one for another Hmm, this is interesting, and something I hadn't thought of before. I'd probably try to solve it with a combination of the newtype pattern (create wrapper types for `MyThing`, like `MyThingJson` and `MyThingToml`) and the Serde flatten attribute. Then you could, for example, add `#[serde(rename_all = "lowercase")]` to the `MyThingJson` struct and `#[serde(rename_all = "snake_case")]` to the `MyThingToml` struct. I haven't tested this though, so I'm not sure if the `rename_all` would flow through the flatten attribute as expected. Worst case you could duplicate the struct (or more likely, find or build a macro to do it for you) rather than use the newtype/flatten technique.
- dangets 7y ago
  This is where I would either a) create 2 separate DTO representations with adapters to the internal domain object or b) have a trait which the 2 specialized structs implement. This conversion point is also a good place to do your validation that the object is in a consistent state. This isn't quite zero-cost - but it does isolate your external serialization code from your internal logic and it is easier to add even more formats in the future. And at least in Java-land is much cleaner than stacking lots and lots of annotations on the same struct.
eggie5 7y ago
I recently saw this strange word Serde in some Hive table create commands...
- dfsegoat 7y ago
  https://cwiki.apache.org/confluence/display/Hive/SerDe https://cwiki.apache.org/confluence/display/Hive/SerDe This was where I first saw it as well. Years ago...
lmkg 7y ago
Pronunciation question: Does Serde have one syllable or two? Looking at the word out of context, it seems the e at the end ought to be silent. But it's derived as an abbreviation from SERialize/DEserialize. Does the author or community have a consensus on one of these being the "correct" form?
- nicoburns 7y ago
  I think it has two, although I wouldn't be confident in telling you what they are. I've always pronounced it sair-day (where sair rhymes with hair).
- gulbrandr 7y ago
  I would say two, I personally pronounce "de" like in DElaware.
- cuddlecake 7y ago
  It's abserde to not use only one syllable! ... Sorry, just trying to be funny.
- wrigby 7y ago
  I'm not sure about the rust package, but in the hardware world SerDes ("serializer / deserializer") is pronounced "sir-deez."
- erickt 7y ago
  Original author of serde here. I call it sir-dee, but the current maintainer, dtolnay, calls it sear-day. But we really don't care how people pronounce it. It's really more of a neither / potato thing: https://www.youtube.com/watch?v=K75g7eRhH9M https://www.youtube.com/watch?v=K75g7eRhH9M
  - mercer 7y ago
    Just wanted to thank you for the great work. Thanks!
- dasKrokodil 7y ago
  To me, pronouncing it seer-dee would make sense, because that's how I'd pronounce seer-ialize / dee-serialize. However, for some reason I still pronounce it sir-dee regardless :)

121 points by JoshMcguigan 7y ago

Serde is one of Rust's killer feature which makes you appreciate the philosophy behind the language. With just a few annotations you have a robust and complete parser for whatever format (JSON, TOML, etc.) automatically built for serializing and deserializing your data structures. And it's fast, and optionnally zero-copy. I've never seen this in other languages.

pjmlp 7y ago
Common Lisp?
- fnord123 7y ago
  Have you got a link to the Common Lisp library that lets you annotate data structures so you can use the annotations to parse or dump data by 3rd party libraries for various formats?
  - pjmlp 7y ago
    No, but given that Common Lisp macro capabilities trump what Rust is capable of, that would be easily achievable. Or now that I think of it, Template Haskell.
jimenezrick 7y ago
Haskell? Data generics isn't really nothing new. We had this technology for years.
- bjz_ 7y ago
  At the same level of performance though?
  - jimenezrick 7y ago
    If you are asking if it's fast? yes it's fast serialization. Not sure which numbers you have at hand, but for me fast means it isn't the bottleneck of my application.
  - ghostwriter 7y ago
    it doesn't have to be to make the point, as Serde isn't on the same level of verbosity as Aeson.
- earthboundkid 7y ago
  Haskell has been around for more than a decade and had huge uptake by language nerds, but the only significant OSS written in it is Pandoc. In contrast, Rust already has Servo and a number of CLI tools, ripgrep being the most prominent.
  - leshow 7y ago
    I'm not sure what you consider 'significant', but to my mind, there are many great haskell open source projects: xmonad, shellcheck, stack, haxl, to name a few. Not to mention how many languages are written in it: Elm, Purescript, Idris, GHC itself.
    - earthboundkid 7y ago
      I almost listed xmonad. I haven’t heard of the other ones. Doesn’t change my point.
  - jimenezrick 7y ago
    Leaving popularity contests aside, i worked in a company using Haskell and i did used a lot these techniques, if that counts \o/ i guess today it only matter how many stars you repo has in github...
- nicoburns 7y ago
  True. Not widely available though. The great thing about Rust is that has a whole of the advantages of Haskell, and C++, and to a lesser extent even languages like Python.
rkangel 7y ago
It is best not to confuse deserialisation and parsing. Parsing is a separate task with separate issues. For JSON deserialisation, Serde is a very convenient API to use an existing JSON parser that someone else has (hand) written. If you needed a new syntax, you'd have to write a new parser. I often implement communications protocols as part of my work. In most cases, Serde doesn't help me much there - I have to parse the protocol myself (e.g. using Nom).
- legulere 7y ago
  What is the difference between parsing and deserializing? With both you have a string representation of a data structure that you read and bring into the native form. Do you call it parsing if the grammar is more complicated?
  - earthboundkid 7y ago
    Serde only helps with moving the values from the parser into a struct. It doesn't help with figuring out the data format.
  - p0llard 7y ago
    Deserialisation (in this context) is the act of decoding data from some "string representation" to a representation suitable for use, typically a native format. Typically, although this depends on the specific encoding, a parser is required as one of the first stages of a deserialiser; normally a parser refers to something which applies the rules of something as least as complex as a CFG, as opposed to a lexer/scanner which typically just applies regular expressions; a combination of the two allows the serialised format to be interpreted. You can show quite easily that JSON (and indeed any CFL) cannot be "parsed" using a lexer alone.
  - specialist 7y ago
    Good question, thanks. I had the vague notion that parsing was for human readable content and deserialization was for computer-to-computer content. Now I tentatively make the distinction that compiling transforms streams into code (stream -> lex -> parse -> magic -> code) whereas deserialization is for data. Making parsing just one of the transformation steps for both compilation and deserialization.
skocznymroczny 7y ago
D also has serialization libraries based on introspection capabilities. Orange is an example of such library https://code.dlang.org/packages/orange https://code.dlang.org/packages/orange . Here's a blog series (in progress) showing how to write a serializer/deserializer from scratch in D: https://bradley.chatha.dev/Home/Blog?post=JsonSerialiser1 https://bradley.chatha.dev/Home/Blog?post=JsonSerialiser1
- JoshMcguigan 7y ago
  Thanks for sharing. I'm not familiar with D, but it looks like compile time type introspection is something which is built into the language, which removes the need for something like the serde derive macro. Is that correct?
  - skocznymroczny 7y ago
    Yes. D metaprogramming is mostly based around template and it gives you compile-time traits which you can use to query structs/classes for their fields, determining if something is an array or a pointer etc. This way, you can write a serialize(foo) function which will work for any type, without needing any derives or marking the type as Serializable like in Java. The closest equivalent to derive in D would be a mixin. You could declare a mixin which autogenerates the toJSON() fromJSON() methods based on the fields of a struct.
chris_overseas 7y ago
Kotlin serialization[0] is similar to this, implemented via a compiler plugin that allows optimal serializers/deserializers to be generated at compile time. [0] https://github.com/Kotlin/kotlinx.serialization https://github.com/Kotlin/kotlinx.serialization
jcl 7y ago
I've read, though, that one downside of Serde is that it tends to generate large binaries -- the cost of performance, I guess. https://raphlinus.github.io/rust/2019/08/21/rust-bloat.html#use-serialization-sparingly https://raphlinus.github.io/rust/2019/08/21/rust-bloat.html#...

kolektiv 7y ago

Serde is very strong indeed (although not the easiest to understand, this would have been useful when I implemented a Serde library for [de]serialising with the AWS DynamoDB AttributeValue format). That said, I've always had a slight problem with this approach to serialisation in general, which is that it makes it very hard to have multiple serialisation approaches for a single type (e.g. I want one casing rule for one target, and a different one for another). This seems effectively impossible with an approach based on annotations/macros. Would love to know how others have dealt with that.

JoshMcguigan 7y ago
> I want one casing rule for one target, and a different one for another Hmm, this is interesting, and something I hadn't thought of before. I'd probably try to solve it with a combination of the newtype pattern (create wrapper types for `MyThing`, like `MyThingJson` and `MyThingToml`) and the Serde flatten attribute. Then you could, for example, add `#[serde(rename_all = "lowercase")]` to the `MyThingJson` struct and `#[serde(rename_all = "snake_case")]` to the `MyThingToml` struct. I haven't tested this though, so I'm not sure if the `rename_all` would flow through the flatten attribute as expected. Worst case you could duplicate the struct (or more likely, find or build a macro to do it for you) rather than use the newtype/flatten technique.
dangets 7y ago
This is where I would either a) create 2 separate DTO representations with adapters to the internal domain object or b) have a trait which the 2 specialized structs implement. This conversion point is also a good place to do your validation that the object is in a consistent state. This isn't quite zero-cost - but it does isolate your external serialization code from your internal logic and it is easier to add even more formats in the future. And at least in Java-land is much cleaner than stacking lots and lots of annotations on the same struct.

eggie5 7y ago

I recently saw this strange word Serde in some Hive table create commands...

dfsegoat 7y ago
https://cwiki.apache.org/confluence/display/Hive/SerDe https://cwiki.apache.org/confluence/display/Hive/SerDe This was where I first saw it as well. Years ago...

lmkg 7y ago

Pronunciation question: Does Serde have one syllable or two? Looking at the word out of context, it seems the e at the end ought to be silent. But it's derived as an abbreviation from SERialize/DEserialize. Does the author or community have a consensus on one of these being the "correct" form?

nicoburns 7y ago
I think it has two, although I wouldn't be confident in telling you what they are. I've always pronounced it sair-day (where sair rhymes with hair).
gulbrandr 7y ago
I would say two, I personally pronounce "de" like in DElaware.
cuddlecake 7y ago
It's abserde to not use only one syllable! ... Sorry, just trying to be funny.
wrigby 7y ago
I'm not sure about the rust package, but in the hardware world SerDes ("serializer / deserializer") is pronounced "sir-deez."
erickt 7y ago
Original author of serde here. I call it sir-dee, but the current maintainer, dtolnay, calls it sear-day. But we really don't care how people pronounce it. It's really more of a neither / potato thing: https://www.youtube.com/watch?v=K75g7eRhH9M https://www.youtube.com/watch?v=K75g7eRhH9M
- mercer 7y ago
  Just wanted to thank you for the great work. Thanks!
dasKrokodil 7y ago
To me, pronouncing it seer-dee would make sense, because that's how I'd pronounce seer-ialize / dee-serialize. However, for some reason I still pronounce it sir-dee regardless :)