5 ms·

Serde is a fantastic part of the Rust ecosystem, and is often credited as a reason people reach for Rust. This power and convenience coming to C++ should be a c

Serde is a fantastic part of the Rust ecosystem, and is often credited as a reason people reach for Rust. This power and convenience coming to C++ should be a cause for celebration, in my mind. I am sad that Rust is missing an opportunity to do similar here, and hope that someone will pick the proposal back up someday. Barry was kind enough to share a draft of this with me, and he inserted this based on some of my feedback: > newer Rust has something called derive macro helper attributes which will make this easier to do. Apparently I am mistaken about this, and basically every Rust procedural macro does what serde does here. I find the documentation for this a bit confusing. I've emailed him to let him know, and please consider this mistake mine, not his!

tux3 2y ago
As a followup, could the helper attributes feature be implemented to work how you thought it did, enough that most proc macros might want to start using it? =) The author does have a good point when they note that the C++ compiler does the parsing and the "serde intermediate representation" for them, it's pretty nice to not have to bring your own parser for everything! Could we give proc macro authors fast compiler-backed helpers that they can use as building blocks for their proc macros?
- steveklabnik 2y ago
  I do not know! I would prefer more energy be put into revitalizing the reflection proposal than doing that, though of course open source is always about “who wants to do the work” and I won’t be doing either.
- mplanchard 2y ago
  Having just written a proc macro, I'd even be happy with some higher level abstractions on top of syn, if anyone knows of any.
the_mitsuhiko 2y ago
I still feel that it was a big mistake in retrospective that we ended up with serde style macros all over the place instead of getting introspection early on. The end result is ungodly amounts of quite convoluted code that in parts is not even compatible with each other. It also greatly restricts what you can do, because you cannot be conditional on the thing you're generating for. I'm not sure if the C++ solution proposed is the right abstraction, but I feel very confident in saying that Rust's is not the gold standard that one should strive for.
- steveklabnik 2y ago
  Full agreement. Proc macros are an accident of history in many ways. I would encourage new language authors to go straight towards reflection instead.
  - cogman10 2y ago
    Really makes me appreciate Java/kotlin's annotation processors. They are much cleaner/clearer APIs for writing compile time generated software and the entire ecosystem has benefited from them.
    - Twisol 2y ago
      I, uh, cannot in good conscience call Java's annotation processing APIs "clean" or "clear"; the way they interact with the multi-round processing model, and in particular make it extremely hard to build well-behaved processors that can cope with parts of the compile-time code graph not existing until later rounds, has baffled me for a long time. Despite their definite flaws, though, I have to agree: compile-time code generation via annotation processing is something I think we should do more of (and invest more time into better tools for it).
      - cogman10 2y ago
        > in particular make it extremely hard to build well-behaved processors that can cope with parts of the compile-time code graph not existing until later rounds, has baffled me for a long time. I'm not exactly sure how you can really make this particular problem better when working with compile time code generation. But what I'd say WRT cleaner and clearer, Java and Kotlin both expose much higher level type information than is available in rust's proc macros. That's mostly what I meant by cleaner and clearer. Without needing to pull in weirdo not-quite-third-party libraries, you can reflect on and generate for all sort of different type information. It seems to me that something like this should be possible with rust given its early transformation into high level bytecode. But I could see why the rust devs have pushed back on doing that as it'd really lock in features they might not want to lock in.
secondcoming 2y ago
I'm not convinced that serialisation should be a first class citizen of a language. Mainly due to issues with unintended breaking changes causing carnage downstream. Protobuf/FlatBuffers et all are way more safe. Especially with C++ where doing something like serialising a bool raises questions about implementation defined behaviour such as: - what bit pattern represents 'true' - what is sizeof(bool) - can something serialised with gcc be deserialised with clang
- steveklabnik 2y ago
  None of this is specific to serialization. This is compile time meta programming. All of those questions are handled by the code that implements serialization, not the language feature that enables that code to be written.
- cryptonector 2y ago
  Serde supports many serialization standards. These issues don't apply.
j-pb 2y ago
> Serde ... is often credited as a reason people reach for Rust. I've heard this narrative a lot, but I'd challenge it. I tend to actively disable serde feature flags and don't use libraries with mandatory serde dependencies. It seems to be one of those things that are good enough™, while still falling short of being actually good. - rusts type system is strong enough to build extremely powerful zerocopy serialization, but people don't explore that space because of serde - macros and especially proc-macros in Rust are horrible, and are only feasible because of the syn crate - syn itself is essentially a hack that has become so entrenched that there is little incentive to improve rusts metaprogramming story - serde adds a ton of compile time whenever used because of the heavy macro usage - the author of serde and syn has been at the center of a ton of drama, being at the root of the keynote incident that essentially stalled comptime reflection (Zig is eating Rusts lunch in that department) and caused a huge setback on rusts inclusivity, shipping binary blobs in serde to make a point and push an RFC (essentially abusing the power he has with serde), being the final descision maker of the latest core::time breakage, all leave a bitter after-taste for someone who doesn't aparently have any official position in the rust foundation with the appropriate accountability, yet holds an enormous amount of hidden power (and being ex-palantir also makes me a lot less inclined to give him the benefit of doubt, that these were all unfortunate misunderstandings) I think the overall impact of serde and syn, including the authors behaviour has been a net negative for the rust ecosystem, both technically, personally, and marketing wise (the rust community is perceived as being unprofessional and overly dramatic). Having written this down like that, makes me realised that I don't just avoid serde, I boycott it.
- aabhay 2y ago
  The fact that most crates are compiled with serde as feature flag and the fact that you can raise a PR on most things that aren’t — are both great cases for Rust having an awesome ecosystem. Just as with async support, serialization support is a free market of ideas.
- steveklabnik 2y ago
  That proc macros have significant drawbacks, and you personally weigh those drawbacks as having a lot of significance, doesn’t change that for many people, the juice is worth the squeeze. 375M downloads don’t lie. I agree with you that there are significant drawbacks, which is why I’d love to see reflection happen in Rust. > being at the root of the keynote incident I don’t believe this to be true, based on what I know from both public and private information. I think a lot of people don’t like dtolnay due to the Palantir thing, and decided to try and inflate his involvement. But even from just public information: blame lies with the decision makers. By his own account, Josh Triplett suggested a demotion, and Sage Griffin set that into motion. They’ve both since apologized. I do not believe they are solely to blame, but if you want to point fingers at individuals, they’re who I would describe as “in the center.” Overall the blame relies on Rust’s continual failure to do governance in a way that’s responsive and legible to its users, and continually ignores the written rules to just do whatever whoever has the soft power wants. This isn’t a unique story in open source, of course. > someone who doesn't aparently have any official position in the rust foundation The Rust Foundation does not govern most of the things you’re upset with. He is a member of the libs-api team in the Project, though. > being ex-palantir I’ll be the first to say “Fuck Palantir” but being upset at people for where they worked a long time ago just doesn’t sit right with me. Do you think that anyone who’s ever worked anywhere objectionable should be branded with that decision for life? Many people don’t like me for working at Cloudflare. Is there no possibility for change in your mind? If we truly want to play that game, you’re gonna have to exclude like half of the industry.
nrclark 2y ago
Tangential, but does anybody else get real npm vibes from the rust ecosystem? Something about the “every productive project depends on this one external package” situation really makes me uneasy. And there are language features like async that can’t even really be used without going to crates.io for a bunch of stuff that really ought to be in the stdlib.
- bryanlarsen 2y ago
  withoutboats mentioned trying to get something like https://github.com/zesterer/pollster https://github.com/zesterer/pollster into the standard library. I think that's a great idea. They don't want to put an executor into the standard library because that would "pick a winner" before the ideas are settled. But pollster allows async crates to be useful to sync code, is obviously not a "winner" and its inclusion in the standard library would force crates to be properly executor-agnostic.
  - zesterer 2y ago
    The problem with this is that pollster doesn't, by itself, provide a signalling mechanism by which bottom-level futures (such as OS-provided IO primitives) can signal the executor. It isn't a "light-weight replacement" for tokio, tokio does a lot of stuff that pollster simply can't because tokio is a reactor + a runtime, whereas pollster is just a runtime. [This page](https://rust-lang.github.io/async-book/08_ecosystem/00_chapter.html#determining-ecosystem-compatibility https://rust-lang.github.io/async-book/08_ecosystem/00_chapt...) gives some nice details about the difference, and hopefully some sense of the sheer difficult that comes with trying to integrate a default reactor into `std` without it falling well short when it comes to extensibility.
- steveklabnik 2y ago
  npm, being one of the most successful package management systems in history, was an express inspiration for Cargo, sure. It’s also not without flaw, and so Cargo and npm do differ in some key ways. Software engineers love to talk about how code re-use is good, and keeping code small and simple is good, and then somehow get upset when a lot of modular, small, reusable code is produced and widely shared.
VyseofArcadia 2y ago
> Serde is a fantastic part of the Rust ecosystem, and is often credited as a reason people reach for Rust. From what little I've seen of it, from a user perspective it seems like a pretty bog standard (de)serialization library. What is so special about it that people would reach for Rust to use it instead of an equivalent library in their language of choice?
- steveklabnik 2y ago
  There isn’t anything super novel about Serde, for sure. But there’s three important things about it, in my mind: 1. It is extremely convenient and pleasant to use. 2. It’s kind of like pandoc: due to its design, it’s not just a serialization library for a specific format, but instead a platform for other libraries. Want to take in some JSON and produce TOML? No issues. Those two libraries are already interoperable. See 1 again. 3. Due to being an extremely old library, as well as a good one, everyone supports it where appropriate. This ubiquity, combined with the flexibility of 2, is a big part of what makes 1 true as well. This doesn’t mean that Serde isn’t perfect, mind you. But it is very good. And with Rust’s reputation as a difficult language, I think sometimes are surprised when things are actually convenient.
- josephg 2y ago
  I think the point here is that C++ notably doesn't have an equivalent to serde. Its very hard to make an equivalent library in C or C++ without some external codegen step, manually writing parsing code, or some horrible macros. Mind you, it sounds like this will change with C++26. Good times!

tux3 2y ago

As a followup, could the helper attributes feature be implemented to work how you thought it did, enough that most proc macros might want to start using it? =) The author does have a good point when they note that the C++ compiler does the parsing and the "serde intermediate representation" for them, it's pretty nice to not have to bring your own parser for everything! Could we give proc macro authors fast compiler-backed helpers that they can use as building blocks for their proc macros?

steveklabnik 2y ago
I do not know! I would prefer more energy be put into revitalizing the reflection proposal than doing that, though of course open source is always about “who wants to do the work” and I won’t be doing either.
mplanchard 2y ago
Having just written a proc macro, I'd even be happy with some higher level abstractions on top of syn, if anyone knows of any.

the_mitsuhiko 2y ago

I still feel that it was a big mistake in retrospective that we ended up with serde style macros all over the place instead of getting introspection early on. The end result is ungodly amounts of quite convoluted code that in parts is not even compatible with each other. It also greatly restricts what you can do, because you cannot be conditional on the thing you're generating for. I'm not sure if the C++ solution proposed is the right abstraction, but I feel very confident in saying that Rust's is not the gold standard that one should strive for.

steveklabnik 2y ago
Full agreement. Proc macros are an accident of history in many ways. I would encourage new language authors to go straight towards reflection instead.
- cogman10 2y ago
  Really makes me appreciate Java/kotlin's annotation processors. They are much cleaner/clearer APIs for writing compile time generated software and the entire ecosystem has benefited from them.
  - Twisol 2y ago
    I, uh, cannot in good conscience call Java's annotation processing APIs "clean" or "clear"; the way they interact with the multi-round processing model, and in particular make it extremely hard to build well-behaved processors that can cope with parts of the compile-time code graph not existing until later rounds, has baffled me for a long time. Despite their definite flaws, though, I have to agree: compile-time code generation via annotation processing is something I think we should do more of (and invest more time into better tools for it).
    - cogman10 2y ago
      > in particular make it extremely hard to build well-behaved processors that can cope with parts of the compile-time code graph not existing until later rounds, has baffled me for a long time. I'm not exactly sure how you can really make this particular problem better when working with compile time code generation. But what I'd say WRT cleaner and clearer, Java and Kotlin both expose much higher level type information than is available in rust's proc macros. That's mostly what I meant by cleaner and clearer. Without needing to pull in weirdo not-quite-third-party libraries, you can reflect on and generate for all sort of different type information. It seems to me that something like this should be possible with rust given its early transformation into high level bytecode. But I could see why the rust devs have pushed back on doing that as it'd really lock in features they might not want to lock in.

secondcoming 2y ago

I'm not convinced that serialisation should be a first class citizen of a language. Mainly due to issues with unintended breaking changes causing carnage downstream. Protobuf/FlatBuffers et all are way more safe. Especially with C++ where doing something like serialising a bool raises questions about implementation defined behaviour such as: - what bit pattern represents 'true' - what is sizeof(bool) - can something serialised with gcc be deserialised with clang

steveklabnik 2y ago
None of this is specific to serialization. This is compile time meta programming. All of those questions are handled by the code that implements serialization, not the language feature that enables that code to be written.
cryptonector 2y ago
Serde supports many serialization standards. These issues don't apply.

j-pb 2y ago

> Serde ... is often credited as a reason people reach for Rust. I've heard this narrative a lot, but I'd challenge it. I tend to actively disable serde feature flags and don't use libraries with mandatory serde dependencies. It seems to be one of those things that are good enough™, while still falling short of being actually good. - rusts type system is strong enough to build extremely powerful zerocopy serialization, but people don't explore that space because of serde - macros and especially proc-macros in Rust are horrible, and are only feasible because of the syn crate - syn itself is essentially a hack that has become so entrenched that there is little incentive to improve rusts metaprogramming story - serde adds a ton of compile time whenever used because of the heavy macro usage - the author of serde and syn has been at the center of a ton of drama, being at the root of the keynote incident that essentially stalled comptime reflection (Zig is eating Rusts lunch in that department) and caused a huge setback on rusts inclusivity, shipping binary blobs in serde to make a point and push an RFC (essentially abusing the power he has with serde), being the final descision maker of the latest core::time breakage, all leave a bitter after-taste for someone who doesn't aparently have any official position in the rust foundation with the appropriate accountability, yet holds an enormous amount of hidden power (and being ex-palantir also makes me a lot less inclined to give him the benefit of doubt, that these were all unfortunate misunderstandings) I think the overall impact of serde and syn, including the authors behaviour has been a net negative for the rust ecosystem, both technically, personally, and marketing wise (the rust community is perceived as being unprofessional and overly dramatic). Having written this down like that, makes me realised that I don't just avoid serde, I boycott it.

aabhay 2y ago
The fact that most crates are compiled with serde as feature flag and the fact that you can raise a PR on most things that aren’t — are both great cases for Rust having an awesome ecosystem. Just as with async support, serialization support is a free market of ideas.
steveklabnik 2y ago
That proc macros have significant drawbacks, and you personally weigh those drawbacks as having a lot of significance, doesn’t change that for many people, the juice is worth the squeeze. 375M downloads don’t lie. I agree with you that there are significant drawbacks, which is why I’d love to see reflection happen in Rust. > being at the root of the keynote incident I don’t believe this to be true, based on what I know from both public and private information. I think a lot of people don’t like dtolnay due to the Palantir thing, and decided to try and inflate his involvement. But even from just public information: blame lies with the decision makers. By his own account, Josh Triplett suggested a demotion, and Sage Griffin set that into motion. They’ve both since apologized. I do not believe they are solely to blame, but if you want to point fingers at individuals, they’re who I would describe as “in the center.” Overall the blame relies on Rust’s continual failure to do governance in a way that’s responsive and legible to its users, and continually ignores the written rules to just do whatever whoever has the soft power wants. This isn’t a unique story in open source, of course. > someone who doesn't aparently have any official position in the rust foundation The Rust Foundation does not govern most of the things you’re upset with. He is a member of the libs-api team in the Project, though. > being ex-palantir I’ll be the first to say “Fuck Palantir” but being upset at people for where they worked a long time ago just doesn’t sit right with me. Do you think that anyone who’s ever worked anywhere objectionable should be branded with that decision for life? Many people don’t like me for working at Cloudflare. Is there no possibility for change in your mind? If we truly want to play that game, you’re gonna have to exclude like half of the industry.

nrclark 2y ago

Tangential, but does anybody else get real npm vibes from the rust ecosystem? Something about the “every productive project depends on this one external package” situation really makes me uneasy. And there are language features like async that can’t even really be used without going to crates.io for a bunch of stuff that really ought to be in the stdlib.

bryanlarsen 2y ago
withoutboats mentioned trying to get something like https://github.com/zesterer/pollster https://github.com/zesterer/pollster into the standard library. I think that's a great idea. They don't want to put an executor into the standard library because that would "pick a winner" before the ideas are settled. But pollster allows async crates to be useful to sync code, is obviously not a "winner" and its inclusion in the standard library would force crates to be properly executor-agnostic.
- zesterer 2y ago
  The problem with this is that pollster doesn't, by itself, provide a signalling mechanism by which bottom-level futures (such as OS-provided IO primitives) can signal the executor. It isn't a "light-weight replacement" for tokio, tokio does a lot of stuff that pollster simply can't because tokio is a reactor + a runtime, whereas pollster is just a runtime. [This page](https://rust-lang.github.io/async-book/08_ecosystem/00_chapter.html#determining-ecosystem-compatibility https://rust-lang.github.io/async-book/08_ecosystem/00_chapt...) gives some nice details about the difference, and hopefully some sense of the sheer difficult that comes with trying to integrate a default reactor into `std` without it falling well short when it comes to extensibility.
steveklabnik 2y ago
npm, being one of the most successful package management systems in history, was an express inspiration for Cargo, sure. It’s also not without flaw, and so Cargo and npm do differ in some key ways. Software engineers love to talk about how code re-use is good, and keeping code small and simple is good, and then somehow get upset when a lot of modular, small, reusable code is produced and widely shared.

VyseofArcadia 2y ago

> Serde is a fantastic part of the Rust ecosystem, and is often credited as a reason people reach for Rust. From what little I've seen of it, from a user perspective it seems like a pretty bog standard (de)serialization library. What is so special about it that people would reach for Rust to use it instead of an equivalent library in their language of choice?

steveklabnik 2y ago
There isn’t anything super novel about Serde, for sure. But there’s three important things about it, in my mind: 1. It is extremely convenient and pleasant to use. 2. It’s kind of like pandoc: due to its design, it’s not just a serialization library for a specific format, but instead a platform for other libraries. Want to take in some JSON and produce TOML? No issues. Those two libraries are already interoperable. See 1 again. 3. Due to being an extremely old library, as well as a good one, everyone supports it where appropriate. This ubiquity, combined with the flexibility of 2, is a big part of what makes 1 true as well. This doesn’t mean that Serde isn’t perfect, mind you. But it is very good. And with Rust’s reputation as a difficult language, I think sometimes are surprised when things are actually convenient.
josephg 2y ago
I think the point here is that C++ notably doesn't have an equivalent to serde. Its very hard to make an equivalent library in C or C++ without some external codegen step, manually writing parsing code, or some horrible macros. Mind you, it sounds like this will change with C++26. Good times!