4 ms·

Serde has started shipping precompiled binaries with no way to opt out

98 points by davikr 3y ago

jamincan 3y ago
And no justification or explanation that I can see. David Tolnay's response: > Thanks for the comments everyone. I'll go ahead and close this. The precompiled implementation is the only supported way to use the macros that are published in serde_derive. If there is implementation work needed in some build tools to accommodate it, someone should feel free to do that work (as I have done for Buck and Bazel, which are tools I use and contribute significantly to) or publish your own fork of the source code under a different name. Separately, regarding the commentary above about security, the best path forward would be for one of the people who cares about this to invest in a Cargo or crates.io RFC around first-class precompiled macros so that there is an approach that would suit your preferences; serde_derive would adopt that when available.
- DoesntMatter22 3y ago
  I don't know what any of that means. I need an ELI 5
  - __s 3y ago
    Serde is a Rust library for serialization. serde_derive allows creating serialzers/deserializers automatically by marking structs. It uses Rust's procedural macros to generate code at compile time. Rust compiles procedural macros at compile time before compiling rest of code. To speed up compillation of code using serde_derive, a recent change was made to implement the procedural macros with a precompiled binary
    - DoesntMatter22 3y ago
      So is this just because server is known to make rust really slow to build? I'm curious why rust doesn't just compile your binaries once and then you wouldn't have to compile them on each build
      - Sprocklem 3y ago
        It generally does cache built crates for future builds on the same system, so it is usually only the initial build on a given system and the build after dependencies are updated that requires rebuilding serde_derive. I honestly don't know why they thought it was worth doing.
      - earthling8118 3y ago
        It very much does this. The compile times are hardly a concern to the vast majority of rust devs in part because of that cache.
  - jgerrish 3y ago
    serde is also used in a LOT of projects and pulled in as a dependency or subdependency often in the crates ecosystem. For organizations with internal transparent reproducible build guidelines or similar rules, this maybe complicates things and introduces uncertainty. So much uncertainty nowadays. It's exhausting. You can take the generous / positive view that this will encourage formalizing reproducible build policies in organizations. Water is wet, the sky is blue.
- gxt 3y ago
  I'll be pinning my versions of serde until this gets reverted, fixed, or forked. It's an extremely surprising move to disregard supply chain concerns in favor of non publicly benchmarked performance improvements.
  - gxt 3y ago
    Correction: It's benchmarked, https://github.com/serde-rs/serde/pull/2514 https://github.com/serde-rs/serde/pull/2514.
mmastrac 3y ago
This is a pretty solid way to motivate the Rust core team to finally solve the proc macro issue. We've been living with the slowness for so long. Give us wasm and precompiled, sandboxed binaries from crates.io! I suspect dtolnay is trading a bit of his outsized influence on the community [1] to light a fire, and I think it's working. [1] https://crates.io/users/dtolnay?sort=downloads https://crates.io/users/dtolnay?sort=downloads
- CodesInChaos 3y ago
  I think the much bigger issue this spotlights is that many foundation crates are maintained by random third parties and not the rust organization. I think this will turn into Rust's "leftpad" incident, unless dtolnay rolls back this change soon.
  - mmastrac 3y ago
    dtolnay is not random by far -- he's one of the most influential crate authors in the ecosystem. If anyone is trustworthy enough to pull a stunt like this, it's him. I would challenge you to find more than a half dozen crate authors with more downloads: https://crates.io/users/dtolnay?sort=downloads https://crates.io/users/dtolnay?sort=downloads These are not `leftpad`-level crates. This is fundamental tech in the Rust ecosystem, especially in proc macro work.
    - CodesInChaos 3y ago
      > If anyone is trustworthy enough to pull a stunt like this I'd turn this around. If somebody as trusted as him, pulls such a stunt, how are we supposed to trust anybody? > These are not `leftpad`-level crates. They are like leftpad, because they're very popular (indirect) dependencies outside the control of the rust organization.
      - mmastrac 3y ago
        If you find malware in any package he publishes I'll bring out the pitchforks but this drama is all pretty much all hysteria at this point. Calling his contributions leftpad is pretty rude, considering the massive difference between complexity and importance of the two I've only had a handful of interactions with him and he's been a stand-up guy. I'll value the human over the outrage here.
38 3y ago
serde is bloated anyway. see for yourself: 146 kB: https://crates.io/crates/serde_json https://crates.io/crates/serde_json 43 kB: https://crates.io/crates/sj https://crates.io/crates/sj 29 kB: https://crates.io/crates/humphrey_json https://crates.io/crates/humphrey_json 7 kB: https://crates.io/crates/rust_json https://crates.io/crates/rust_json
- slikrick 3y ago
  kB of code doesn't quantify bloat. most of the time those kB are documentation, which far outweigh any sort of bloatedness for a majority of people.

98 points by davikr 3y ago

And no justification or explanation that I can see. David Tolnay's response: > Thanks for the comments everyone. I'll go ahead and close this. The precompiled implementation is the only supported way to use the macros that are published in serde_derive. If there is implementation work needed in some build tools to accommodate it, someone should feel free to do that work (as I have done for Buck and Bazel, which are tools I use and contribute significantly to) or publish your own fork of the source code under a different name. Separately, regarding the commentary above about security, the best path forward would be for one of the people who cares about this to invest in a Cargo or crates.io RFC around first-class precompiled macros so that there is an approach that would suit your preferences; serde_derive would adopt that when available.

DoesntMatter22 3y ago
I don't know what any of that means. I need an ELI 5
- __s 3y ago
  Serde is a Rust library for serialization. serde_derive allows creating serialzers/deserializers automatically by marking structs. It uses Rust's procedural macros to generate code at compile time. Rust compiles procedural macros at compile time before compiling rest of code. To speed up compillation of code using serde_derive, a recent change was made to implement the procedural macros with a precompiled binary
  - DoesntMatter22 3y ago
    So is this just because server is known to make rust really slow to build? I'm curious why rust doesn't just compile your binaries once and then you wouldn't have to compile them on each build
    - Sprocklem 3y ago
      It generally does cache built crates for future builds on the same system, so it is usually only the initial build on a given system and the build after dependencies are updated that requires rebuilding serde_derive. I honestly don't know why they thought it was worth doing.
    - earthling8118 3y ago
      It very much does this. The compile times are hardly a concern to the vast majority of rust devs in part because of that cache.
- jgerrish 3y ago
  serde is also used in a LOT of projects and pulled in as a dependency or subdependency often in the crates ecosystem. For organizations with internal transparent reproducible build guidelines or similar rules, this maybe complicates things and introduces uncertainty. So much uncertainty nowadays. It's exhausting. You can take the generous / positive view that this will encourage formalizing reproducible build policies in organizations. Water is wet, the sky is blue.
gxt 3y ago
I'll be pinning my versions of serde until this gets reverted, fixed, or forked. It's an extremely surprising move to disregard supply chain concerns in favor of non publicly benchmarked performance improvements.
- gxt 3y ago
  Correction: It's benchmarked, https://github.com/serde-rs/serde/pull/2514 https://github.com/serde-rs/serde/pull/2514.

mmastrac 3y ago

This is a pretty solid way to motivate the Rust core team to finally solve the proc macro issue. We've been living with the slowness for so long. Give us wasm and precompiled, sandboxed binaries from crates.io! I suspect dtolnay is trading a bit of his outsized influence on the community [1] to light a fire, and I think it's working. [1] https://crates.io/users/dtolnay?sort=downloads https://crates.io/users/dtolnay?sort=downloads

CodesInChaos 3y ago
I think the much bigger issue this spotlights is that many foundation crates are maintained by random third parties and not the rust organization. I think this will turn into Rust's "leftpad" incident, unless dtolnay rolls back this change soon.
- mmastrac 3y ago
  dtolnay is not random by far -- he's one of the most influential crate authors in the ecosystem. If anyone is trustworthy enough to pull a stunt like this, it's him. I would challenge you to find more than a half dozen crate authors with more downloads: https://crates.io/users/dtolnay?sort=downloads https://crates.io/users/dtolnay?sort=downloads These are not `leftpad`-level crates. This is fundamental tech in the Rust ecosystem, especially in proc macro work.
  - CodesInChaos 3y ago
    > If anyone is trustworthy enough to pull a stunt like this I'd turn this around. If somebody as trusted as him, pulls such a stunt, how are we supposed to trust anybody? > These are not `leftpad`-level crates. They are like leftpad, because they're very popular (indirect) dependencies outside the control of the rust organization.
    - mmastrac 3y ago
      If you find malware in any package he publishes I'll bring out the pitchforks but this drama is all pretty much all hysteria at this point. Calling his contributions leftpad is pretty rude, considering the massive difference between complexity and importance of the two I've only had a handful of interactions with him and he's been a stand-up guy. I'll value the human over the outrage here.

38 3y ago

serde is bloated anyway. see for yourself: 146 kB: https://crates.io/crates/serde_json https://crates.io/crates/serde_json 43 kB: https://crates.io/crates/sj https://crates.io/crates/sj 29 kB: https://crates.io/crates/humphrey_json https://crates.io/crates/humphrey_json 7 kB: https://crates.io/crates/rust_json https://crates.io/crates/rust_json

slikrick 3y ago
kB of code doesn't quantify bloat. most of the time those kB are documentation, which far outweigh any sort of bloatedness for a majority of people.