18 ms·

Serde json has 3gb of dependencies once you do a build for debug and a build for release. Use serde on a few active projects and you run out of disk space. I do

Serde json has 3gb of dependencies once you do a build for debug and a build for release. Use serde on a few active projects and you run out of disk space. I don’t know why json parsing needs 3gb of dependencies. I’m all for code reuse but Serde for json is a bit of a dogs breakfast when it comes to dependencies. all you need is an exploit in on of those dependencies and half of the rust ecosystem is vulnerable. Rust should have Jason built in.

ninkendo 2y ago
It has 5 dependencies, one of which is optional, and another is serde itself: https://github.com/serde-rs/json/blob/master/Cargo.toml https://github.com/serde-rs/json/blob/master/Cargo.toml indexmap = { version = "2.2.3", optional = true } itoa = "1.0" memchr = { version = "2", default-features = false } ryu = "1.0" serde = { version = "1.0.194", default-features = false } I don’t think you’re measuring what you think you’re measuring when you say it has 3GB of dependencies. But I can’t say for sure because you don’t provide any evidence for it, you just declare it as true. If I were to guess, I’d say you’re doing a lot of #[derive(Serialize, Deserialize)] and it’s generating tons of code (derive does code generation, after all) and you’re measuring the total size of your target directory after all of this. But this is just a guess… other commenters have shown that a simple build produces code on the order of tens of MB…
- deleted 2y ago
  [deleted]
hermanradtke 2y ago
Please show your work. I cannot reproduce "3gb of dependencies". Here is my test: Cargo.toml [package] name = "serde-test" version = "0.1.0" edition = "2021" [dependencies] serde = { version = "1.0.208", features = ["derive"] } serde_json = "1.0.127" src/main.rs use serde::Deserialize; #[derive(Deserialize)] struct Foo { bar: String, } fn main() { let foo: Foo = serde_json::from_str("\"bar\": \"baz\"").unwrap(); println!("{}", foo.bar); } $ cargo build && cargo build --release && du -sh target ... 78M target
- tredre3 2y ago
  I arrive at almost the same result as you, with 76MB. I've also checked .cargo, .rustup, and my various cache folders (just in case) and haven't found any additional disk usage. OP is clearly mistaken.
  - inferiorhuman 2y ago
    The first thing that jumps out is that the code example doesn't work. The next thing is that the example merely calls cargo build. Using an IDE of any sort will typically invoke rust-analyzer which will bloat the target directory quite a bit. I've also found that stale build artifacts tend to chew up a lot of space (especially if you're trying to measure the typically smaller release builds). Beyond that, none of the serde features that will tend to generate a ton of code are being used. So yeah a minimal example won't use a lot of space but if you start to use the bells and whistles serde brings you will definitely bloat your target directory. I expect a typical rust project to take around 3–4 gigs for build artifacts depending.
    - wtetzner 2y ago
      > So yeah a minimal example won't use a lot of space but if you start to use the bells and whistles serde brings you will definitely bloat your target directory. Which seems orthogonal to the number of dependencies?
    - hermanradtke 2y ago
      > The first thing that jumps out is that the code example doesn't work. Good catch. I forgot the braces. It does not change the target directory size in a significant way. As for your other comments: sure! We can have a real conversation about rust-analyzer and other serde features (though I am not sure which specific features you are referring to) causing the target directory to increase drastically in size. However, a sensationalist comment that claims the _dependencies_ are 3gb appears to be misleading at best.
- cedws 2y ago
  Jesus, 78MB is still a lot for such a simple program.
38 2y ago
this 100%. serde is a bloated monster, its sad that its the popular JSON, because all it does is make Rust look bad in my opinion. here are some smaller options: https://lib.rs/crates/humphrey_json https://lib.rs/crates/humphrey_json https://lib.rs/crates/rust_json https://lib.rs/crates/rust_json https://lib.rs/crates/sj https://lib.rs/crates/sj
- skitter 2y ago
  merde_json should also be relatively small.
- INGSOCIALITE 2y ago
  can rust use the json-c library?
  - makeitshine 2y ago
    I'd assume you could use bindgen and create bindings no problem.
  - rapsey 2y ago
    People use rust for its memory safety.
- duped 2y ago
  The value prop isn't serde_json, it's automatically generated serializers and deserializers for structured data without needing an extra codegen step like with protobufs/capnproto, plus all that machinery decoupled from the actual data format you're reading. It essentially generates a massive amount of code that you need to write anyway, at the cost of code size and compile time. And a lot of people are happy to make that trade off. I wouldn't call that a "bloated monster" because of that. Also, none of those options are alternatives to serde_json, unless you restrict yourself to serde_json::Value - which no one does in practice.
  - 38 2y ago
    > none of those options are alternatives to serde_json, unless you restrict yourself to serde_json::Value - which no one does in practice. check your facts, all the above options have derive support, serde is not special in that.
sweca 2y ago
I swear the target folder for literally any project of any scale is at least several GB in size.
- gnuvince 2y ago
  I get a progress bar when I run `cargo clean` because it's so large.
- aldanor 2y ago
  For a work project, my recent cargo clean removed 90 GB.
  - sweca 2y ago
    jaw drops
purplesyringa 2y ago
> Rust should have Jason built in. I don't think this is a reasonable approach. That's just a way to introduce bloat. Importantly, std does not differ from other crates, except for stability guarantees, so there would be no positive here. All it does is link the library's release cycle to the compiler's. (In fact, rustc-serialize used to be built-in, so Rust tried to go that way.) But also, serde_json isn't large by default. I'm not sure where you are getting those numbers from. serde_json isn't large, serde isn't large. They both have very low MSRVs few other crates support, so in all truth they can't even have many dependencies.
- kelnos 2y ago
  > That's just a way to introduce bloat. I don't think "bloat" is the issue; as I'm sure you know, Rust programs only contain code for the features of the stdlib they use; if it had JSON support and it wasn't used, the linker would omit that code from the final binary. (Granted, that would make the linker work a little harder.) More at issue is maintenance burden. Data formats come and go over time. In 10 or 20 years when JSON has faded to XML's level of relevance, the Rust stdlib team shouldn't have to continue maintaining a JSON parser.
- wiseowise 2y ago
  > I don't think this is a reasonable approach. That's just a way to introduce bloat. Can this meme die already? The fact that out of the box install of Rust can’t parse JSON is a joke, and you know it.
  - hombre_fatal 2y ago
    I don’t want to wait on language releases to get updates to json, regex, etc. Nor do I want a crappy stdlib impl of something to become widespread just because it comes out of the box like Go’s worst of breed html templating and http “routing”.
    - wiseowise 2y ago
      Somehow Python and JS can get away with json in std lib, but thing that builds binaries can’t? How often does it even need to be updated to parse freaking json?
Klonoar 2y ago
What kind of machine are you developing on that runs out of space that quickly…?
pornel 2y ago
Rust emits unreasonable amount of debug information. It's so freakishly large, I expect it's just a bug. Anything you compile will dump gigabytes into the target folder, but that's not representative of the final product (after stripping the debug info, or at least using a toned-down verbosity setting).
- hinkley 2y ago
  Does it need a more compact representation of its debug info?
- duped 2y ago
  Most of your target folder isn't debug info, but stale build artifacts because Cargo doesn't do any garbage collection.
- khuey 2y ago
  > Rust emits unreasonable amount of debug information. It's so freakishly large, I expect it's just a bug. Rust relies on the linker (via -ffunction-sections and -gc-sections) to delete functions that aren't ever used but the linker isn't capable of removing the corresponding debug info. https://github.com/rust-lang/rust/issues/56068 https://github.com/rust-lang/rust/issues/56068
jwells89 2y ago
Built in JSON encoding/decoding is one of the things I’ve enjoyed about Swift. It’s nice when it’s not necessary to shop around for libraries for common needs like that.
- throwup238 2y ago
  Almost nobody is shopping around Rust JSON libraries unless they need some specific feature not provided by serde and serde_json. They are the default everyone reaches for.
  - bscphil 2y ago
    The fact that a JSON parser is the sort of ecological niche where you naturally get a single best library which does basically what everyone wants it to do with minimal dependencies is exactly the argument for putting it in the stdlib, though.
    - throwup238 2y ago
      That would require moving the entirety of serde and its traits (or equivalent) into the standard library. Considering how long it takes to stabilize basic things in the stdlib, I think that’s a terrible idea. IMO Rust has the best tradeoff between stdlib and ecosystem libraries I’ve ever seen in a (compiled) language, and that includes serde/serde_json.
  - wiseowise 2y ago
    The moment you NEED to include a library to parse some basic JSON file - you’ve lost already.
    - ModernMech 2y ago
      Based on your other reply about JSON being a "basic" feature I assume you do a lot of work with JSON. What you need to understand is not everyone works with JSON, and for them it's a feature to not have JSON parsing code in their binaries. It's not a loss for them.
troad 2y ago
Dependency bloat is an issue with Rust in general. The dependency trees for any meaty Rust project quickly become pretty horrifying. Auditing all these dependencies is infeasible, and my level of confidence in a lot of them is fairly low. I worked with Rust for a few years, and with the benefit of a few years' experience, I don't think I'll be touching Rust again until the ecosystem matures a great deal (which will only come with significant corporate adoption), or if I need something for a no-std, no-deps, strictly-a-C-replacement kind of project. (Though Zig might edge out Rust for this use case once it stabilises.)
- samatman 2y ago
  > Though Zig might edge out Rust for this use case once it stabilises. Zig has a meaningful advantage in the context of this discussion: lazy compilation. The compiler won't even semantically analyze a block of code unless the target requires that to happen. Currently, dependencies are more eager than they really need to be, but making them just as lazy as compilation is on the roadmap. Lazy compilation means no tree shaking is needed, and it means that the actual build-graph of a program can be traced on a fine-grained level. We might not be able to audit a hundred dependencies, but auditing the actual used code from all those dependencies might be more practical. This is well positioned to handle a common pattern: `small-lib` provides some useful stuff, but also has extensions for working with `huge-framework`, and it needs to have `huge-framework` as a dependency to do so. Currently this means that the build system will fetch `huge-framework`, but if we can get it lazy enough, even that won't have to happen unless the code consuming `small-lib` touches the `huge-framework` dependency, which won't happen unless the program itself needs `huge-framework`. Existing build systems don't do such a great job with that kind of structure, and the culprit is eagerness.
- echelon 2y ago
  > The dependency trees for any meaty Rust project quickly become pretty horrifying. s/Rust// This is really no different from any other language. At least Rust, with Cargo, makes it easy to scan your dependencies. And many notable Rust projects attempt to keep third party dependencies to a minimum. C++ gives you absolutely nothing to work with. Other languages with package managers don't keep dependency trees shallow. You're holding Rust up to a standard that nothing meets.
tinrab 2y ago
From crates.io, `serde` is a 76.4 KiB dependency. And from what I've seen looking through the code, it's pretty minimal.
the__alchemist 2y ago
I just added `serde_json` to my small GUI program. It alone increased the binary size from 6Mb to 8Mb; unacceptable for what it does.

ninkendo 2y ago

It has 5 dependencies, one of which is optional, and another is serde itself: https://github.com/serde-rs/json/blob/master/Cargo.toml https://github.com/serde-rs/json/blob/master/Cargo.toml indexmap = { version = "2.2.3", optional = true } itoa = "1.0" memchr = { version = "2", default-features = false } ryu = "1.0" serde = { version = "1.0.194", default-features = false } I don’t think you’re measuring what you think you’re measuring when you say it has 3GB of dependencies. But I can’t say for sure because you don’t provide any evidence for it, you just declare it as true. If I were to guess, I’d say you’re doing a lot of #[derive(Serialize, Deserialize)] and it’s generating tons of code (derive does code generation, after all) and you’re measuring the total size of your target directory after all of this. But this is just a guess… other commenters have shown that a simple build produces code on the order of tens of MB…

deleted 2y ago
[deleted]

hermanradtke 2y ago

Please show your work. I cannot reproduce "3gb of dependencies". Here is my test: Cargo.toml [package] name = "serde-test" version = "0.1.0" edition = "2021" [dependencies] serde = { version = "1.0.208", features = ["derive"] } serde_json = "1.0.127" src/main.rs use serde::Deserialize; #[derive(Deserialize)] struct Foo { bar: String, } fn main() { let foo: Foo = serde_json::from_str("\"bar\": \"baz\"").unwrap(); println!("{}", foo.bar); } $ cargo build && cargo build --release && du -sh target ... 78M target

tredre3 2y ago
I arrive at almost the same result as you, with 76MB. I've also checked .cargo, .rustup, and my various cache folders (just in case) and haven't found any additional disk usage. OP is clearly mistaken.
- inferiorhuman 2y ago
  The first thing that jumps out is that the code example doesn't work. The next thing is that the example merely calls cargo build. Using an IDE of any sort will typically invoke rust-analyzer which will bloat the target directory quite a bit. I've also found that stale build artifacts tend to chew up a lot of space (especially if you're trying to measure the typically smaller release builds). Beyond that, none of the serde features that will tend to generate a ton of code are being used. So yeah a minimal example won't use a lot of space but if you start to use the bells and whistles serde brings you will definitely bloat your target directory. I expect a typical rust project to take around 3–4 gigs for build artifacts depending.
  - wtetzner 2y ago
    > So yeah a minimal example won't use a lot of space but if you start to use the bells and whistles serde brings you will definitely bloat your target directory. Which seems orthogonal to the number of dependencies?
  - hermanradtke 2y ago
    > The first thing that jumps out is that the code example doesn't work. Good catch. I forgot the braces. It does not change the target directory size in a significant way. As for your other comments: sure! We can have a real conversation about rust-analyzer and other serde features (though I am not sure which specific features you are referring to) causing the target directory to increase drastically in size. However, a sensationalist comment that claims the _dependencies_ are 3gb appears to be misleading at best.
cedws 2y ago
Jesus, 78MB is still a lot for such a simple program.

38 2y ago

this 100%. serde is a bloated monster, its sad that its the popular JSON, because all it does is make Rust look bad in my opinion. here are some smaller options: https://lib.rs/crates/humphrey_json https://lib.rs/crates/humphrey_json https://lib.rs/crates/rust_json https://lib.rs/crates/rust_json https://lib.rs/crates/sj https://lib.rs/crates/sj

skitter 2y ago
merde_json should also be relatively small.
INGSOCIALITE 2y ago
can rust use the json-c library?
- makeitshine 2y ago
  I'd assume you could use bindgen and create bindings no problem.
- rapsey 2y ago
  People use rust for its memory safety.
duped 2y ago
The value prop isn't serde_json, it's automatically generated serializers and deserializers for structured data without needing an extra codegen step like with protobufs/capnproto, plus all that machinery decoupled from the actual data format you're reading. It essentially generates a massive amount of code that you need to write anyway, at the cost of code size and compile time. And a lot of people are happy to make that trade off. I wouldn't call that a "bloated monster" because of that. Also, none of those options are alternatives to serde_json, unless you restrict yourself to serde_json::Value - which no one does in practice.
- 38 2y ago
  > none of those options are alternatives to serde_json, unless you restrict yourself to serde_json::Value - which no one does in practice. check your facts, all the above options have derive support, serde is not special in that.

sweca 2y ago

I swear the target folder for literally any project of any scale is at least several GB in size.

gnuvince 2y ago
I get a progress bar when I run `cargo clean` because it's so large.
aldanor 2y ago
For a work project, my recent cargo clean removed 90 GB.
- sweca 2y ago
  jaw drops

purplesyringa 2y ago

> Rust should have Jason built in. I don't think this is a reasonable approach. That's just a way to introduce bloat. Importantly, std does not differ from other crates, except for stability guarantees, so there would be no positive here. All it does is link the library's release cycle to the compiler's. (In fact, rustc-serialize used to be built-in, so Rust tried to go that way.) But also, serde_json isn't large by default. I'm not sure where you are getting those numbers from. serde_json isn't large, serde isn't large. They both have very low MSRVs few other crates support, so in all truth they can't even have many dependencies.

kelnos 2y ago
> That's just a way to introduce bloat. I don't think "bloat" is the issue; as I'm sure you know, Rust programs only contain code for the features of the stdlib they use; if it had JSON support and it wasn't used, the linker would omit that code from the final binary. (Granted, that would make the linker work a little harder.) More at issue is maintenance burden. Data formats come and go over time. In 10 or 20 years when JSON has faded to XML's level of relevance, the Rust stdlib team shouldn't have to continue maintaining a JSON parser.
wiseowise 2y ago
> I don't think this is a reasonable approach. That's just a way to introduce bloat. Can this meme die already? The fact that out of the box install of Rust can’t parse JSON is a joke, and you know it.
- hombre_fatal 2y ago
  I don’t want to wait on language releases to get updates to json, regex, etc. Nor do I want a crappy stdlib impl of something to become widespread just because it comes out of the box like Go’s worst of breed html templating and http “routing”.
  - wiseowise 2y ago
    Somehow Python and JS can get away with json in std lib, but thing that builds binaries can’t? How often does it even need to be updated to parse freaking json?

Klonoar 2y ago

What kind of machine are you developing on that runs out of space that quickly…?

pornel 2y ago

Rust emits unreasonable amount of debug information. It's so freakishly large, I expect it's just a bug. Anything you compile will dump gigabytes into the target folder, but that's not representative of the final product (after stripping the debug info, or at least using a toned-down verbosity setting).

hinkley 2y ago
Does it need a more compact representation of its debug info?
duped 2y ago
Most of your target folder isn't debug info, but stale build artifacts because Cargo doesn't do any garbage collection.
khuey 2y ago
> Rust emits unreasonable amount of debug information. It's so freakishly large, I expect it's just a bug. Rust relies on the linker (via -ffunction-sections and -gc-sections) to delete functions that aren't ever used but the linker isn't capable of removing the corresponding debug info. https://github.com/rust-lang/rust/issues/56068 https://github.com/rust-lang/rust/issues/56068

jwells89 2y ago

Built in JSON encoding/decoding is one of the things I’ve enjoyed about Swift. It’s nice when it’s not necessary to shop around for libraries for common needs like that.

throwup238 2y ago
Almost nobody is shopping around Rust JSON libraries unless they need some specific feature not provided by serde and serde_json. They are the default everyone reaches for.
- bscphil 2y ago
  The fact that a JSON parser is the sort of ecological niche where you naturally get a single best library which does basically what everyone wants it to do with minimal dependencies is exactly the argument for putting it in the stdlib, though.
  - throwup238 2y ago
    That would require moving the entirety of serde and its traits (or equivalent) into the standard library. Considering how long it takes to stabilize basic things in the stdlib, I think that’s a terrible idea. IMO Rust has the best tradeoff between stdlib and ecosystem libraries I’ve ever seen in a (compiled) language, and that includes serde/serde_json.
- wiseowise 2y ago
  The moment you NEED to include a library to parse some basic JSON file - you’ve lost already.
  - ModernMech 2y ago
    Based on your other reply about JSON being a "basic" feature I assume you do a lot of work with JSON. What you need to understand is not everyone works with JSON, and for them it's a feature to not have JSON parsing code in their binaries. It's not a loss for them.

troad 2y ago

Dependency bloat is an issue with Rust in general. The dependency trees for any meaty Rust project quickly become pretty horrifying. Auditing all these dependencies is infeasible, and my level of confidence in a lot of them is fairly low. I worked with Rust for a few years, and with the benefit of a few years' experience, I don't think I'll be touching Rust again until the ecosystem matures a great deal (which will only come with significant corporate adoption), or if I need something for a no-std, no-deps, strictly-a-C-replacement kind of project. (Though Zig might edge out Rust for this use case once it stabilises.)

samatman 2y ago
> Though Zig might edge out Rust for this use case once it stabilises. Zig has a meaningful advantage in the context of this discussion: lazy compilation. The compiler won't even semantically analyze a block of code unless the target requires that to happen. Currently, dependencies are more eager than they really need to be, but making them just as lazy as compilation is on the roadmap. Lazy compilation means no tree shaking is needed, and it means that the actual build-graph of a program can be traced on a fine-grained level. We might not be able to audit a hundred dependencies, but auditing the actual used code from all those dependencies might be more practical. This is well positioned to handle a common pattern: `small-lib` provides some useful stuff, but also has extensions for working with `huge-framework`, and it needs to have `huge-framework` as a dependency to do so. Currently this means that the build system will fetch `huge-framework`, but if we can get it lazy enough, even that won't have to happen unless the code consuming `small-lib` touches the `huge-framework` dependency, which won't happen unless the program itself needs `huge-framework`. Existing build systems don't do such a great job with that kind of structure, and the culprit is eagerness.
echelon 2y ago
> The dependency trees for any meaty Rust project quickly become pretty horrifying. s/Rust// This is really no different from any other language. At least Rust, with Cargo, makes it easy to scan your dependencies. And many notable Rust projects attempt to keep third party dependencies to a minimum. C++ gives you absolutely nothing to work with. Other languages with package managers don't keep dependency trees shallow. You're holding Rust up to a standard that nothing meets.

tinrab 2y ago

From crates.io, `serde` is a 76.4 KiB dependency. And from what I've seen looking through the code, it's pretty minimal.

the__alchemist 2y ago

I just added `serde_json` to my small GUI program. It alone increased the binary size from 6Mb to 8Mb; unacceptable for what it does.