7 ms·
I sped up serde_json strings by 20%
- spense 2y agoawesome that serde moves so quickly. i just ran across simdutf8 and realized the pr for simd-enabled uft8 parsing is coming up on 5 years: https://github.com/rust-lang/rust/issues/68455 https://github.com/rust-lang/rust/issues/68455
- zadokshi 2y agoSerde json has 3gb of dependencies once you do a build for debug and a build for release. Use serde on a few active projects and you run out of disk space. I don’t know why json parsing needs 3gb of dependencies. I’m all for code reuse but Serde for json is a bit of a dogs breakfast when it comes to dependencies. all you need is an exploit in on of those dependencies and half of the rust ecosystem is vulnerable. Rust should have Jason built in.
- ninkendo 2y agoIt has 5 dependencies, one of which is optional, and another is serde itself: https://github.com/serde-rs/json/blob/master/Cargo.toml https://github.com/serde-rs/json/blob/master/Cargo.toml indexmap = { version = "2.2.3", optional = true } itoa = "1.0" memchr = { version = "2", default-features = false } ryu = "1.0" serde = { version = "1.0.194", default-features = false } I don’t think you’re measuring what you think you’re measuring when you say it has 3GB of dependencies. But I can’t say for sure because you don’t provide any evidence for it, you just declare it as true. If I were to guess, I’d say you’re doing a lot of #[derive(Serialize, Deserialize)] and it’s generating tons of code (derive does code generation, after all) and you’re measuring the total size of your target directory after all of this. But this is just a guess… other commenters have shown that a simple build produces code on the order of tens of MB…
- deleted 2y ago[deleted]
- hermanradtke 2y agoPlease show your work. I cannot reproduce "3gb of dependencies". Here is my test: Cargo.toml [package] name = "serde-test" version = "0.1.0" edition = "2021" [dependencies] serde = { version = "1.0.208", features = ["derive"] } serde_json = "1.0.127" src/main.rs use serde::Deserialize; #[derive(Deserialize)] struct Foo { bar: String, } fn main() { let foo: Foo = serde_json::from_str("\"bar\": \"baz\"").unwrap(); println!("{}", foo.bar); } $ cargo build && cargo build --release && du -sh target ... 78M target
- tredre3 2y agoI arrive at almost the same result as you, with 76MB. I've also checked .cargo, .rustup, and my various cache folders (just in case) and haven't found any additional disk usage. OP is clearly mistaken.
- inferiorhuman 2y agoThe first thing that jumps out is that the code example doesn't work. The next thing is that the example merely calls cargo build. Using an IDE of any sort will typically invoke rust-analyzer which will bloat the target directory quite a bit. I've also found that stale build artifacts tend to chew up a lot of space (especially if you're trying to measure the typically smaller release builds). Beyond that, none of the serde features that will tend to generate a ton of code are being used. So yeah a minimal example won't use a lot of space but if you start to use the bells and whistles serde brings you will definitely bloat your target directory. I expect a typical rust project to take around 3–4 gigs for build artifacts depending.
- wtetzner 2y ago> So yeah a minimal example won't use a lot of space but if you start to use the bells and whistles serde brings you will definitely bloat your target directory. Which seems orthogonal to the number of dependencies?
- hermanradtke 2y ago
- 38 2y agothis 100%. serde is a bloated monster, its sad that its the popular JSON, because all it does is make Rust look bad in my opinion. here are some smaller options: https://lib.rs/crates/humphrey_json https://lib.rs/crates/humphrey_json https://lib.rs/crates/rust_json https://lib.rs/crates/rust_json https://lib.rs/crates/sj https://lib.rs/crates/sj
- skitter 2y agomerde_json should also be relatively small.
- INGSOCIALITE 2y agocan rust use the json-c library?
- makeitshine 2y agoI'd assume you could use bindgen and create bindings no problem.
- rapsey 2y agoPeople use rust for its memory safety.
- duped 2y agoThe value prop isn't serde_json, it's automatically generated serializers and deserializers for structured data without needing an extra codegen step like with protobufs/capnproto, plus all that machinery decoupled from the actual data format you're reading. It essentially generates a massive amount of code that you need to write anyway, at the cost of code size and compile time. And a lot of people are happy to make that trade off. I wouldn't call that a "bloated monster" because of that. Also, none of those options are alternatives to serde_json, unless you restrict yourself to serde_json::Value - which no one does in practice.
- 38 2y ago> none of those options are alternatives to serde_json, unless you restrict yourself to serde_json::Value - which no one does in practice. check your facts, all the above options have derive support, serde is not special in that.
- sweca 2y agoI swear the target folder for literally any project of any scale is at least several GB in size.
- purplesyringa 2y ago> Rust should have Jason built in. I don't think this is a reasonable approach. That's just a way to introduce bloat. Importantly, std does not differ from other crates, except for stability guarantees, so there would be no positive here. All it does is link the library's release cycle to the compiler's. (In fact, rustc-serialize used to be built-in, so Rust tried to go that way.) But also, serde_json isn't large by default. I'm not sure where you are getting those numbers from. serde_json isn't large, serde isn't large. They both have very low MSRVs few other crates support, so in all truth they can't even have many dependencies.
- kelnos 2y ago> That's just a way to introduce bloat. I don't think "bloat" is the issue; as I'm sure you know, Rust programs only contain code for the features of the stdlib they use; if it had JSON support and it wasn't used, the linker would omit that code from the final binary. (Granted, that would make the linker work a little harder.) More at issue is maintenance burden. Data formats come and go over time. In 10 or 20 years when JSON has faded to XML's level of relevance, the Rust stdlib team shouldn't have to continue maintaining a JSON parser.
- wiseowise 2y ago> I don't think this is a reasonable approach. That's just a way to introduce bloat. Can this meme die already? The fact that out of the box install of Rust can’t parse JSON is a joke, and you know it.
- hombre_fatal 2y agoI don’t want to wait on language releases to get updates to json, regex, etc. Nor do I want a crappy stdlib impl of something to become widespread just because it comes out of the box like Go’s worst of breed html templating and http “routing”.
- wiseowise 2y agoSomehow Python and JS can get away with json in std lib, but thing that builds binaries can’t? How often does it even need to be updated to parse freaking json?
- Klonoar 2y agoWhat kind of machine are you developing on that runs out of space that quickly…?
- pornel 2y agoRust emits unreasonable amount of debug information. It's so freakishly large, I expect it's just a bug. Anything you compile will dump gigabytes into the target folder, but that's not representative of the final product (after stripping the debug info, or at least using a toned-down verbosity setting).
- hinkley 2y agoDoes it need a more compact representation of its debug info?
- duped 2y agoMost of your target folder isn't debug info, but stale build artifacts because Cargo doesn't do any garbage collection.
- khuey 2y ago> Rust emits unreasonable amount of debug information. It's so freakishly large, I expect it's just a bug. Rust relies on the linker (via -ffunction-sections and -gc-sections) to delete functions that aren't ever used but the linker isn't capable of removing the corresponding debug info. https://github.com/rust-lang/rust/issues/56068 https://github.com/rust-lang/rust/issues/56068
- jwells89 2y agoBuilt in JSON encoding/decoding is one of the things I’ve enjoyed about Swift. It’s nice when it’s not necessary to shop around for libraries for common needs like that.
- throwup238 2y agoAlmost nobody is shopping around Rust JSON libraries unless they need some specific feature not provided by serde and serde_json. They are the default everyone reaches for.
- bscphil 2y agoThe fact that a JSON parser is the sort of ecological niche where you naturally get a single best library which does basically what everyone wants it to do with minimal dependencies is exactly the argument for putting it in the stdlib, though.
- throwup238 2y agoThat would require moving the entirety of serde and its traits (or equivalent) into the standard library. Considering how long it takes to stabilize basic things in the stdlib, I think that’s a terrible idea. IMO Rust has the best tradeoff between stdlib and ecosystem libraries I’ve ever seen in a (compiled) language, and that includes serde/serde_json.
- wiseowise 2y agoThe moment you NEED to include a library to parse some basic JSON file - you’ve lost already.
- ModernMech 2y agoBased on your other reply about JSON being a "basic" feature I assume you do a lot of work with JSON. What you need to understand is not everyone works with JSON, and for them it's a feature to not have JSON parsing code in their binaries. It's not a loss for them.
- troad 2y agoDependency bloat is an issue with Rust in general. The dependency trees for any meaty Rust project quickly become pretty horrifying. Auditing all these dependencies is infeasible, and my level of confidence in a lot of them is fairly low. I worked with Rust for a few years, and with the benefit of a few years' experience, I don't think I'll be touching Rust again until the ecosystem matures a great deal (which will only come with significant corporate adoption), or if I need something for a no-std, no-deps, strictly-a-C-replacement kind of project. (Though Zig might edge out Rust for this use case once it stabilises.)
- samatman 2y ago> Though Zig might edge out Rust for this use case once it stabilises. Zig has a meaningful advantage in the context of this discussion: lazy compilation. The compiler won't even semantically analyze a block of code unless the target requires that to happen. Currently, dependencies are more eager than they really need to be, but making them just as lazy as compilation is on the roadmap. Lazy compilation means no tree shaking is needed, and it means that the actual build-graph of a program can be traced on a fine-grained level. We might not be able to audit a hundred dependencies, but auditing the actual used code from all those dependencies might be more practical. This is well positioned to handle a common pattern: `small-lib` provides some useful stuff, but also has extensions for working with `huge-framework`, and it needs to have `huge-framework` as a dependency to do so. Currently this means that the build system will fetch `huge-framework`, but if we can get it lazy enough, even that won't have to happen unless the code consuming `small-lib` touches the `huge-framework` dependency, which won't happen unless the program itself needs `huge-framework`. Existing build systems don't do such a great job with that kind of structure, and the culprit is eagerness.
- echelon 2y ago> The dependency trees for any meaty Rust project quickly become pretty horrifying. s/Rust// This is really no different from any other language. At least Rust, with Cargo, makes it easy to scan your dependencies. And many notable Rust projects attempt to keep third party dependencies to a minimum. C++ gives you absolutely nothing to work with. Other languages with package managers don't keep dependency trees shallow. You're holding Rust up to a standard that nothing meets.
- tinrab 2y agoFrom crates.io, `serde` is a 76.4 KiB dependency. And from what I've seen looking through the code, it's pretty minimal.
- the__alchemist 2y agoI just added `serde_json` to my small GUI program. It alone increased the binary size from 6Mb to 8Mb; unacceptable for what it does.
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- Sytten 2y agoThe utf-8 tricks make me very nervous since I have seen too many attacks with parser confusion. I for with serde for correctness not speed. I hope this was fuzzed all the way with a bunch of invalid utf-8 strings.
- dwattttt 2y agoLuckily utf-8 structure is _very_ trivial compared to the average parser. Not to say there can't be bugs, but that the internal states of a parser shouldn't be large, and can be exhaustively tested.
- hinkley 2y agoThis is the sort of space where I’d like to see a fuzzer.
- hsbauauvhabzb 2y agoAny bugs you can point to that come to mind of this class?
- eesmith 2y agohttps://en.wikipedia.org/wiki/UTF-8#Invalid_sequences_and_error_handling https://en.wikipedia.org/wiki/UTF-8#Invalid_sequences_and_er... > Many of the first UTF-8 decoders would decode these, ignoring incorrect bits and accepting overlong results. Carefully crafted invalid UTF-8 could make them either skip or create ASCII characters such as NUL, slash, or quotes. Invalid UTF-8 has been used to bypass security validations in high-profile products including Microsoft's IIS web server[26] and Apache's Tomcat servlet container.[27] RFC 3629 states "Implementations of the decoding algorithm MUST protect against decoding invalid sequences."
- zorked 2y ago> Teaching to _think_ is just as important as teaching to code, but this is seldom done Oh, the arrogance of thinking that the other person doesn't think.
- loa_in_ 2y agoIt isn't what author argues at all. It's about teaching how to think: that you have to do some research and it's not a step you can skip; having done your research doesn't free you from having to draw your own conclusions too. Skipping either of those steps is easy but wrong.
- ramon156 2y ago> is seldom done It's this part
- purplesyringa 2y agoPerhaps the wording was off on my part. What I meant is not that people don't think, it's that people seldom teach others to think, at least in web articles. Most posts of such format I have seen are "we did this and got this", not "we tried this, it failed because of this, then we figured out something else might work and it worked after these modifications".
- chipdart 2y ago> Most posts of such format I have seen are "we did this and got this", not "we tried this, it failed because of this, then we figured out something else might work and it worked after these modifications". That doesn't resemble anything remotely related to teaching how to think. You're just logging your trial and error process, which is exactly what each and every single developer goes through on a daily basis. What exactly do you think other developers do?
- __s 2y agoAgain, they aren't saying developers don't think. They're talking about blogging I had this issue at PeerDB where we'd blog about some dev, when I wrote it'd be a stream of consciousness trying to communicate the mood, frustration, & flailing process. It wouldn't get published, in favor of blogs with clearer product messaging
- s_Hogg 2y agoVery strong jart feel about this person's blog, that was a nice read > We would need to reinvent the wheel, but this is quite neat if you think about it. Is this real or ironic though? I read it and started laughing at the writer but the rest of the page seems quite heavy on self-deprecation
- vecplane 2y agoWhat does jart mean?
- IggleSniggle 2y agoHN username for an extremely talented software engineer and software-engineering communicator, justine.lol. Probably most known around here for her cross-platform C code (cosmopolitan) and relatedly redbean, a zip file and tool that is also an executable file-server-file that hosts itself and can produce other such self-hosting cross platform executable zip file servers.
- wffurr 2y agoA commonly posted author: https://justine.lol/ https://justine.lol/
- wtetzner 2y agoI think it means the approach is quite neat, not the fact that it requires reinventing the wheel.