11 ms·

This isn’t the way to speed up Rust compile times

128 points by cpach 3y ago

deleted 3y ago
[deleted]
titaniumtown 3y ago
very different format to the blog with different characters chiming in. interesting!
phamilton 3y ago
The Pre-RFC that came out of this situation is pretty interesting: https://internals.rust-lang.org/t/pre-rfc-sandboxed-deterministic-reproducible-efficient-wasm-compilation-of-proc-macros/19359 https://internals.rust-lang.org/t/pre-rfc-sandboxed-determin... Some points I hadn't considered: - Current compile times aren't too bad, but compile times are a factor in how macros are developed. Making that less of an issue will likely make the macro ecosystem richer and more ambitious. - Precompiled binaries can be compiled with optimizations, which anecdotally overcome the potential perf hit from running in WASM - Nondeterminism in macros (especially randomness) has bit a lot of people and a more controlled execution environment is very beneficial - Sandboxing doesn't have to be WASM, it's just the most capable, currently available option. - Comments on the pre-RFC are exploring solving this in rustup/cargo instead of crates.io and that is also promising. In general, I appreciate how this is creating awareness and driving exploration of solutions. In particular I really like the notion that there is currently implicit pressure on macro authors to make them somewhat quick to compile and releasing that pressure will enable innovation.
- mid-kid 3y ago
  Please don't ship wasm binaries in cargo please don't ship wasm binaries in cargo please don't ship wasm binaries in cargo please don't ship wasm binaries in cargo please don-
  - MrJohz 3y ago
    Why not? WASM with optimisations will mostly run faster than binaries without optimisations, which is the current strategy. Plus you won't need the compile step that's necessary at the moment. Distributing arbitrary binaries right now is bad because it's not clear what those binaries contain, but the WASM environment is completely sandboxed, so it wouldn't be possible for them to read files they weren't meant to or phone home with telemetry. The Cargo project already does some amount of automatic building e.g. of docs pages - this could be extended to proc macro crates to ensure that the uploaded binaries must be built from the uploaded source files. Alternatively, making sure that reproducible binaries are standard would help a lot from a trust perspective. This isn't suitable for every situation, but as a tool to reduce the compile times of some of the bigger and slower macro crates, it sounds like a great option to have.
brigadier132 3y ago
I know a lot of people feel very strongly about this thing and thankfully Dtolnay reversed course but if there is anyone out there that has social media mob mentality please don't harass him. A lot of the packages I use have his name associated with them and if we get the classic "I'm leaving open source because of the toxicity" and he is the author the rust community will be screwed.
- rewmie 3y ago
  Let's see if the Rust community learned anything from the actix-web fiasco.
  - deleted 3y ago
    [deleted]
  - Dowwie 3y ago
    Rage blogging is never by contributors
jamamp 3y ago
Swift's `Codable`[0] system seems very similar to serde in this regard. Structs that you define can be marked with the Codable protocol, and the Swift compiler automatically generates code during compilation that encodes from and decodes to the structs properties, with the option for you to customize it using CodingKeys for different properties names or completely custom coding behavior. It seems built-in to Swift, as opposed to a dynamically executed crate like with serde. I wonder how it's implemented in Swift and if it leads to any significant slowdowns. [0] https://developer.apple.com/documentation/foundation/archives_and_serialization/encoding_and_decoding_custom_types https://developer.apple.com/documentation/foundation/archive...
- raphlinus 3y ago
  Last I checked[1], Swift JSON performance was terrible, well over an order of magnitude slower than serde. It's possible it's been improved since then but if so I didn't find any announcement. [1]: https://github.com/xi-editor/xi-mac/issues/102 https://github.com/xi-editor/xi-mac/issues/102
  - viktorcode 3y ago
    The problem wasn't Swift Codable, the problem was Apple's implementation of JSONEncoder / Decoder on top of Obj-C runtime, which created unnecessary round trips. It was only fully addressed in Swift 5.9.
    - plorkyeran 3y ago
      Both are problems, and it's far from fully addressed in Swift 5.9. The new Swift implementation of JSONEncoding is "only" 2-5x faster than the old one, so while it's a big improvement it's still very slow. Codable itself is also very performance-hostile, and a no-op encoder which fully walks the input but doesn't produce any output will be slower than serde producing json.
      - viktorcode 3y ago
        Can you get into more details on why do you think Codable is performance hostile?
dundarious 3y ago
On topic: I agree that upstream-provided 3rd party binaries are not a good solution to this particular problem. That is the type of solution that works in a closed environment like a corporate CI system, but it should not be the default. Off topic: I don't understand why the article repeatedly says the Rust compiler or procedural macros are already fast, even "plenty fast". Aren't they slower or about as slow as C++, which is notorious for being frustratingly slow, especially for local, non-distributed builds? "Plenty fast" would mean fast enough for some simple use-case. e.g., "1G ethernet is plenty fast for a home network, but data centers can definitely benefit from faster links". i.e., fast enough you won't notice anything faster when doing your average user's daily activities, but certainly not fast enough for all users or activities. The Rust compiler is then in no way "plenty fast". It has many benefits, and lots of hard work has gone into it, even optimizing it, but everyone would notice and benefit from it being any amount faster.
- rewmie 3y ago
  > On topic: I agree that 3rd party binaries are not a good solution to this particular problem. That is the type of solution that works in a closed environment like a corporate CI system, but it should not be the default. Why? I mean, OS distributions making available whole catalogs of prebuilt binaries is a time-tested solution solution that's in place for decades. What leads you to believe that this is suddenly undesirable?
  - dundarious 3y ago
    I think there's a very obvious distinction between the kind of "3rd party" that is your chosen OS distribution and the kind of 3rd party that is "some person on github or crates.io", and I didn't feel the need to clarify this as I thought it was obvious. I've clarified it to "upstream-provided 3rd party" in the original post.
    - formerly_proven 3y ago
      Either way you're running their code, I don't understand the outrage here, frankly. People who look at the code they're running and the changes they're pulling in of course noticed this, but I seriously doubt even 1% of the rageposters are in that set of rust users.
incrudible 3y ago
I like the Go style better. I probably will not ever care about more than one or two encodings and being explicit is an upside in that regard. I do however really care about compile times. If performance really matters, I would not go for a generic serialization approach in the first place.
- 1letterunixname 3y ago
  Apples and oranges. Go doesn't care about release or dev builds. Nothing about it applies here. Rust needs specific toolchain improvements to cache partial compilations for reuse. sccache can cache some results, but it's not a comprehensive solution. What specific proposal would replace serde? I'd like to know.
alphanullmeric 3y ago
The issue is the reliance on macros in the first place, which half the time are just a shitty substitute for overloading, variadics and inheritance that can’t be nested, doesn’t have editor highlighting, and takes longer to compile. Macros should be a last resort and not a first resort, “rust macros are better” isn’t a good counter argument. Rust’s big government philosophy of forcing design choices onto the programmer yet again comes back to bite it.
- pie_flavor 3y ago
  Please describe how you would implement the functionality of the serde::Serialize macro with overloading, variadics, and inheritance.
  - alphanullmeric 3y ago
    I’m referring to things like println, which isn’t a macro in any other language. If you want to stick with the serde example, then look at the diy name mangling in the deserializer trait: https://docs.rs/serde/latest/serde/trait.Deserializer.html https://docs.rs/serde/latest/serde/trait.Deserializer.html Or the copy and pasting in Axum: https://github.com/tokio-rs/axum/blob/24f0f3eae8054c7a495cd364087f2dd7fa8b87e0/axum-extra/src/handler/mod.rs#L142 https://github.com/tokio-rs/axum/blob/24f0f3eae8054c7a495cd3...
    - nightfly 3y ago
      Println is also much more complicated and expensive at runtime in "any other language"
      - alphanullmeric 3y ago
        Except for C++.
        pie_flavor 3y ago
        You mean the language which, in every major implementation, that particular function is a macro?
charcircuit 3y ago
>If we're going to be trusting some random guy's binaries, I think we are in the right to demand that it is byte-for-byte reproducible on commodity hardware without having to reverse-engineer the build process and figure out which nightly version of the compiler is being used to compile this binary blob that will be run everywhere. David Tolnay is not a random guy.
- CUViper 3y ago
  Then extrapolate "random guy trying to follow dtolnay's lead."
  - IshKebab 3y ago
    Well then complain about that, not this. And anyway it doesn't really matter. As soon as you use anyone's crates you're more or less completely trusting them. It's not difficult to hide malware in a Rust crate even if you don't ship a binary. And... come on. David Tolnay came up with Watt. He's clearly not intending to ship a binary forever - the long term solution is WASM. This author comes across as an annoying naysayer - everything is impossible, even things that have already been done like WASI.
    - est31 3y ago
      wasm is also a binary format, and I wouldn't like it to be included into the .crate file either. The .crate files should ideally only include source code, to preserve the source first nature of crates.io (even if it's never been officially confirmed). Rustdoc is also automatically built by docs.rs and nobody distributes it in their .crate file. I think the same should be done for wasm proc macros, too: they should be built by public infrastructure, and then people can opt into using binaries provided by that infrastructure to do their development, and if they want also opt towards using native binaries instead of wasm. But the binaries, including wasm, should only be a cache.
      - IshKebab 3y ago
        > I think the same should be done for wasm proc macros, too: they should be built by public infrastructure, and then people can opt into using binaries provided by that infrastructure to do their development That's obviously how it would work. Read dtonlay's proposal. Crates.io would compile the WASM.
pie_flavor 3y ago
If you skipped over the header, like I did originally, it bears repeating: This change was reverted a couple days ago.
jeremyjh 3y ago
[flagged]
- nightfly 3y ago
  What ad are you talking about?
  - jeremyjh 3y ago
    If you have an ad-blocker enabled, the first paragraph of the page asks you to turn it off or donate. I have no problem with creators being compensated for their work, or asking for that compensation, but I found it confusing.
    - SushiHippie 3y ago
      uBlock origin also blocks the adblock detection, so I couldn't see neither the ads nor the paragraph that talks about disabling the blocker.
- dang 3y ago
  "Please don't pick the most provocative thing in an article or post to complain about in the thread. Find something interesting to respond to instead." "Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
galkk 3y ago
Totally unrelated. But generated picture at the beginning of the article, the one with space needle, robs me very wrong way, as real landscape doesn’t look like it. It’s like seeing Golden Gate Bridge in New York.
- xena 3y ago
  Author here. Yeah, the model I use (Anything v3) has a really weird way of generating the Space Needle in particular. I'm originally from Seattle and I really miss the pacific northwest, so I like to put the Space Needle in my AI generations because of how much it can stick out or clash with the rest of the image. In the future I'm gonna use more of my own photography like this: https://pony.social/@cadey/110956174300162810 https://pony.social/@cadey/110956174300162810, I'm just building up a library of viable photos.
  - scubbo 3y ago
    What's the story behind "-amputee" in the arguments? Were such images so common that it was necessary to exclude them!?
    - xena 3y ago
      If you don't include it, sometimes you get otherwise perfect images that have only half of a leg in a really distracting way. I usually prune negative arguments (my usual prompts are close to a kilobyte long including negative prompt elements), but I thought it would be amusing to leave it in this time.
      - scubbo 3y ago
        Huh. You inspired me to finally get around to installing a local Midjourney-like[0], and, yeah, what do you know - the first result I got from those prompts _without_ the `-amputee` negative prompt was this horrifying (though SFW, despite what Imgur claims) monstrosity[1] [0] https://github.com/AUTOMATIC1111/stable-diffusion-webui https://github.com/AUTOMATIC1111/stable-diffusion-webui [1] https://imgur.com/a/bMp5aAB https://imgur.com/a/bMp5aAB
Guvante 3y ago
In case anyone missed the first line like I did this has been resolved and no longer works like this.
lamontcg 3y ago
> However, this means that the most commonly used crate is shipping an arbitrary binary for production builds without any way to opt-out. The nokogiri rubygem has been shipping a prebuilt windows dll for probably more than a decade. The sun has not imploded into a black hole at any point in the meantime. > If we're going to be trusting some random guy's binaries, I think we are in the right to demand that it is byte-for-byte reproducible on commodity hardware without having to reverse-engineer the build process and figure out which nightly version of the compiler is being used to compile this binary blob that will be run everywhere. I have some really, really fucking bad news about the O/S distros most of you all use...
- yw3410 3y ago
  The author uses NixOS.
  - chungy 3y ago
    Debian and Fedora have likewise both been pushing reproducible builds.
    - lamontcg 3y ago
      I used to use a FreeBSD desktop at work back in like 2002. And that is a reasonable thing for those projects to try to achieve. Still we've been downloading binaries for decades and the world hasn't ended, this issue never warranted people treating it like nuclear war was imminent and hurling abuse into Dtolnay's github issues like he was a child pornographer. There's a way to have a discussion about this being the wrong direction without doing that. And I'd encourage everyone to take an honest audit of how many precompiled binaries are in their lives, even if they've switched to NixOS and how much stuff they trust downloading from the internet. Very few people achieve an RMS-level of purity in their technological lives.
      - jacquesm 3y ago
        This is all true, but even if you don't care about reproducible builds that doesn't mean that nobody cares about reproducible builds. And the people that do care about such things care about them very much. To the point that if you don't offer it to them that they'll go somewhere else.
KolmogorovComp 3y ago
Note that the pre-compiled binary blob the blog post is referring to has since been removed [0]. [0]: https://github.com/serde-rs/serde/pull/2590 https://github.com/serde-rs/serde/pull/2590
- xena 3y ago
  I thought I made that pretty clear in my article. Do I need to bring back the blink tag or something to make it more obvious in the future?
  - KolmogorovComp 3y ago
    I indeed missed the header, as other readers hear. I'd advise against using the characters' bubbles to provide important heads-up, as they're often used for additional insights or humorous opinion, and so can be easily missed by quick readers.
  - gpm 3y ago
    I missed it because the header to me looks like > Something about ads > Title > Something I skipped > Something about patreon I dismissed it as "yet another part of the headers" and not as "part of the article". I mean, to be fair, I was already aware it was removed from serde, but I'm pretty sure I didn't notice that box in the article the first time I read through it.
  - capableweb 3y ago
    The top of the article looks like a screenshot of a chat, where someone said something, then some text and then the same person said something. I basically skipped that whole part to look for the article beginning, and when I didn't find the title, I skimmed back up. The title is kind of hidden as it is right now, and the "notice" on the top doesn't look like a notice. Maybe try making it more different than other elements on the website, and put it below the article title rather than above?
  - deleted 3y ago
    [deleted]
  - 3y ago
DisgracePlacard 3y ago
> If we're going to be trusting some random guy's binaries, I think we are in the right to demand that it is byte-for-byte reproducible on commodity hardware I don't think anyone has a right to demand anything of the project. The MIT license specifically has the whole "THIS SOFTWARE IS PROVIDED AS IS" spiel for a reason. Thinking that you can demand anything of an open source developer who, afaik, has no responsibilities in relationship towards you, is a rather toxic mindset that should be kept out of open source.
- saagarjha 3y ago
  That’s not true. I can, for example, demand that an open source project not bundle malware. The maintainer may not listen to me but I reserve the right to demand it and I don’t see how this is toxic at all.
- pessimizer 3y ago
  The demands are provided "as is," too, so you have no right to complain about them.
- zer8k 3y ago
  "No one has any right to complain because muh license agreement" What a tiring argument. Why complain about anything? Why do anything differently? Maybe it's because people have invested time, money, and effort into supporting and using the project and because of your stupid decision they now have to do more work. There were a lot of comments about people showing up that monday to pin+vendor the old SerDe. Others planned to remove it entirely in favor of other libraries. Almost like actions have consequences even if your "license agreement" says otherwise. > is a rather toxic mindset that should be kept out of open source. I wish I lived in your echo chamber where everyone who uses your stuff is just totally, like, accepting of your garbage ideas. The precompiled binary idea was a classic garbage idea. Refusing to make it reproducible was just the corn on top of the cowpie. The author made a major screw up, doubled down, tripled down, and then finally gave in. The author has no idea what scale and scope of project this is used in. They also clearly did no evaluation on potential damage OR ask for feedback before moving it into mainline. For a library with 3M+ downloads this was the what third? Fourth? Classic ego-driven folly. You know what else shouldn't exist in open source? Ego tripping morons.
say_it_as_it_is 3y ago
[flagged]
- dang 3y ago
  Please don't do this here.
1letterunixname 3y ago
Caching needs to be brought to the ecosystem rather than point non-solutions. While sccache is a partial solution available now from the outside, the toolchain needs adjustment internally to deduplicate work.
- epage 3y ago
  I've been exploring local, cross-project caching for cargo and my biggest concern is users won't get enough cache hits. With maximal versions and a arbitrary set of dependencies being updated when adding a new dependency. all projects on a system can end up in very different states.
- pjmlp 3y ago
  That is the whole issue, as scache is more like a workaround versus a proper solution, supported out of the box in cargo, like in Java, .NET, C++ (vcpkg and cargo do bin libs), node,...
user6723 3y ago
[flagged]
- xena 3y ago
  [flagged]
swsieber 3y ago
There's a few other options that I think in combination would make things much better. 1) Make using proc macros in your own application code faster: Encourage better macro re-export hygiene. Basically, one of the reasons serde + serde_derive is so slow is because for serde re-exports serde_derive to compile before it itself can be built. A solution (I didn't come up with this myself) would be to have a crate that re-exports serde and serde_derive together; see top comment herehttps://www.reddit.com/r/rust/comments/1602eah/associated_proc_macro_pattern/ https://www.reddit.com/r/rust/comments/1602eah/associated_pr... 2) Make using libraries that use proc macros faster: Publish pre-expanded source code, such that no proc macros run for dependencies. Some work would have to be done around conditional compilation to make things just work (TM), but I think it could be done. (Who knows, maybe it'd be really hard to make the compiler deal with expanding proc macros on stuff behind conditional values. Also, there'd be issues regarding macro hygiene. Both solvable, but I'm not sure how much effort they'd take).
anacrolix 3y ago
this article style is very hard to read.

128 points by cpach 3y ago

deleted 3y ago

[deleted]

titaniumtown 3y ago

very different format to the blog with different characters chiming in. interesting!

phamilton 3y ago

The Pre-RFC that came out of this situation is pretty interesting: https://internals.rust-lang.org/t/pre-rfc-sandboxed-deterministic-reproducible-efficient-wasm-compilation-of-proc-macros/19359 https://internals.rust-lang.org/t/pre-rfc-sandboxed-determin... Some points I hadn't considered: - Current compile times aren't too bad, but compile times are a factor in how macros are developed. Making that less of an issue will likely make the macro ecosystem richer and more ambitious. - Precompiled binaries can be compiled with optimizations, which anecdotally overcome the potential perf hit from running in WASM - Nondeterminism in macros (especially randomness) has bit a lot of people and a more controlled execution environment is very beneficial - Sandboxing doesn't have to be WASM, it's just the most capable, currently available option. - Comments on the pre-RFC are exploring solving this in rustup/cargo instead of crates.io and that is also promising. In general, I appreciate how this is creating awareness and driving exploration of solutions. In particular I really like the notion that there is currently implicit pressure on macro authors to make them somewhat quick to compile and releasing that pressure will enable innovation.

mid-kid 3y ago
Please don't ship wasm binaries in cargo please don't ship wasm binaries in cargo please don't ship wasm binaries in cargo please don't ship wasm binaries in cargo please don-
- MrJohz 3y ago
  Why not? WASM with optimisations will mostly run faster than binaries without optimisations, which is the current strategy. Plus you won't need the compile step that's necessary at the moment. Distributing arbitrary binaries right now is bad because it's not clear what those binaries contain, but the WASM environment is completely sandboxed, so it wouldn't be possible for them to read files they weren't meant to or phone home with telemetry. The Cargo project already does some amount of automatic building e.g. of docs pages - this could be extended to proc macro crates to ensure that the uploaded binaries must be built from the uploaded source files. Alternatively, making sure that reproducible binaries are standard would help a lot from a trust perspective. This isn't suitable for every situation, but as a tool to reduce the compile times of some of the bigger and slower macro crates, it sounds like a great option to have.

brigadier132 3y ago

I know a lot of people feel very strongly about this thing and thankfully Dtolnay reversed course but if there is anyone out there that has social media mob mentality please don't harass him. A lot of the packages I use have his name associated with them and if we get the classic "I'm leaving open source because of the toxicity" and he is the author the rust community will be screwed.

rewmie 3y ago
Let's see if the Rust community learned anything from the actix-web fiasco.
- deleted 3y ago
  [deleted]
- Dowwie 3y ago
  Rage blogging is never by contributors

jamamp 3y ago

Swift's `Codable`[0] system seems very similar to serde in this regard. Structs that you define can be marked with the Codable protocol, and the Swift compiler automatically generates code during compilation that encodes from and decodes to the structs properties, with the option for you to customize it using CodingKeys for different properties names or completely custom coding behavior. It seems built-in to Swift, as opposed to a dynamically executed crate like with serde. I wonder how it's implemented in Swift and if it leads to any significant slowdowns. [0] https://developer.apple.com/documentation/foundation/archives_and_serialization/encoding_and_decoding_custom_types https://developer.apple.com/documentation/foundation/archive...

raphlinus 3y ago
Last I checked[1], Swift JSON performance was terrible, well over an order of magnitude slower than serde. It's possible it's been improved since then but if so I didn't find any announcement. [1]: https://github.com/xi-editor/xi-mac/issues/102 https://github.com/xi-editor/xi-mac/issues/102
- viktorcode 3y ago
  The problem wasn't Swift Codable, the problem was Apple's implementation of JSONEncoder / Decoder on top of Obj-C runtime, which created unnecessary round trips. It was only fully addressed in Swift 5.9.
  - plorkyeran 3y ago
    Both are problems, and it's far from fully addressed in Swift 5.9. The new Swift implementation of JSONEncoding is "only" 2-5x faster than the old one, so while it's a big improvement it's still very slow. Codable itself is also very performance-hostile, and a no-op encoder which fully walks the input but doesn't produce any output will be slower than serde producing json.
    - viktorcode 3y ago
      Can you get into more details on why do you think Codable is performance hostile?

dundarious 3y ago

On topic: I agree that upstream-provided 3rd party binaries are not a good solution to this particular problem. That is the type of solution that works in a closed environment like a corporate CI system, but it should not be the default. Off topic: I don't understand why the article repeatedly says the Rust compiler or procedural macros are already fast, even "plenty fast". Aren't they slower or about as slow as C++, which is notorious for being frustratingly slow, especially for local, non-distributed builds? "Plenty fast" would mean fast enough for some simple use-case. e.g., "1G ethernet is plenty fast for a home network, but data centers can definitely benefit from faster links". i.e., fast enough you won't notice anything faster when doing your average user's daily activities, but certainly not fast enough for all users or activities. The Rust compiler is then in no way "plenty fast". It has many benefits, and lots of hard work has gone into it, even optimizing it, but everyone would notice and benefit from it being any amount faster.

rewmie 3y ago
> On topic: I agree that 3rd party binaries are not a good solution to this particular problem. That is the type of solution that works in a closed environment like a corporate CI system, but it should not be the default. Why? I mean, OS distributions making available whole catalogs of prebuilt binaries is a time-tested solution solution that's in place for decades. What leads you to believe that this is suddenly undesirable?
- dundarious 3y ago
  I think there's a very obvious distinction between the kind of "3rd party" that is your chosen OS distribution and the kind of 3rd party that is "some person on github or crates.io", and I didn't feel the need to clarify this as I thought it was obvious. I've clarified it to "upstream-provided 3rd party" in the original post.
  - formerly_proven 3y ago
    Either way you're running their code, I don't understand the outrage here, frankly. People who look at the code they're running and the changes they're pulling in of course noticed this, but I seriously doubt even 1% of the rageposters are in that set of rust users.

incrudible 3y ago

I like the Go style better. I probably will not ever care about more than one or two encodings and being explicit is an upside in that regard. I do however really care about compile times. If performance really matters, I would not go for a generic serialization approach in the first place.

1letterunixname 3y ago
Apples and oranges. Go doesn't care about release or dev builds. Nothing about it applies here. Rust needs specific toolchain improvements to cache partial compilations for reuse. sccache can cache some results, but it's not a comprehensive solution. What specific proposal would replace serde? I'd like to know.

alphanullmeric 3y ago

The issue is the reliance on macros in the first place, which half the time are just a shitty substitute for overloading, variadics and inheritance that can’t be nested, doesn’t have editor highlighting, and takes longer to compile. Macros should be a last resort and not a first resort, “rust macros are better” isn’t a good counter argument. Rust’s big government philosophy of forcing design choices onto the programmer yet again comes back to bite it.

pie_flavor 3y ago
Please describe how you would implement the functionality of the serde::Serialize macro with overloading, variadics, and inheritance.
- alphanullmeric 3y ago
  I’m referring to things like println, which isn’t a macro in any other language. If you want to stick with the serde example, then look at the diy name mangling in the deserializer trait: https://docs.rs/serde/latest/serde/trait.Deserializer.html https://docs.rs/serde/latest/serde/trait.Deserializer.html Or the copy and pasting in Axum: https://github.com/tokio-rs/axum/blob/24f0f3eae8054c7a495cd364087f2dd7fa8b87e0/axum-extra/src/handler/mod.rs#L142 https://github.com/tokio-rs/axum/blob/24f0f3eae8054c7a495cd3...
  - nightfly 3y ago
    Println is also much more complicated and expensive at runtime in "any other language"
    - alphanullmeric 3y ago
      Except for C++.
      - pie_flavor 3y ago
        You mean the language which, in every major implementation, that particular function is a macro?

charcircuit 3y ago

>If we're going to be trusting some random guy's binaries, I think we are in the right to demand that it is byte-for-byte reproducible on commodity hardware without having to reverse-engineer the build process and figure out which nightly version of the compiler is being used to compile this binary blob that will be run everywhere. David Tolnay is not a random guy.

CUViper 3y ago
Then extrapolate "random guy trying to follow dtolnay's lead."
- IshKebab 3y ago
  Well then complain about that, not this. And anyway it doesn't really matter. As soon as you use anyone's crates you're more or less completely trusting them. It's not difficult to hide malware in a Rust crate even if you don't ship a binary. And... come on. David Tolnay came up with Watt. He's clearly not intending to ship a binary forever - the long term solution is WASM. This author comes across as an annoying naysayer - everything is impossible, even things that have already been done like WASI.
  - est31 3y ago
    wasm is also a binary format, and I wouldn't like it to be included into the .crate file either. The .crate files should ideally only include source code, to preserve the source first nature of crates.io (even if it's never been officially confirmed). Rustdoc is also automatically built by docs.rs and nobody distributes it in their .crate file. I think the same should be done for wasm proc macros, too: they should be built by public infrastructure, and then people can opt into using binaries provided by that infrastructure to do their development, and if they want also opt towards using native binaries instead of wasm. But the binaries, including wasm, should only be a cache.
    - IshKebab 3y ago
      > I think the same should be done for wasm proc macros, too: they should be built by public infrastructure, and then people can opt into using binaries provided by that infrastructure to do their development That's obviously how it would work. Read dtonlay's proposal. Crates.io would compile the WASM.

pie_flavor 3y ago

If you skipped over the header, like I did originally, it bears repeating: This change was reverted a couple days ago.

jeremyjh 3y ago

[flagged]

nightfly 3y ago
What ad are you talking about?
- jeremyjh 3y ago
  If you have an ad-blocker enabled, the first paragraph of the page asks you to turn it off or donate. I have no problem with creators being compensated for their work, or asking for that compensation, but I found it confusing.
  - SushiHippie 3y ago
    uBlock origin also blocks the adblock detection, so I couldn't see neither the ads nor the paragraph that talks about disabling the blocker.
dang 3y ago
"Please don't pick the most provocative thing in an article or post to complain about in the thread. Find something interesting to respond to instead." "Please don't complain about tangential annoyances—e.g. article or website formats, name collisions, or back-button breakage. They're too common to be interesting." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

galkk 3y ago

Totally unrelated. But generated picture at the beginning of the article, the one with space needle, robs me very wrong way, as real landscape doesn’t look like it. It’s like seeing Golden Gate Bridge in New York.

xena 3y ago
Author here. Yeah, the model I use (Anything v3) has a really weird way of generating the Space Needle in particular. I'm originally from Seattle and I really miss the pacific northwest, so I like to put the Space Needle in my AI generations because of how much it can stick out or clash with the rest of the image. In the future I'm gonna use more of my own photography like this: https://pony.social/@cadey/110956174300162810 https://pony.social/@cadey/110956174300162810, I'm just building up a library of viable photos.
- scubbo 3y ago
  What's the story behind "-amputee" in the arguments? Were such images so common that it was necessary to exclude them!?
  - xena 3y ago
    If you don't include it, sometimes you get otherwise perfect images that have only half of a leg in a really distracting way. I usually prune negative arguments (my usual prompts are close to a kilobyte long including negative prompt elements), but I thought it would be amusing to leave it in this time.
    - scubbo 3y ago
      Huh. You inspired me to finally get around to installing a local Midjourney-like[0], and, yeah, what do you know - the first result I got from those prompts _without_ the `-amputee` negative prompt was this horrifying (though SFW, despite what Imgur claims) monstrosity[1] [0] https://github.com/AUTOMATIC1111/stable-diffusion-webui https://github.com/AUTOMATIC1111/stable-diffusion-webui [1] https://imgur.com/a/bMp5aAB https://imgur.com/a/bMp5aAB

Guvante 3y ago

In case anyone missed the first line like I did this has been resolved and no longer works like this.

lamontcg 3y ago

> However, this means that the most commonly used crate is shipping an arbitrary binary for production builds without any way to opt-out. The nokogiri rubygem has been shipping a prebuilt windows dll for probably more than a decade. The sun has not imploded into a black hole at any point in the meantime. > If we're going to be trusting some random guy's binaries, I think we are in the right to demand that it is byte-for-byte reproducible on commodity hardware without having to reverse-engineer the build process and figure out which nightly version of the compiler is being used to compile this binary blob that will be run everywhere. I have some really, really fucking bad news about the O/S distros most of you all use...

yw3410 3y ago
The author uses NixOS.
- chungy 3y ago
  Debian and Fedora have likewise both been pushing reproducible builds.
  - lamontcg 3y ago
    I used to use a FreeBSD desktop at work back in like 2002. And that is a reasonable thing for those projects to try to achieve. Still we've been downloading binaries for decades and the world hasn't ended, this issue never warranted people treating it like nuclear war was imminent and hurling abuse into Dtolnay's github issues like he was a child pornographer. There's a way to have a discussion about this being the wrong direction without doing that. And I'd encourage everyone to take an honest audit of how many precompiled binaries are in their lives, even if they've switched to NixOS and how much stuff they trust downloading from the internet. Very few people achieve an RMS-level of purity in their technological lives.
    - jacquesm 3y ago
      This is all true, but even if you don't care about reproducible builds that doesn't mean that nobody cares about reproducible builds. And the people that do care about such things care about them very much. To the point that if you don't offer it to them that they'll go somewhere else.

KolmogorovComp 3y ago

Note that the pre-compiled binary blob the blog post is referring to has since been removed [0]. [0]: https://github.com/serde-rs/serde/pull/2590 https://github.com/serde-rs/serde/pull/2590

xena 3y ago
I thought I made that pretty clear in my article. Do I need to bring back the blink tag or something to make it more obvious in the future?
- KolmogorovComp 3y ago
  I indeed missed the header, as other readers hear. I'd advise against using the characters' bubbles to provide important heads-up, as they're often used for additional insights or humorous opinion, and so can be easily missed by quick readers.
- gpm 3y ago
  I missed it because the header to me looks like > Something about ads > Title > Something I skipped > Something about patreon I dismissed it as "yet another part of the headers" and not as "part of the article". I mean, to be fair, I was already aware it was removed from serde, but I'm pretty sure I didn't notice that box in the article the first time I read through it.
- capableweb 3y ago
  The top of the article looks like a screenshot of a chat, where someone said something, then some text and then the same person said something. I basically skipped that whole part to look for the article beginning, and when I didn't find the title, I skimmed back up. The title is kind of hidden as it is right now, and the "notice" on the top doesn't look like a notice. Maybe try making it more different than other elements on the website, and put it below the article title rather than above?
- deleted 3y ago
  [deleted]
- 3y ago

DisgracePlacard 3y ago

> If we're going to be trusting some random guy's binaries, I think we are in the right to demand that it is byte-for-byte reproducible on commodity hardware I don't think anyone has a right to demand anything of the project. The MIT license specifically has the whole "THIS SOFTWARE IS PROVIDED AS IS" spiel for a reason. Thinking that you can demand anything of an open source developer who, afaik, has no responsibilities in relationship towards you, is a rather toxic mindset that should be kept out of open source.

saagarjha 3y ago
That’s not true. I can, for example, demand that an open source project not bundle malware. The maintainer may not listen to me but I reserve the right to demand it and I don’t see how this is toxic at all.
pessimizer 3y ago
The demands are provided "as is," too, so you have no right to complain about them.
zer8k 3y ago
"No one has any right to complain because muh license agreement" What a tiring argument. Why complain about anything? Why do anything differently? Maybe it's because people have invested time, money, and effort into supporting and using the project and because of your stupid decision they now have to do more work. There were a lot of comments about people showing up that monday to pin+vendor the old SerDe. Others planned to remove it entirely in favor of other libraries. Almost like actions have consequences even if your "license agreement" says otherwise. > is a rather toxic mindset that should be kept out of open source. I wish I lived in your echo chamber where everyone who uses your stuff is just totally, like, accepting of your garbage ideas. The precompiled binary idea was a classic garbage idea. Refusing to make it reproducible was just the corn on top of the cowpie. The author made a major screw up, doubled down, tripled down, and then finally gave in. The author has no idea what scale and scope of project this is used in. They also clearly did no evaluation on potential damage OR ask for feedback before moving it into mainline. For a library with 3M+ downloads this was the what third? Fourth? Classic ego-driven folly. You know what else shouldn't exist in open source? Ego tripping morons.

say_it_as_it_is 3y ago

[flagged]

dang 3y ago
Please don't do this here.

1letterunixname 3y ago

Caching needs to be brought to the ecosystem rather than point non-solutions. While sccache is a partial solution available now from the outside, the toolchain needs adjustment internally to deduplicate work.

epage 3y ago
I've been exploring local, cross-project caching for cargo and my biggest concern is users won't get enough cache hits. With maximal versions and a arbitrary set of dependencies being updated when adding a new dependency. all projects on a system can end up in very different states.
pjmlp 3y ago
That is the whole issue, as scache is more like a workaround versus a proper solution, supported out of the box in cargo, like in Java, .NET, C++ (vcpkg and cargo do bin libs), node,...

user6723 3y ago

[flagged]

xena 3y ago
[flagged]

swsieber 3y ago

There's a few other options that I think in combination would make things much better. 1) Make using proc macros in your own application code faster: Encourage better macro re-export hygiene. Basically, one of the reasons serde + serde_derive is so slow is because for serde re-exports serde_derive to compile before it itself can be built. A solution (I didn't come up with this myself) would be to have a crate that re-exports serde and serde_derive together; see top comment herehttps://www.reddit.com/r/rust/comments/1602eah/associated_proc_macro_pattern/ https://www.reddit.com/r/rust/comments/1602eah/associated_pr... 2) Make using libraries that use proc macros faster: Publish pre-expanded source code, such that no proc macros run for dependencies. Some work would have to be done around conditional compilation to make things just work (TM), but I think it could be done. (Who knows, maybe it'd be really hard to make the compiler deal with expanding proc macros on stuff behind conditional values. Also, there'd be issues regarding macro hygiene. Both solvable, but I'm not sure how much effort they'd take).

anacrolix 3y ago

this article style is very hard to read.