4 ms·

I'm curious whether those with more professional exposure to rust are concerned about this or think it's noisy for no reason? I'd also be curious if anyone has

I'm curious whether those with more professional exposure to rust are concerned about this or think it's noisy for no reason? I'd also be curious if anyone has any good explanations as to why a choice like this would be made?

adastra22 3y ago
It’s all in the article.
- Icathian 3y ago
  I read the article, and the linked GitHub issue, and posted it. I'm curious if people who use this for a living and aren't some reporter pasting snippets agree with this framing. The variety of responses here now suggests many do not.
  - adastra22 3y ago
    Ok, well I'm a professional rust developer and I agree with the other Rust developers directly quoted in the article, of which there were many. But this isn't a Rust developer issue so much as a distro and security issue. Most Rust developers don't have to concern themselves with supply chain attacks or security audibility, so to them this probably seems like a weird thing to focus on. But as someone who does have to worry about such things, trading off the ability to audit your supply chain for a small, one-time compilation boost on your first compile is a _terrible_ trade-off.
    - Icathian 3y ago
      Thanks for your perspective. I appreciate it.
dcan 3y ago
The tinfoil hat explanation is that the developer is breaking all projects that depend on the default package manager cargo, forcing them to move to buck or bazel - projects that the developer of serde-derive has invested work into. This is not noisy for no reason - the expectation is all dependencies are compiled from source on your computer, and here the developer forces the use of a binary program, with no alternative.
- philbo 3y ago
  As someone who hasn't used Rust seriously since 2018, I'm tangentially curious what are the improvements (real or perceived) that buck and bazel offer over cargo? I didn't know there were alternative package managers, and cargo always seemed pretty nice to me.
  - philbo 3y ago
    Answering my own question, sorry, looks like I may have got the wrong end of the stick here. Bazel and buck seem like language-agnostic build tools, rather than rust-specific package managers?
    - monocasa 3y ago
      > Bazel and buck seem like language-agnostic build tools, rather than rust-specific package managers? Yeah. Bazel is the sanitized version of Google's internal build system for most of their projects (known internally as blaze). Buck is Facebook's equivalent.
  - MrJohz 3y ago
    The previous comment is pretty paranoid thinking, so I wouldn't take it seriously. That said, Bazel and Buck are good for handling projects "at scale" (e.g. company wide monorepos at mid-to-large companies), or with a lot of different needs (e.g. sources in C++, Rust, Python, and Javascript that need to be combined together). They're not really an alternative to Cargo, and more the next step up for when Cargo isn't sufficient any more. But because they're dealing with much more complicated problems, they're also more complicated to set up and use, so if you're not running into issues with Cargo, you probably don't need to go in that direction (and possibly never will).
Ygg2 3y ago
Not a Rust developer by trade, but from the offending pull request (https://github.com/serde-rs/serde/pull/2514#issue-1810688422 https://github.com/serde-rs/serde/pull/2514#issue-1810688422) it seems to offer 10x compile performance on first compile with small 3ms overhead on incremental compile. Is it worth it? Maybe. Downloading blobs isn't ideal but could be made easier in the future. Ideally a hack like this wouldn't be needed and we'd have comptime serde.
- joshka 3y ago
  10x sounds great until you look at this as an absolute value. It's 10 seconds. That would be huge if it were a a runtime cost, but this is a one time cost at compile time, and this build will be cached unless you're upgrading serde (or its dependencies). The value of that 10 seconds compared to the cost is a faustian bargain.
  - Nullabillity 3y ago
    And even that 10s is assuming that you use no other proc macros that would bring in the same dependencies anyway (very unlikely).
devjab 3y ago
We mainly use high level languages like TypeScript or Python because they work well for us. We do use C/C++ from time to time though, both for high performance requiring bottlenecks and for embedded since we do work with solar plant technology. A couple of years ago we looked into Rust and found it a combination of nice and wanting. It’s sort of great for low level programming and not really worth your time for anything else. Or at least that’s what we decided. We did adopt it and build a few solar inverter applications with Rust where we might’ve used C/C++ before, and it’s a decent language and nice to work with. But then a few months back, or maybe it was longer, some drama hit the ecosystem. Again it was so unimportant to us that I can’t even begin to remember what it was about, but similar to this Serde “drama” it speaks about an immature ecosystem and community. From a purely risk management offset we decided to stop using Rust for the time being. Which probably sounds more dramatic than it is, because we don’t really care about a lot of these issues and we’re probably never even going to be affected by them. But why would we want to spend time and energy on a technology which might change a lot? Well, the quick answer is that we won’t. Take this Serde issue as an example, it’s a very used package. I’m not sure exactly what people use it for, I assume that it’s for things we would likely build in Python, but even with 170 million users it’s main maintainer seems to be an issue. I even agree with the maintainer, but if the Rust ecosystem doesn’t have someone who can fork it and maintain a non-binary version to meet the wants and needs or all these noise makers, well then that’s not exactly a mature ecosystem. So I think the answer is that it’s mostly just noise, but it’s noise you probably don’t want to have to deal with unless you’re invested in Rust itself.
duped 3y ago
> I'd also be curious if anyone has any good explanations as to why a choice like this would be made? It has to do with how Rust builds are structured. The compilation unit is the crate, and the result of compiling it depends on the source code, the version of hte compiler, the compiler options, the target triple, and the set of features (think #ifdef FEATURE_FLAG). Since there's exponential blowup of possible combinations of a single crate, the largest package registry (crates.io) does not cache compiled crates. It only caches their metadata, and when Cargo (the package manager) does dependency resolution, it uses that metadata alone - and your build system (also Cargo, most of the time) will need to actually build all those artifacts. Seems pretty reasonable, right? There's a special kind of crate called a proc_macro crate (procedural macros). When you use these crates you're not adding their code to your compiled output - you're creating a compiler extension - the compiled crate is loaded by the compiler, and then used to generate code in your crate, which is then compiled by the compiler. One of the reasons Rust build times (particularly clean builds) is that they first have to compile all these compiler extensions and then use them to generate the code that the compiler compiles. These crates are super useful however, like serde, which generates serializers/deserializers that can be used for a wide variety of encodings (XML, JSON, TOML, various binary encodings, and so on). It would seem that the same rules apply for normal crates - there's exponential blowup of compiler version, target triple, and features - but that's not really true. proc_macro crates usually don't have features, the compiler ABI is stable enough that they probably won't break across versions, and almost everyone and their mother is compiling on a handful of host systems (x86_64-linux being the most important one). So while it's true that crates-io probably shouldn't cache every crate for every permutation of compiler/target/feature/etc, a few crates are so pervasive and used in such a sparing manner that it should be possible for the registry to host those In fact some build systems can do this automatically. But Cargo can't. But it does have a pre-build feature (build.rs) that works like proc macros, and can pull down a pre-compiled proc macro crate, so widely used ones can use this hack to get around limitations of cargo. My personal take is that the reason that crates-io and Cargo haven't been upgraded to do this, despite years of people complaining, and known solutions existing, is mostly personal/political. There aren't many people willing to put in the work to do it, and the teams haven't been open to contribution of this kind for a long time. For example, they do not believe that Cargo is a build system, and a lot of this is "out of scope" for cargo.

adastra22 3y ago

It’s all in the article.

Icathian 3y ago
I read the article, and the linked GitHub issue, and posted it. I'm curious if people who use this for a living and aren't some reporter pasting snippets agree with this framing. The variety of responses here now suggests many do not.
- adastra22 3y ago
  Ok, well I'm a professional rust developer and I agree with the other Rust developers directly quoted in the article, of which there were many. But this isn't a Rust developer issue so much as a distro and security issue. Most Rust developers don't have to concern themselves with supply chain attacks or security audibility, so to them this probably seems like a weird thing to focus on. But as someone who does have to worry about such things, trading off the ability to audit your supply chain for a small, one-time compilation boost on your first compile is a _terrible_ trade-off.
  - Icathian 3y ago
    Thanks for your perspective. I appreciate it.

dcan 3y ago

The tinfoil hat explanation is that the developer is breaking all projects that depend on the default package manager cargo, forcing them to move to buck or bazel - projects that the developer of serde-derive has invested work into. This is not noisy for no reason - the expectation is all dependencies are compiled from source on your computer, and here the developer forces the use of a binary program, with no alternative.

philbo 3y ago
As someone who hasn't used Rust seriously since 2018, I'm tangentially curious what are the improvements (real or perceived) that buck and bazel offer over cargo? I didn't know there were alternative package managers, and cargo always seemed pretty nice to me.
- philbo 3y ago
  Answering my own question, sorry, looks like I may have got the wrong end of the stick here. Bazel and buck seem like language-agnostic build tools, rather than rust-specific package managers?
  - monocasa 3y ago
    > Bazel and buck seem like language-agnostic build tools, rather than rust-specific package managers? Yeah. Bazel is the sanitized version of Google's internal build system for most of their projects (known internally as blaze). Buck is Facebook's equivalent.
- MrJohz 3y ago
  The previous comment is pretty paranoid thinking, so I wouldn't take it seriously. That said, Bazel and Buck are good for handling projects "at scale" (e.g. company wide monorepos at mid-to-large companies), or with a lot of different needs (e.g. sources in C++, Rust, Python, and Javascript that need to be combined together). They're not really an alternative to Cargo, and more the next step up for when Cargo isn't sufficient any more. But because they're dealing with much more complicated problems, they're also more complicated to set up and use, so if you're not running into issues with Cargo, you probably don't need to go in that direction (and possibly never will).

Ygg2 3y ago

Not a Rust developer by trade, but from the offending pull request (https://github.com/serde-rs/serde/pull/2514#issue-1810688422 https://github.com/serde-rs/serde/pull/2514#issue-1810688422) it seems to offer 10x compile performance on first compile with small 3ms overhead on incremental compile. Is it worth it? Maybe. Downloading blobs isn't ideal but could be made easier in the future. Ideally a hack like this wouldn't be needed and we'd have comptime serde.

joshka 3y ago
10x sounds great until you look at this as an absolute value. It's 10 seconds. That would be huge if it were a a runtime cost, but this is a one time cost at compile time, and this build will be cached unless you're upgrading serde (or its dependencies). The value of that 10 seconds compared to the cost is a faustian bargain.
- Nullabillity 3y ago
  And even that 10s is assuming that you use no other proc macros that would bring in the same dependencies anyway (very unlikely).

devjab 3y ago

We mainly use high level languages like TypeScript or Python because they work well for us. We do use C/C++ from time to time though, both for high performance requiring bottlenecks and for embedded since we do work with solar plant technology. A couple of years ago we looked into Rust and found it a combination of nice and wanting. It’s sort of great for low level programming and not really worth your time for anything else. Or at least that’s what we decided. We did adopt it and build a few solar inverter applications with Rust where we might’ve used C/C++ before, and it’s a decent language and nice to work with. But then a few months back, or maybe it was longer, some drama hit the ecosystem. Again it was so unimportant to us that I can’t even begin to remember what it was about, but similar to this Serde “drama” it speaks about an immature ecosystem and community. From a purely risk management offset we decided to stop using Rust for the time being. Which probably sounds more dramatic than it is, because we don’t really care about a lot of these issues and we’re probably never even going to be affected by them. But why would we want to spend time and energy on a technology which might change a lot? Well, the quick answer is that we won’t. Take this Serde issue as an example, it’s a very used package. I’m not sure exactly what people use it for, I assume that it’s for things we would likely build in Python, but even with 170 million users it’s main maintainer seems to be an issue. I even agree with the maintainer, but if the Rust ecosystem doesn’t have someone who can fork it and maintain a non-binary version to meet the wants and needs or all these noise makers, well then that’s not exactly a mature ecosystem. So I think the answer is that it’s mostly just noise, but it’s noise you probably don’t want to have to deal with unless you’re invested in Rust itself.

duped 3y ago

> I'd also be curious if anyone has any good explanations as to why a choice like this would be made? It has to do with how Rust builds are structured. The compilation unit is the crate, and the result of compiling it depends on the source code, the version of hte compiler, the compiler options, the target triple, and the set of features (think #ifdef FEATURE_FLAG). Since there's exponential blowup of possible combinations of a single crate, the largest package registry (crates.io) does not cache compiled crates. It only caches their metadata, and when Cargo (the package manager) does dependency resolution, it uses that metadata alone - and your build system (also Cargo, most of the time) will need to actually build all those artifacts. Seems pretty reasonable, right? There's a special kind of crate called a proc_macro crate (procedural macros). When you use these crates you're not adding their code to your compiled output - you're creating a compiler extension - the compiled crate is loaded by the compiler, and then used to generate code in your crate, which is then compiled by the compiler. One of the reasons Rust build times (particularly clean builds) is that they first have to compile all these compiler extensions and then use them to generate the code that the compiler compiles. These crates are super useful however, like serde, which generates serializers/deserializers that can be used for a wide variety of encodings (XML, JSON, TOML, various binary encodings, and so on). It would seem that the same rules apply for normal crates - there's exponential blowup of compiler version, target triple, and features - but that's not really true. proc_macro crates usually don't have features, the compiler ABI is stable enough that they probably won't break across versions, and almost everyone and their mother is compiling on a handful of host systems (x86_64-linux being the most important one). So while it's true that crates-io probably shouldn't cache every crate for every permutation of compiler/target/feature/etc, a few crates are so pervasive and used in such a sparing manner that it should be possible for the registry to host those In fact some build systems can do this automatically. But Cargo can't. But it does have a pre-build feature (build.rs) that works like proc macros, and can pull down a pre-compiled proc macro crate, so widely used ones can use this hack to get around limitations of cargo. My personal take is that the reason that crates-io and Cargo haven't been upgraded to do this, despite years of people complaining, and known solutions existing, is mostly personal/political. There aren't many people willing to put in the work to do it, and the teams haven't been open to contribution of this kind for a long time. For example, they do not believe that Cargo is a build system, and a lot of this is "out of scope" for cargo.