5 ms·

> On topic: I agree that 3rd party binaries are not a good solution to this particular problem. That is the type of solution that works in a closed environment

by rewmie 3y ago

> On topic: I agree that 3rd party binaries are not a good solution to this particular problem. That is the type of solution that works in a closed environment like a corporate CI system, but it should not be the default. Why? I mean, OS distributions making available whole catalogs of prebuilt binaries is a time-tested solution solution that's in place for decades. What leads you to believe that this is suddenly undesirable?

dundarious 3y ago
I think there's a very obvious distinction between the kind of "3rd party" that is your chosen OS distribution and the kind of 3rd party that is "some person on github or crates.io", and I didn't feel the need to clarify this as I thought it was obvious. I've clarified it to "upstream-provided 3rd party" in the original post.
- formerly_proven 3y ago
  Either way you're running their code, I don't understand the outrage here, frankly. People who look at the code they're running and the changes they're pulling in of course noticed this, but I seriously doubt even 1% of the rageposters are in that set of rust users.
  - dundarious 3y ago
    I'm not outraged. I dislike it, and it would be a "con" when considering any use of the project. Binaries are relatively inscrutable, and there are massive differences in ease of doing something nefarious or careless with OS binaries, and upstream checked in source code, and upstream checked in binaries. That's just one or two reasons why I think this approach is not at all commonplace in open settings, and is generally limited to closed and controlled settings like corporate environments. A more typical and reasonable approach would be having a dependency on such a binary that you get however you like: OS, build from upstream source, upstream binary, etc., and where the default is not the latter. And now if you allow me to get a bit off-topic again: to me, having to "innovate" like this is a major sign that this project knows it is very slow, and knows it's unlikely to ever not be very slow.
- rewmie 3y ago
  > I think there's a very obvious distinction between the kind of "3rd party" that is your chosen OS distribution and the kind of 3rd party that is "some person on github or crates.io" That tells you more about crates.io than consuming prebuilt binaries. Some Linux distro ship both source code packages and prebuilt binaries. This has not been a problem for the past two decades. Why is it suddenly a problem with Rust's ecosystem?
  - tyg13 3y ago
    crates.io is just a platform for people to make their Rust code distributable; it's not curated in any way really (other than removing obvious malware). An OS distribution is explicitly curated.
    - rewmie 3y ago
      I never saw anyone complain that crates.io isn't curated. Why is this suddenly an issue when discussing shipping prebuilt binaries? If it's somehow ok to download the source code as-is, why is it a problem if crates.io builds it for you? Some problems have solutions, but people need to seek for solutions instead of collecting problems.
      - tyg13 3y ago
        I'm not complaining that it's not curated, specifically because it's exclusively a source code package directory. Auditing source code for safety is hard, doing so with a binary is much harder.
        elsjaako 3y ago
        I wouldn't mind if rust had reproducible builds, and the binaries had to be built+signed by both the original author and crates.io. But how the article describes it seems sketchy
- Joker_vD 3y ago
  It's a 3rd party package that rustc and cargo already internally depend on themselves. If you're okay getting those distributed to you as binaries, this package too may as well come with a precompiled binary.
bsder 3y ago
Because it demonstrates a failure of the Rust ecosystem. Proc macros have effectively become fundamental to Rust. This is unfortunate, but, given that it is reality, they should get compiler support. Part of what is making them so slow is that they need to process a firehose of information through a drinking straw. Placing proc macros into the compiler would not only speed them up but also remove the 50+ crate dependency chain that gets pulled in. Serde, while slow, has a more fundamental Rust problem. Serde demonstrates that Rust needs some more abstractions that it currently does not have. The "orphan rule" means that you, dear user, cannot take my crate and Serde and compose them without a large amount of copypasta boilerplate. A better composition story would speed up the compilation of Serde as it would have to do far less work. Rust has fundamental problems to solve, and lately only seems capable of stopgaps instead of progress. This is unfortunate as it will wind up ceding the field to something like Carbon.
- veber-alex 3y ago
  proc macros are user written code, how can you "place them into the compiler"? What does the orphan rule have to do with the compilation speed of serde? How does it reduce the work it needs to do?
  - bsder 3y ago
    You have to pulverize the macro into syntax atoms and then reassemble it piece by piece into an AST that the compiler needs. All the processing to do all of that should be a part of the compiler--not a smattering of 50 crates (syn being the most oppressive one) and a weak compiler API. Serde has to enumerate the universe even for types that wind up not used. Those types then need to be discarded--which is problematic if it takes a non-trivial amount of time to enumerate or construct them (say: by using proc macros). Part of this is the specification that must be done because Rust can't do compile-time reflection/introspection part of which is blocked because of the way Rust resolves types--different versions of crates wind up with different types even if the structures are structurally identical because Rust can't do compile time reflection and relies on things like the Orphan rule. I am not saying that any of these things are easy. They are in fact hard. However, without solving them, Rust compilation times will always be garbage. I'm beginning to think that the folks who claimed that compiler speed is the single most important design criterion were right. It seems like you can't go back and retrofit "faster compiler time" afterward.

by rewmie 3y ago

dundarious 3y ago

I think there's a very obvious distinction between the kind of "3rd party" that is your chosen OS distribution and the kind of 3rd party that is "some person on github or crates.io", and I didn't feel the need to clarify this as I thought it was obvious. I've clarified it to "upstream-provided 3rd party" in the original post.

formerly_proven 3y ago
Either way you're running their code, I don't understand the outrage here, frankly. People who look at the code they're running and the changes they're pulling in of course noticed this, but I seriously doubt even 1% of the rageposters are in that set of rust users.
- dundarious 3y ago
  I'm not outraged. I dislike it, and it would be a "con" when considering any use of the project. Binaries are relatively inscrutable, and there are massive differences in ease of doing something nefarious or careless with OS binaries, and upstream checked in source code, and upstream checked in binaries. That's just one or two reasons why I think this approach is not at all commonplace in open settings, and is generally limited to closed and controlled settings like corporate environments. A more typical and reasonable approach would be having a dependency on such a binary that you get however you like: OS, build from upstream source, upstream binary, etc., and where the default is not the latter. And now if you allow me to get a bit off-topic again: to me, having to "innovate" like this is a major sign that this project knows it is very slow, and knows it's unlikely to ever not be very slow.
rewmie 3y ago
> I think there's a very obvious distinction between the kind of "3rd party" that is your chosen OS distribution and the kind of 3rd party that is "some person on github or crates.io" That tells you more about crates.io than consuming prebuilt binaries. Some Linux distro ship both source code packages and prebuilt binaries. This has not been a problem for the past two decades. Why is it suddenly a problem with Rust's ecosystem?
- tyg13 3y ago
  crates.io is just a platform for people to make their Rust code distributable; it's not curated in any way really (other than removing obvious malware). An OS distribution is explicitly curated.
  - rewmie 3y ago
    I never saw anyone complain that crates.io isn't curated. Why is this suddenly an issue when discussing shipping prebuilt binaries? If it's somehow ok to download the source code as-is, why is it a problem if crates.io builds it for you? Some problems have solutions, but people need to seek for solutions instead of collecting problems.
    - tyg13 3y ago
      I'm not complaining that it's not curated, specifically because it's exclusively a source code package directory. Auditing source code for safety is hard, doing so with a binary is much harder.
      - elsjaako 3y ago
        I wouldn't mind if rust had reproducible builds, and the binaries had to be built+signed by both the original author and crates.io. But how the article describes it seems sketchy
Joker_vD 3y ago
It's a 3rd party package that rustc and cargo already internally depend on themselves. If you're okay getting those distributed to you as binaries, this package too may as well come with a precompiled binary.

bsder 3y ago

Because it demonstrates a failure of the Rust ecosystem. Proc macros have effectively become fundamental to Rust. This is unfortunate, but, given that it is reality, they should get compiler support. Part of what is making them so slow is that they need to process a firehose of information through a drinking straw. Placing proc macros into the compiler would not only speed them up but also remove the 50+ crate dependency chain that gets pulled in. Serde, while slow, has a more fundamental Rust problem. Serde demonstrates that Rust needs some more abstractions that it currently does not have. The "orphan rule" means that you, dear user, cannot take my crate and Serde and compose them without a large amount of copypasta boilerplate. A better composition story would speed up the compilation of Serde as it would have to do far less work. Rust has fundamental problems to solve, and lately only seems capable of stopgaps instead of progress. This is unfortunate as it will wind up ceding the field to something like Carbon.

veber-alex 3y ago
proc macros are user written code, how can you "place them into the compiler"? What does the orphan rule have to do with the compilation speed of serde? How does it reduce the work it needs to do?
- bsder 3y ago
  You have to pulverize the macro into syntax atoms and then reassemble it piece by piece into an AST that the compiler needs. All the processing to do all of that should be a part of the compiler--not a smattering of 50 crates (syn being the most oppressive one) and a weak compiler API. Serde has to enumerate the universe even for types that wind up not used. Those types then need to be discarded--which is problematic if it takes a non-trivial amount of time to enumerate or construct them (say: by using proc macros). Part of this is the specification that must be done because Rust can't do compile-time reflection/introspection part of which is blocked because of the way Rust resolves types--different versions of crates wind up with different types even if the structures are structurally identical because Rust can't do compile time reflection and relies on things like the Orphan rule. I am not saying that any of these things are easy. They are in fact hard. However, without solving them, Rust compilation times will always be garbage. I'm beginning to think that the folks who claimed that compiler speed is the single most important design criterion were right. It seems like you can't go back and retrofit "faster compiler time" afterward.