9 ms·

Rust is alright as a language, but cargo is extremely scary. I hope a culture of coding in rust outside of the cargo "ecosystem" develops. The current situati

Rust is alright as a language, but cargo is extremely scary. I hope a culture of coding in rust outside of the cargo "ecosystem" develops. The current situation is alienating to reproducibility-oriented developers.

Ar-Curunir 5y ago
I hope that never ever happens. We don't need a C-like mess where every library has a bespoke build process that doesn't work across OSes.
- proto_lambda 5y ago
  OP seems to be conflating the concepts of cargo (the rust build system with dependency management functionality) and crates.io (the main and default package repository today). It is totally possible to use cargo without ever touching crates.io, but if you need any dependencies you'll of course have to provide them through some other means (local file system, git repositories, or a custom package repository).
  - mort96 5y ago
    Cargo defaults to downloading stuff from crates.io. I certainly wish cargo was just a build system which doesn't care where you get your packages from, but that's not what it is.
    - steveklabnik 5y ago
      Cargo does not care where you get your packages from, but it does have a default place. If you don't like it, it is easy to change.
      - mort96 5y ago
        That... is caring where I get packages from. Something like make or cmake doesn't care at all how packages end up on my drive. Almost everyone uses cargo in a way which makes cargo automatically download them from the web.
        steveklabnik 5y ago
        If you're making that argument, you could easily turn it around: make or cmake do care where your pacakges are; they by default only build things on your local drive. The point is: all options are available for all systems, so suggesting any workflow doesn't work with any of these is just incorrect.
        mort96 5y ago
        My point is that building code coming from my local drive is much less problematic than automatically downloading the code from some random website. And I never said any workflow doesn't work with Cargo. I said I wished it didn't download code (by default) and that it didn't care how code ended up on my drive, that it just worked like a normal build system. That you can use Cargo in that way doesn't help.
        saghm 5y ago
        > My point is that building code coming from my local drive is much less problematic than automatically downloading the code from some random website. How does the code get on your hard drive? I'd imagine from downloading it from some random website (arguably _more_ random since it's less likely to be a centralized place like crates.io). Or you could vendor the dependencies so that they're included when you get the source code for the thing you're working on, but as mentioned throughout this discussion, cargo lets you do that too.
  - estebank 5y ago
    I could totally see the Linux Kernel setting up their own alternative Rust package mirror that cargo uses: you get the ease of use of cargo while also getting the desired level of curation. It could either be treated as owned packages, only projects written by kernel contributors, or it could be the linux distro packager model, where blessed crates at a point in time are curated, but not necessarily audited beyond some smoke tests.
- enriquto 5y ago
  > where every library has a bespoke build process doesn't work across OSes cargo sounds pretty much like a new bespoke build process that doesn't even work across different languages on the same OS
  - sanderjd 5y ago
    Yes. It turns out this works much better than the situation previously. The bespoke-per language model turns out to have fewer drawbacks than the bespoke-per-library-and-OS model. Note: fewer drawbacks, not none. I would not have predicted this a priori, for what it's worth, and I was very critical of and first one of these I became familiar with (rvm), but it turns out to work well in practice. I still think cross-language efforts (like bazel and I'm sure there are others, maybe nix?) seem generally better, but I suspect there is some fairly good reason they are less widely used.
    - howinteresting 5y ago
      The problem with bazel and nix is that they are totalizing build systems. They want everything to bend to their worldview, and ask for an all-in commitment. Cargo etc require somewhat less of a commitment.
      - sanderjd 5y ago
        Right, but it seems like it would probably make more sense to pick a totalizing system, if you know you'll have a polyglot environment, which seems pretty inevitable for any business (as opposed to just like a side project). But I think there are probably good reasons this isn't the most common way to do things, and I just don't know them.
        howinteresting 5y ago
        Many businesses do pick totalizing systems -- many medium-size or above corporations use bazel, and a few use nix. There's also some tooling to convert Cargo.toml to bazel and nix build files, though I don't know how well that works. I think the problem is that bazel and nix are just not that easy to get started with. If you're using them you're likely to have a team (or at least one person) working full-time on them, since bending everything to a singular worldview involves a lot of work.
cillian64 5y ago
If you particularly want to replicate the common C situation where your dependencies are in-tree or you just don’t have external dependencies and write everything yourself then you are totally welcome to do that, even with Cargo. Nobody is forcing you to use external dependencies. If your problem is other people writing software with external dependencies, then I guess you have different priorities to them.
- nyanpasu64 5y ago
  My problem is dependencies taking on dependencies taking on hundreds of dependencies, making it more difficult to find a self-contained dependency which solves a problem, compared to C++ (outside of Boost or ffmpeg and such).
  - cillian64 5y ago
    That’s fair, and I hope we see crates being more considerate over what deps they pull in (or people making alternatives with minimal dependencies).
gspr 5y ago
I wholeheartedly agree. I love Rust, it's the most fun I've had with any programming language (barring perhaps Haskell). But I still run cargo in offline mode with crates.io disabled, pointing cargo to /usr/share/cargo/registry for dependencies (that's where Debian's librust-*-dev packages get installed, the only dependencies I accept in my Rust projects).
- maxerickson 5y ago
  So you can change 2 settings to get the behavior you want? Sounds like they did a decent job anticipating that use case!
  - enriquto 5y ago
    Let's just make "unsafe" the default in Rust, and add a new "safe" keyword to anticipate the use case of overly concerned users like gspr.
  - gspr 5y ago
    Absolutely (although see note below)! I don't have any gripes with the tooling from a technical perspective. I do have gripes from a cultural perspective, though. Addendum: Until quite recently, this was quite cumbersome. It also meant that all cargo invocations (by the same user) would use that override, always. It meant that compiling someone else's project became quite the hassle. Or compiling a project that mostly uses system dependencies, but some crates.io deps. But the situation is improving.
  - steveklabnik 5y ago
    To say it explicitly: "I don't have any internet" was a hard constraint on everything from the beginning. Firefox's build requires it. Most distros require it. Some of it was a bit awkward to actually use this way in the early days, but those harsh edges have been since sanded off.
pornel 5y ago
Cargo and crates.io have been specifically designed to be reliably reproducible, and had a chance to learn from npm's mistakes to not repeat them. Cargo binary projects have Cargo.lock by default with checksums of all dependencies used. crates.io doesn't allow changing past releases, and has a policy of not deleting any crates unless legally required (user-accesible "yank" hides, but doesn't delete). Crates.io index is a git repository with full history of all changes to the registry, so you can recreate its state at any point in time (in case you lost your Cargo.lock, you can reliably remake one from the past). And on top of that there's `cargo vendor` command that makes a local offline copy of everything you use, so you can fully archive a Cargo project and rebuild it any time later.
nyoomboom 5y ago
How does cargo prevent reproducible builds?
- enriquto 5y ago
  The default behavior of cargo is to download stuff from the internet. This may be the least reproducible thing ever. I'm honestly astonished that programmers of a language that is deemed to be "safe by default" thought that this behavior was acceptable in any form, not to say the default. If downloading things at build time is somehow necessary, it should be an obscure option behind a flag with a scary name, like --extremely-unsafe-i-know-what-i-am-doing, that prompted the user with a small turing test every time that it is run. Cargo is just bonkers, it doesn't matter at all if it is "convenient" or not. Convenience before basic safety and reproducibility is contrary to the spirit of the language itself. It's as if bounds checking in the language was deferred to a third party that you need to "trust" in order to believe that you won't have segmentation faults.
  - deleted 5y ago
    [deleted]
  - thecrm 5y ago
    It doesn't just download random things. Cargo generates a Cargo.lock file with checksums and will make sure that those checksums match when building later on. It's about as safe as vendoring all dependencies while being far easier to work with (though tools like cargo-vendor do exist, of course). Edit: for things like the kernel, vendoring dependencies is still probably not a bad idea, of course
    - yw3410 5y ago
      You're forgetting about custom build scripts. Thankfully most of the core ones have moved off cloning dependencies for ffi purposes (think cloning an alsa-lib version for ffi), but it used to be super common.
    - humanrebar 5y ago
fulafel 5y ago
For other repro curious readers, this seems like a good entry point to follow reproducibility efforts: https://github.com/rust-secure-code/wg/issues/28 https://github.com/rust-secure-code/wg/issues/28
CraigJPerry 5y ago
Is reproducability achiveable? If you preserved an immutable tag of the source code and all its dependencies, a copy of the compiler version used and all build flags, then you’ve still got some big holes in your ability to reproduce a binary: 1. OS version & patches installed 2. OS configuration 3. Hardware used (processors can have weird subtle bugs, microcode can affect execution behaviour, etc etc) 4. Transient issues - the golden copy to be reproduced for some post event investigation might have contained a bit flip leading to impossible to reproduce verification signatures Etc etc Or is reproducability just a spectrum and you try to get further along it with some careful attention to detail, rather than an absolute to be achieved? If the latter are you not cheaper just archiving binaries and tagging them with the source + deps + compiler + arch used to build them? Thats a 5 minute job to setup in your CI process and costs comparatively little to maintain vs wasting expensive human brains chasing down a futile goal.
- IshKebab 5y ago
  Reproducibility is absolutely achievable. It is difficult with languages that have C system dependencies like C and Rust though. Usually you have to effectively check the compiler and system headers into your repo, which isn't ideal. For something like Go it's trivial though. If you're using pure Go code then reproducible builds are pretty much as simple as "compile using Go 1.xx". I don't understand why you think 3. and 4. are issues. Bit flips are very unlikely at compilation scale, and the hardware you use to compile something shouldn't affect the output. In either case you can just compile it twice and on different hardware and compare the result to confirm.
- JonChesterfield 5y ago
  It's normally taken as binary + transitive dependencies are bitwise identical on multiple builds. Needs a copy of a deterministic compiler along with your source. Whether the build runs the same on different machines is orthogonal to consistently rebuilding it.
- enriquto 5y ago
  Reproducibility is certainly achievable, and an important point for some people. Look for example at guix, that bootstraps explicitly with what they call "the maxwell equations of software". Imposing a web-based build system, into the kernel of all places, is a kick to the face to people who care about reproducibility.
nomoreusernames 5y ago
why do we have to have soooo many package managers? makes me cry inside a bit.
- humanrebar 5y ago
  Yeah, especially language specific ones. It seems like the world wants to be polyglot. Do we really want to have all the encryption algorithms reimplemented in all the languages? The answer there is often "use FFI". But if we're all going to use C APIs anyway, then shouldn't our package managers support C? But C has probably the most awkward build system culture of any language, complicating the job of packaging quite a bit. A lot of language specific package management systems get lift from offloading the ugly C support problems to other layers. See python and "manylinux" for instance.
  - nomoreusernames 5y ago
    you are right about C. i mean i do get why we have so many. and i use them so often that its ingrained in me. and how to fix them when i mess up. but sometimes i notice how many libs we are talking about. honestly it is kinda ridiculous. i know how hard it is to manage one repo and keep it safe. maybe im too old. young ones now just download go binaries and run them.
- pjmlp 5y ago
  Because not everyone is using Linux distributions, and even those cannot agree in which system packaging they want to have.

Ar-Curunir 5y ago

I hope that never ever happens. We don't need a C-like mess where every library has a bespoke build process that doesn't work across OSes.

proto_lambda 5y ago
OP seems to be conflating the concepts of cargo (the rust build system with dependency management functionality) and crates.io (the main and default package repository today). It is totally possible to use cargo without ever touching crates.io, but if you need any dependencies you'll of course have to provide them through some other means (local file system, git repositories, or a custom package repository).
- mort96 5y ago
  Cargo defaults to downloading stuff from crates.io. I certainly wish cargo was just a build system which doesn't care where you get your packages from, but that's not what it is.
  - steveklabnik 5y ago
    Cargo does not care where you get your packages from, but it does have a default place. If you don't like it, it is easy to change.
    - mort96 5y ago
      That... is caring where I get packages from. Something like make or cmake doesn't care at all how packages end up on my drive. Almost everyone uses cargo in a way which makes cargo automatically download them from the web.
      - steveklabnik 5y ago
        If you're making that argument, you could easily turn it around: make or cmake do care where your pacakges are; they by default only build things on your local drive. The point is: all options are available for all systems, so suggesting any workflow doesn't work with any of these is just incorrect.
        mort96 5y ago
        My point is that building code coming from my local drive is much less problematic than automatically downloading the code from some random website. And I never said any workflow doesn't work with Cargo. I said I wished it didn't download code (by default) and that it didn't care how code ended up on my drive, that it just worked like a normal build system. That you can use Cargo in that way doesn't help.
        saghm 5y ago
        > My point is that building code coming from my local drive is much less problematic than automatically downloading the code from some random website. How does the code get on your hard drive? I'd imagine from downloading it from some random website (arguably _more_ random since it's less likely to be a centralized place like crates.io). Or you could vendor the dependencies so that they're included when you get the source code for the thing you're working on, but as mentioned throughout this discussion, cargo lets you do that too.
- estebank 5y ago
  I could totally see the Linux Kernel setting up their own alternative Rust package mirror that cargo uses: you get the ease of use of cargo while also getting the desired level of curation. It could either be treated as owned packages, only projects written by kernel contributors, or it could be the linux distro packager model, where blessed crates at a point in time are curated, but not necessarily audited beyond some smoke tests.
enriquto 5y ago
> where every library has a bespoke build process doesn't work across OSes cargo sounds pretty much like a new bespoke build process that doesn't even work across different languages on the same OS
- sanderjd 5y ago
  Yes. It turns out this works much better than the situation previously. The bespoke-per language model turns out to have fewer drawbacks than the bespoke-per-library-and-OS model. Note: fewer drawbacks, not none. I would not have predicted this a priori, for what it's worth, and I was very critical of and first one of these I became familiar with (rvm), but it turns out to work well in practice. I still think cross-language efforts (like bazel and I'm sure there are others, maybe nix?) seem generally better, but I suspect there is some fairly good reason they are less widely used.
  - howinteresting 5y ago
    The problem with bazel and nix is that they are totalizing build systems. They want everything to bend to their worldview, and ask for an all-in commitment. Cargo etc require somewhat less of a commitment.
    - sanderjd 5y ago
      Right, but it seems like it would probably make more sense to pick a totalizing system, if you know you'll have a polyglot environment, which seems pretty inevitable for any business (as opposed to just like a side project). But I think there are probably good reasons this isn't the most common way to do things, and I just don't know them.
      - howinteresting 5y ago
        Many businesses do pick totalizing systems -- many medium-size or above corporations use bazel, and a few use nix. There's also some tooling to convert Cargo.toml to bazel and nix build files, though I don't know how well that works. I think the problem is that bazel and nix are just not that easy to get started with. If you're using them you're likely to have a team (or at least one person) working full-time on them, since bending everything to a singular worldview involves a lot of work.

cillian64 5y ago

If you particularly want to replicate the common C situation where your dependencies are in-tree or you just don’t have external dependencies and write everything yourself then you are totally welcome to do that, even with Cargo. Nobody is forcing you to use external dependencies. If your problem is other people writing software with external dependencies, then I guess you have different priorities to them.

nyanpasu64 5y ago
My problem is dependencies taking on dependencies taking on hundreds of dependencies, making it more difficult to find a self-contained dependency which solves a problem, compared to C++ (outside of Boost or ffmpeg and such).
- cillian64 5y ago
  That’s fair, and I hope we see crates being more considerate over what deps they pull in (or people making alternatives with minimal dependencies).

gspr 5y ago

I wholeheartedly agree. I love Rust, it's the most fun I've had with any programming language (barring perhaps Haskell). But I still run cargo in offline mode with crates.io disabled, pointing cargo to /usr/share/cargo/registry for dependencies (that's where Debian's librust-*-dev packages get installed, the only dependencies I accept in my Rust projects).

maxerickson 5y ago
So you can change 2 settings to get the behavior you want? Sounds like they did a decent job anticipating that use case!
- enriquto 5y ago
  Let's just make "unsafe" the default in Rust, and add a new "safe" keyword to anticipate the use case of overly concerned users like gspr.
- gspr 5y ago
  Absolutely (although see note below)! I don't have any gripes with the tooling from a technical perspective. I do have gripes from a cultural perspective, though. Addendum: Until quite recently, this was quite cumbersome. It also meant that all cargo invocations (by the same user) would use that override, always. It meant that compiling someone else's project became quite the hassle. Or compiling a project that mostly uses system dependencies, but some crates.io deps. But the situation is improving.
- steveklabnik 5y ago
  To say it explicitly: "I don't have any internet" was a hard constraint on everything from the beginning. Firefox's build requires it. Most distros require it. Some of it was a bit awkward to actually use this way in the early days, but those harsh edges have been since sanded off.

pornel 5y ago

Cargo and crates.io have been specifically designed to be reliably reproducible, and had a chance to learn from npm's mistakes to not repeat them. Cargo binary projects have Cargo.lock by default with checksums of all dependencies used. crates.io doesn't allow changing past releases, and has a policy of not deleting any crates unless legally required (user-accesible "yank" hides, but doesn't delete). Crates.io index is a git repository with full history of all changes to the registry, so you can recreate its state at any point in time (in case you lost your Cargo.lock, you can reliably remake one from the past). And on top of that there's `cargo vendor` command that makes a local offline copy of everything you use, so you can fully archive a Cargo project and rebuild it any time later.

nyoomboom 5y ago

How does cargo prevent reproducible builds?

enriquto 5y ago
The default behavior of cargo is to download stuff from the internet. This may be the least reproducible thing ever. I'm honestly astonished that programmers of a language that is deemed to be "safe by default" thought that this behavior was acceptable in any form, not to say the default. If downloading things at build time is somehow necessary, it should be an obscure option behind a flag with a scary name, like --extremely-unsafe-i-know-what-i-am-doing, that prompted the user with a small turing test every time that it is run. Cargo is just bonkers, it doesn't matter at all if it is "convenient" or not. Convenience before basic safety and reproducibility is contrary to the spirit of the language itself. It's as if bounds checking in the language was deferred to a third party that you need to "trust" in order to believe that you won't have segmentation faults.
- deleted 5y ago
  [deleted]
- thecrm 5y ago
  It doesn't just download random things. Cargo generates a Cargo.lock file with checksums and will make sure that those checksums match when building later on. It's about as safe as vendoring all dependencies while being far easier to work with (though tools like cargo-vendor do exist, of course). Edit: for things like the kernel, vendoring dependencies is still probably not a bad idea, of course
  - yw3410 5y ago
    You're forgetting about custom build scripts. Thankfully most of the core ones have moved off cloning dependencies for ffi purposes (think cloning an alsa-lib version for ffi), but it used to be super common.
  - humanrebar 5y ago

fulafel 5y ago

For other repro curious readers, this seems like a good entry point to follow reproducibility efforts: https://github.com/rust-secure-code/wg/issues/28 https://github.com/rust-secure-code/wg/issues/28

CraigJPerry 5y ago

Is reproducability achiveable? If you preserved an immutable tag of the source code and all its dependencies, a copy of the compiler version used and all build flags, then you’ve still got some big holes in your ability to reproduce a binary: 1. OS version & patches installed 2. OS configuration 3. Hardware used (processors can have weird subtle bugs, microcode can affect execution behaviour, etc etc) 4. Transient issues - the golden copy to be reproduced for some post event investigation might have contained a bit flip leading to impossible to reproduce verification signatures Etc etc Or is reproducability just a spectrum and you try to get further along it with some careful attention to detail, rather than an absolute to be achieved? If the latter are you not cheaper just archiving binaries and tagging them with the source + deps + compiler + arch used to build them? Thats a 5 minute job to setup in your CI process and costs comparatively little to maintain vs wasting expensive human brains chasing down a futile goal.

IshKebab 5y ago
Reproducibility is absolutely achievable. It is difficult with languages that have C system dependencies like C and Rust though. Usually you have to effectively check the compiler and system headers into your repo, which isn't ideal. For something like Go it's trivial though. If you're using pure Go code then reproducible builds are pretty much as simple as "compile using Go 1.xx". I don't understand why you think 3. and 4. are issues. Bit flips are very unlikely at compilation scale, and the hardware you use to compile something shouldn't affect the output. In either case you can just compile it twice and on different hardware and compare the result to confirm.
JonChesterfield 5y ago
It's normally taken as binary + transitive dependencies are bitwise identical on multiple builds. Needs a copy of a deterministic compiler along with your source. Whether the build runs the same on different machines is orthogonal to consistently rebuilding it.
enriquto 5y ago
Reproducibility is certainly achievable, and an important point for some people. Look for example at guix, that bootstraps explicitly with what they call "the maxwell equations of software". Imposing a web-based build system, into the kernel of all places, is a kick to the face to people who care about reproducibility.

nomoreusernames 5y ago

why do we have to have soooo many package managers? makes me cry inside a bit.

humanrebar 5y ago
Yeah, especially language specific ones. It seems like the world wants to be polyglot. Do we really want to have all the encryption algorithms reimplemented in all the languages? The answer there is often "use FFI". But if we're all going to use C APIs anyway, then shouldn't our package managers support C? But C has probably the most awkward build system culture of any language, complicating the job of packaging quite a bit. A lot of language specific package management systems get lift from offloading the ugly C support problems to other layers. See python and "manylinux" for instance.
- nomoreusernames 5y ago
  you are right about C. i mean i do get why we have so many. and i use them so often that its ingrained in me. and how to fix them when i mess up. but sometimes i notice how many libs we are talking about. honestly it is kinda ridiculous. i know how hard it is to manage one repo and keep it safe. maybe im too old. young ones now just download go binaries and run them.
pjmlp 5y ago
Because not everyone is using Linux distributions, and even those cannot agree in which system packaging they want to have.