10 ms·

Release engineering is exhausting so here's cargo-dist

262 points by ag_dubs 4y ago

oconnor663 4y ago
Bend the curve!
ZeroCool2u 4y ago
"Alright so I've given you The Pitch of how this should work, but how does it actually work today? Well, yes, I wouldn't announce this if not -- but there's a lot of future work to do! Actually no I take that back, it's definitely perfect. Those familiar with my broader oeuvre know this will absolutely be a perfectly crafted demo that avoids all the sharp corners of cargo-dist!" One of the most honest paragraphs ever written. Seriously though, great tool and great write up. I hope something like this lands as an official cargo feature. Coming from mostly Python land at work with crazy dependencies for TF and PyTorch GPU support (On Windows sometimes!) makes me super jealous.
- Yoric 4y ago
  Ouch. Python on Windows? I sympathize.
IshKebab 4y ago
> Congrats kid you're A Release Engineer now and your life is hell. Enjoy debugging basic typos on a remote machine with 20 minute latency because you sure can't run those Github CI bash-scripts-in-yaml files locally! Yes! Why is this accepted?? Gitlab has a way of running CI locally (for Docker based builds anyway; who knows about Windows or Mac) but a) it doesn't support the same features at the "proper" one (even basic ones like the `default` key) and b) they deprecated it! Ok in fairness they've stated in a random comment that they won't remove it before providing an alternative.... But still, how is this not a core feature of all CI systems?
- sedatk 4y ago
  You can run GitHub workflows locally with act: https://github.com/nektos/act https://github.com/nektos/act
  - ollien 4y ago
    This is far from perfect, IME. The big problem I have (and maybe there's a solution I don't know about) is that there's no easy way to test getting event data, unless you wanna rebuild the events yourself (which is hardly reliable if the problem is something like verifying a conditional that enables/disables a stage).
    - adrienthebo 4y ago
      Seconding this; act has a lot of potential but misses a number of features such as support for deployment environment variables (eg `${{ var.DEPLOY_SPECIFIC_ENV_VAR }}`) and only recently added support for reusable workflows (https://github.com/nektos/act/issues/826 https://github.com/nektos/act/issues/826). It looks like fine software and the maintainers deserve praise for their work but it's not yet a drop-in replacement for GitHub Actions.
  - frodowtf 4y ago
    Act is okay, but the runner image behaves quite differently than a GitHub runner would. The original image would just be too big for reasonable local workflows. Also artifacts don't seem to be supported.
    - netr0ute 4y ago
      > The original image would just be too big for reasonable local workflows. Is there a way so I can still do this? I have terabytes of SSD just waiting to be used.
- wizzwizz4 4y ago
  > But still, how is this not a core feature of all CI systems? Vendor lock-in, presumably.
  - WorldMaker 4y ago
    In multiple directions, too. It's easier to build the CI system itself if you are only targeting one class of servers/one means of hosting servers/one specific "cloud".
- capableweb 4y ago
  CircleCI solved this so many years ago, allow users to SSH into the environment the build happens. Workflow is something like: - Guess together a random CI or CD workflow, this is just to kick off the process. You can also start with a empty config. - Something fails, get SSH host+port to connect to - Enter environment and manually do everything you want to be able to automatically do - Execute `history` and copy output, trim it into something nicer and put in your config
- brundolf 4y ago
  What we need is a standard CI language/config format across vendors (stuffing a custom DSL inside YAML doesn't count) That would allow for a tooling ecosystem of static checkers, offline runners, IDE integrations, etc etc, and would also cut down on the learning barrier every time you switch to a new company that uses a different vendor
- angio 4y ago
  If you're using nix to build your repo, it's worth adding scripts for releases etc and run them locally and as part of the ci https://determinate.systems/posts/nix-github-actions https://determinate.systems/posts/nix-github-actions
- nsteel 4y ago
  If you just want to solve fundamental build issues (rather than say, uploading artefacts etc), I open a PR, make the silly small edits/experiments using the in-browser file editor, then it runs the CI each time. If I ever get it working, I then squash all the crappy debug I did to get there. Miles from ideal but a slight improvement.
Shared404 4y ago
> (Did you know the warning you get on Windows isn't about code signing, but is actually just a special flag Windows' builtin unzipping tool sets on all executables it extracts?) My jaw hit the floor here.
- Gankra 4y ago
  Correction on this someone else sent me: The check of interest is for a Mark Of The Web[0] flag that Windows includes in file system metadata. The builtin unzipping utility just faithfully propagates this flag to the files it unpacks. Other utilities like 7zip are unlikely to do this propagation (effectively clearing it). But yeah either way it has nothing to do with code signing! [0]: https://nolongerset.com/mark-of-the-web-details/ https://nolongerset.com/mark-of-the-web-details/
  - jakub_g 4y ago
    If someone's interested with more details about MoTW, EricLaw (long time MSFT engineer at IE and Edge teams) got you covered: https://textslashplain.com/2016/04/04/downloads-and-the-mark-of-the-web/ https://textslashplain.com/2016/04/04/downloads-and-the-mark... https://textslashplain.com/2022/12/02/mark-of-the-web-additional-guidance/ https://textslashplain.com/2022/12/02/mark-of-the-web-additi...
    - Zababa 4y ago
      Interesting, thank you for sharing!
  - chatmasta 4y ago
    macOS has a similar feature with Gatekeeper, which bit me when preparing a Pyinstaller binary for Mac. The flag doesn't get added when you download a file with curl, but it does when you download it through a web browser, which can cause difficult to debug issues with binaries downloaded from GitHub releases. You can remove this flag with the xattr command: xattr -d com.apple.quarantine the_quarantined_binary I wrote up the details of this in a PR [0] where I last dealt with it. [0] https://github.com/splitgraph/sgr/pull/656 https://github.com/splitgraph/sgr/pull/656
MuffinFlavored 4y ago
I think this would benefit from an example repo that shows just Cargo.toml for a simple src/main.rs with `fn main() { println!("Hello, world!"); }` project with the simplest needed .github/workflows/foo.yaml possible to actually use this. If it was in the article and I missed it I apologize.
- Gankra 4y ago
  The "way-too-quickstart" is the minimal example: https://github.com/axodotdev/cargo-dist#way-too-quick-start https://github.com/axodotdev/cargo-dist#way-too-quick-start A key feature of cargo-dist is that cargo dist init --ci=github should simply set everything up for an arbitrary* Rust workspace and you just check in the results. * Not sufficiently tested for all the wild workspaces you can build, but a "boring" one with a bunch of libraries supporting one binary should work -- that's what cargo-dist's own workspace is, and it's self-hosting without any configuration. Any time I want to update the bootstrap dist I just install that version of cargo-dist on my machine and run `cargo dist generate-ci github --installer=...` to completely overwrite the ci with the latest impl.
  - Gankra 4y ago
    Just to elaborate on this a bit: as discussed in the Concepts section of the docs[0] the core of cargo-dist absolutely supports workspaces with multiple binaries, and will chunk them out into their own distinct logical applications and provide builds/metadata for all them. However this isn't fully supported by the actual github CI integration yet[1], as I haven't implemented proper support for detecting that you're only trying to publish a new version of only one of the applications (or none of them!), and it doesn't properly merge the release notes if you're trying to publish multiple ones at once. I never build workspaces like that so I'm waiting for someone who does to chime in with the behaviour they want (since there's lots of defensible choices and I only have so many waking hours to implement stuff). [0]: https://github.com/axodotdev/cargo-dist/#concepts https://github.com/axodotdev/cargo-dist/#concepts [1]: https://github.com/axodotdev/cargo-dist/issues/69 https://github.com/axodotdev/cargo-dist/issues/69
  - MuffinFlavored 4y ago
chatmasta 4y ago
I'm really not a fan of the "download the prebuilt binary from github releases" workflow that's been proliferating along with the popularity of Rust. It seems like a step backward in terms of secure package management, and it's surprising to me that Rust doesn't offer a more out-of-box experience for this, instead encouraging users to build from source. I understand the arguments for this, and I even find some of them convincing - namely, that the inverse problem of opaque intermediate binaries and object files would be much worse as it would cause a cascade of brittle builds and barely any packages would work. But the fact remains that end users want to download a binary, and the common approach to this is to simply publish them to GitHub actions. Surely Cargo could offer a better option than this, while retaining the ability to build all packages from source (maybe you can only publish binaries to Cargo _in addition_ to the source files... or maybe Cargo.toml could include a link to GitHub releases, and Cargo could include a command for downloading directly from there.) In the meantime, I've been considering publishing Rust binaries to npm (in addition to GitHub releases). I got the idea from esbuild, which is written in Go but distributed via npm. What do people think of this approach? Here's a recent blog post [0] describing the setup. [0] https://blog.orhun.dev/packaging-rust-for-npm/ https://blog.orhun.dev/packaging-rust-for-npm/
- CuriousCosmic 4y ago
  Worth considering using nix with cargo. Of course it still involves a lot of "download from github" or "download from nix cache" but reproducibility + tight source hash pinning helps guarantee provenance.
- ag_dubs 4y ago
  we actually agree and are working on this! github releases are just an easy initial target, and makes our tool a drop-in replacement for the kinds of things people are already doing. longer-term we'd like to see something more robust, and cargo-dist is the first cog in that machine. i have personally packaged and published many rust devtools on npm (cloudflare's wrangler, apollo's rover, wasm-pack) but that was largely because they were targeted at a javascript developer audience. as a former npm registry engineer i'm curious what you find to be the particular value of publishing to npm? installing node is actually very unpleasant and then getting global installs to work is also... very unpleasant. i think it works well for people already in that ecosystem but i think we can build something better for a more agnostic audience that can give a similar out-of-box exp without requiring a centralized registry. would love to learn more about your perspective!
jph 4y ago
Awesome thank you. I'm adding it to the cargo favorites list: https://github.com/sixarm/cargo-install-favorites https://github.com/sixarm/cargo-install-favorites Tiny feedback: - Can you consider changing "git add ." to be explicit e.g. "git add Cargo.toml .github/workflows/release.yml"? - How about modifying Cargo.toml to add cargo-dist as a dev-dependency? I know it's not strictly necessary; it's just very helpful for typical collaboration.
- Gankra 4y ago
  Do you mean this kind of dep? https://rust-lang.github.io/rfcs/3028-cargo-binary-dependencies.html https://rust-lang.github.io/rfcs/3028-cargo-binary-dependenc... That's an interesting thought, I'm not sure I've ever seen someone employ that as a pattern. Actually no wait, I thought cargo bin-deps specifically gave the developer no way to manually invoke it (i.e. there's no equivalent functionality to npm's npx)? Without that, what use would the dependency be?
  - jph 4y ago
    Yes, where that link describes "[dev-dependencies]" and "[build-dependencies]". I use the dev-dependencies section often, and the build-dependencies rarely. And anyone here, please correct my understanding if there's a better way to do what I'm describing. For me, these sections are an easy way to be explicit with collaborators that the project needs the deps installed in order to work on the project, and does not want/need the deps to be compiled into the release. My #1 use case is to have a collaborator start with a git clone, then run "cargo build", and have cargo download and cache everything that's needed to work on the project and release it. My #2 use case is to deal with semver, so a project can be explicit that it needs a greater cargo-dist version in the future. To your point about cargo bin deps akin to npx, yes, your understanding matches mine i.e. not available yet. I do advocate for that feature to be added because it's helpful for local environments. Cargo does offer "default-run" which shows there an awareness of a developer preferring specific local exectuables-- maybe Cargo can/will add more like that for deps?
samsquire 4y ago
There's so much work to do to release software. Kind of explains why everything is a website.
- SOLAR_FIELDS 4y ago
  I think that this article and the discussion around it are more of a condemnation of the way GitHub Actions and similar software works rather than a generic “releasing software is hard”. One of the first things this guy mentioned in the article is that he went down this route because GHA sucks and he can’t run it locally (I know about act, but it ain’t a solution for everything) I realize you’ve probably thought about this a lot since I know you as the guy who wrote “mazzle” and posted about it here a few months ago. I wish more CI systems worked closer to the thing you designed.
  - samsquire 4y ago
    Thanks for remembering me :-) I would like things to run locally by default and then deployed to the cloud where they run. Should be easier to debug problems if I can get the code to my machine and investigate issues with tools that my computer has such as "strace", "perf" and debug logging that I liberally apply to the build script. In production we would have log aggregation and log search (such as ELK stack) and it is a good habit to get into the perspective of debugging production via tooling. But CICD feels before that tooling in the pipeline. You could wire up your CICD to log to ELK but I would prefer local deployable software. I think my focus on automating things means I want to be capable of seeing how the thing works without relying on a deployed black box in the cloud and using assumptions of how it works rather than direct investigation. One of my journal entries is almost a lamentation of all the things that need to be done to release and use software. This is that entry: https://github.com/samsquire/ideas4#5-permanent-softwareplatformlanguage--software-subscriptions-and-outsourceable-stack https://github.com/samsquire/ideas4#5-permanent-softwareplat... I wonder if software could be deployed more like a URL that has all the information to configure a virtual machine. Docker over URL or something.
  - howinteresting 4y ago
    Woman, she. In general, please don't unnecessarily gender (verb) people whose gender (noun) you don't know. Using "they" has been fine since the 13th century.
Chmouel 4y ago
i really enjoy goreleaser https://github.com/goreleaser/goreleaser/ https://github.com/goreleaser/goreleaser/ and use it with rust https://github.com/chmouel/snazy/blob/main/.goreleaser.yaml#L11 https://github.com/chmouel/snazy/blob/main/.goreleaser.yaml#... combined with build matrix https://github.com/chmouel/snazy/blob/main/.github/workflows/releaser.yaml#L33 https://github.com/chmouel/snazy/blob/main/.github/workflows...
- ag_dubs 4y ago
  definitely inspired by goreleaser! it's a great project
braincode 4y ago
How does this tool differ from release-plz?: https://github.com/marcoIeni/release-plz https://github.com/marcoIeni/release-plz
ElijahLynn 4y ago
This article does a really good job of setting up the "why" right in the beginning.
Animats 4y ago
OK, after reading through all that, I still can't tell if this can generate a Windows installer. Generating an installer is mentioned, but the examples seem to just generate a .tar file. Or maybe a .zip file. That's not what non-programmer Windows users expect. Rust does a good job of generating Windows binaries cross-platform. But the tools for generating Windows installers are not yet cross-plaform. Does this project improve that situation?
- Animats 4y ago
  The actual project is at [1]. It looks like the closest thing you can get to an installer is an 'executable zip'. [1] https://github.com/axodotdev/cargo-dist/ https://github.com/axodotdev/cargo-dist/
loveparade 4y ago
Nearly all of these problems (except for maybe the Windows one) are solved by publishing a nix flake with your application and telling people to install that instead. Not everyone uses nix, but I'd rather push its adoption than trying to build these kind of solutions for all possible language ecosystems when a general solution already exists and works great.
- deleted 4y ago
  [deleted]
- bin_bash 4y ago
  Unbelievable. So if they don't use Nix they simply don't deserve to use the software? I really wish people wouldn't get so zealous all the time about their favorite technology.
  - loveparade 4y ago
    Nix builds binaries. You don't need nix to run binaries built using nix. Just like you don't need Cargo to run binaries built using cargo. This is about developer tooling.
    - yencabulator 4y ago
      Nix normally binary edits /nix/store paths all over the place in the usual built binaries, those won't exist without nix installed & dependencies downloaded.
    - howinteresting 4y ago
      We should generally be pushing in the direction of developer tooling being available to more developers on more platforms. One of Rust's great strengths is good Windows support. Not supporting native Windows makes Nix a non-starter in many of these discussions.
      - loveparade 4y ago
        It's interesting how many people comment on nix without seeming to know anything about it. Why? Nix doesn't replace cargo. It would use cargo under the hood. If Cargo supports Windows builds, those builds will work with nix. However, nix (probably) doesn't give you anything additional in this case.
docandrew 4y ago
Really impressive work. The description of “release engineering” really hit home in regards to the slow feedback cycle with everything CI/CD. Also, seeing the initial attempt fail because of a missing C dependency is a common problem with a lot of language-specific package managers.
bin_bash 4y ago
If you happen to install a lot of things with cargo, check out cargo-binstall: https://github.com/cargo-bins/cargo-binstall https://github.com/cargo-bins/cargo-binstall It'll fetch the binary release from the repo so you don't have to compile it yourself.
Yoric 4y ago
Wow, that looks extremely useful! Thanks!

262 points by ag_dubs 4y ago

Bend the curve!

"Alright so I've given you The Pitch of how this should work, but how does it actually work today? Well, yes, I wouldn't announce this if not -- but there's a lot of future work to do! Actually no I take that back, it's definitely perfect. Those familiar with my broader oeuvre know this will absolutely be a perfectly crafted demo that avoids all the sharp corners of cargo-dist!" One of the most honest paragraphs ever written. Seriously though, great tool and great write up. I hope something like this lands as an official cargo feature. Coming from mostly Python land at work with crazy dependencies for TF and PyTorch GPU support (On Windows sometimes!) makes me super jealous.

Yoric 4y ago
Ouch. Python on Windows? I sympathize.

IshKebab 4y ago

> Congrats kid you're A Release Engineer now and your life is hell. Enjoy debugging basic typos on a remote machine with 20 minute latency because you sure can't run those Github CI bash-scripts-in-yaml files locally! Yes! Why is this accepted?? Gitlab has a way of running CI locally (for Docker based builds anyway; who knows about Windows or Mac) but a) it doesn't support the same features at the "proper" one (even basic ones like the `default` key) and b) they deprecated it! Ok in fairness they've stated in a random comment that they won't remove it before providing an alternative.... But still, how is this not a core feature of all CI systems?

sedatk 4y ago
You can run GitHub workflows locally with act: https://github.com/nektos/act https://github.com/nektos/act
- ollien 4y ago
  This is far from perfect, IME. The big problem I have (and maybe there's a solution I don't know about) is that there's no easy way to test getting event data, unless you wanna rebuild the events yourself (which is hardly reliable if the problem is something like verifying a conditional that enables/disables a stage).
  - adrienthebo 4y ago
    Seconding this; act has a lot of potential but misses a number of features such as support for deployment environment variables (eg `${{ var.DEPLOY_SPECIFIC_ENV_VAR }}`) and only recently added support for reusable workflows (https://github.com/nektos/act/issues/826 https://github.com/nektos/act/issues/826). It looks like fine software and the maintainers deserve praise for their work but it's not yet a drop-in replacement for GitHub Actions.
- frodowtf 4y ago
  Act is okay, but the runner image behaves quite differently than a GitHub runner would. The original image would just be too big for reasonable local workflows. Also artifacts don't seem to be supported.
  - netr0ute 4y ago
    > The original image would just be too big for reasonable local workflows. Is there a way so I can still do this? I have terabytes of SSD just waiting to be used.
wizzwizz4 4y ago
> But still, how is this not a core feature of all CI systems? Vendor lock-in, presumably.
- WorldMaker 4y ago
  In multiple directions, too. It's easier to build the CI system itself if you are only targeting one class of servers/one means of hosting servers/one specific "cloud".
capableweb 4y ago
CircleCI solved this so many years ago, allow users to SSH into the environment the build happens. Workflow is something like: - Guess together a random CI or CD workflow, this is just to kick off the process. You can also start with a empty config. - Something fails, get SSH host+port to connect to - Enter environment and manually do everything you want to be able to automatically do - Execute `history` and copy output, trim it into something nicer and put in your config
brundolf 4y ago
What we need is a standard CI language/config format across vendors (stuffing a custom DSL inside YAML doesn't count) That would allow for a tooling ecosystem of static checkers, offline runners, IDE integrations, etc etc, and would also cut down on the learning barrier every time you switch to a new company that uses a different vendor
angio 4y ago
If you're using nix to build your repo, it's worth adding scripts for releases etc and run them locally and as part of the ci https://determinate.systems/posts/nix-github-actions https://determinate.systems/posts/nix-github-actions
nsteel 4y ago
If you just want to solve fundamental build issues (rather than say, uploading artefacts etc), I open a PR, make the silly small edits/experiments using the in-browser file editor, then it runs the CI each time. If I ever get it working, I then squash all the crappy debug I did to get there. Miles from ideal but a slight improvement.

Shared404 4y ago

> (Did you know the warning you get on Windows isn't about code signing, but is actually just a special flag Windows' builtin unzipping tool sets on all executables it extracts?) My jaw hit the floor here.

Gankra 4y ago
Correction on this someone else sent me: The check of interest is for a Mark Of The Web[0] flag that Windows includes in file system metadata. The builtin unzipping utility just faithfully propagates this flag to the files it unpacks. Other utilities like 7zip are unlikely to do this propagation (effectively clearing it). But yeah either way it has nothing to do with code signing! [0]: https://nolongerset.com/mark-of-the-web-details/ https://nolongerset.com/mark-of-the-web-details/
- jakub_g 4y ago
  If someone's interested with more details about MoTW, EricLaw (long time MSFT engineer at IE and Edge teams) got you covered: https://textslashplain.com/2016/04/04/downloads-and-the-mark-of-the-web/ https://textslashplain.com/2016/04/04/downloads-and-the-mark... https://textslashplain.com/2022/12/02/mark-of-the-web-additional-guidance/ https://textslashplain.com/2022/12/02/mark-of-the-web-additi...
  - Zababa 4y ago
    Interesting, thank you for sharing!
- chatmasta 4y ago
  macOS has a similar feature with Gatekeeper, which bit me when preparing a Pyinstaller binary for Mac. The flag doesn't get added when you download a file with curl, but it does when you download it through a web browser, which can cause difficult to debug issues with binaries downloaded from GitHub releases. You can remove this flag with the xattr command: xattr -d com.apple.quarantine the_quarantined_binary I wrote up the details of this in a PR [0] where I last dealt with it. [0] https://github.com/splitgraph/sgr/pull/656 https://github.com/splitgraph/sgr/pull/656

MuffinFlavored 4y ago

I think this would benefit from an example repo that shows just Cargo.toml for a simple src/main.rs with `fn main() { println!("Hello, world!"); }` project with the simplest needed .github/workflows/foo.yaml possible to actually use this. If it was in the article and I missed it I apologize.

Gankra 4y ago
The "way-too-quickstart" is the minimal example: https://github.com/axodotdev/cargo-dist#way-too-quick-start https://github.com/axodotdev/cargo-dist#way-too-quick-start A key feature of cargo-dist is that cargo dist init --ci=github should simply set everything up for an arbitrary* Rust workspace and you just check in the results. * Not sufficiently tested for all the wild workspaces you can build, but a "boring" one with a bunch of libraries supporting one binary should work -- that's what cargo-dist's own workspace is, and it's self-hosting without any configuration. Any time I want to update the bootstrap dist I just install that version of cargo-dist on my machine and run `cargo dist generate-ci github --installer=...` to completely overwrite the ci with the latest impl.
- Gankra 4y ago
  Just to elaborate on this a bit: as discussed in the Concepts section of the docs[0] the core of cargo-dist absolutely supports workspaces with multiple binaries, and will chunk them out into their own distinct logical applications and provide builds/metadata for all them. However this isn't fully supported by the actual github CI integration yet[1], as I haven't implemented proper support for detecting that you're only trying to publish a new version of only one of the applications (or none of them!), and it doesn't properly merge the release notes if you're trying to publish multiple ones at once. I never build workspaces like that so I'm waiting for someone who does to chime in with the behaviour they want (since there's lots of defensible choices and I only have so many waking hours to implement stuff). [0]: https://github.com/axodotdev/cargo-dist/#concepts https://github.com/axodotdev/cargo-dist/#concepts [1]: https://github.com/axodotdev/cargo-dist/issues/69 https://github.com/axodotdev/cargo-dist/issues/69
- MuffinFlavored 4y ago

chatmasta 4y ago

I'm really not a fan of the "download the prebuilt binary from github releases" workflow that's been proliferating along with the popularity of Rust. It seems like a step backward in terms of secure package management, and it's surprising to me that Rust doesn't offer a more out-of-box experience for this, instead encouraging users to build from source. I understand the arguments for this, and I even find some of them convincing - namely, that the inverse problem of opaque intermediate binaries and object files would be much worse as it would cause a cascade of brittle builds and barely any packages would work. But the fact remains that end users want to download a binary, and the common approach to this is to simply publish them to GitHub actions. Surely Cargo could offer a better option than this, while retaining the ability to build all packages from source (maybe you can only publish binaries to Cargo _in addition_ to the source files... or maybe Cargo.toml could include a link to GitHub releases, and Cargo could include a command for downloading directly from there.) In the meantime, I've been considering publishing Rust binaries to npm (in addition to GitHub releases). I got the idea from esbuild, which is written in Go but distributed via npm. What do people think of this approach? Here's a recent blog post [0] describing the setup. [0] https://blog.orhun.dev/packaging-rust-for-npm/ https://blog.orhun.dev/packaging-rust-for-npm/

CuriousCosmic 4y ago
Worth considering using nix with cargo. Of course it still involves a lot of "download from github" or "download from nix cache" but reproducibility + tight source hash pinning helps guarantee provenance.
ag_dubs 4y ago
we actually agree and are working on this! github releases are just an easy initial target, and makes our tool a drop-in replacement for the kinds of things people are already doing. longer-term we'd like to see something more robust, and cargo-dist is the first cog in that machine. i have personally packaged and published many rust devtools on npm (cloudflare's wrangler, apollo's rover, wasm-pack) but that was largely because they were targeted at a javascript developer audience. as a former npm registry engineer i'm curious what you find to be the particular value of publishing to npm? installing node is actually very unpleasant and then getting global installs to work is also... very unpleasant. i think it works well for people already in that ecosystem but i think we can build something better for a more agnostic audience that can give a similar out-of-box exp without requiring a centralized registry. would love to learn more about your perspective!

jph 4y ago

Awesome thank you. I'm adding it to the cargo favorites list: https://github.com/sixarm/cargo-install-favorites https://github.com/sixarm/cargo-install-favorites Tiny feedback: - Can you consider changing "git add ." to be explicit e.g. "git add Cargo.toml .github/workflows/release.yml"? - How about modifying Cargo.toml to add cargo-dist as a dev-dependency? I know it's not strictly necessary; it's just very helpful for typical collaboration.

Gankra 4y ago
Do you mean this kind of dep? https://rust-lang.github.io/rfcs/3028-cargo-binary-dependencies.html https://rust-lang.github.io/rfcs/3028-cargo-binary-dependenc... That's an interesting thought, I'm not sure I've ever seen someone employ that as a pattern. Actually no wait, I thought cargo bin-deps specifically gave the developer no way to manually invoke it (i.e. there's no equivalent functionality to npm's npx)? Without that, what use would the dependency be?
- jph 4y ago
  Yes, where that link describes "[dev-dependencies]" and "[build-dependencies]". I use the dev-dependencies section often, and the build-dependencies rarely. And anyone here, please correct my understanding if there's a better way to do what I'm describing. For me, these sections are an easy way to be explicit with collaborators that the project needs the deps installed in order to work on the project, and does not want/need the deps to be compiled into the release. My #1 use case is to have a collaborator start with a git clone, then run "cargo build", and have cargo download and cache everything that's needed to work on the project and release it. My #2 use case is to deal with semver, so a project can be explicit that it needs a greater cargo-dist version in the future. To your point about cargo bin deps akin to npx, yes, your understanding matches mine i.e. not available yet. I do advocate for that feature to be added because it's helpful for local environments. Cargo does offer "default-run" which shows there an awareness of a developer preferring specific local exectuables-- maybe Cargo can/will add more like that for deps?

samsquire 4y ago

There's so much work to do to release software. Kind of explains why everything is a website.

SOLAR_FIELDS 4y ago
I think that this article and the discussion around it are more of a condemnation of the way GitHub Actions and similar software works rather than a generic “releasing software is hard”. One of the first things this guy mentioned in the article is that he went down this route because GHA sucks and he can’t run it locally (I know about act, but it ain’t a solution for everything) I realize you’ve probably thought about this a lot since I know you as the guy who wrote “mazzle” and posted about it here a few months ago. I wish more CI systems worked closer to the thing you designed.
- samsquire 4y ago
  Thanks for remembering me :-) I would like things to run locally by default and then deployed to the cloud where they run. Should be easier to debug problems if I can get the code to my machine and investigate issues with tools that my computer has such as "strace", "perf" and debug logging that I liberally apply to the build script. In production we would have log aggregation and log search (such as ELK stack) and it is a good habit to get into the perspective of debugging production via tooling. But CICD feels before that tooling in the pipeline. You could wire up your CICD to log to ELK but I would prefer local deployable software. I think my focus on automating things means I want to be capable of seeing how the thing works without relying on a deployed black box in the cloud and using assumptions of how it works rather than direct investigation. One of my journal entries is almost a lamentation of all the things that need to be done to release and use software. This is that entry: https://github.com/samsquire/ideas4#5-permanent-softwareplatformlanguage--software-subscriptions-and-outsourceable-stack https://github.com/samsquire/ideas4#5-permanent-softwareplat... I wonder if software could be deployed more like a URL that has all the information to configure a virtual machine. Docker over URL or something.
- howinteresting 4y ago
  Woman, she. In general, please don't unnecessarily gender (verb) people whose gender (noun) you don't know. Using "they" has been fine since the 13th century.

Chmouel 4y ago

i really enjoy goreleaser https://github.com/goreleaser/goreleaser/ https://github.com/goreleaser/goreleaser/ and use it with rust https://github.com/chmouel/snazy/blob/main/.goreleaser.yaml#L11 https://github.com/chmouel/snazy/blob/main/.goreleaser.yaml#... combined with build matrix https://github.com/chmouel/snazy/blob/main/.github/workflows/releaser.yaml#L33 https://github.com/chmouel/snazy/blob/main/.github/workflows...

ag_dubs 4y ago
definitely inspired by goreleaser! it's a great project

braincode 4y ago

How does this tool differ from release-plz?: https://github.com/marcoIeni/release-plz https://github.com/marcoIeni/release-plz

ElijahLynn 4y ago

This article does a really good job of setting up the "why" right in the beginning.

Animats 4y ago

OK, after reading through all that, I still can't tell if this can generate a Windows installer. Generating an installer is mentioned, but the examples seem to just generate a .tar file. Or maybe a .zip file. That's not what non-programmer Windows users expect. Rust does a good job of generating Windows binaries cross-platform. But the tools for generating Windows installers are not yet cross-plaform. Does this project improve that situation?

Animats 4y ago
The actual project is at [1]. It looks like the closest thing you can get to an installer is an 'executable zip'. [1] https://github.com/axodotdev/cargo-dist/ https://github.com/axodotdev/cargo-dist/

loveparade 4y ago

Nearly all of these problems (except for maybe the Windows one) are solved by publishing a nix flake with your application and telling people to install that instead. Not everyone uses nix, but I'd rather push its adoption than trying to build these kind of solutions for all possible language ecosystems when a general solution already exists and works great.

deleted 4y ago
[deleted]
bin_bash 4y ago
Unbelievable. So if they don't use Nix they simply don't deserve to use the software? I really wish people wouldn't get so zealous all the time about their favorite technology.
- loveparade 4y ago
  Nix builds binaries. You don't need nix to run binaries built using nix. Just like you don't need Cargo to run binaries built using cargo. This is about developer tooling.
  - yencabulator 4y ago
    Nix normally binary edits /nix/store paths all over the place in the usual built binaries, those won't exist without nix installed & dependencies downloaded.
  - howinteresting 4y ago
    We should generally be pushing in the direction of developer tooling being available to more developers on more platforms. One of Rust's great strengths is good Windows support. Not supporting native Windows makes Nix a non-starter in many of these discussions.
    - loveparade 4y ago
      It's interesting how many people comment on nix without seeming to know anything about it. Why? Nix doesn't replace cargo. It would use cargo under the hood. If Cargo supports Windows builds, those builds will work with nix. However, nix (probably) doesn't give you anything additional in this case.

docandrew 4y ago

Really impressive work. The description of “release engineering” really hit home in regards to the slow feedback cycle with everything CI/CD. Also, seeing the initial attempt fail because of a missing C dependency is a common problem with a lot of language-specific package managers.

bin_bash 4y ago

If you happen to install a lot of things with cargo, check out cargo-binstall: https://github.com/cargo-bins/cargo-binstall https://github.com/cargo-bins/cargo-binstall It'll fetch the binary release from the repo so you don't have to compile it yourself.

Yoric 4y ago

Wow, that looks extremely useful! Thanks!