5 ms·

Will this play nicely with the package management tools OSes already have, or is this going to end up being yet another source for files/packages to accumulate

Will this play nicely with the package management tools OSes already have, or is this going to end up being yet another source for files/packages to accumulate that are outside the view of well-documented and designed administrative tools?

sanderjd 12y ago
I think it would be just as easy to package rust programs using OS native package management as anything else, and I'm sure packages will be made for popular things for common package managers. But OS native package managers are a royal pain to use when actively developing on a project that has some dependencies, and bespoke Makefiles are a very imperfect solution. Flipping your question around a bit: will package manager creators and maintainers ever develop better solutions to the use case of development rather than system administration so that we don't have to keep creating these language-specific tools?
- kibwen 12y ago
  This is an insightful point. Nix is the only package manager I'm aware of that seems like it could fit the bill: https://nixos.org/nix/ https://nixos.org/nix/
  - eddyb 12y ago
    They already have automated Haskell packages from cabal, it shouldn't be hard to integrate cargo with nix once it's more stable. You wouldn't even require upstream NixOS packages, just place built cargo packages in the nix store, using it like a cache. Then upstream NixOS channels could start accumulating cargo packages, making cargo dependency "builds" faster.
  - sanderjd 12y ago
    Yeah, nix does look pretty awesome and aware of (even actively designed for )both the system administration and development use cases. If it becomes the native package manager for some popular operating systems, I may have to eat my words about language-specific solutions.
- Sanddancer 12y ago
  Have you taken a look any at the FreeBSD ports system? It uses makefiles to manage pretty much any software install you can think of. Additionally, there's a pretty good infrastructure there for managing custom build trees, etc. Additionally, they also have things like BSDpan that let you use arbitrary Perl modules with the package management system.
  - sanderjd 12y ago
    I've used ports, but I've never developed a project using it, so maybe I should try that out. But "uses makefiles" does not make me particularly optimistic for its pleasantness. I really prefer (and think it's been proven possible to build) declarative, rather than imperative, systems for managing dependencies.
ithkuil 12y ago
I had the impression this fills the same role as bower, mpm (non global) dependency or plain custom vendor dirs handled with your favourite git module paraphernalia: your build is the place your dependencies live (+ possible cache directory somewhere else). And, these you use this packaging system to manage your build, not the deployment of your (iirc statically linked) build artifacts. While technically the cache directory is a place where files can accumulate outside the view of a well documented and designed administrative tools, this is common problem shared with many tools including your favourite browser.
deleted 12y ago
[deleted]
alextgordon 12y ago
Ha! There is nothing well-designed about the mutually incompatible, political hell that is OS package managers. The best solution (as taken by npm, virtualenv and others) is to install libraries locally to the project that is building them. That way, package management becomes the sole concern of the build system. "Accumulation" is a good thing, it means each project has the exact version of a package that it was tested with, not some later version that a sysadmin decides it "probably should work with".
- bryanlarsen 12y ago
  OS package managers make two assumptions: - that packages follow semver - that the OS packagers are in a better position to test package combinations. If the author releases a new version of libfoo, and A, B and C in an OS repo depend on libfoo, then the OS packagers do not release a new version of libfoo until the tests for A, B & C pass. These are two good assumptions, and the language package world would be in much better shape if they followed those assumptions too.
  - steveklabnik 12y ago
    Cargo assumes the former.
  - pjmlp 12y ago
    Which OS are "OS package managers"? There is a world out there greater than GNU/Linux.
- Sanddancer 12y ago
  Which is why we have testing and staging environments to make these changes on first. Conversely, when everything is pulling in their own version of a library, you run into situations like the zlib buffer overflow, where you had huge numbers of programs that needed to be rebuilt because no one used system-packaged libraries. Obsolete and vulnerable software is a liability, and not having tools which can readily tell what software needs to be updated makes quite a few peoples' jobs quite a bit harder.
awj 12y ago
Until the various OS package managers stop pretending they are a universe unto themselves, this problem will continue. Languages have to distribute libraries across all of their host systems, so focusing on support for any particular package manager (or small subset thereof) doesn't buy much. Hopefully the cargo team can come up with a solution that works a little better here, but I wouldn't hold my breath.
dmacvicar 12y ago
Each language having its own tools is fine and I doubt OSes are against that. Those tools just need to support a very specific set of features or scenarios to make packaging easy: Understand that enterprise OSes are not built on developer Macbooks. Enterprise distros have reproducible builds on automated systems with chroots that contain only what the package needs, no network access and sometimes the build happens inside a virtual machine. Its is almost ridiculous how after maven forgot the topic of being "offline" almost every tool released afterward has done the same mistake. Understand that Linux distributions sell support and that means being able to rebuild a patched version of a failed package. So whatever dependencies are pinned in Cargo.toml or Gemfile is irrelevant. The OS maker will override them as a result of testing, patching or simply to share one dependency across multiple packages. Distros can't afford having one package per git revision used on a popular distro and then afford to fix the same security issue in all of them. So having "cargo build" be able to override dependencies and instead look on the installed libraries with a command line switch or env variable helps the packager not having to rewrite the Cargo.toml in the package recipe. Maven was probably the first tool that made packaging almost impossible and completely ignored the use-case of patching a component of the chain and be able to rebuild it completely. Semantic versioning is great news, because it allows you to replace the dependencies for a common semantic version (not exactly the same). For integrating with the C libraries not much needs to be done. If you support pkg-config then you cover 85% of the cases.

sanderjd 12y ago

I think it would be just as easy to package rust programs using OS native package management as anything else, and I'm sure packages will be made for popular things for common package managers. But OS native package managers are a royal pain to use when actively developing on a project that has some dependencies, and bespoke Makefiles are a very imperfect solution. Flipping your question around a bit: will package manager creators and maintainers ever develop better solutions to the use case of development rather than system administration so that we don't have to keep creating these language-specific tools?

kibwen 12y ago
This is an insightful point. Nix is the only package manager I'm aware of that seems like it could fit the bill: https://nixos.org/nix/ https://nixos.org/nix/
- eddyb 12y ago
  They already have automated Haskell packages from cabal, it shouldn't be hard to integrate cargo with nix once it's more stable. You wouldn't even require upstream NixOS packages, just place built cargo packages in the nix store, using it like a cache. Then upstream NixOS channels could start accumulating cargo packages, making cargo dependency "builds" faster.
- sanderjd 12y ago
  Yeah, nix does look pretty awesome and aware of (even actively designed for )both the system administration and development use cases. If it becomes the native package manager for some popular operating systems, I may have to eat my words about language-specific solutions.
Sanddancer 12y ago
Have you taken a look any at the FreeBSD ports system? It uses makefiles to manage pretty much any software install you can think of. Additionally, there's a pretty good infrastructure there for managing custom build trees, etc. Additionally, they also have things like BSDpan that let you use arbitrary Perl modules with the package management system.
- sanderjd 12y ago
  I've used ports, but I've never developed a project using it, so maybe I should try that out. But "uses makefiles" does not make me particularly optimistic for its pleasantness. I really prefer (and think it's been proven possible to build) declarative, rather than imperative, systems for managing dependencies.

ithkuil 12y ago

I had the impression this fills the same role as bower, mpm (non global) dependency or plain custom vendor dirs handled with your favourite git module paraphernalia: your build is the place your dependencies live (+ possible cache directory somewhere else). And, these you use this packaging system to manage your build, not the deployment of your (iirc statically linked) build artifacts. While technically the cache directory is a place where files can accumulate outside the view of a well documented and designed administrative tools, this is common problem shared with many tools including your favourite browser.

deleted 12y ago

[deleted]

alextgordon 12y ago

Ha! There is nothing well-designed about the mutually incompatible, political hell that is OS package managers. The best solution (as taken by npm, virtualenv and others) is to install libraries locally to the project that is building them. That way, package management becomes the sole concern of the build system. "Accumulation" is a good thing, it means each project has the exact version of a package that it was tested with, not some later version that a sysadmin decides it "probably should work with".

bryanlarsen 12y ago
OS package managers make two assumptions: - that packages follow semver - that the OS packagers are in a better position to test package combinations. If the author releases a new version of libfoo, and A, B and C in an OS repo depend on libfoo, then the OS packagers do not release a new version of libfoo until the tests for A, B & C pass. These are two good assumptions, and the language package world would be in much better shape if they followed those assumptions too.
- steveklabnik 12y ago
  Cargo assumes the former.
- pjmlp 12y ago
  Which OS are "OS package managers"? There is a world out there greater than GNU/Linux.
Sanddancer 12y ago
Which is why we have testing and staging environments to make these changes on first. Conversely, when everything is pulling in their own version of a library, you run into situations like the zlib buffer overflow, where you had huge numbers of programs that needed to be rebuilt because no one used system-packaged libraries. Obsolete and vulnerable software is a liability, and not having tools which can readily tell what software needs to be updated makes quite a few peoples' jobs quite a bit harder.

awj 12y ago

Until the various OS package managers stop pretending they are a universe unto themselves, this problem will continue. Languages have to distribute libraries across all of their host systems, so focusing on support for any particular package manager (or small subset thereof) doesn't buy much. Hopefully the cargo team can come up with a solution that works a little better here, but I wouldn't hold my breath.

dmacvicar 12y ago

Each language having its own tools is fine and I doubt OSes are against that. Those tools just need to support a very specific set of features or scenarios to make packaging easy: Understand that enterprise OSes are not built on developer Macbooks. Enterprise distros have reproducible builds on automated systems with chroots that contain only what the package needs, no network access and sometimes the build happens inside a virtual machine. Its is almost ridiculous how after maven forgot the topic of being "offline" almost every tool released afterward has done the same mistake. Understand that Linux distributions sell support and that means being able to rebuild a patched version of a failed package. So whatever dependencies are pinned in Cargo.toml or Gemfile is irrelevant. The OS maker will override them as a result of testing, patching or simply to share one dependency across multiple packages. Distros can't afford having one package per git revision used on a popular distro and then afford to fix the same security issue in all of them. So having "cargo build" be able to override dependencies and instead look on the installed libraries with a command line switch or env variable helps the packager not having to rewrite the Cargo.toml in the package recipe. Maven was probably the first tool that made packaging almost impossible and completely ignored the use-case of patching a component of the chain and be able to rebuild it completely. Semantic versioning is great news, because it allows you to replace the dependencies for a common semantic version (not exactly the same). For integrating with the C libraries not much needs to be done. If you support pkg-config then you cover 85% of the cases.