15 ms·

The best C and [insert any language here] package manager is Nix. Or rather, it should be[1]. I'm fairly confident something like it will be. If you haven't, pl

The best C and [insert any language here] package manager is Nix. Or rather, it should be[1]. I'm fairly confident something like it will be. If you haven't, please do try it out (or maybe don't[1], but at least read the paper[2] and let its ideas sink in). I honestly just sigh and think "yet another package manager and they still haven't learned..." whenever I see a new one now. [1]: Unfortunately documentation is famously bad, the repo that holds package declarations is full of undocumented conventions, there often isn't a "blessed" way of doing things, and there are some other warts, such as all the difficulty one may experience from having to learn a new functional, lazy, dynamically typed language. I'm not aware of active, coordinated efforts to improve on these fronts, except for new languages such as Nickel. [2]: https://edolstra.github.io/pubs/phd-thesis.pdf https://edolstra.github.io/pubs/phd-thesis.pdf (don't be discouraged by the number of pages, it's fine if you read just the first part)

pjmlp 5y ago
Assuming GNU/Linux is the only thing C developers care about.
- tadfisher 5y ago
  Nix also runs on Darwin and BSD.
  - pjmlp 5y ago
    Ok a bit better, doesn't change the fact it doesn't run everywhere where there is a C compiler.
    - _k9eq 5y ago
      Not everyone using C is writing software that turns on all platforms supported by any C compiler?
      - pjmlp 5y ago
        Not everyone using C cares about UNIX FOSS clones.
      - bregma 5y ago
        Not everyone is writing C software to run only on their own laptop.
        tadfisher 5y ago
        I'm confused, do we expect end-users to install a language package manager to use software? Nix would only be used wherever the software is built.
- zeec123 5y ago
  Yes, today nix does not run everywhere C is used. So what? More platforms can be supported in the future.
  - pjmlp 5y ago
    Then it isn't a solution worth spending any time on.
    - zeec123 5y ago
      So if not all and every platform is supported from day 1, then “it isn’t a solution worth spending any time on”?
      - pjmlp 5y ago
        Yes, given that is keeps being advocated as a solution to solve all problems.
- shadowofneptune 5y ago
  It's such a deep issue with the C and C++ ecosystem that it's probably what will finally lead systems programming to other languages. It wasn't an issue until the Internet, and still isn't an issue if you are linking against binaries, but definitely is a problem now. Windows shouldn't be treated like a freak, something too outside the norm to support compiling on.
  - pjmlp 5y ago
    Windows isn't the only other OS out there, plenty of them on IoT, mainframes, game consoles.
throwaway984393 5y ago
Nix is an academic's solution to a large problem space based on a single principle ignoring several real limitations in order to shoehorn itself into an existing box of a problem. Of the many problems it has are assumptions about development models, deployment models, and operational models. Moreover, it seeks to be a monolithic proprietary solution rather than a collection of loosely-coupled layers that can be interchanged. But what irks me is that it doesn't fundamentally change anything. There are clearly major inherent limitations in how software is developed that led us here. Nix is intended to find complicated ways to work around the problems in those development patterns. But we can't call out the pink elephant. Rather, let's make pink wallpaper, pink sofas, pink chairs, pink rugs, and then say that we've solved the problem, because you can barely even see the elephant now. The problem isn't packaging. The problem is software design itself. The dependency hell, the conflicting versions, the filesystem hash trees, the DSL. It all exists because the software itself has no way to express its own compatibility with other functions or dependencies. We just kick the can down the road to a random package manager and hope for the best. But clearly the solution needs to happen in the software, not in how we install it.
- willhinsa 5y ago
  Is there an example of this being done the "right" way? I'm interested in learning more about this.
  - throwaway984393 5y ago
    Glibc allows specifying the version of a function at call time, so that you can call a specific version of a function in a library. This means that regardless of when you released some software, you can pin how it works, so even if you upgrade the library 10 years later, it will work the same way (as long as the library keeps the old version of the function). This needs to be extended, made easier, and every programming language should do it by default. The filesystem is also a limitation. We need to refer to versions in files and directories in the same way, due to application dependence on files. This can be accomplished probably just by adding versions in paths and then defining version compatibility in code. Would be nice if the filesystems all got versioning built in, though, so we could just call file functions with version arguments rather than hacking around path names. Data formats and protocols also need explicit versions for all operations, so they can advertise their compatibility and thus be backwards and forwards compatible. We must also be able to execute software with a specific combination of versioned functions at exec time. This requires kernel modifications / new syscalls. Libraries and applications should keep old versions of functions indefinitely, and only load code when needed. This will cause some consequences which we need solutions for, such as how to handle the reduced disk space and shared memory. It will require rethinking how applications are built, stored, and executed. All of this rethinking of software design requires a lot of effort. But it would provide a world of benefits that we currently need hacks for. We could upgrade randomly to bleeding edge software without any change to existing applications; any software interacting would continue to function as before. But newer function versions would be available and could be enabled via feature flags. We would no longer need containers, maybe not even package managers. Software could be upgraded continuously and remain perfectly stable. Users could define what versions they will execute depending on their wishes. And legacy software could be supported for decades, maybe centuries, with no extra effort.
infogulch 5y ago
Dynamic typing irks me too. Apparently you can write Dhall that compiles to nix, effectively making it a statically typed language on top of nix.
dotancohen 5y ago
> The best C and [insert any language here] package manager is Nix. I raise you a docker. Seriously, the reproducibility and ability to just share a Dockerfile is amazing, especially with version pinning.
- ModernMech 5y ago
  I have a colleague who insists that that docker is the C++ package manager. I dunno, I've tried to see his point of view but as a usability issue it seems like far too much complexity to just use Docker as a package manager. Compared to something like Cargo for Rust, Docker is incredibly complicated and nags you all the time about installing upgrades. And my understanding is that Docker has non-zero performance impact, right?
  - dotancohen 5y ago
    Docker performance impact might as well be zero. You're actually using the system kernel, with the application's filesystem (thus dependencies) and other resources (CPU, memory, network access) namespaced away from the rest of the system. So there is no e.g. mapping on top of the extant kernel mapping. I've not noticed and performance impact, though obviously my workflow isn't your workflow. Seriously, try it. I personally use docker-compose, even for single containers.
    - pornel 5y ago
      Docker is reasonably performant only on Linux. On macOS it's a fat slow VM that bogs down the whole machine (unless you make the VM even slower by giving it fewer CPU cores), has a horrendously slow network file system, and it's managed by a needy Electron app. And don't forget that all new Macs are now ARM-based, so x86-based Docker images don't work. My gripe with C is that everyone pretends it's portable, but only care about porting to one (their) platform. It's easy: just use pkg-config/docker/nix. Windows? macOS? Who cares!? Just run a Linux VM!
    - duped 5y ago
      > Docker performance impact might as well be zero. But god help you if you care about MacOS and Windows
wocram 5y ago
Nix is nice as a catch-all, and especially good when languages like C and C++ don't have popular package managers. It's definitely not better than cargo or pip when dealing with just Rust or Python, but it's definitely relevant when you're building multiple languages. I would love to see an evolution of nixpkgs with more of a starlark-style for package declarations.
- jonringer117 5y ago
  This likely wouldn't work on practice. Nix is aggressively lazy, so if you just want a single package, you don't have to evaluate the builds of all 60k+ packages, just the package you wanted and its dependencies. This just might be me as a functional-programming-maximalist, but I do not like imperative languages for configuration management. You implicitly have to hold the execution state of all actions to know what is going on.
  - wocram 5y ago
    BUILD file evaluation is very similar to nix expression evaluation. I think the bazel 'analysis' phase is equivalent to the nix derivation expansion step Maybe nix is more lazy, but I would be surprised if nix ends up doing substantially less work than bazel when a single target in a large graph is built.
- johnisgood 5y ago
  Does cargo re-use the same packages between projects? I am asking because cargo is the only build system capable of making me run out of space.
  - wocram 5y ago
    It can if you use a cargo workspace to build all of your projects.
- smilliken 5y ago
  For managing python dependencies, nix avoids all the headaches of pip, poetry, etc. Builds always work, and one has confidence that they have exactly the same bits as everyone else. nix-shell gives you a virtualenv that also has all your non-python dependencies as well. Instead of dealing with 10 different package managers across a codebase, there's just one.

pjmlp 5y ago

Assuming GNU/Linux is the only thing C developers care about.

tadfisher 5y ago
Nix also runs on Darwin and BSD.
- pjmlp 5y ago
  Ok a bit better, doesn't change the fact it doesn't run everywhere where there is a C compiler.
  - _k9eq 5y ago
    Not everyone using C is writing software that turns on all platforms supported by any C compiler?
    - pjmlp 5y ago
      Not everyone using C cares about UNIX FOSS clones.
    - bregma 5y ago
      Not everyone is writing C software to run only on their own laptop.
      - tadfisher 5y ago
        I'm confused, do we expect end-users to install a language package manager to use software? Nix would only be used wherever the software is built.
zeec123 5y ago
Yes, today nix does not run everywhere C is used. So what? More platforms can be supported in the future.
- pjmlp 5y ago
  Then it isn't a solution worth spending any time on.
  - zeec123 5y ago
    So if not all and every platform is supported from day 1, then “it isn’t a solution worth spending any time on”?
    - pjmlp 5y ago
      Yes, given that is keeps being advocated as a solution to solve all problems.
shadowofneptune 5y ago
It's such a deep issue with the C and C++ ecosystem that it's probably what will finally lead systems programming to other languages. It wasn't an issue until the Internet, and still isn't an issue if you are linking against binaries, but definitely is a problem now. Windows shouldn't be treated like a freak, something too outside the norm to support compiling on.
- pjmlp 5y ago
  Windows isn't the only other OS out there, plenty of them on IoT, mainframes, game consoles.

throwaway984393 5y ago

Nix is an academic's solution to a large problem space based on a single principle ignoring several real limitations in order to shoehorn itself into an existing box of a problem. Of the many problems it has are assumptions about development models, deployment models, and operational models. Moreover, it seeks to be a monolithic proprietary solution rather than a collection of loosely-coupled layers that can be interchanged. But what irks me is that it doesn't fundamentally change anything. There are clearly major inherent limitations in how software is developed that led us here. Nix is intended to find complicated ways to work around the problems in those development patterns. But we can't call out the pink elephant. Rather, let's make pink wallpaper, pink sofas, pink chairs, pink rugs, and then say that we've solved the problem, because you can barely even see the elephant now. The problem isn't packaging. The problem is software design itself. The dependency hell, the conflicting versions, the filesystem hash trees, the DSL. It all exists because the software itself has no way to express its own compatibility with other functions or dependencies. We just kick the can down the road to a random package manager and hope for the best. But clearly the solution needs to happen in the software, not in how we install it.

willhinsa 5y ago
Is there an example of this being done the "right" way? I'm interested in learning more about this.
- throwaway984393 5y ago
  Glibc allows specifying the version of a function at call time, so that you can call a specific version of a function in a library. This means that regardless of when you released some software, you can pin how it works, so even if you upgrade the library 10 years later, it will work the same way (as long as the library keeps the old version of the function). This needs to be extended, made easier, and every programming language should do it by default. The filesystem is also a limitation. We need to refer to versions in files and directories in the same way, due to application dependence on files. This can be accomplished probably just by adding versions in paths and then defining version compatibility in code. Would be nice if the filesystems all got versioning built in, though, so we could just call file functions with version arguments rather than hacking around path names. Data formats and protocols also need explicit versions for all operations, so they can advertise their compatibility and thus be backwards and forwards compatible. We must also be able to execute software with a specific combination of versioned functions at exec time. This requires kernel modifications / new syscalls. Libraries and applications should keep old versions of functions indefinitely, and only load code when needed. This will cause some consequences which we need solutions for, such as how to handle the reduced disk space and shared memory. It will require rethinking how applications are built, stored, and executed. All of this rethinking of software design requires a lot of effort. But it would provide a world of benefits that we currently need hacks for. We could upgrade randomly to bleeding edge software without any change to existing applications; any software interacting would continue to function as before. But newer function versions would be available and could be enabled via feature flags. We would no longer need containers, maybe not even package managers. Software could be upgraded continuously and remain perfectly stable. Users could define what versions they will execute depending on their wishes. And legacy software could be supported for decades, maybe centuries, with no extra effort.

infogulch 5y ago

Dynamic typing irks me too. Apparently you can write Dhall that compiles to nix, effectively making it a statically typed language on top of nix.

dotancohen 5y ago

> The best C and [insert any language here] package manager is Nix. I raise you a docker. Seriously, the reproducibility and ability to just share a Dockerfile is amazing, especially with version pinning.

ModernMech 5y ago
I have a colleague who insists that that docker is the C++ package manager. I dunno, I've tried to see his point of view but as a usability issue it seems like far too much complexity to just use Docker as a package manager. Compared to something like Cargo for Rust, Docker is incredibly complicated and nags you all the time about installing upgrades. And my understanding is that Docker has non-zero performance impact, right?
- dotancohen 5y ago
  Docker performance impact might as well be zero. You're actually using the system kernel, with the application's filesystem (thus dependencies) and other resources (CPU, memory, network access) namespaced away from the rest of the system. So there is no e.g. mapping on top of the extant kernel mapping. I've not noticed and performance impact, though obviously my workflow isn't your workflow. Seriously, try it. I personally use docker-compose, even for single containers.
  - pornel 5y ago
    Docker is reasonably performant only on Linux. On macOS it's a fat slow VM that bogs down the whole machine (unless you make the VM even slower by giving it fewer CPU cores), has a horrendously slow network file system, and it's managed by a needy Electron app. And don't forget that all new Macs are now ARM-based, so x86-based Docker images don't work. My gripe with C is that everyone pretends it's portable, but only care about porting to one (their) platform. It's easy: just use pkg-config/docker/nix. Windows? macOS? Who cares!? Just run a Linux VM!
  - duped 5y ago
    > Docker performance impact might as well be zero. But god help you if you care about MacOS and Windows

wocram 5y ago

Nix is nice as a catch-all, and especially good when languages like C and C++ don't have popular package managers. It's definitely not better than cargo or pip when dealing with just Rust or Python, but it's definitely relevant when you're building multiple languages. I would love to see an evolution of nixpkgs with more of a starlark-style for package declarations.

jonringer117 5y ago
This likely wouldn't work on practice. Nix is aggressively lazy, so if you just want a single package, you don't have to evaluate the builds of all 60k+ packages, just the package you wanted and its dependencies. This just might be me as a functional-programming-maximalist, but I do not like imperative languages for configuration management. You implicitly have to hold the execution state of all actions to know what is going on.
- wocram 5y ago
  BUILD file evaluation is very similar to nix expression evaluation. I think the bazel 'analysis' phase is equivalent to the nix derivation expansion step Maybe nix is more lazy, but I would be surprised if nix ends up doing substantially less work than bazel when a single target in a large graph is built.
johnisgood 5y ago
Does cargo re-use the same packages between projects? I am asking because cargo is the only build system capable of making me run out of space.
- wocram 5y ago
  It can if you use a cargo workspace to build all of your projects.
smilliken 5y ago
For managing python dependencies, nix avoids all the headaches of pip, poetry, etc. Builds always work, and one has confidence that they have exactly the same bits as everyone else. nix-shell gives you a virtualenv that also has all your non-python dependencies as well. Instead of dealing with 10 different package managers across a codebase, there's just one.