6 ms·

Nix is an academic's solution to a large problem space based on a single principle ignoring several real limitations in order to shoehorn itself into an existin

Nix is an academic's solution to a large problem space based on a single principle ignoring several real limitations in order to shoehorn itself into an existing box of a problem. Of the many problems it has are assumptions about development models, deployment models, and operational models. Moreover, it seeks to be a monolithic proprietary solution rather than a collection of loosely-coupled layers that can be interchanged. But what irks me is that it doesn't fundamentally change anything. There are clearly major inherent limitations in how software is developed that led us here. Nix is intended to find complicated ways to work around the problems in those development patterns. But we can't call out the pink elephant. Rather, let's make pink wallpaper, pink sofas, pink chairs, pink rugs, and then say that we've solved the problem, because you can barely even see the elephant now. The problem isn't packaging. The problem is software design itself. The dependency hell, the conflicting versions, the filesystem hash trees, the DSL. It all exists because the software itself has no way to express its own compatibility with other functions or dependencies. We just kick the can down the road to a random package manager and hope for the best. But clearly the solution needs to happen in the software, not in how we install it.

willhinsa 5y ago
Is there an example of this being done the "right" way? I'm interested in learning more about this.
- throwaway984393 5y ago
  Glibc allows specifying the version of a function at call time, so that you can call a specific version of a function in a library. This means that regardless of when you released some software, you can pin how it works, so even if you upgrade the library 10 years later, it will work the same way (as long as the library keeps the old version of the function). This needs to be extended, made easier, and every programming language should do it by default. The filesystem is also a limitation. We need to refer to versions in files and directories in the same way, due to application dependence on files. This can be accomplished probably just by adding versions in paths and then defining version compatibility in code. Would be nice if the filesystems all got versioning built in, though, so we could just call file functions with version arguments rather than hacking around path names. Data formats and protocols also need explicit versions for all operations, so they can advertise their compatibility and thus be backwards and forwards compatible. We must also be able to execute software with a specific combination of versioned functions at exec time. This requires kernel modifications / new syscalls. Libraries and applications should keep old versions of functions indefinitely, and only load code when needed. This will cause some consequences which we need solutions for, such as how to handle the reduced disk space and shared memory. It will require rethinking how applications are built, stored, and executed. All of this rethinking of software design requires a lot of effort. But it would provide a world of benefits that we currently need hacks for. We could upgrade randomly to bleeding edge software without any change to existing applications; any software interacting would continue to function as before. But newer function versions would be available and could be enabled via feature flags. We would no longer need containers, maybe not even package managers. Software could be upgraded continuously and remain perfectly stable. Users could define what versions they will execute depending on their wishes. And legacy software could be supported for decades, maybe centuries, with no extra effort.
  - carapace 5y ago
    Have you seen Unison lang? It sounds like it overlaps (at least) with what you're talking about. > Unison’s core idea is that code is immutable and identified by its content. This lets us reimagine many aspects of how a programming language works. We simplify codebase management — Unison has no builds, no dependency conflicts, and renaming things is trivial. https://www.unisonweb.org/ https://www.unisonweb.org/ FWIW, I agree with you that the way we do things now is, uh, not the best of all possible worlds. I think Nix (and Guix) is a step in the right direction, though, but they don't go far enough.
  - virchau13 5y ago
    > you can call a specific version of a function in a library A library has some functions. Later, the library gets refactored, and those functions are thereby rewritten; but since no changes have actually been made, the version number stays the same. However, there exists some legacy code that horrendously breaks when given a non-ABI-compatible function from this library, _even though_ the functionality and version of those functions the same. Oh dear! What can we do? Well, clearly exact binary compatibility is important, so someone has an idea: we'll specify functions with exact hashes of their code instead of versioning, so this problem doesn't happen. This is all well and good, but then the library decides to update some internal utility subroutine, changing its hash, but _not_ changing its call sites. Oh dear, now some other functions have hashes that are still the same, but they have different functionality due to that utility update, and legacy code breaks horribly again! In order to make sure this doesn't happen, someone comes up with the idea that a function's hash should be dependent on the hashes of all of the functions, variables, etc. it depends on _and_ its code, so all of the "inputs" that go towards defining the function end up updating its hash, thereby insuring legacy code stability. Now we run into another problem: where do we put all of these functions? They need to be stored somewhere on the system so applications can have access to them, so we'll put them in a repository that we'll call a "store". Now we can do cool stuff with shared stores such as "only load code when needed"! Oh dear, now the user sitting at the shell of the system wants to run programs too. Our function infrastructure works so well, why not extend it to programs too? Some programs really like filesystems and environments and depend on specific structures. We can't really encode these structures as an "input" to our programs' hashes, because they could be highly dynamic (or even undescribable), so instead we can put software into a little box where it sees the filesystem structure it wants, so it is happy, and the box itself could be specified with input programs/functions in the same way that programs or functions can! Hmm.. we seem to be recreating this "hash" structure that depends on inputs a lot. We need a common name for this... hmm, how about "derivations"? Congratulations, you have just reinvented Nix! The solutions in your comment that you talk about are things that Nix already was designed to provide. The only discrepancy is that Nix doesn't actually provide hashes for individual functions, rather they work for libraries, but this is because of a limitation on how libraries work: that they can have shared, global, mutable state, making the idea of individual function versions useless. The main problem why Nix doesn't obviously seem to solve these problems is that Nix's tools are not very well designed. For example, it's very non-obvious how to specify older versions of packages. However, flakes makes this easier, and although I don't believe _Nix_ specifically will be the solution, the holy grail at the end of the road will be very similar to Nix in core ideas.
slabity 5y ago
You're stating a lot of interesting opinions, but I do not see any arguments to help describe why you feel that way. How exactly is Nix monolithic or proprietary? And what is complicated about it? It seems pretty simple to me, if not at odds with how other package managers do things. It's one of the few package managers that can successfully work across multiple Linux distributions, some BSD systems, MacOS, and even Windows for some software. In my opinion it's the least proprietary or monolithic package managers around, the other being Guix which follows the same principles. But I think the biggest issue is in your last paragraph where you don't describe why or how the solution needs to happen in the software itself rather than at a higher level.
- goosedragons 5y ago
  If you need something that exists outside the repository it can get complicated. Even playing within the repository can have pitfalls at least on Guix which is what I've mainly used (like having a package appear in two profiles can cause issues that are non-obvious or like using "guix shell gcc binutils" instead of " guix shell gcc-toolchain"). Some of that has to do with the fact of what an expected *nix install is supposed to be and expectations of what should be installed on a users system and so are not marked as dependencies or requirements when they really are. I am curious about how a system designed from the ground up with this in mind might be better.
ramses0 5y ago
Suppose: 1) foo.lang: class foo { func bar(a) { return b } } 2) baz.lang: import foo && print( foo.bar(1) ) Generally `foo` needs to have unit tests (self-tests), but `baz` needs to have "acceptance tests" for `foo` because `baz` imports `foo` (external validation / fit-for-purpose). Blah blah "law of demeter" aka: self.do_foo_bar() { foo.bar(1) }. I've come to the mental model that code should test itself, and (some) code should test its dependencies. Code fundamentally can't test code that uses it, but can attempt to be "well-behaved" and share (via documentation or checked interface) how it can be / should be used. eg: `foo_test.lang: assert( foo.bar(1) == 42 );` But if `baz` calls `foo.bar(2)` or `foo.bar(0.3)` or `foo.bar("four")`, then `foo` cannot possibly be responsible for how `baz` interoperates with it in all cases. "Software has no way to express its own compatibility with other functions or dependencies". I've heard that at a certain point all internal behavior can turn out to "bake" and be depended on by an external user. An extreme example is `*.append(...)` having O(n), O(log n), O(n^2) performance characteristics in either memory or speed. Some (ab-)use means that you may get a bug report b/c your performance characteristics changed in a way that the _user_ of your code didn't expect, and I struggle to see how anything other than snapshotting the world [never mind network-call dependencies] would ever "fix" things? I think it's an impossible problem: 1) code can test itself 2) by definition, code can't [exhaustively] test how it is used by "others" 3) therefore: any changes to dependent code are potentially "breaking" changes from the perspective of the user of the dependent code (even bugfixes!) 4) a generalized "really good" outcome is possible (eg: imagine something like TypeScript but for dependency management), but once you loop around to point #1 again... you either exactly match like a puzzle piece (eg: frozen dependency), or there's some sort of wiggle-room where any sort of change to the dependency has the _potential_ to break the user, or there is guaranteed slack/wastage in between the two dependencies (ie: size, performance, memory, operating range, etc) where changes can be relatively safely made. 5) therefore: code that uses a dependency should have acceptance tests (fit-for-purpose) of how the dependency behaves. We're relying on version numbers and transitive dependencies at the moment, but I can't imagine anything other than "freeze everything" being a viable, generalized solution?
- DenseComet 5y ago
  It feels like the solution to most of this is static typing, not writing a ton of tests by hand for every single dependency. Observable changes in a dependency's logic should be caught by your own unit tests. Changes in dependency function names or signatures should be caught by the compiler.
_k9eq 5y ago
The ability to start a shell with all the right dependencies configured, without requiring root access or permanently installing new dependencies, seems like something useful.
- misnome 5y ago
  unless you need specific versions of dependencies? The last time I looked to try and port any of our own software the nix answer to needing specific versions dependencies seemed to be "you are holding it wrong".
  - grumbel 5y ago
    Nix has no problems installing a specific version of a software. You can either override an existing package or just build your own: https://nixos.wiki/wiki/Overlays#Overriding_a_version https://nixos.wiki/wiki/Overlays#Overriding_a_version With Nix flakes it's even easier and you can just mix and match inputs as you want. For example if you want to replace libfoo with your own fork you just do something like: $ nix build --override-input libfoo /home/juser/src/libfoo And the best part is that none of this requires any changes to your system. If you need multiple different specific versions at once, Nix can just do it, as every software is sitting around in /nix/store/{HASH} and there is no risk of anything colliding.
- bogwog 5y ago
  You can also do that with rootless containers via Podman. There’s even a convenient wrapper for that workflow made by Red Hat called Toolbox (https://github.com/containers/toolbox https://github.com/containers/toolbox)

willhinsa 5y ago

Is there an example of this being done the "right" way? I'm interested in learning more about this.

throwaway984393 5y ago
Glibc allows specifying the version of a function at call time, so that you can call a specific version of a function in a library. This means that regardless of when you released some software, you can pin how it works, so even if you upgrade the library 10 years later, it will work the same way (as long as the library keeps the old version of the function). This needs to be extended, made easier, and every programming language should do it by default. The filesystem is also a limitation. We need to refer to versions in files and directories in the same way, due to application dependence on files. This can be accomplished probably just by adding versions in paths and then defining version compatibility in code. Would be nice if the filesystems all got versioning built in, though, so we could just call file functions with version arguments rather than hacking around path names. Data formats and protocols also need explicit versions for all operations, so they can advertise their compatibility and thus be backwards and forwards compatible. We must also be able to execute software with a specific combination of versioned functions at exec time. This requires kernel modifications / new syscalls. Libraries and applications should keep old versions of functions indefinitely, and only load code when needed. This will cause some consequences which we need solutions for, such as how to handle the reduced disk space and shared memory. It will require rethinking how applications are built, stored, and executed. All of this rethinking of software design requires a lot of effort. But it would provide a world of benefits that we currently need hacks for. We could upgrade randomly to bleeding edge software without any change to existing applications; any software interacting would continue to function as before. But newer function versions would be available and could be enabled via feature flags. We would no longer need containers, maybe not even package managers. Software could be upgraded continuously and remain perfectly stable. Users could define what versions they will execute depending on their wishes. And legacy software could be supported for decades, maybe centuries, with no extra effort.
- carapace 5y ago
  Have you seen Unison lang? It sounds like it overlaps (at least) with what you're talking about. > Unison’s core idea is that code is immutable and identified by its content. This lets us reimagine many aspects of how a programming language works. We simplify codebase management — Unison has no builds, no dependency conflicts, and renaming things is trivial. https://www.unisonweb.org/ https://www.unisonweb.org/ FWIW, I agree with you that the way we do things now is, uh, not the best of all possible worlds. I think Nix (and Guix) is a step in the right direction, though, but they don't go far enough.
- virchau13 5y ago
  > you can call a specific version of a function in a library A library has some functions. Later, the library gets refactored, and those functions are thereby rewritten; but since no changes have actually been made, the version number stays the same. However, there exists some legacy code that horrendously breaks when given a non-ABI-compatible function from this library, _even though_ the functionality and version of those functions the same. Oh dear! What can we do? Well, clearly exact binary compatibility is important, so someone has an idea: we'll specify functions with exact hashes of their code instead of versioning, so this problem doesn't happen. This is all well and good, but then the library decides to update some internal utility subroutine, changing its hash, but _not_ changing its call sites. Oh dear, now some other functions have hashes that are still the same, but they have different functionality due to that utility update, and legacy code breaks horribly again! In order to make sure this doesn't happen, someone comes up with the idea that a function's hash should be dependent on the hashes of all of the functions, variables, etc. it depends on _and_ its code, so all of the "inputs" that go towards defining the function end up updating its hash, thereby insuring legacy code stability. Now we run into another problem: where do we put all of these functions? They need to be stored somewhere on the system so applications can have access to them, so we'll put them in a repository that we'll call a "store". Now we can do cool stuff with shared stores such as "only load code when needed"! Oh dear, now the user sitting at the shell of the system wants to run programs too. Our function infrastructure works so well, why not extend it to programs too? Some programs really like filesystems and environments and depend on specific structures. We can't really encode these structures as an "input" to our programs' hashes, because they could be highly dynamic (or even undescribable), so instead we can put software into a little box where it sees the filesystem structure it wants, so it is happy, and the box itself could be specified with input programs/functions in the same way that programs or functions can! Hmm.. we seem to be recreating this "hash" structure that depends on inputs a lot. We need a common name for this... hmm, how about "derivations"? Congratulations, you have just reinvented Nix! The solutions in your comment that you talk about are things that Nix already was designed to provide. The only discrepancy is that Nix doesn't actually provide hashes for individual functions, rather they work for libraries, but this is because of a limitation on how libraries work: that they can have shared, global, mutable state, making the idea of individual function versions useless. The main problem why Nix doesn't obviously seem to solve these problems is that Nix's tools are not very well designed. For example, it's very non-obvious how to specify older versions of packages. However, flakes makes this easier, and although I don't believe _Nix_ specifically will be the solution, the holy grail at the end of the road will be very similar to Nix in core ideas.

slabity 5y ago

You're stating a lot of interesting opinions, but I do not see any arguments to help describe why you feel that way. How exactly is Nix monolithic or proprietary? And what is complicated about it? It seems pretty simple to me, if not at odds with how other package managers do things. It's one of the few package managers that can successfully work across multiple Linux distributions, some BSD systems, MacOS, and even Windows for some software. In my opinion it's the least proprietary or monolithic package managers around, the other being Guix which follows the same principles. But I think the biggest issue is in your last paragraph where you don't describe why or how the solution needs to happen in the software itself rather than at a higher level.

goosedragons 5y ago
If you need something that exists outside the repository it can get complicated. Even playing within the repository can have pitfalls at least on Guix which is what I've mainly used (like having a package appear in two profiles can cause issues that are non-obvious or like using "guix shell gcc binutils" instead of " guix shell gcc-toolchain"). Some of that has to do with the fact of what an expected *nix install is supposed to be and expectations of what should be installed on a users system and so are not marked as dependencies or requirements when they really are. I am curious about how a system designed from the ground up with this in mind might be better.

ramses0 5y ago

Suppose: 1) foo.lang: class foo { func bar(a) { return b } } 2) baz.lang: import foo && print( foo.bar(1) ) Generally `foo` needs to have unit tests (self-tests), but `baz` needs to have "acceptance tests" for `foo` because `baz` imports `foo` (external validation / fit-for-purpose). Blah blah "law of demeter" aka: self.do_foo_bar() { foo.bar(1) }. I've come to the mental model that code should test itself, and (some) code should test its dependencies. Code fundamentally can't test code that uses it, but can attempt to be "well-behaved" and share (via documentation or checked interface) how it can be / should be used. eg: `foo_test.lang: assert( foo.bar(1) == 42 );` But if `baz` calls `foo.bar(2)` or `foo.bar(0.3)` or `foo.bar("four")`, then `foo` cannot possibly be responsible for how `baz` interoperates with it in all cases. "Software has no way to express its own compatibility with other functions or dependencies". I've heard that at a certain point all internal behavior can turn out to "bake" and be depended on by an external user. An extreme example is `*.append(...)` having O(n), O(log n), O(n^2) performance characteristics in either memory or speed. Some (ab-)use means that you may get a bug report b/c your performance characteristics changed in a way that the _user_ of your code didn't expect, and I struggle to see how anything other than snapshotting the world [never mind network-call dependencies] would ever "fix" things? I think it's an impossible problem: 1) code can test itself 2) by definition, code can't [exhaustively] test how it is used by "others" 3) therefore: any changes to dependent code are potentially "breaking" changes from the perspective of the user of the dependent code (even bugfixes!) 4) a generalized "really good" outcome is possible (eg: imagine something like TypeScript but for dependency management), but once you loop around to point #1 again... you either exactly match like a puzzle piece (eg: frozen dependency), or there's some sort of wiggle-room where any sort of change to the dependency has the _potential_ to break the user, or there is guaranteed slack/wastage in between the two dependencies (ie: size, performance, memory, operating range, etc) where changes can be relatively safely made. 5) therefore: code that uses a dependency should have acceptance tests (fit-for-purpose) of how the dependency behaves. We're relying on version numbers and transitive dependencies at the moment, but I can't imagine anything other than "freeze everything" being a viable, generalized solution?

DenseComet 5y ago
It feels like the solution to most of this is static typing, not writing a ton of tests by hand for every single dependency. Observable changes in a dependency's logic should be caught by your own unit tests. Changes in dependency function names or signatures should be caught by the compiler.

_k9eq 5y ago

The ability to start a shell with all the right dependencies configured, without requiring root access or permanently installing new dependencies, seems like something useful.

misnome 5y ago
unless you need specific versions of dependencies? The last time I looked to try and port any of our own software the nix answer to needing specific versions dependencies seemed to be "you are holding it wrong".
- grumbel 5y ago
  Nix has no problems installing a specific version of a software. You can either override an existing package or just build your own: https://nixos.wiki/wiki/Overlays#Overriding_a_version https://nixos.wiki/wiki/Overlays#Overriding_a_version With Nix flakes it's even easier and you can just mix and match inputs as you want. For example if you want to replace libfoo with your own fork you just do something like: $ nix build --override-input libfoo /home/juser/src/libfoo And the best part is that none of this requires any changes to your system. If you need multiple different specific versions at once, Nix can just do it, as every software is sitting around in /nix/store/{HASH} and there is no risk of anything colliding.
bogwog 5y ago
You can also do that with rootless containers via Podman. There’s even a convenient wrapper for that workflow made by Red Hat called Toolbox (https://github.com/containers/toolbox https://github.com/containers/toolbox)