6 ms·

It seems like Nixpkgs aims to minimize the number of package versions in use at one time. Not just nix, most package managers do, it seems (i.e. you wouldn't ex

It seems like Nixpkgs aims to minimize the number of package versions in use at one time. Not just nix, most package managers do, it seems (i.e. you wouldn't expect to find different minor versions of Nginx in Debian, would you?) So by that same logic, there is only one version of Django 4. It is definitely possible with Nix to use the precise versions of what's in your requirements.txt, but I'm not sure if the Nixpkgs maintainers would allow all that extra duplication upstream.

scandinavian 3y ago
I get what you are saying, but nothing you said works in practice for python packages, so not sure that I actually learned anything. Is it fair to summize that python applications with python dependencies do not really work well as nix packages and shouldn't be used?
- nrabulinski 3y ago
  Sounds like the problem is with Python maintainers who don’t understand that breaking changes should only be made between major versions. If that’s not possible though then as sibling comment said - you can override the dependencies and the nix maintainer should make sure the package works as expected
  - hfkwer 3y ago
    Sounds like the problem could also be with Nix maintainers who don't understand that "semver" is not a universal law of nature and that not all projects and ecosystems follow it. This kind of blanket dismissal can cut both ways. Semver (the website and "spec") was created in 2009 by some guy. It's not an RFC, a standard, or anything like that. Yes, it gained widespread adoption. Yes, the guy in question is a cofounder of GitHub. So what? You cannot force it upon everyone. Python is about 20 years older than semver. Django is several years older. Should the whole ecosystem change their conventions because it's more convenient for a few people?
    - nrabulinski 3y ago
      Except Django site says that a.b are feature releases which should be backwards compatible except for specific exceptions. If their software truly breaks “with every update to django_4” then it’s either a problem on Django’s side or a problem in how said person uses Django
      - hfkwer 3y ago
        I don't know if it's deliberate or a communication/comprehension problem, but you're misquoting Django's release process https://docs.djangoproject.com/en/dev/internals/release-process/ https://docs.djangoproject.com/en/dev/internals/release-proc... > * Versions are numbered in the form A.B or A.B.C. > * A.B is the feature release version number. Each version will be mostly backwards compatible with the previous release. Exceptions to this rule will be listed in the release notes. > * C is the patch release version number, which is incremented for bugfix and security releases. These releases will be 100% backwards-compatible with the previous patch release. The only exception is when a security or data loss issue can’t be fixed without breaking backwards-compatibility. If this happens, the release notes will provide detailed upgrade instructions. Going from "mostly backwards compatible with the previous release. Exceptions to this rule will be listed" to "should be backwards compatible except for specific exceptions" is quite the stretch. There are no "specific exceptions": incompatibilities can be anywhere and you need to read the release notes to know where. In semver, a minor version increment is backwards-compatible, no exception, no ifs or buts. If you want to shoehorn Django's release process into "semver", then act as if the product is called "Django 4". If the version is "Django v4.X.Y", then X is the major version number, Y is the minor version number, and there is no patch version. It should be version in Nix as "django4 vX.Y.0".
        nrabulinski 3y ago
        They clearly say “exceptions to this rule will be listed in the release notes” meaning that backwards compatibility is the rule. There’d be no exceptions if there was no rule hence I said they “should” be backwards compatible except for specific exceptions, which shall be noted in the release notes.
- hamandcheese 3y ago
  > but nothing you said works in practice for python packages How do transitive dependencies in the Python ecosystem work, then? I assume Django works with multiple versions of python and bcrypt. I assume pandas works with multiple versions of scipy. Is there no semantic versioning? If everything requires an exact version, how do you prevent everything from grinding to a halt? > Is it fair to summize that python applications with python dependencies do not really work well as nix packages and shouldn't be used? Let's not conflate Nix and Nixpkgs. Nixpkgs has its reasons for minimizing redundant packages, however it is certainly possible to package your app with Nix and use the exact specified dependencies.
  - scandinavian 3y ago
    > How do transitive dependencies in the Python ecosystem work, then? Not very well. > how do you prevent everything from grinding to a halt? I don't have a good answer for you. > Is there no semantic versioning? You can read django release process here [1], not sure how it's relevant. I'm not the maintainer of django, but of a project using django. Would it be better if all software was perfect, had no bugs and used perfect semantic versioning? Yes, I would say so. Is that a requirement for using nixpkgs? > Nixpkgs has its reasons for minimizing redundant packages, however it is certainly possible to package your app with Nix and use the exact specified dependencies. I'm not packaging it, someone else is, it breaks and they come to the project to raise invalid bug reports. [1] https://docs.djangoproject.com/en/dev/internals/release-process/ https://docs.djangoproject.com/en/dev/internals/release-proc...
    - hamandcheese 3y ago
      > not sure how it's relevant. Well you said earlier that nothing I said works in practice for python packages. My only point is that it must work at some level in the python ecosystem, else the ecosystem would collapse. Anyways, it sounds like you're unhappy that someone did a bad job packaging your application. That sucks. Elsewhere in this thread someone mentioned that there isn't a strict single version policy in nixpkgs, so this can probably be easily fixed. I'd suggest filing a bug in Nixpkgs.
- ruuda 3y ago
  No, that is not a fair summary; Nix is the nicest way to manage Python packages that I have found thus far.
  - ParetoOptimal 3y ago
    I'm assuming you try to keep all dependencies on the nixpkgs version?
    - Cu3PO42 3y ago
      That's not really necessary. Pulling in arbitrary versions of packages from PyPi is fairly easy, if a bit verbose.
    - ruuda 3y ago
      For packages that I need to pin to specific versions, that's also easy to do with Nix.
- eptcyka 3y ago
  What's the point of a minor version change if it's breaking? Does Django not have a versioning policy that enforces non-breaking changes between minor versions?
  - scandinavian 3y ago
    That's my fault for writing minor, as that's what it would be in semver. I should have written feature release. You can read the release process here. https://docs.djangoproject.com/en/dev/internals/release-process/ https://docs.djangoproject.com/en/dev/internals/release-proc...
- KirillPanov 3y ago
  > python dependencies do not really work well Yes, that is exactly correct.
- eulenteufel 3y ago
  In this case I think it is important to distinguish nix (the package manager) and nixpkgs (the popular package repository / distribution used with nix). Packaging python applications with nix is doable, but you have to specify the exact versions of your dependencies and for that you can't easily use nixpkgs. Nixpkgs tries to keep a minimum number of packages (like Arch or Debian as well), so each of the dependencies will typically only occur with one minor version for each release of nixpkgs. We could still use the nixpkgs to build our application but we have to override each of our dependencies to the right version, but that approach can get quiet tedious for a large number of dependencies. Fortunately there are tools to automatically generate your dependencies from a requirements.txt such as mach-nix or pip2nix.
- SuperSandro2000 3y ago
  > Is it fair to summize that python applications with python dependencies do not really work well as nix packages and shouldn't be used? No, applications that are properly maintained work as they should and this can be ensured with tests and e2e tests.
  - lolinder 3y ago
    This is such a condescending attitude. What you mean is applications that are maintained the way that you and the Nix developers think an application should be maintained. It's incredibly naive for a package manager as ambitious as Nix to assume semver. I'm a big fan of semver myself, but the vast majority of software projects follow it imperfectly or not at all, and for good reason—it's nearly impossible to follow it perfectly, because even bugs are part of your API. Every project I've worked on has eventually had something break on a version upgrade because we were depending on something that was later decided to be a bug (but at the time was just how it worked). Elm can mostly get away with enforcing semver because they designed it that way at the language level, but Nix wants to manage dependencies written in all languages and ecosystems, which have dramatically different versioning practices.
    - SuperSandro2000 3y ago
      Nixpkgs does not assume Denver that's why we run if possible the package's tests, our own tests and build dependent packages to make sure the most obvious breakages are noticed before things are even merged.
      - lolinder 3y ago
        Ah, I thought you were saying that if we all just used e2e tests to ensure we didn't make breaking changes in minor versions, we'd be fine. I didn't realize you were talking about the downstream package's tests. I do still take issue with your insinuation that it's the package maintainers' poor practices that are at fault here. The real world is a messy, complex place and "best practices" don't translate well from situation to situation. OP didn't ask for their package to be included in Nix. Presumably OP's system works for them and for their use case, but whoever created the Nix package made assumptions that turned out to be flawed. It's not fair of you to say that those bad assumptions are OP's fault because their package isn't "properly maintained" and doesn't "work as it should". Someone (you?) made a bad assumption. Don't cast blame for that on someone who only knows Nix exists because it sends phony bug reports their way.
traxys 3y ago
I packaged some python applications in nixpkgs, and it seems the consensus is to try and relax the dependency so that the globally packaged version is used, but if it fails the you can override the version yourself. Though this is not done through the requirements.txt because that file does not have enough information (no integrity hash for example).
lolinder 3y ago
Doesn't this auto-upgrade behavior punch straight through the reproducibility Nix is supposed to be giving you? It's not exactly a functional build system if the results you get depend on when you download the dependencies. (I mean, I guess you could say that time is an input to the function, but that seems to miss the point.)
- assbuttbuttass 3y ago
  You can think of the function inputs as: 1. All the package definitions in nixpkgs 2. Any external sources When a package is updated in nixpkgs, input #1 changes.
  - lolinder 3y ago
    I mean, I get that, but that means that the reproducibility of my build depends on the whims of the nixpkgs maintainers, it's not a property guaranteed by the package manager.
    - tymscar 3y ago
      You can however define inputs that are not the whole of nixpkgs. You would use something like this and you would pin it to a very exact version and hash of a package: https://nixos.org/guides/nix-pills/nixpkgs-overriding-packages.html https://nixos.org/guides/nix-pills/nixpkgs-overriding-packag...
    - deleted 3y ago
      [deleted]
    - pxc 3y ago
      The goal of a downstream Linux distribution is never to reproduce whatever builds you run on your own machine as an upstream developer. It's to produce a collection of installable software that meets various constraints and goals, like cohesion (can all be installed and managed uniformly), minimal size, easy/manageable security updates, integration (compatibility and so on). That can involve things like building the software against particular library versions mandated by downstream needs or even patching it. Some distros try hard to avoid patching upstream and some don't, and in all distros there may be cases where other priorities take precedence over the value of leaving upstream untouched. In the case of Nixpkgs and Python, the community wants to maintain a collection of Python libraries that are all interoperable, and Python doesn't support vendorization well enough to allow multiple versions of the same library in a single Python process, which is one reason for preferring singular versions of most Python libraries in Nixpkgs. The other factor is likely just reducing the maintenance across Nixpkgs by maintaining as few redundant versions within the tree as possible. If you want to control/determine the entire runtime your end users use, you have to do the packaging work required to ship them that runtime with some tooling that's capable of the reproducibility you desire. Python doesn't have one a reproducible package manager, so your options are basically creating your own Nix package (probably as a flake.nix in your repo), Docker, and Flatpak. That said, it's perfectly possibly to include multiple minor releases of Django 4 in a single snapshot of the Nixpkgs tree and maybe that should be done. Have you talked with the maintainers of your downstream package of Nixpkgs to let them know Django breaks things on minor releases, and so using different versions of Django 4 interchangeably is not tested or supported in your application?
pxc 3y ago
> It is definitely possible with Nix to use the precise versions of what's in your requirements.txt, but I'm not sure if the Nixpkgs maintainers would allow all that extra duplication upstream. They do for end user applications, but not for Python libraries. The libraries in Nixpkgs are expected to be interoperable, which requires converging certain versions because otherwise transitive dependencies on varying library versions mean that libraries used together are subject to serious, mysterious bugs. But applications packaged in Nixpkgs can pull in an exact set of libraries of their own if that's what it takes for them to run reliably.

scandinavian 3y ago

I get what you are saying, but nothing you said works in practice for python packages, so not sure that I actually learned anything. Is it fair to summize that python applications with python dependencies do not really work well as nix packages and shouldn't be used?

nrabulinski 3y ago
Sounds like the problem is with Python maintainers who don’t understand that breaking changes should only be made between major versions. If that’s not possible though then as sibling comment said - you can override the dependencies and the nix maintainer should make sure the package works as expected
- hfkwer 3y ago
  Sounds like the problem could also be with Nix maintainers who don't understand that "semver" is not a universal law of nature and that not all projects and ecosystems follow it. This kind of blanket dismissal can cut both ways. Semver (the website and "spec") was created in 2009 by some guy. It's not an RFC, a standard, or anything like that. Yes, it gained widespread adoption. Yes, the guy in question is a cofounder of GitHub. So what? You cannot force it upon everyone. Python is about 20 years older than semver. Django is several years older. Should the whole ecosystem change their conventions because it's more convenient for a few people?
  - nrabulinski 3y ago
    Except Django site says that a.b are feature releases which should be backwards compatible except for specific exceptions. If their software truly breaks “with every update to django_4” then it’s either a problem on Django’s side or a problem in how said person uses Django
    - hfkwer 3y ago
      I don't know if it's deliberate or a communication/comprehension problem, but you're misquoting Django's release process https://docs.djangoproject.com/en/dev/internals/release-process/ https://docs.djangoproject.com/en/dev/internals/release-proc... > * Versions are numbered in the form A.B or A.B.C. > * A.B is the feature release version number. Each version will be mostly backwards compatible with the previous release. Exceptions to this rule will be listed in the release notes. > * C is the patch release version number, which is incremented for bugfix and security releases. These releases will be 100% backwards-compatible with the previous patch release. The only exception is when a security or data loss issue can’t be fixed without breaking backwards-compatibility. If this happens, the release notes will provide detailed upgrade instructions. Going from "mostly backwards compatible with the previous release. Exceptions to this rule will be listed" to "should be backwards compatible except for specific exceptions" is quite the stretch. There are no "specific exceptions": incompatibilities can be anywhere and you need to read the release notes to know where. In semver, a minor version increment is backwards-compatible, no exception, no ifs or buts. If you want to shoehorn Django's release process into "semver", then act as if the product is called "Django 4". If the version is "Django v4.X.Y", then X is the major version number, Y is the minor version number, and there is no patch version. It should be version in Nix as "django4 vX.Y.0".
      - nrabulinski 3y ago
        They clearly say “exceptions to this rule will be listed in the release notes” meaning that backwards compatibility is the rule. There’d be no exceptions if there was no rule hence I said they “should” be backwards compatible except for specific exceptions, which shall be noted in the release notes.
hamandcheese 3y ago
> but nothing you said works in practice for python packages How do transitive dependencies in the Python ecosystem work, then? I assume Django works with multiple versions of python and bcrypt. I assume pandas works with multiple versions of scipy. Is there no semantic versioning? If everything requires an exact version, how do you prevent everything from grinding to a halt? > Is it fair to summize that python applications with python dependencies do not really work well as nix packages and shouldn't be used? Let's not conflate Nix and Nixpkgs. Nixpkgs has its reasons for minimizing redundant packages, however it is certainly possible to package your app with Nix and use the exact specified dependencies.
- scandinavian 3y ago
  > How do transitive dependencies in the Python ecosystem work, then? Not very well. > how do you prevent everything from grinding to a halt? I don't have a good answer for you. > Is there no semantic versioning? You can read django release process here [1], not sure how it's relevant. I'm not the maintainer of django, but of a project using django. Would it be better if all software was perfect, had no bugs and used perfect semantic versioning? Yes, I would say so. Is that a requirement for using nixpkgs? > Nixpkgs has its reasons for minimizing redundant packages, however it is certainly possible to package your app with Nix and use the exact specified dependencies. I'm not packaging it, someone else is, it breaks and they come to the project to raise invalid bug reports. [1] https://docs.djangoproject.com/en/dev/internals/release-process/ https://docs.djangoproject.com/en/dev/internals/release-proc...
  - hamandcheese 3y ago
    > not sure how it's relevant. Well you said earlier that nothing I said works in practice for python packages. My only point is that it must work at some level in the python ecosystem, else the ecosystem would collapse. Anyways, it sounds like you're unhappy that someone did a bad job packaging your application. That sucks. Elsewhere in this thread someone mentioned that there isn't a strict single version policy in nixpkgs, so this can probably be easily fixed. I'd suggest filing a bug in Nixpkgs.
ruuda 3y ago
No, that is not a fair summary; Nix is the nicest way to manage Python packages that I have found thus far.
- ParetoOptimal 3y ago
  I'm assuming you try to keep all dependencies on the nixpkgs version?
  - Cu3PO42 3y ago
    That's not really necessary. Pulling in arbitrary versions of packages from PyPi is fairly easy, if a bit verbose.
  - ruuda 3y ago
    For packages that I need to pin to specific versions, that's also easy to do with Nix.
eptcyka 3y ago
What's the point of a minor version change if it's breaking? Does Django not have a versioning policy that enforces non-breaking changes between minor versions?
- scandinavian 3y ago
  That's my fault for writing minor, as that's what it would be in semver. I should have written feature release. You can read the release process here. https://docs.djangoproject.com/en/dev/internals/release-process/ https://docs.djangoproject.com/en/dev/internals/release-proc...
KirillPanov 3y ago
> python dependencies do not really work well Yes, that is exactly correct.
eulenteufel 3y ago
In this case I think it is important to distinguish nix (the package manager) and nixpkgs (the popular package repository / distribution used with nix). Packaging python applications with nix is doable, but you have to specify the exact versions of your dependencies and for that you can't easily use nixpkgs. Nixpkgs tries to keep a minimum number of packages (like Arch or Debian as well), so each of the dependencies will typically only occur with one minor version for each release of nixpkgs. We could still use the nixpkgs to build our application but we have to override each of our dependencies to the right version, but that approach can get quiet tedious for a large number of dependencies. Fortunately there are tools to automatically generate your dependencies from a requirements.txt such as mach-nix or pip2nix.
SuperSandro2000 3y ago
> Is it fair to summize that python applications with python dependencies do not really work well as nix packages and shouldn't be used? No, applications that are properly maintained work as they should and this can be ensured with tests and e2e tests.
- lolinder 3y ago
  This is such a condescending attitude. What you mean is applications that are maintained the way that you and the Nix developers think an application should be maintained. It's incredibly naive for a package manager as ambitious as Nix to assume semver. I'm a big fan of semver myself, but the vast majority of software projects follow it imperfectly or not at all, and for good reason—it's nearly impossible to follow it perfectly, because even bugs are part of your API. Every project I've worked on has eventually had something break on a version upgrade because we were depending on something that was later decided to be a bug (but at the time was just how it worked). Elm can mostly get away with enforcing semver because they designed it that way at the language level, but Nix wants to manage dependencies written in all languages and ecosystems, which have dramatically different versioning practices.
  - SuperSandro2000 3y ago
    Nixpkgs does not assume Denver that's why we run if possible the package's tests, our own tests and build dependent packages to make sure the most obvious breakages are noticed before things are even merged.
    - lolinder 3y ago
      Ah, I thought you were saying that if we all just used e2e tests to ensure we didn't make breaking changes in minor versions, we'd be fine. I didn't realize you were talking about the downstream package's tests. I do still take issue with your insinuation that it's the package maintainers' poor practices that are at fault here. The real world is a messy, complex place and "best practices" don't translate well from situation to situation. OP didn't ask for their package to be included in Nix. Presumably OP's system works for them and for their use case, but whoever created the Nix package made assumptions that turned out to be flawed. It's not fair of you to say that those bad assumptions are OP's fault because their package isn't "properly maintained" and doesn't "work as it should". Someone (you?) made a bad assumption. Don't cast blame for that on someone who only knows Nix exists because it sends phony bug reports their way.

traxys 3y ago

I packaged some python applications in nixpkgs, and it seems the consensus is to try and relax the dependency so that the globally packaged version is used, but if it fails the you can override the version yourself. Though this is not done through the requirements.txt because that file does not have enough information (no integrity hash for example).

lolinder 3y ago

Doesn't this auto-upgrade behavior punch straight through the reproducibility Nix is supposed to be giving you? It's not exactly a functional build system if the results you get depend on when you download the dependencies. (I mean, I guess you could say that time is an input to the function, but that seems to miss the point.)

assbuttbuttass 3y ago
You can think of the function inputs as: 1. All the package definitions in nixpkgs 2. Any external sources When a package is updated in nixpkgs, input #1 changes.
- lolinder 3y ago
  I mean, I get that, but that means that the reproducibility of my build depends on the whims of the nixpkgs maintainers, it's not a property guaranteed by the package manager.
  - tymscar 3y ago
    You can however define inputs that are not the whole of nixpkgs. You would use something like this and you would pin it to a very exact version and hash of a package: https://nixos.org/guides/nix-pills/nixpkgs-overriding-packages.html https://nixos.org/guides/nix-pills/nixpkgs-overriding-packag...
  - deleted 3y ago
    [deleted]
  - pxc 3y ago
    The goal of a downstream Linux distribution is never to reproduce whatever builds you run on your own machine as an upstream developer. It's to produce a collection of installable software that meets various constraints and goals, like cohesion (can all be installed and managed uniformly), minimal size, easy/manageable security updates, integration (compatibility and so on). That can involve things like building the software against particular library versions mandated by downstream needs or even patching it. Some distros try hard to avoid patching upstream and some don't, and in all distros there may be cases where other priorities take precedence over the value of leaving upstream untouched. In the case of Nixpkgs and Python, the community wants to maintain a collection of Python libraries that are all interoperable, and Python doesn't support vendorization well enough to allow multiple versions of the same library in a single Python process, which is one reason for preferring singular versions of most Python libraries in Nixpkgs. The other factor is likely just reducing the maintenance across Nixpkgs by maintaining as few redundant versions within the tree as possible. If you want to control/determine the entire runtime your end users use, you have to do the packaging work required to ship them that runtime with some tooling that's capable of the reproducibility you desire. Python doesn't have one a reproducible package manager, so your options are basically creating your own Nix package (probably as a flake.nix in your repo), Docker, and Flatpak. That said, it's perfectly possibly to include multiple minor releases of Django 4 in a single snapshot of the Nixpkgs tree and maybe that should be done. Have you talked with the maintainers of your downstream package of Nixpkgs to let them know Django breaks things on minor releases, and so using different versions of Django 4 interchangeably is not tested or supported in your application?

pxc 3y ago

> It is definitely possible with Nix to use the precise versions of what's in your requirements.txt, but I'm not sure if the Nixpkgs maintainers would allow all that extra duplication upstream. They do for end user applications, but not for Python libraries. The libraries in Nixpkgs are expected to be interoperable, which requires converging certain versions because otherwise transitive dependencies on varying library versions mean that libraries used together are subject to serious, mysterious bugs. But applications packaged in Nixpkgs can pull in an exact set of libraries of their own if that's what it takes for them to run reliably.