7 ms·

I get what you are saying, but nothing you said works in practice for python packages, so not sure that I actually learned anything. Is it fair to summize that

I get what you are saying, but nothing you said works in practice for python packages, so not sure that I actually learned anything. Is it fair to summize that python applications with python dependencies do not really work well as nix packages and shouldn't be used?

nrabulinski 3y ago
Sounds like the problem is with Python maintainers who don’t understand that breaking changes should only be made between major versions. If that’s not possible though then as sibling comment said - you can override the dependencies and the nix maintainer should make sure the package works as expected
- hfkwer 3y ago
  Sounds like the problem could also be with Nix maintainers who don't understand that "semver" is not a universal law of nature and that not all projects and ecosystems follow it. This kind of blanket dismissal can cut both ways. Semver (the website and "spec") was created in 2009 by some guy. It's not an RFC, a standard, or anything like that. Yes, it gained widespread adoption. Yes, the guy in question is a cofounder of GitHub. So what? You cannot force it upon everyone. Python is about 20 years older than semver. Django is several years older. Should the whole ecosystem change their conventions because it's more convenient for a few people?
  - nrabulinski 3y ago
    Except Django site says that a.b are feature releases which should be backwards compatible except for specific exceptions. If their software truly breaks “with every update to django_4” then it’s either a problem on Django’s side or a problem in how said person uses Django
    - hfkwer 3y ago
      I don't know if it's deliberate or a communication/comprehension problem, but you're misquoting Django's release process https://docs.djangoproject.com/en/dev/internals/release-process/ https://docs.djangoproject.com/en/dev/internals/release-proc... > * Versions are numbered in the form A.B or A.B.C. > * A.B is the feature release version number. Each version will be mostly backwards compatible with the previous release. Exceptions to this rule will be listed in the release notes. > * C is the patch release version number, which is incremented for bugfix and security releases. These releases will be 100% backwards-compatible with the previous patch release. The only exception is when a security or data loss issue can’t be fixed without breaking backwards-compatibility. If this happens, the release notes will provide detailed upgrade instructions. Going from "mostly backwards compatible with the previous release. Exceptions to this rule will be listed" to "should be backwards compatible except for specific exceptions" is quite the stretch. There are no "specific exceptions": incompatibilities can be anywhere and you need to read the release notes to know where. In semver, a minor version increment is backwards-compatible, no exception, no ifs or buts. If you want to shoehorn Django's release process into "semver", then act as if the product is called "Django 4". If the version is "Django v4.X.Y", then X is the major version number, Y is the minor version number, and there is no patch version. It should be version in Nix as "django4 vX.Y.0".
      - nrabulinski 3y ago
        They clearly say “exceptions to this rule will be listed in the release notes” meaning that backwards compatibility is the rule. There’d be no exceptions if there was no rule hence I said they “should” be backwards compatible except for specific exceptions, which shall be noted in the release notes.
        scandinavian 3y ago
        Not sure how this conversation is productive, but there's never been a X.Y release of django without noted backwards incompatible changes to my knowledge. Just imagine that djangos X.Y releases are semvers major releases, not much more to it than that.
        lolinder 3y ago
  - dlahoda 3y ago
    i use nix for 2 years. never was thinkin or relied on semver. exactly nix allows me not to consider semver as something realy existing or working.
hamandcheese 3y ago
> but nothing you said works in practice for python packages How do transitive dependencies in the Python ecosystem work, then? I assume Django works with multiple versions of python and bcrypt. I assume pandas works with multiple versions of scipy. Is there no semantic versioning? If everything requires an exact version, how do you prevent everything from grinding to a halt? > Is it fair to summize that python applications with python dependencies do not really work well as nix packages and shouldn't be used? Let's not conflate Nix and Nixpkgs. Nixpkgs has its reasons for minimizing redundant packages, however it is certainly possible to package your app with Nix and use the exact specified dependencies.
- scandinavian 3y ago
  > How do transitive dependencies in the Python ecosystem work, then? Not very well. > how do you prevent everything from grinding to a halt? I don't have a good answer for you. > Is there no semantic versioning? You can read django release process here [1], not sure how it's relevant. I'm not the maintainer of django, but of a project using django. Would it be better if all software was perfect, had no bugs and used perfect semantic versioning? Yes, I would say so. Is that a requirement for using nixpkgs? > Nixpkgs has its reasons for minimizing redundant packages, however it is certainly possible to package your app with Nix and use the exact specified dependencies. I'm not packaging it, someone else is, it breaks and they come to the project to raise invalid bug reports. [1] https://docs.djangoproject.com/en/dev/internals/release-process/ https://docs.djangoproject.com/en/dev/internals/release-proc...
  - hamandcheese 3y ago
    > not sure how it's relevant. Well you said earlier that nothing I said works in practice for python packages. My only point is that it must work at some level in the python ecosystem, else the ecosystem would collapse. Anyways, it sounds like you're unhappy that someone did a bad job packaging your application. That sucks. Elsewhere in this thread someone mentioned that there isn't a strict single version policy in nixpkgs, so this can probably be easily fixed. I'd suggest filing a bug in Nixpkgs.
ruuda 3y ago
No, that is not a fair summary; Nix is the nicest way to manage Python packages that I have found thus far.
- ParetoOptimal 3y ago
  I'm assuming you try to keep all dependencies on the nixpkgs version?
  - Cu3PO42 3y ago
    That's not really necessary. Pulling in arbitrary versions of packages from PyPi is fairly easy, if a bit verbose.
  - ruuda 3y ago
    For packages that I need to pin to specific versions, that's also easy to do with Nix.
eptcyka 3y ago
What's the point of a minor version change if it's breaking? Does Django not have a versioning policy that enforces non-breaking changes between minor versions?
- scandinavian 3y ago
  That's my fault for writing minor, as that's what it would be in semver. I should have written feature release. You can read the release process here. https://docs.djangoproject.com/en/dev/internals/release-process/ https://docs.djangoproject.com/en/dev/internals/release-proc...
KirillPanov 3y ago
> python dependencies do not really work well Yes, that is exactly correct.
eulenteufel 3y ago
In this case I think it is important to distinguish nix (the package manager) and nixpkgs (the popular package repository / distribution used with nix). Packaging python applications with nix is doable, but you have to specify the exact versions of your dependencies and for that you can't easily use nixpkgs. Nixpkgs tries to keep a minimum number of packages (like Arch or Debian as well), so each of the dependencies will typically only occur with one minor version for each release of nixpkgs. We could still use the nixpkgs to build our application but we have to override each of our dependencies to the right version, but that approach can get quiet tedious for a large number of dependencies. Fortunately there are tools to automatically generate your dependencies from a requirements.txt such as mach-nix or pip2nix.
SuperSandro2000 3y ago
> Is it fair to summize that python applications with python dependencies do not really work well as nix packages and shouldn't be used? No, applications that are properly maintained work as they should and this can be ensured with tests and e2e tests.
- lolinder 3y ago
  This is such a condescending attitude. What you mean is applications that are maintained the way that you and the Nix developers think an application should be maintained. It's incredibly naive for a package manager as ambitious as Nix to assume semver. I'm a big fan of semver myself, but the vast majority of software projects follow it imperfectly or not at all, and for good reason—it's nearly impossible to follow it perfectly, because even bugs are part of your API. Every project I've worked on has eventually had something break on a version upgrade because we were depending on something that was later decided to be a bug (but at the time was just how it worked). Elm can mostly get away with enforcing semver because they designed it that way at the language level, but Nix wants to manage dependencies written in all languages and ecosystems, which have dramatically different versioning practices.
  - SuperSandro2000 3y ago
    Nixpkgs does not assume Denver that's why we run if possible the package's tests, our own tests and build dependent packages to make sure the most obvious breakages are noticed before things are even merged.
    - lolinder 3y ago
      Ah, I thought you were saying that if we all just used e2e tests to ensure we didn't make breaking changes in minor versions, we'd be fine. I didn't realize you were talking about the downstream package's tests. I do still take issue with your insinuation that it's the package maintainers' poor practices that are at fault here. The real world is a messy, complex place and "best practices" don't translate well from situation to situation. OP didn't ask for their package to be included in Nix. Presumably OP's system works for them and for their use case, but whoever created the Nix package made assumptions that turned out to be flawed. It's not fair of you to say that those bad assumptions are OP's fault because their package isn't "properly maintained" and doesn't "work as it should". Someone (you?) made a bad assumption. Don't cast blame for that on someone who only knows Nix exists because it sends phony bug reports their way.

nrabulinski 3y ago

Sounds like the problem is with Python maintainers who don’t understand that breaking changes should only be made between major versions. If that’s not possible though then as sibling comment said - you can override the dependencies and the nix maintainer should make sure the package works as expected

hfkwer 3y ago
Sounds like the problem could also be with Nix maintainers who don't understand that "semver" is not a universal law of nature and that not all projects and ecosystems follow it. This kind of blanket dismissal can cut both ways. Semver (the website and "spec") was created in 2009 by some guy. It's not an RFC, a standard, or anything like that. Yes, it gained widespread adoption. Yes, the guy in question is a cofounder of GitHub. So what? You cannot force it upon everyone. Python is about 20 years older than semver. Django is several years older. Should the whole ecosystem change their conventions because it's more convenient for a few people?
- nrabulinski 3y ago
  Except Django site says that a.b are feature releases which should be backwards compatible except for specific exceptions. If their software truly breaks “with every update to django_4” then it’s either a problem on Django’s side or a problem in how said person uses Django
  - hfkwer 3y ago
    I don't know if it's deliberate or a communication/comprehension problem, but you're misquoting Django's release process https://docs.djangoproject.com/en/dev/internals/release-process/ https://docs.djangoproject.com/en/dev/internals/release-proc... > * Versions are numbered in the form A.B or A.B.C. > * A.B is the feature release version number. Each version will be mostly backwards compatible with the previous release. Exceptions to this rule will be listed in the release notes. > * C is the patch release version number, which is incremented for bugfix and security releases. These releases will be 100% backwards-compatible with the previous patch release. The only exception is when a security or data loss issue can’t be fixed without breaking backwards-compatibility. If this happens, the release notes will provide detailed upgrade instructions. Going from "mostly backwards compatible with the previous release. Exceptions to this rule will be listed" to "should be backwards compatible except for specific exceptions" is quite the stretch. There are no "specific exceptions": incompatibilities can be anywhere and you need to read the release notes to know where. In semver, a minor version increment is backwards-compatible, no exception, no ifs or buts. If you want to shoehorn Django's release process into "semver", then act as if the product is called "Django 4". If the version is "Django v4.X.Y", then X is the major version number, Y is the minor version number, and there is no patch version. It should be version in Nix as "django4 vX.Y.0".
    - nrabulinski 3y ago
      They clearly say “exceptions to this rule will be listed in the release notes” meaning that backwards compatibility is the rule. There’d be no exceptions if there was no rule hence I said they “should” be backwards compatible except for specific exceptions, which shall be noted in the release notes.
      - scandinavian 3y ago
        Not sure how this conversation is productive, but there's never been a X.Y release of django without noted backwards incompatible changes to my knowledge. Just imagine that djangos X.Y releases are semvers major releases, not much more to it than that.
      - lolinder 3y ago
- dlahoda 3y ago
  i use nix for 2 years. never was thinkin or relied on semver. exactly nix allows me not to consider semver as something realy existing or working.

hamandcheese 3y ago

> but nothing you said works in practice for python packages How do transitive dependencies in the Python ecosystem work, then? I assume Django works with multiple versions of python and bcrypt. I assume pandas works with multiple versions of scipy. Is there no semantic versioning? If everything requires an exact version, how do you prevent everything from grinding to a halt? > Is it fair to summize that python applications with python dependencies do not really work well as nix packages and shouldn't be used? Let's not conflate Nix and Nixpkgs. Nixpkgs has its reasons for minimizing redundant packages, however it is certainly possible to package your app with Nix and use the exact specified dependencies.

scandinavian 3y ago
> How do transitive dependencies in the Python ecosystem work, then? Not very well. > how do you prevent everything from grinding to a halt? I don't have a good answer for you. > Is there no semantic versioning? You can read django release process here [1], not sure how it's relevant. I'm not the maintainer of django, but of a project using django. Would it be better if all software was perfect, had no bugs and used perfect semantic versioning? Yes, I would say so. Is that a requirement for using nixpkgs? > Nixpkgs has its reasons for minimizing redundant packages, however it is certainly possible to package your app with Nix and use the exact specified dependencies. I'm not packaging it, someone else is, it breaks and they come to the project to raise invalid bug reports. [1] https://docs.djangoproject.com/en/dev/internals/release-process/ https://docs.djangoproject.com/en/dev/internals/release-proc...
- hamandcheese 3y ago
  > not sure how it's relevant. Well you said earlier that nothing I said works in practice for python packages. My only point is that it must work at some level in the python ecosystem, else the ecosystem would collapse. Anyways, it sounds like you're unhappy that someone did a bad job packaging your application. That sucks. Elsewhere in this thread someone mentioned that there isn't a strict single version policy in nixpkgs, so this can probably be easily fixed. I'd suggest filing a bug in Nixpkgs.

ruuda 3y ago

No, that is not a fair summary; Nix is the nicest way to manage Python packages that I have found thus far.

ParetoOptimal 3y ago
I'm assuming you try to keep all dependencies on the nixpkgs version?
- Cu3PO42 3y ago
  That's not really necessary. Pulling in arbitrary versions of packages from PyPi is fairly easy, if a bit verbose.
- ruuda 3y ago
  For packages that I need to pin to specific versions, that's also easy to do with Nix.

eptcyka 3y ago

What's the point of a minor version change if it's breaking? Does Django not have a versioning policy that enforces non-breaking changes between minor versions?

scandinavian 3y ago
That's my fault for writing minor, as that's what it would be in semver. I should have written feature release. You can read the release process here. https://docs.djangoproject.com/en/dev/internals/release-process/ https://docs.djangoproject.com/en/dev/internals/release-proc...

KirillPanov 3y ago

> python dependencies do not really work well Yes, that is exactly correct.

eulenteufel 3y ago

In this case I think it is important to distinguish nix (the package manager) and nixpkgs (the popular package repository / distribution used with nix). Packaging python applications with nix is doable, but you have to specify the exact versions of your dependencies and for that you can't easily use nixpkgs. Nixpkgs tries to keep a minimum number of packages (like Arch or Debian as well), so each of the dependencies will typically only occur with one minor version for each release of nixpkgs. We could still use the nixpkgs to build our application but we have to override each of our dependencies to the right version, but that approach can get quiet tedious for a large number of dependencies. Fortunately there are tools to automatically generate your dependencies from a requirements.txt such as mach-nix or pip2nix.

SuperSandro2000 3y ago

> Is it fair to summize that python applications with python dependencies do not really work well as nix packages and shouldn't be used? No, applications that are properly maintained work as they should and this can be ensured with tests and e2e tests.

lolinder 3y ago
This is such a condescending attitude. What you mean is applications that are maintained the way that you and the Nix developers think an application should be maintained. It's incredibly naive for a package manager as ambitious as Nix to assume semver. I'm a big fan of semver myself, but the vast majority of software projects follow it imperfectly or not at all, and for good reason—it's nearly impossible to follow it perfectly, because even bugs are part of your API. Every project I've worked on has eventually had something break on a version upgrade because we were depending on something that was later decided to be a bug (but at the time was just how it worked). Elm can mostly get away with enforcing semver because they designed it that way at the language level, but Nix wants to manage dependencies written in all languages and ecosystems, which have dramatically different versioning practices.
- SuperSandro2000 3y ago
  Nixpkgs does not assume Denver that's why we run if possible the package's tests, our own tests and build dependent packages to make sure the most obvious breakages are noticed before things are even merged.
  - lolinder 3y ago
    Ah, I thought you were saying that if we all just used e2e tests to ensure we didn't make breaking changes in minor versions, we'd be fine. I didn't realize you were talking about the downstream package's tests. I do still take issue with your insinuation that it's the package maintainers' poor practices that are at fault here. The real world is a messy, complex place and "best practices" don't translate well from situation to situation. OP didn't ask for their package to be included in Nix. Presumably OP's system works for them and for their use case, but whoever created the Nix package made assumptions that turned out to be flawed. It's not fair of you to say that those bad assumptions are OP's fault because their package isn't "properly maintained" and doesn't "work as it should". Someone (you?) made a bad assumption. Don't cast blame for that on someone who only knows Nix exists because it sends phony bug reports their way.