4 ms·

> As an observer it looks like Python's package ecosystem in particular is a giant mess. Most people would say it is. The better question is how to avoid such

by jnxx 4y ago

> As an observer it looks like Python's package ecosystem in particular is a giant mess. Most people would say it is. The better question is how to avoid such a mess. My take is that maintaining backward compatibility is a core principle which needs to be strictly observed to solve that problem. And yeah, that has also become a cultural issue with Python, as the Python2/3 breakage shows. So, could one sum it up in that Nix magnifies unsolved compatibility issues in packaging systems? Because if there were a single core Python distribution, like say, Anaconda, but nothing else, these issues would not exist. Of course, people can avoid the issues if they only use a handful of packages. But putting all into a single channel makes the problem much more acute.

infogulch 4y ago
> Nix magnifies unsolved compatibility issues in packaging systems ... putting all into a single channel makes the problem much more acute Well said.
lamontcg 4y ago
> My take is that maintaining backward compatibility is a core principle which needs to be strictly observed to solve that problem. Given that software developers never guess the correct design up front this means that you always have architecturally buggy software, and a bunch of complaining about why buggy-looking edge conditions are never fixed. There has to be some kind of release valve for software to evolve and break backwards compatibility.
- infogulch 4y ago
  Yes they've invented that: It's called bumping the major version.
  - lamontcg 4y ago
    The parent comment is complaining about Python 2/3 breakage which was a major version.
    - infogulch 4y ago
      Not exactly, they're saying that the Python 2/3 breakage debacle became a model for how packages are maintained in general.
      - lamontcg 4y ago
        Well they should clarify because they cite the example of the 2/3 breakage which is a major version break, and they're not complaining about packages just violating SemVer so I don't think your interpretation makes sense. It also certainly wasn't what I was responding to, and responding with "hurr durr major versions" like I've never heard of them before is just mildly insulting (and kind of insulting to the parent comment by proxy)
        jnxx 4y ago
        > they're saying that the Python 2/3 breakage debacle became a model for how packages are maintained in general. As explained above, semver is not a solution. And it is often not really followed. For example, boost is breaking backwards compatibility some times, and this is causing problems, last not least because boost Python bindings are used in so many projects.
        jnxx 4y ago
        > they're not complaining about packages just violating SemVer so I don't think your interpretation makes sense. Packages regularly violate semver, to the degree it becomes a cargo cult. It is funny that at the one hand side, people say that keeping backward compatibility is too difficult for normal package authors and contributors, and on the other hand side suggesting that using semver would improve this. In order to actually use semver, one needs to know what backward compatibility is, what kind of changes break it, and how to make sure that this breakage does not happen. It is not that difficult. But also, semver require stability against an API, so one absolutely needs to have a clearly documented API of some sort, because otherwise, if there is absolutely nothing specific you promise, how could one expect you to keep it? And further more, major packages should actually respect semver if they claim to use it, and not do breaking changes with minor version numbers, like for example boost does. Actually, I think if somebody uses a three-element version number and does not strictly adhere to semver, it should come in a popup box in front of every download link, because these three-part version numbers somehow imply that the package uses semver, and this in some cases (like boost) is a false promise. And before somebody throws in that it is not *his* package that is breaking semver, but some dependency that his package happens to use: No. If you use dependencies, you are responsible for their behavior, because otherwise, one could always shift the blame somewhere else. If a dependent package breaks backward compatibility and your package is a library package, including it is a breaking change, because backwards (in)compatibility of dependencies which have visible effects (as is the case in all Python library modules, as has been discussed) is a transitive property which travels up the dependency graph. If you include visible breaking changes, then your package introduces a breaking change, and cannot honour semver without bumping the major version number.
  - jnxx 4y ago
    > Yes they've invented that: It's called bumping the major version. Rich Hickey's Talk: https://www.youtube.com/watch?v=oyLBGkS5ICk https://www.youtube.com/watch?v=oyLBGkS5ICk explains why this is not a solution. It makes a difference, yes. But it is the difference between "the incompatible changes in my library are going to break your application" and "the incompatible changes in my library are going to break your application, and I am telling you this beforehand".
- btschaegg 4y ago
  I don't disagree, but it doesn't only have to go in this one direction. One of the most interesting things about Rust for example is how it tackles experimental implementations and has concepts in the compiler etc. that make unstable language features "first class". I'd say this will definitely yield better results than "well a couple of guys hacked around on some prototype forks of a compiler, and now we're stuck with the result". Of course, they also make very impressive backwards compatibility guarantees for stable stuff (cf. Rust's "editions").
  - lamontcg 4y ago
    Rust has corporate sponsorship and a very experienced team of developers. You won't get that level of attention to detail and commitment to getting the design right up front, and the willingness to maintain old APIs in the name of backwards compatibility in a single-person open source project published into a package manager done on someone's free time. So you are probably arguing for very thick standard libraries which are maintained by the core language team, which is corporate sponsored, and a reduction in reliance on open source packages. That also means as well that we shouldn't tolerate "shaming" of projects for taking a long time to fix and merge features since 95% of the work will be required to be done up front in thinking about the right shape of APIs. I'm cool with all of that as long as the whole package comes along. The idea that a bunch of solo, unpaid open source maintainers are going to be doing good API design up front and maintaining perfect backcompat, while being incredibly responsive to PRs from the community is kind of "unicorn farts" levels of not going to happen in the real world. You sort of get what you pay for, and a bunch of unpaid solo volunteers are going to need to make breaking changes to fix their old mistakes and abandon maintaining their old tech debt. And if you paid nothing for it, really you're getting more than you deserve in that deal.
    - btschaegg 4y ago
      > So you are probably arguing for very thick standard libraries which are maintained by the core language team, […] No, I'm not arguing that Nix should do anything in particular. All I was saying is that the "we have to get it right the first time without much feedback" way obviously isn't the only one and there's empirical evidence of other working models. As for PRs and so on you're really putting words in my mouth, and frankly, I don't like it. Just so you know. I have never made PRs to core Nix, but for Nixpkgs I have only had a good experience so far. --- Edit: Or are we really talking Python? In that case, I could even less comment on PRs. But: Python has a large stdlib (it's "batteries included", after all). But also, Python often has found good ways to deal with its warts. And I hope I don't have to argue that Python3k wasn't worth the trouble, right? And frankly, there, I'd argue that growing to the point that python has you'll have to reexamine some more ways to gather data about community interest, for example. From the outside, the process around the walrus operator and Guido leaving the BDFL post looks like a prime issue of either not having enough "wild information" early on or of the final decision ignoring a vocal part of a huge language community.
- jnxx 4y ago
  > There has to be some kind of release valve for software to evolve and break backwards compatibility. You kinda insinuate that breaking backwards compatibility is kinda necessary at times. This is not the case. Projects like * the Linux kernel, or * the GNU C library, or * the Numeric -> Numpy transition around Python 2.0, or * Common Lisp (which is much older than Python) adopting Unicode are good examples that this is not necessary. It is not true that you have to break backward compatibility. There are domains where breaking backwards compatibility in libraries is not acceptable at all, like vendor libraries in industrial automation. You don't throw away a 15-year old printing machine or a chemical plant just because the vendor of the automation software is tired of supporting its old interfaces. That might sound strong, but others have expressed it more strongly. Read this: https://linuxreviews.org/WE_DO_NOT_BREAK_USERSPACE https://linuxreviews.org/WE_DO_NOT_BREAK_USERSPACE How it is done? It starts with well-designed interfaces. And when interfaces are changed, the old interfaces are kept and become special cases of the new ones. Numeric/Numpy is a good example. Here is a talk, brilliant as always, by Rich Hickey which explains why and how: https://www.youtube.com/watch?v=oyLBGkS5ICk https://www.youtube.com/watch?v=oyLBGkS5ICk It is highly relevant to Nix and Guix. Python3 could have gone the same way - keeping the interpreter compatible to Python2 code, making the semantics dependent on whether a source file has a *.py or a *.py3 extension, and so on. It would have been more work but the transition would have been nearly painless, and I guess much faster. Support for old stuff does not need to go on forever - for example, Linux does not support any more Intel 386 CPUs. It boils down to whether keeping stuff backwards-compatible is a goal of the project leaders or not.
  - lamontcg 4y ago
    The problem with most open source software that is in package managers is that it is usually done by one person. It isn't started by someone with a decade of interface design, it is often their first large important project, they DONT do the well designed interfaces because they haven't made the mistakes in interface design yet which they'll eventually learn from. And then when it comes to backwards compatibility it is cheap for you to say they should just support their old interfaces forever, but that has a cost and creates more friction going forwards for the projected. When it is one person working on open source who isn't getting paid, that is all somewhat unreasonable to expect and you just won't ever get it. In your world what you'll wind up with instead of back compat breaking changes is just abandoned and rotting software as maintainers give up. You could do this by arguing that languages need very thick and well-designed standard libraries, which means that hopefully there's a large business supporting the library and there are teams of reasonably well paid software engineers who are doing the design work up front for everything. You should be explicit about that though. I'm kind of not surprised that you cite one of Linus' asshole rants to LKML as well. Try screaming that at a single-person open source maintainer and watch them decide it just isn't worth it any more and quit on the project entirely. If you want that then don't use anything outside of your language's standard library and don't use package managers and contributed source code at all. Write everything else yourself, no dependencies, no worries about backwards compatibility breaks.
zaphirplane 4y ago
Is the discussion here that Nix doesn’t support multiple side by side installations of different versions making it difficult to install two packages depending on the same package but different versions ? I thought in Nix you can have that, and python can have venv for independent library versions, is it that no one has done the work to combine the two ?
- pxc 4y ago
  Nope. All Python programs in Nix are effectively venv'd, so-to-speak. The problem is that if two packages A & B each depend on the same library L but they require different versions of it, any Python code that imports both as libraries will end up with two, potentially incompatible, copies of the same library in its import path. And Python doesn't support this, so it uses the same version of L with all the Python code running in that process. So now depending on whether some package P which requires both A and B imports A first or B first, A will end up using B's version of L or vice-versa. This sometimes does nothing, and it sometimes causes very weird, very subtle breakage. Thus for all of the Python libraries in Nixpkgs to be usable in any combination by any package in Nixpkgs, there can only be one version of each Python library in Nixpkgs. Once your package collection is large enough, you start actually encountering versioning conflicts as described above in the transitive Python dependencies of your end-user application packages. That's why Linux distros run into these integration issues. Application developers generally don't because their applications development environments are much smaller.
- SuperSandro2000 4y ago
  > I thought in Nix you can have that, and python can have venv for independent library versions, is it that no one has done the work to combine the two ? A python environment is effectively a different take on a venv. Python packages is one giant python environment. For programs that life outside of python packages but use the packages from there we are free to apply overrides how we want. So it is possible to have different versions of a package in a different python environment.

by jnxx 4y ago

infogulch 4y ago

> Nix magnifies unsolved compatibility issues in packaging systems ... putting all into a single channel makes the problem much more acute Well said.

lamontcg 4y ago

> My take is that maintaining backward compatibility is a core principle which needs to be strictly observed to solve that problem. Given that software developers never guess the correct design up front this means that you always have architecturally buggy software, and a bunch of complaining about why buggy-looking edge conditions are never fixed. There has to be some kind of release valve for software to evolve and break backwards compatibility.

infogulch 4y ago
Yes they've invented that: It's called bumping the major version.
- lamontcg 4y ago
  The parent comment is complaining about Python 2/3 breakage which was a major version.
  - infogulch 4y ago
    Not exactly, they're saying that the Python 2/3 breakage debacle became a model for how packages are maintained in general.
    - lamontcg 4y ago
      Well they should clarify because they cite the example of the 2/3 breakage which is a major version break, and they're not complaining about packages just violating SemVer so I don't think your interpretation makes sense. It also certainly wasn't what I was responding to, and responding with "hurr durr major versions" like I've never heard of them before is just mildly insulting (and kind of insulting to the parent comment by proxy)
      - jnxx 4y ago
        > they're saying that the Python 2/3 breakage debacle became a model for how packages are maintained in general. As explained above, semver is not a solution. And it is often not really followed. For example, boost is breaking backwards compatibility some times, and this is causing problems, last not least because boost Python bindings are used in so many projects.
      - jnxx 4y ago
        > they're not complaining about packages just violating SemVer so I don't think your interpretation makes sense. Packages regularly violate semver, to the degree it becomes a cargo cult. It is funny that at the one hand side, people say that keeping backward compatibility is too difficult for normal package authors and contributors, and on the other hand side suggesting that using semver would improve this. In order to actually use semver, one needs to know what backward compatibility is, what kind of changes break it, and how to make sure that this breakage does not happen. It is not that difficult. But also, semver require stability against an API, so one absolutely needs to have a clearly documented API of some sort, because otherwise, if there is absolutely nothing specific you promise, how could one expect you to keep it? And further more, major packages should actually respect semver if they claim to use it, and not do breaking changes with minor version numbers, like for example boost does. Actually, I think if somebody uses a three-element version number and does not strictly adhere to semver, it should come in a popup box in front of every download link, because these three-part version numbers somehow imply that the package uses semver, and this in some cases (like boost) is a false promise. And before somebody throws in that it is not *his* package that is breaking semver, but some dependency that his package happens to use: No. If you use dependencies, you are responsible for their behavior, because otherwise, one could always shift the blame somewhere else. If a dependent package breaks backward compatibility and your package is a library package, including it is a breaking change, because backwards (in)compatibility of dependencies which have visible effects (as is the case in all Python library modules, as has been discussed) is a transitive property which travels up the dependency graph. If you include visible breaking changes, then your package introduces a breaking change, and cannot honour semver without bumping the major version number.
- jnxx 4y ago
  > Yes they've invented that: It's called bumping the major version. Rich Hickey's Talk: https://www.youtube.com/watch?v=oyLBGkS5ICk https://www.youtube.com/watch?v=oyLBGkS5ICk explains why this is not a solution. It makes a difference, yes. But it is the difference between "the incompatible changes in my library are going to break your application" and "the incompatible changes in my library are going to break your application, and I am telling you this beforehand".
btschaegg 4y ago
I don't disagree, but it doesn't only have to go in this one direction. One of the most interesting things about Rust for example is how it tackles experimental implementations and has concepts in the compiler etc. that make unstable language features "first class". I'd say this will definitely yield better results than "well a couple of guys hacked around on some prototype forks of a compiler, and now we're stuck with the result". Of course, they also make very impressive backwards compatibility guarantees for stable stuff (cf. Rust's "editions").
- lamontcg 4y ago
  Rust has corporate sponsorship and a very experienced team of developers. You won't get that level of attention to detail and commitment to getting the design right up front, and the willingness to maintain old APIs in the name of backwards compatibility in a single-person open source project published into a package manager done on someone's free time. So you are probably arguing for very thick standard libraries which are maintained by the core language team, which is corporate sponsored, and a reduction in reliance on open source packages. That also means as well that we shouldn't tolerate "shaming" of projects for taking a long time to fix and merge features since 95% of the work will be required to be done up front in thinking about the right shape of APIs. I'm cool with all of that as long as the whole package comes along. The idea that a bunch of solo, unpaid open source maintainers are going to be doing good API design up front and maintaining perfect backcompat, while being incredibly responsive to PRs from the community is kind of "unicorn farts" levels of not going to happen in the real world. You sort of get what you pay for, and a bunch of unpaid solo volunteers are going to need to make breaking changes to fix their old mistakes and abandon maintaining their old tech debt. And if you paid nothing for it, really you're getting more than you deserve in that deal.
  - btschaegg 4y ago
    > So you are probably arguing for very thick standard libraries which are maintained by the core language team, […] No, I'm not arguing that Nix should do anything in particular. All I was saying is that the "we have to get it right the first time without much feedback" way obviously isn't the only one and there's empirical evidence of other working models. As for PRs and so on you're really putting words in my mouth, and frankly, I don't like it. Just so you know. I have never made PRs to core Nix, but for Nixpkgs I have only had a good experience so far. --- Edit: Or are we really talking Python? In that case, I could even less comment on PRs. But: Python has a large stdlib (it's "batteries included", after all). But also, Python often has found good ways to deal with its warts. And I hope I don't have to argue that Python3k wasn't worth the trouble, right? And frankly, there, I'd argue that growing to the point that python has you'll have to reexamine some more ways to gather data about community interest, for example. From the outside, the process around the walrus operator and Guido leaving the BDFL post looks like a prime issue of either not having enough "wild information" early on or of the final decision ignoring a vocal part of a huge language community.
jnxx 4y ago
> There has to be some kind of release valve for software to evolve and break backwards compatibility. You kinda insinuate that breaking backwards compatibility is kinda necessary at times. This is not the case. Projects like * the Linux kernel, or * the GNU C library, or * the Numeric -> Numpy transition around Python 2.0, or * Common Lisp (which is much older than Python) adopting Unicode are good examples that this is not necessary. It is not true that you have to break backward compatibility. There are domains where breaking backwards compatibility in libraries is not acceptable at all, like vendor libraries in industrial automation. You don't throw away a 15-year old printing machine or a chemical plant just because the vendor of the automation software is tired of supporting its old interfaces. That might sound strong, but others have expressed it more strongly. Read this: https://linuxreviews.org/WE_DO_NOT_BREAK_USERSPACE https://linuxreviews.org/WE_DO_NOT_BREAK_USERSPACE How it is done? It starts with well-designed interfaces. And when interfaces are changed, the old interfaces are kept and become special cases of the new ones. Numeric/Numpy is a good example. Here is a talk, brilliant as always, by Rich Hickey which explains why and how: https://www.youtube.com/watch?v=oyLBGkS5ICk https://www.youtube.com/watch?v=oyLBGkS5ICk It is highly relevant to Nix and Guix. Python3 could have gone the same way - keeping the interpreter compatible to Python2 code, making the semantics dependent on whether a source file has a *.py or a *.py3 extension, and so on. It would have been more work but the transition would have been nearly painless, and I guess much faster. Support for old stuff does not need to go on forever - for example, Linux does not support any more Intel 386 CPUs. It boils down to whether keeping stuff backwards-compatible is a goal of the project leaders or not.
- lamontcg 4y ago
  The problem with most open source software that is in package managers is that it is usually done by one person. It isn't started by someone with a decade of interface design, it is often their first large important project, they DONT do the well designed interfaces because they haven't made the mistakes in interface design yet which they'll eventually learn from. And then when it comes to backwards compatibility it is cheap for you to say they should just support their old interfaces forever, but that has a cost and creates more friction going forwards for the projected. When it is one person working on open source who isn't getting paid, that is all somewhat unreasonable to expect and you just won't ever get it. In your world what you'll wind up with instead of back compat breaking changes is just abandoned and rotting software as maintainers give up. You could do this by arguing that languages need very thick and well-designed standard libraries, which means that hopefully there's a large business supporting the library and there are teams of reasonably well paid software engineers who are doing the design work up front for everything. You should be explicit about that though. I'm kind of not surprised that you cite one of Linus' asshole rants to LKML as well. Try screaming that at a single-person open source maintainer and watch them decide it just isn't worth it any more and quit on the project entirely. If you want that then don't use anything outside of your language's standard library and don't use package managers and contributed source code at all. Write everything else yourself, no dependencies, no worries about backwards compatibility breaks.

zaphirplane 4y ago

Is the discussion here that Nix doesn’t support multiple side by side installations of different versions making it difficult to install two packages depending on the same package but different versions ? I thought in Nix you can have that, and python can have venv for independent library versions, is it that no one has done the work to combine the two ?

pxc 4y ago
Nope. All Python programs in Nix are effectively venv'd, so-to-speak. The problem is that if two packages A & B each depend on the same library L but they require different versions of it, any Python code that imports both as libraries will end up with two, potentially incompatible, copies of the same library in its import path. And Python doesn't support this, so it uses the same version of L with all the Python code running in that process. So now depending on whether some package P which requires both A and B imports A first or B first, A will end up using B's version of L or vice-versa. This sometimes does nothing, and it sometimes causes very weird, very subtle breakage. Thus for all of the Python libraries in Nixpkgs to be usable in any combination by any package in Nixpkgs, there can only be one version of each Python library in Nixpkgs. Once your package collection is large enough, you start actually encountering versioning conflicts as described above in the transitive Python dependencies of your end-user application packages. That's why Linux distros run into these integration issues. Application developers generally don't because their applications development environments are much smaller.
SuperSandro2000 4y ago
> I thought in Nix you can have that, and python can have venv for independent library versions, is it that no one has done the work to combine the two ? A python environment is effectively a different take on a venv. Python packages is one giant python environment. For programs that life outside of python packages but use the packages from there we are free to apply overrides how we want. So it is possible to have different versions of a package in a different python environment.