4 ms·

Should You Use Upper Bound Version Constraints?

39 points by BerislavLopac 5y ago

travisd 5y ago
I enjoy Go’s approach. This is a cultural thing more than a function of what the language itself imposes (and there are certainly many exceptions), but usually major version = revolutionary change (might as well be a new library). Major versions are taken as opportunities to refine what is known about the problem space and fix bad abstractions. Maybe a remmenant from the pre-module days.
- jrockway 5y ago
  Yeah, Go's approach seems to be doing fairly well in practice. There are occasional problems (logrus's author changing the capitalization of their name, azure-autorest), but overall it seems to work. One thing that's nice is that library changes are scoped to a top-level module. So if you have two projects and update a module that both projects use in one of your projects, you don't update the libraries used by the other project. And, fresh installs pick the exact version the go.mod file specifies; version substitution logic only shows up when you depend on module A and module B that both depend on module C and A and B specify a different minor version of C. This is as annoying as it is in Python or Node, but at least more limited in scope. (A project I work on suffered greatly with this problem. We depended on Kubernetes, and had a tiny bit of code that reads logs from Loki. Loki's client depends on Prometheus, which depends on Kubernetes. So the version of Kubernetes we could use was limited by that selection, and we had no control beyond forking the Loki library. If you're a module author and want to avoid inflicting this pain upon your users; a couple things went wrong here. A long time ago Kubernetes' client-go used go module versions matching the public version of Kubernetes. 10 corresponded with 1.10, 11 corresponded with 1.11, etc. Go doesn't consider these two compatible, but they basically are. They switched the version numbers to be 0.10, 0.11, etc. resolving that problem. Our dependency on Loki was for the 1.x branch, which was too old to benefit from this. The other thing authors can do is not bundle their clients and servers. It's convenient, since your tests probably like using the client to talk to the server. But if you can break that dependency, then you probably only have a dependency on net/http and not all of Kubernetes, which is nice for your clients. Oh well!)
  - travisd 5y ago
    I’ve had to fork a dependency (Kaniko). That wasn’t the main reason at the time, but it did end up helping that (since kaniko depends on a specific docker version). But at the very least, forking is relatively easy (git subtree plus replace in go.mod).
resonious 5y ago
I've noticed a similar problem with C/C++ projects that use -werror. It effectively puts an upper bound on the compiler version used to compile the library when new warnings are added since the last time the library was updated.
- cozzyd 5y ago
  Yes it is generally a bad idea to use -Werror in a release build. Fortunately this is easy to fix unlike the library version capping conflict.
- Too 5y ago
  In Cmake mark third party libraries as SYSTEM. I think this translates to -isystem instead of -i and should not generate warnings.
zebracanevra 5y ago
If a library requires pyparsing>=3 and another library requests pyparsing<3, that’s the end, you are out of business. This article is very much only applicable to Python and other package managers that don't scope dependencies. I think nuget (.NET) also works like this. In Nodejs land, if you depend on pkg v3, and another dependency needs v2, they'll happily co-exist with no trouble, as long as those v2 and v3 didn't need to talk to each other. If you did only want one version of a package, that is also supported with peerDependencies.
- cozzyd 5y ago
  Could scoped dependencies even work in Python? Especially if there is any native C code involved (which there will be) I imagine you may get symbol conflicts when trying to load two different versions of the same shared sofile.
- masklinn 5y ago
  The article covers that just a few paragraphs below.
  - deleted 5y ago
    [deleted]
  - Wowfunhappy 5y ago
    ...I'm probably going to be very sheepish once I find it, but I took a quick look back at the article and I couldn't find this? (I'm not the GP, but I had the same question as they did.)
    - masklinn 5y ago
      There is a section called “Python is not JavaScript”, the first subsection of which is “technical differences”. In it the author explains that javascript/npm uses local dependencies, so transitive conflicts are not an issue, and it clearly states that this “invalidates several of the arguments above”.
Too 5y ago
Instead of expecting every library to follow best practices, can this be solved in the package manager instead. First of all, require all non-lower constraints to include a mandatory reason field. Heck, while you’re at it, mandate a reason-field for including the package in the first place, package.json doesn’t even allow comments for describing why a package is included, sigh. Second, downstream applications should be allowed to easily force an override from an upstream constraint. The constraint might have been added because it broke feature foo in the library but I only use bar so I can accept that incompatibility. In fact pip’s separation of requirements.txt and constraints.txt does play quite nicely with this model. Libraries could, in theory, provide a constraints.txt file with a tested combination of packages, which downstream applications are free to completely ignore or to use in pip install -c. In practice it’s a fair bit of administration so might be tough to apply. Would have liked to to see some more mentions of constraints.txt mentioned more in the article btw, this is the pip alternative to lock-files, but it is a rather new feature so I guess that’s why author advices pip-tools and frozen requirements.txt instead.
- genuine_smiles 5y ago
  > Second, downstream applications should be allowed to easily force an override from an upstream constraint. The constraint might have been added because it broke feature foo in the library but I only use bar so I can accept that incompatibility. Yarn has this feature. See “resolutions” in its package.json documentation: https://yarnpkg.com/configuration/manifest https://yarnpkg.com/configuration/manifest
- onion2k 5y ago
  package.json doesn’t even allow comments for describing why a package is included, sigh It sort of does. NPM ignores properties it doesn't understand, so you can easily add something like "dependencyReasons": { "react": "Make website go brr" } if you want to. I imagine your actual complaint is that there isn't a -m flag to add a comment when you install something, or a way to list those comments when you list the packages, or a way to enforce the user always adds a comment. It's true that those are missing, and that they'd be very useful, but you can still do it manually if it's important enough to you.
- 5y ago
rob_c 5y ago
Had a similar conversation with academics who seem to think it's a great idea to change upper and lower limits for every release based upon what they ran their test-suite on. Frankly as you'd imagine it leads to more requests to expand the test-suite or remove the limits because there is always love group using an "untested therefore unsupported" external library that they can't/don't control. Min version yes. Massive ABI/API breakage then sure have a max version, but set it to something like 2.9999 if you know your code doesn't work with 3 not 2.7.9.11 because that was the last release at the time of writing... Basically most coders make bad repo maintainers.

39 points by BerislavLopac 5y ago

I enjoy Go’s approach. This is a cultural thing more than a function of what the language itself imposes (and there are certainly many exceptions), but usually major version = revolutionary change (might as well be a new library). Major versions are taken as opportunities to refine what is known about the problem space and fix bad abstractions. Maybe a remmenant from the pre-module days.

jrockway 5y ago
Yeah, Go's approach seems to be doing fairly well in practice. There are occasional problems (logrus's author changing the capitalization of their name, azure-autorest), but overall it seems to work. One thing that's nice is that library changes are scoped to a top-level module. So if you have two projects and update a module that both projects use in one of your projects, you don't update the libraries used by the other project. And, fresh installs pick the exact version the go.mod file specifies; version substitution logic only shows up when you depend on module A and module B that both depend on module C and A and B specify a different minor version of C. This is as annoying as it is in Python or Node, but at least more limited in scope. (A project I work on suffered greatly with this problem. We depended on Kubernetes, and had a tiny bit of code that reads logs from Loki. Loki's client depends on Prometheus, which depends on Kubernetes. So the version of Kubernetes we could use was limited by that selection, and we had no control beyond forking the Loki library. If you're a module author and want to avoid inflicting this pain upon your users; a couple things went wrong here. A long time ago Kubernetes' client-go used go module versions matching the public version of Kubernetes. 10 corresponded with 1.10, 11 corresponded with 1.11, etc. Go doesn't consider these two compatible, but they basically are. They switched the version numbers to be 0.10, 0.11, etc. resolving that problem. Our dependency on Loki was for the 1.x branch, which was too old to benefit from this. The other thing authors can do is not bundle their clients and servers. It's convenient, since your tests probably like using the client to talk to the server. But if you can break that dependency, then you probably only have a dependency on net/http and not all of Kubernetes, which is nice for your clients. Oh well!)
- travisd 5y ago
  I’ve had to fork a dependency (Kaniko). That wasn’t the main reason at the time, but it did end up helping that (since kaniko depends on a specific docker version). But at the very least, forking is relatively easy (git subtree plus replace in go.mod).

resonious 5y ago

I've noticed a similar problem with C/C++ projects that use -werror. It effectively puts an upper bound on the compiler version used to compile the library when new warnings are added since the last time the library was updated.

cozzyd 5y ago
Yes it is generally a bad idea to use -Werror in a release build. Fortunately this is easy to fix unlike the library version capping conflict.
Too 5y ago
In Cmake mark third party libraries as SYSTEM. I think this translates to -isystem instead of -i and should not generate warnings.

zebracanevra 5y ago

If a library requires pyparsing>=3 and another library requests pyparsing<3, that’s the end, you are out of business. This article is very much only applicable to Python and other package managers that don't scope dependencies. I think nuget (.NET) also works like this. In Nodejs land, if you depend on pkg v3, and another dependency needs v2, they'll happily co-exist with no trouble, as long as those v2 and v3 didn't need to talk to each other. If you did only want one version of a package, that is also supported with peerDependencies.

cozzyd 5y ago
Could scoped dependencies even work in Python? Especially if there is any native C code involved (which there will be) I imagine you may get symbol conflicts when trying to load two different versions of the same shared sofile.
masklinn 5y ago
The article covers that just a few paragraphs below.
- deleted 5y ago
  [deleted]
- Wowfunhappy 5y ago
  ...I'm probably going to be very sheepish once I find it, but I took a quick look back at the article and I couldn't find this? (I'm not the GP, but I had the same question as they did.)
  - masklinn 5y ago
    There is a section called “Python is not JavaScript”, the first subsection of which is “technical differences”. In it the author explains that javascript/npm uses local dependencies, so transitive conflicts are not an issue, and it clearly states that this “invalidates several of the arguments above”.

Too 5y ago

Instead of expecting every library to follow best practices, can this be solved in the package manager instead. First of all, require all non-lower constraints to include a mandatory reason field. Heck, while you’re at it, mandate a reason-field for including the package in the first place, package.json doesn’t even allow comments for describing why a package is included, sigh. Second, downstream applications should be allowed to easily force an override from an upstream constraint. The constraint might have been added because it broke feature foo in the library but I only use bar so I can accept that incompatibility. In fact pip’s separation of requirements.txt and constraints.txt does play quite nicely with this model. Libraries could, in theory, provide a constraints.txt file with a tested combination of packages, which downstream applications are free to completely ignore or to use in pip install -c. In practice it’s a fair bit of administration so might be tough to apply. Would have liked to to see some more mentions of constraints.txt mentioned more in the article btw, this is the pip alternative to lock-files, but it is a rather new feature so I guess that’s why author advices pip-tools and frozen requirements.txt instead.

genuine_smiles 5y ago
> Second, downstream applications should be allowed to easily force an override from an upstream constraint. The constraint might have been added because it broke feature foo in the library but I only use bar so I can accept that incompatibility. Yarn has this feature. See “resolutions” in its package.json documentation: https://yarnpkg.com/configuration/manifest https://yarnpkg.com/configuration/manifest
onion2k 5y ago
package.json doesn’t even allow comments for describing why a package is included, sigh It sort of does. NPM ignores properties it doesn't understand, so you can easily add something like "dependencyReasons": { "react": "Make website go brr" } if you want to. I imagine your actual complaint is that there isn't a -m flag to add a comment when you install something, or a way to list those comments when you list the packages, or a way to enforce the user always adds a comment. It's true that those are missing, and that they'd be very useful, but you can still do it manually if it's important enough to you.
5y ago

rob_c 5y ago

Had a similar conversation with academics who seem to think it's a great idea to change upper and lower limits for every release based upon what they ran their test-suite on. Frankly as you'd imagine it leads to more requests to expand the test-suite or remove the limits because there is always love group using an "untested therefore unsupported" external library that they can't/don't control. Min version yes. Massive ABI/API breakage then sure have a max version, but set it to something like 2.9999 if you know your code doesn't work with 3 not 2.7.9.11 because that was the last release at the time of writing... Basically most coders make bad repo maintainers.