8 ms·
Package Management: The problem with using version ranges
- kronos29296 9y agoSomething like Haskell Stack fixes a lot of problems with versions but it is not suitable for all situations. It is a lot of work to produce a curated set of packages that work. But most of the time things just work so people work around it the rest of the time.
- wereHamster 9y agoIt's not stack but stackage you are referring to. Stackage puts considerable effort on the whole community to ensure that packages remain compatible, and forces package publishers (developers) to update their packages in a timely manner (otherwise the package will be dropped from the snapshot). That works in the Haskell community, but I'd be hard pressed to believe any javascript developer who has a package on npm.org would put the same amount of effort into keeping it up to date.
- kronos29296 9y agoPackage manager is stack and the online repo is hosted at stackage. You manage packages from stackage using stack. You probably can't do it in a larger community like JS or Python because it is too big. Haskell is at the right size to do such things.
- wereHamster 9y agoNot all Haskell packages (those in hackage.org) are in the stackage snapshots. Stackage is opt-in, there must be a maintainer willing to resolve the occasional build failures. That means there is no such thing as 'too big', because the number of all existing packages in a ecosystem doesn't matter. It's the commitment that counts, and that's what I don't see in the JS community.
- chriswarbo 9y agoSuch curated sets make sense for "core functionality", like text handling, streaming, encoding/decoding, etc., which are used by many 'mainstream' applications (Web backends, console programs, etc.). Yet they raise the barrier to entry by placing a larger burden on developers. For example, there's a long tail of one-off packages on Hackage that are really valuable to keep around, even if they're not actively maintained and bumping versions would cause breaking changes. I'm thinking of all the various logic frameworks, theorem provers, etc. which are out there; maybe to accompany some paper from the 90s. They're certainly not "core functionality", but every now and then it may turn out that one of them is exactly what is needed in some app/library. Every few years the major repos may drop the last working version of GHC, so someone may have to step up and do the required fixes, but that's few and far between. These days I use a combination of Hackage (all versions of all packages), Cabal (to solve dependencies), tinc (to make Cabal deterministic) and Nix (to cache everything).
- sjrd 9y agoHow come this is not already obvious to everyone? I completely agree with the article. Version ranges are evil. Avoid them at all cost.
- yuchi 9y agoThat’s not right. Ranges are powerful and important, but you need a locking mechanism too otherwise you‘ll loose control. Npm and Yarn got finally right and dependency management in the npm ecosystem is now a breeze.
- andybak 9y ago> How come this is not already obvious to everyone? Because other people have different priorities and have come to a different cost/benefit ratio value than you?
- alexscheelmeyer 9y agoOne reason is that many developers work with package managers / languages that are INCAPABLE of handling multiple versions of the same dependency. Without this you quickly get version deadlocks without semver. Another reason is that many developers are obsessing on getting the latest version of their dependencies for fear of security issues or just missing out on the latest and greatest - and they often completely ignore retesting the application since they now have someone to blame if it fails (that other developer should not have pushed the breaking change with a minor version bump!) I agree with you that it should be the standard to have fixed versions and update your dependencies at a time of your choosing so that everything can get tested properly - but it seems to be an uphill battle.
- eeZah7Ux 9y ago> many developers are obsessing on getting the latest version of their dependencies for fear of security issues Getting the latest version is how you get new vulnerabilities. Various software distributors, including some Linux distros let software bake in for this reason and can be even faster than the upstreams in developing and applying patches to known vulnerabilities. Also, unfixed but known vulnerabilities are less dangerous: security and system engineers can work around them, also IDS/IPS can detect and often block attacks.
- _pmf_ 9y agoThis is about using open version ranges. Closed ranges are slightly less evil, though errors can creep in, too.
- yuchi 9y agoVery nice point. If you look at all npm examples in the article it shows the >= M.m.b pattern which is never found in the wild. You’ll find ~M.m.b for VERY old packages and ^M.m.b for modern ones (last 4 years). Those let the minor and build increase, but not the major.
- yuchi 9y agoPlease note: I can speak only for the Node.js npm/Yarn ecosystem. This article is terribly outdated. Version ranges are great and the only way to manage correctly your dependencies without having tons of duplicates. Also they are a security guarantee, because with them you can get security fixes “automatically”. Yet there have been problems, that’s why both Yarn and npm@5 implement a lock file which gives you all the guarantees of a range-less package management with the power of a range-based dependency tree. If module developers following this (IMHO not lucid) article start bundling specific versions in their npm packages everyone will suffer. It’s as stupid as it get. Please, do your research before calling out on practices.
- rdsubhas 9y agonode+npm5/yarn, ruby+gemfile, rust+cargo (and maybe a few more) are the best examples for the whole workflow around packaging. Semver ranges (dependency 1.0.x) along with version pinning (lock files that represent the currently installed versions) are the best-case scenario. One can avoid the exercise of "bumping versions" every once in a while (just upgrade all packages within supported semver ranges), get the same semver compatibility guarantees, makes sure all other team members are using same versions, and most importantly the CI process is reproduceable and can be sped up with reliable, non-broken dependency caching. The elephant in the room is Java. Despite its maturity, both mvn and gradle have poor support for ranges and pretty much no concept of lock files. Everyone is left to "bumping versions" by hand all the time, and snapshot versions are pretty much hell and lead to irreproduceable builds. Vendoring the whole tree is only an optional add-on to semver ranges. You can't vendor a whole tree and resolve all nested dependencies if everyone is using directly pinned versions! I think the article might have a fundamental misunderstanding between versioning as a library author and as a user. TL;DR - Semver ranges are great when coupled with version pinning and lock files. If not, there are some trade-offs, and as long as people are aware of the trade-offs, its still better than no ranges.
- rowyourboat 9y agoSnapshot versions are great. We wish npm would have them. Only use them for packages you are developing yourself, of course, and never use them outside of development.
- lucb1e 9y agoI can only imagine the amount of new version conflicts that this would generate (I've already seen more than I'd like) if we would no longer go "1.8.0 has a feature that we depend on, so 1.8.0 and newer". Every time every dependency gets an update, every dependent package would need updating too (or at least its metadata). Hell no. Perhaps go for a range + last tested version, e.g. "1.9.7 worked" so when 2.0 comes out and breaks, you know what to use instead.
- yuchi 9y agoThat’s how semver ranges work, in fact saying `^1.3.12` means “I tested for v1.3.12 and I want all non-breaking updates that come afterwards”.
- HurrdurrHodor 9y agoWell, if you already know which updates are non-breaking, of course you have solved this problem.
- lucb1e 9y agoIndeed. If you change the output of "ls", you know everything that uses ls will break and you can bump a major version. But if you change e.g. the date format to include a timezone (visible when -l is specified), does such a change really warrant a major version bump? It's just a minor change but it might still break things. Or maybe if you refactor some stuff in curl, the curl-dev package might suddenly be incompatible with something that depended on it.
- flukus 9y agoIf specific version are used then you can end up with multiple versions of the same dependency which can cause additional problems if they're passed around. In compiled languages you end up with errors like "can't assign typeA to typeA because they are different types", when they are different versions of the same type. Not sure how node would handle the same type of error but I'm guessing it would involve subtle compatibility issues.
- yuchi 9y agoYou would have no errors except when dependencies expect to be “the only one”, in “peer dependency” style.
- molsson 9y agoIn npm all packages have their own dependencies so specifying a specific version sort of works. In Maven for example the dependency tree is flattened by the package manager so that at runtime there can only be a single version of libA. This is just how Java 8 and below works, you simply cannot import two versions of com.example.Foo at the same time. This will change in Java 9 with Jigsaw though. OSGi was a way to workaround this problem in <= Java 8. npm also has a "flattening" feature to save disk space but this is a completely different thing, an implementation detail that is never visible to the packages themselves.
- SanderMak 9y agoJust want to point out that Jigsaw/Java 9 will not support multiple versions out-of-the-box. (disclosure: I'm currently authoring Java 9 Modularity for O'Reilly)
- infinity0 9y agoYet another package management "advice" article from the point-of-view of a developer and not a maintainer. You try maintaining an ecosystem where every dependency constraint is a hard specific version. You'll spend 80% of your time fiddling with bumping versions. Fuck that shit.
- k__ 9y agoWell, someone has to do the work. But version pinning is probably the best way at the moment. This way the maintainers can use ranges and the developers can pin what they really need in the end-product. There are more developers than maintainers, so this scales much better.
- eeZah7Ux 9y ago> There are more developers than maintainers, so this scales much better. Wrong comparison. The majority of software is built, deployed, maintained and used by entirely different organizations over years. Companies doing end-to-end CD and running their software only internally are the minority. This kind of bad practices from few developers turns into countless hours of work to maintain systems years later.
- mrweasel 9y agoIt also completely ignores security issues. It's worthless to require a library or package be a specific version number, if that version is behind on security fixes. Yes, the developer might like a library, middleware, application server or whatever to be a specific version, but that ignores that in a production environment you can be behind on security fixes. It's completely reasonable to lock your code a specific major release of some dependency, assuming it's still being patched, but minor release should be something your patch management solution just applies. From an operational point of view, I think the original approach of Go, where HEAD of a dependency was just pulled from the repository of the dependency, is the right way to go. If something break, you fix it, because it needs to be done a some point anyway. I see this all the time with Java middleware. Some developer specifies that we need JBoss version X.Y.Z and not patch unless they say so. At the same time we're required to patch the operation system with the latest security patches. Well what's the point of that if the only service exposed to the Internet is the only thing not being patched?
- bmn__ 9y agoPaul Draper from TFA totally ignored the perspective of a packager. It would be a completely unacceptable situation for application A1 to depend on DB abstraction library v0.8651, A2 on v0.8653, A3 on v0.8670, A4 on v0.87… How do you ship this to your distro users? Almost always there is no easy way to install several libraries of the same name but different versions in parallel, and it's never supported by the distro auto-packaging tools. Patching to use the latest version of the dependencies and verifying the patched version works ok is a huge burden on package maintainers. The application author knows the code much better! The current solution of specifying minimum versions of dependencies exists not because we like the suffering caused by occasional breakage. No, it exists because it is a practical, working solution for the real world that is the best balanced trade-off.
- codedokode 9y agoThe problem is that nobody really tests all the versions they specify in a dependency list. They just install the dependencies, and if everything works, commit the file. In PHP we usually have a version range in libraries' dependencies and lock file that specifies exact dependency versions for the entire application. So you can be sure that you get tested combinantion of libraries.
- bmn__ 9y agoI'm pretty sure lockfiles get ignored by PHP packagers as a practical matter.
- mattacular 9y agoCan you elaborate on what you mean? Composer is the most popular PHP packager and its two core commands - update and install - make updates to and install from the composer.lock file respectively
- eeZah7Ux 9y ago> The problem is that nobody really tests all the versions they specify in a dependency list Some do. Some others target the versions shipped in a stable Linux distribution. And even if the upstream don't test some combinations of versions, distributions will test what they ship.
- codedokode 9y agoPHP's package manager (composer) has a lock file that contains exact versions of packages and can be commited to the filesystem. So we don't have problems with version ranges.
- ryanbrunner 9y agoI feel like this is describing a solved problem, and what's more, the author knows the solution. Lock files eliminate the issue of mysterious build failures and allow almost trivial creation of automated canary builds to detect problems early.
- jacques_chester 9y agoThis is right: unless you use exact versions, ideally vendored or cached in a local package server, you do not have even basic reproducibility. However. The reason people use ranges is because they don't want to handle the administrative burden of tracking upstream changes and updating their software. The OWASP Top 10 (2017 draft) shows that using known-vulnerable components is a common security weakness in software. Nobody sets out to do this, but the flipside of having access to massive troves of dependencies is that you have dozens, hundreds, perhaps thousands of dependencies you might not know about. I've worked on Cloud Foundry Buildpacks. A large amount of effort has gone into exactly this problem: tracking upstream versions, pulling new ones immediately, building them and testing them. But it's at varying levels of sophistication. For some dependencies, you get structured data. NodeJS publish an index.json file[0] which is fairly trivial to keep up with. But for other sources, which I will leave unnamed, it's necessary to do flaky, unreliable HTML parsing to detect new releases. Data about what's been found so far is kept in a ci-robots[1] repo to ensure atomicity and simplicity (I'd prefer a Concourse resource that emits versions one by one, but that's just me). What's needed is a uniform way to describe available versions of software packages. Uniform across all the major tributaries of versions: package managers, distributions, the works. If it becomes possible, as Buildpacks have after a great deal of effort, to automate fetching and testing new dependencies, then the need for ranges vanishes. You can rely on your CI/CD to run each dependency change through its paces and update the firmly fixed list for you. [0] https://nodejs.org/dist/index.json https://nodejs.org/dist/index.json [1] https://github.com/cloudfoundry/public-buildpacks-ci-robots https://github.com/cloudfoundry/public-buildpacks-ci-robots
- baq 9y agothere's one solution in the article that's valid: version lock files. these are a must have for any reasonable deployment strategy. now when you talk about actually developing the software, this is a completely different set of requirements and that one is best served with very coarse grained (but safe) version ranges. different tool for a different use case.
- ex_amazon_sde 9y agoIt's sad to see many people quoting npm as a good example while version ranging has been mastered by package managers on Linux and earlier Unix systems for decades.
- u801e 9y agoWhat I would like to see is support for existing package managers like rpm and deb that could be mirrored from these language repositories (e.g. Python, Ruby, Perl, etc). This would be more difficult for packages that have external language bindings, but a source rpm/deb package could work).