4 ms·
Expecting library authors to get their design entirely correct the first time round isn't reasonable. When libraries make backwards-incompatible changes it's us
by simonw 5y ago
Expecting library authors to get their design entirely correct the first time round isn't reasonable. When libraries make backwards-incompatible changes it's usually because the original design was flawed in a way that made continuing to support it, while adding new features, infeasible.
You're welcome to evaluate and select libraries with the best possible track record for avoiding backwards incompatible changes - I do that myself. But, especially for libraries that are relatively early in their development process, having a hard "no breakages" policy isn't realistic.
I still think semver is the best fix for this. If you make a backwards-incompatible change, you bump the major version number and you extensively document the change.
If you're consuming libraries, the best way to address this problem is with solid test coverage. That way any time there's a new library version you can lean on your test suite to help you understand if it's going to break anything for you or not.
- wahern 5y ago> Expecting library authors to get their design entirely correct the first time round isn't reasonable. People shouldn't be publishing such libraries, nor should people be using them. With the exception of my first few FOSS projects, all of my open source libraries represent the 2nd, 3rd, or even 4th incarnation of a concept. That's just being responsible--don't knowingly litter the ecosystem with junk. I take a similar view at work. People are too quick to break out components into modules and libraries, long before they have a firm notion of what a proper interface should look like. If you can't commit to a stable API, then often times it's better to just copy whole functions and files between projects and let them evolve naturally. It takes time to see where that evolution goes, and which commonalities emerge in the implementation and surrounding application. But what about sharing bug and security fixes, you ask? Well, without a stable API you'll invariably end up with different projects stuck using different versions, anyhow. That's why people invented Docker :( What you end up with isn't an actually shared library, just the pretense of sharing, plus a bunch of unnecessary build complexity.
- simonw 5y ago> People shouldn't be publishing such libraries, nor should people be using them. That's why we have pre-1.0 version numbers (and alphas and betas) - they let us release code early for people to provide feedback before we've committed to a non-breaking API. Even with a stable 1.0 release things change. People using the library may come up with logical feature requests that the original design didn't consider, and which require backwards incompatible changes. That's when you bump to a 2.0 release. My dream project would have a version number of something like 1.237 - meaning there have been 237 new features releases since the 1.0 release without backwards compatibility breaks. My hat is off to any project that pulls that off though, it's a rare accomplishment!
- neolog 5y agoIf the new feature is incompatible with the old library, it's a different library. It should get a different name. If you're all out of names, just append a 2, as in jinja2 or psycopg2. No need to break existing software or to make it impossible to install the new library alongside the existing library.
- klodolph 5y agoThat may not be necessary, depending on how everything works. When I use a v2 of a Go library, it doesn’t prevent me from also using v1 somewhere else in the same project.
- neolog 5y agoGP is a python dev, so I'm talking python to him.
- p1necone 5y agoYou're basically suggesting reimplementing semver in a clunkier way. Just bump the major version but continue supporting the old major version with security releases etc. This is a solved problem, it's just that some library maintainers don't practice proper communication of breaking changes. If your breaking changes are a total paradigm shift in the intended usage of the library/API then sure, it should be a new library. But quite often those breaking changes are just things like "field Foo is now required when performing process Bar" in order to prevent some sort of invalid state from being reached. In those cases communicating the breaking changes via versioning is the way to go imo.
- p1necone 5y agoGetting a design right the first time is hard, getting a design right without any users is really hard. Also look into how many massive libraries basically started as someones weekend project for fun. I posit that the quality of the open source ecosystem would go down, not up if we tried to somehow enforce these kinds of standards on public projects.
- edejong 5y agoIs Linux unreasonable? The kernel has been backwards-compatible to user-land from the start. Or TeX, C, RS232?
- rot13xor 5y agoSame with x86. Even modern x86 CPUs boots in real mode so you can run a baremetal 8086 binary from the 1980s.
- zmmmmm 5y ago> The kernel has been backwards-compatible to user-land from the start Maybe at the source level ... at the binary driver level there is no stability at all, and it's a huge reason why the driver scene for linux still languishes in so many areas. Now there are reasons linux has chosen this model but it nonetheless demonstrates this is not a "free" decision without any consequences.
- jnxx 5y ago> at the binary driver level there is no stability at all You are evading the argument, what we were talking about was stability of APIs in respect to the userland code. Linux never promised binary compatibility at driver level. Also, Linux has far better stability in hardware support than competing systems. If a relative tells me they have a photographic slide scanner or printer that stopped to work with some Windows x.y release, chances are quite good that it is continued to be supported by Linux.
- simonw 5y agoI didn't say it made the projects themselves unreasonable, I said that projecting those expectations on other projects is unreasonable. Those are all inspiring examples of projects that went truly above and beyond the norm and achieved great success as a result.
- lmm 5y agoI don't know about RS232, but Linux, TeX and C all ended up really inelegant and hard to work with. Maybe the backward compatibility is worth it in your case; personally I'm far happier using something that has been ruthlessly refactored to keep everything clean even if it means I have to update things on my side sometimes.
- jnxx 5y ago> When libraries make backwards-incompatible changes it's usually because the original design was flawed in a way that made continuing to support it, while adding new features, infeasible. I think that "infeasible" is rarely true. Let's take, as a well-known example, Python3. Well, the python developers wanted to change the default string representation, the fact that "print" was a statement, and the behavior of integer division, but this absolutely could have been done by things such as a new syntax for Unicode strings (such as, huh, u"hello"), making integer division dependent on "import from __future__"statements, using a *.py3 file extension for breaking new syntax, and so on. It would have been more work for developers, true, and this was ultimately their decision, but it was not technically required. What developers of library and infrastructure need to realize is that very soon there is far more code which depends on established APIs and that breaking this code is, on absolute terms, more work, and is disruptive in collective terms. It is like a tree where most of the biomass is in the leaves, not in the trunk. For example, there is a Wiki software named MoinMoin Wiki, written in Python, and supporting multiple markup formats, which is written in Python. It is popular and also used for the Python wiki, so you could be assured it is used in infrastruccture. But there is no Python 3 port because the developers did not had time for that, and in this respect the breaking changes of Python 3 are disruptive. There are also many scientific code bases written in Python 2 which will never be ported to Python 3, because there is simply no funding and no scientific career for such work. And this becomes very visible in larger projects which depend on a lot of libraries which in turn have transitive dependencies on multiple levels. Just look at the full dependency graph of something like Gazebo, Tensorflow or Gimp. The former might well have several hundred dependencies As soon as one of there libraries demands that a newer version of the dependency is used, and this dependency introduces breaking change, it is likely that another dependency in the same project is broken by it. And telling to users "to just fix and upgrade" is not going to work, it does not scale, and people will at some point stop to upgrade your stuff altogether. Imagine the Linux kernel worked this way! No sane person would use it.