7 ms·

The semver trick (2019)

109 points by jcbhmr 2y ago

hinkley 2y ago
> The cause of the difficulty was the large number of crates using types from these libraries in their public API. This has bitten me on the ass so many times. Exposing third party types means you have to upgrade the library simultaneously across the entire application, even if the language allows you to load different versions of the same library in different parts of the app. And if you’re trying to skirt Conway, having multiple teams sharing a single deployable, you’re gonna have a bad time. Because good luck getting all teams to agree to drop everything and upgrade next sprint. I’d much rather my coworkers use libraries instead of NIH, but you have to be very careful how you use them. It gives me a little sympathy for microservices, but that’s often just swapping in a devil you don’t know.
- sethammons 2y ago
  > And if you’re trying to skirt Conway, having multiple teams sharing a single deployable, you’re gonna have a bad time. Because good luck getting all teams to agree to drop everything and upgrade next sprint. this is why I'm not on team monorepo. Let other teams vendor their dependencies and upgrade on their own time. Just don't go trying to release N versions. Measure who is on an old version and work with them to update. But again, this slow team allows the other N-1 teams to adopt the new version and provide value to the org. for those of you who are pro monorepo and have worked in one with dozens of teams in different orgs who have competing priorities and can't upgrade a dependency in lock-step, how have you solved this?
  - frizlab 2y ago
    Wouldn’t subtrees fix this instead of a mono-repo? Or even (gasp) submodules?
    - dallasg3 2y ago
      I was looking into Paket for this on .NET. https://github.com/fsprojects/Paket https://github.com/fsprojects/Paket
    - osigurdson 2y ago
      You mentioned sub modules - you are now on call until someone else mentions them again.
      - frizlab 2y ago
        I’m fine with that :)
  - mgaunard 2y ago
    1. ensure you have continuous integration against both stable and latest versions of everything, with the stable set of all transitive dependencies defined on a per-project basis, and what gets used for releases. 2. people are expected to fix the latest build whenever they have spare cycles. The incentive is that it is part of the pipeline for code reviews and can only be bypassed by the project owner. This means you'll need to support both stable and latest concurrently (bumping stable is also an option to simplify the requirement). 3. if you added conditional support for old stable/new latest and latest was eventually made stable, you can refactor that code.
  - osigurdson 2y ago
    Does monorepo confer that all projects use the same dependencies? Binary dependencies seem orthogonal to how source code is managed.
    - adastra22 2y ago
      Usually, yes. Monorepo means all dependencies are checked into the same repo. Even external dependencies (when source available) are checked in.
      - osigurdson 2y ago
        Usually we are using package managers and just referencing packages. However, even in the hypothetical where I am downloading binaries and checking them in, I still don't see the relationship to the monorepo. I can have a monorepo with 100 folders representing 100 programs, each of these can have completely separate dependencies, languages, etc. Dependency versions matter when they are all part of the same program or library. Then you have the diamond dependency issue and related problems. However, none of this is related maintaining code in a monorepo. One program can be maintain in 10 different repos (impractical but possible) and still have the diamond dependency issue.
    - hinkley 2y ago
      I’ve done monorepo, separate compilation units before. It has its own set of issues but it cuts down on circular dependencies and makes more sense than having sixty repositories checked out. Those are hell.
- mgaunard 2y ago
  Using third-party libraries is usually a bad idea, and must be done so very carefully. If you do, onboard it onto your own build system, and pin it to a version you're going to provide support for and ensure compatibility with. You should never have multiple versions of the same code in objects that end up linked together, that's a recipe for ODR.
  - progval 2y ago
    Concrete example: the Apache Arrow format is designed to allow passing arrays across languages and libraries without conversion; so many public functions/methods (and FFIs) will have Arrow arrays in their signature. However, the Rust Arrow library bumps its major version every few months because it changes methods and auxiliary types, without changing the memory layout of arrays. But this means you can't have two dependencies (eg. an ORC parser and a CSV serializer) that depend on different versions of Arrow if you want to pass arrays from one dependency's functions to the other's. And vendoring like you mention won't help, because the Rust compiler wouldn't recognize both vendored Arrow crates as compatible.
    - mgaunard 2y ago
      Use a single version of the arrow library, and build everything against that one. I don't see where the problem is. I don't use Rust or third-party build or packaging systems -- I usually recommend that people don't do so either, but I understand that cargo is part of the appeal of Rust. I'd say just build whatever you need to do, this way you won't be limited by arbitrary restrictions others have decided.
      - hinkley 2y ago
        Security patches say no, doesn’t work that way.
      - progval 2y ago
        I can't always do that because dependency A depends dependency B that depends on arrow 52, and dependency C depends on arrow 53. And sure, I can soft-fork dependency B to add support for arrow 53, and dependency A to depend on my soft-fork of dependency B, but it quickly becomes messy. Especially when in parallel I send unrelated patches to A and/or B.
- do_not_redeem 2y ago
  Microservices only solve this because you have to serialize objects on the wire, no? You could accomplish the same within an app or lib by accepting a String and calling `serde_json::from_str` on it, avoiding the problems of microservices. Or more realistically, define your own struct and convert it to/from the dependency type at the interface boundary.
  - hinkley 2y ago
    Process boundaries tend to increase friction to exposing internal implementation details like this. They don’t fix the problem but they do discourage it.
    - deleted 2y ago
      [deleted]
- alexchamberlain 2y ago
  If you have a single deployable(1), I think you need to have a person/team responsible for the whole. They need to be trusted to work on any of the code to make the whole build, without overstepping into micromanaging the whole code base. It's a tough balance. (1) it has to be said I don't have much experience with mono-builds, but having brought together many services ( micro or not) from different teams and backgrounds, I wish for a little more coordination and governance.
tantalor 2y ago
Not knowing rust, this is pretty meaningless to me. Is this trick relevant outside of rust?
- Xylakant 2y ago
  Theoretically, it can be. However, it’s only relevant in specific circumstances that Rust happens to provide, for example one of the requirements is that the package manager / build system allows you to have two different versions of the same dependencies in a single binary - which is comparatively rare.
- jerf 2y ago
  I've used something very similar in Go. I can't point to it because it's an internal repository, but it allowed for relatively smooth co-existence between a v1 and a v2 of a particular package that had one aspect that was hard-to-upgrade, but also, generally rarely used compared to the rest. The rest of the types were fully identical because v1 simply re-exported the v2 types, so Go only complains if you try to cross the streams with the specifically-changed but rarely-used types, which is exactly the complaint I want. In the case of Go I'm not sure if it's important that v1 specifically re-export the v2; I think it would work equally well either way. But it's at least very similar.
- adastra22 2y ago
  Rust has a nice semver-based package management tool called cargo. One thing rust/cargo does really well is packaging up different versions of the same dependency for the same build. E.g. your project can update to widget=2.0 while some of your downstream dependencies are still stuck on the 1.x release branch. Cargo's semver support means that those dependencies can be automatically upgrade to support newer 1.x releases, but they will not work with 2.x without the package maintainer's intervention. In the mean time it just compiles and links to both, while keeping them isolated. Usually this is great. You avoid dependency hell while only trading off some compiled binary size. Sometimes it fails when a package must have only one version, e.g. because you're passing around data types from that package to/from your dependencies which use the old version, but part of rust/cargo's trick is that types drawn from different versions of the same package are distinct types. So you get a compile-time type check error. Solution: the widget package maintainer pushes out a final 1.x release which itself depends on widget=2, and is simply a shim that re-exports the API. This semver-violating trickery is basically a stealth upgrade that forces all upstream widget=1 pegged crates onto the new widget=2 release branch. This trick is generic and relevant outside of rust, at least so far as other package managers user semver-aware dependency management.
jwilk 2y ago
Discussed in 2020: https://news.ycombinator.com/item?id=24020254 https://news.ycombinator.com/item?id=24020254 (43 comments)
ghssds 2y ago
>To the extent that it constitutes copyrightable work, the idea of depending on a future version of the same library is licensed under the CC0 1.0 Universal license (LICENSE-CC0) and may be used without attribution. An idea isn't copyrightable work; only the text explaining it is. Please don't propagate the idea that ideas are copyrightable.
- deleted 2y ago
  [deleted]
- conartist6 2y ago
  Great point! Most people also aren't aware of the principle of convergence, which is why something like a specific semver pattern could not be the subject of copyright. This is the same principle that means that mathematical equations or basic API usage examples are not and cannot be the subject of copyright.
- ironhaven 2y ago
  David tolnay should have filed a defensive software patent to be more precise then!
  - conartist6 2y ago
    It isn't clear to me that there is any such thing as a "defensive patent." I've looked into trying to get such a thing for my work but my understanding is that it would offer me exactly 0 protection for being sued over what I have made.
nektro 2y ago
this problem also doesn't happen when you build with HEAD and manage it with lockfiles
- adastra22 2y ago
  This problem can arise in your dependencies regardless of your use of lockfiles.
cortesoft 2y ago
Am I misunderstanding this, or will it break dependencies that actually rely on the “rarely used api”? Seems to me like this trick is just to get the compiler to ignore semver?
- aldonius 2y ago
  I see it not so much as getting the compiler to ignore semver but rather giving it more granular information about the nature of the changes. As I understand it: - Anything which is breaking-changed in any way for 0.3.0 (e.g. in the article a type change from i32 to u32) is the exact same code in 0.2.1 as it was in 0.2.0. - Anything which stays the same for 0.3.0 is re-exported in 0.2.1 (from 0.3.0). The code is replaced with a `pub use ...;` - Anything which moves around in the submodule hierarchy (but is otherwise unchanged) in 0.3.0 is re-exported in 0.2.1, from where it used to be. If crate Baz depends on something in Foo 0.2.x which changes in Foo 0.3.0, then when (if!) Baz updates to Foo 0.3.0 it will need to deal with that. But Baz can upgrade to Foo 0.2.1 without concern, and any types unchanged in Foo 0.3.0 will be interoperable.
- felurx 2y ago
  You only re-export the unchanged parts. EVFILT_AIO is the same in 0.2.0 and 0.2.1 (which is different - in practice and to the compiler - to EVFILT_AIO in 0.3). (Only) c_void is the same in 0.2.0, 0.2.1 and 0.3 (again, in practice and to the compiler.)
  - cpeterso 2y ago
    And the user can postpone making their code work with u32 EVFILT_AIO until they update from v0.2.* to v0.3.
wpollock 2y ago
Isn't this issue caused by having multiple unrelated APIs in a single package? If they were in different crates, a breaking change on one would not affect users of the others.

109 points by jcbhmr 2y ago

> The cause of the difficulty was the large number of crates using types from these libraries in their public API. This has bitten me on the ass so many times. Exposing third party types means you have to upgrade the library simultaneously across the entire application, even if the language allows you to load different versions of the same library in different parts of the app. And if you’re trying to skirt Conway, having multiple teams sharing a single deployable, you’re gonna have a bad time. Because good luck getting all teams to agree to drop everything and upgrade next sprint. I’d much rather my coworkers use libraries instead of NIH, but you have to be very careful how you use them. It gives me a little sympathy for microservices, but that’s often just swapping in a devil you don’t know.

sethammons 2y ago
> And if you’re trying to skirt Conway, having multiple teams sharing a single deployable, you’re gonna have a bad time. Because good luck getting all teams to agree to drop everything and upgrade next sprint. this is why I'm not on team monorepo. Let other teams vendor their dependencies and upgrade on their own time. Just don't go trying to release N versions. Measure who is on an old version and work with them to update. But again, this slow team allows the other N-1 teams to adopt the new version and provide value to the org. for those of you who are pro monorepo and have worked in one with dozens of teams in different orgs who have competing priorities and can't upgrade a dependency in lock-step, how have you solved this?
- frizlab 2y ago
  Wouldn’t subtrees fix this instead of a mono-repo? Or even (gasp) submodules?
  - dallasg3 2y ago
    I was looking into Paket for this on .NET. https://github.com/fsprojects/Paket https://github.com/fsprojects/Paket
  - osigurdson 2y ago
    You mentioned sub modules - you are now on call until someone else mentions them again.
    - frizlab 2y ago
      I’m fine with that :)
- mgaunard 2y ago
  1. ensure you have continuous integration against both stable and latest versions of everything, with the stable set of all transitive dependencies defined on a per-project basis, and what gets used for releases. 2. people are expected to fix the latest build whenever they have spare cycles. The incentive is that it is part of the pipeline for code reviews and can only be bypassed by the project owner. This means you'll need to support both stable and latest concurrently (bumping stable is also an option to simplify the requirement). 3. if you added conditional support for old stable/new latest and latest was eventually made stable, you can refactor that code.
- osigurdson 2y ago
  Does monorepo confer that all projects use the same dependencies? Binary dependencies seem orthogonal to how source code is managed.
  - adastra22 2y ago
    Usually, yes. Monorepo means all dependencies are checked into the same repo. Even external dependencies (when source available) are checked in.
    - osigurdson 2y ago
      Usually we are using package managers and just referencing packages. However, even in the hypothetical where I am downloading binaries and checking them in, I still don't see the relationship to the monorepo. I can have a monorepo with 100 folders representing 100 programs, each of these can have completely separate dependencies, languages, etc. Dependency versions matter when they are all part of the same program or library. Then you have the diamond dependency issue and related problems. However, none of this is related maintaining code in a monorepo. One program can be maintain in 10 different repos (impractical but possible) and still have the diamond dependency issue.
  - hinkley 2y ago
    I’ve done monorepo, separate compilation units before. It has its own set of issues but it cuts down on circular dependencies and makes more sense than having sixty repositories checked out. Those are hell.
mgaunard 2y ago
Using third-party libraries is usually a bad idea, and must be done so very carefully. If you do, onboard it onto your own build system, and pin it to a version you're going to provide support for and ensure compatibility with. You should never have multiple versions of the same code in objects that end up linked together, that's a recipe for ODR.
- progval 2y ago
  Concrete example: the Apache Arrow format is designed to allow passing arrays across languages and libraries without conversion; so many public functions/methods (and FFIs) will have Arrow arrays in their signature. However, the Rust Arrow library bumps its major version every few months because it changes methods and auxiliary types, without changing the memory layout of arrays. But this means you can't have two dependencies (eg. an ORC parser and a CSV serializer) that depend on different versions of Arrow if you want to pass arrays from one dependency's functions to the other's. And vendoring like you mention won't help, because the Rust compiler wouldn't recognize both vendored Arrow crates as compatible.
  - mgaunard 2y ago
    Use a single version of the arrow library, and build everything against that one. I don't see where the problem is. I don't use Rust or third-party build or packaging systems -- I usually recommend that people don't do so either, but I understand that cargo is part of the appeal of Rust. I'd say just build whatever you need to do, this way you won't be limited by arbitrary restrictions others have decided.
    - hinkley 2y ago
      Security patches say no, doesn’t work that way.
    - progval 2y ago
      I can't always do that because dependency A depends dependency B that depends on arrow 52, and dependency C depends on arrow 53. And sure, I can soft-fork dependency B to add support for arrow 53, and dependency A to depend on my soft-fork of dependency B, but it quickly becomes messy. Especially when in parallel I send unrelated patches to A and/or B.
do_not_redeem 2y ago
Microservices only solve this because you have to serialize objects on the wire, no? You could accomplish the same within an app or lib by accepting a String and calling `serde_json::from_str` on it, avoiding the problems of microservices. Or more realistically, define your own struct and convert it to/from the dependency type at the interface boundary.
- hinkley 2y ago
  Process boundaries tend to increase friction to exposing internal implementation details like this. They don’t fix the problem but they do discourage it.
  - deleted 2y ago
    [deleted]
alexchamberlain 2y ago
If you have a single deployable(1), I think you need to have a person/team responsible for the whole. They need to be trusted to work on any of the code to make the whole build, without overstepping into micromanaging the whole code base. It's a tough balance. (1) it has to be said I don't have much experience with mono-builds, but having brought together many services ( micro or not) from different teams and backgrounds, I wish for a little more coordination and governance.

tantalor 2y ago

Not knowing rust, this is pretty meaningless to me. Is this trick relevant outside of rust?

Xylakant 2y ago
Theoretically, it can be. However, it’s only relevant in specific circumstances that Rust happens to provide, for example one of the requirements is that the package manager / build system allows you to have two different versions of the same dependencies in a single binary - which is comparatively rare.
jerf 2y ago
I've used something very similar in Go. I can't point to it because it's an internal repository, but it allowed for relatively smooth co-existence between a v1 and a v2 of a particular package that had one aspect that was hard-to-upgrade, but also, generally rarely used compared to the rest. The rest of the types were fully identical because v1 simply re-exported the v2 types, so Go only complains if you try to cross the streams with the specifically-changed but rarely-used types, which is exactly the complaint I want. In the case of Go I'm not sure if it's important that v1 specifically re-export the v2; I think it would work equally well either way. But it's at least very similar.
adastra22 2y ago
Rust has a nice semver-based package management tool called cargo. One thing rust/cargo does really well is packaging up different versions of the same dependency for the same build. E.g. your project can update to widget=2.0 while some of your downstream dependencies are still stuck on the 1.x release branch. Cargo's semver support means that those dependencies can be automatically upgrade to support newer 1.x releases, but they will not work with 2.x without the package maintainer's intervention. In the mean time it just compiles and links to both, while keeping them isolated. Usually this is great. You avoid dependency hell while only trading off some compiled binary size. Sometimes it fails when a package must have only one version, e.g. because you're passing around data types from that package to/from your dependencies which use the old version, but part of rust/cargo's trick is that types drawn from different versions of the same package are distinct types. So you get a compile-time type check error. Solution: the widget package maintainer pushes out a final 1.x release which itself depends on widget=2, and is simply a shim that re-exports the API. This semver-violating trickery is basically a stealth upgrade that forces all upstream widget=1 pegged crates onto the new widget=2 release branch. This trick is generic and relevant outside of rust, at least so far as other package managers user semver-aware dependency management.

jwilk 2y ago

Discussed in 2020: https://news.ycombinator.com/item?id=24020254 https://news.ycombinator.com/item?id=24020254 (43 comments)

ghssds 2y ago

>To the extent that it constitutes copyrightable work, the idea of depending on a future version of the same library is licensed under the CC0 1.0 Universal license (LICENSE-CC0) and may be used without attribution. An idea isn't copyrightable work; only the text explaining it is. Please don't propagate the idea that ideas are copyrightable.

deleted 2y ago
[deleted]
conartist6 2y ago
Great point! Most people also aren't aware of the principle of convergence, which is why something like a specific semver pattern could not be the subject of copyright. This is the same principle that means that mathematical equations or basic API usage examples are not and cannot be the subject of copyright.
ironhaven 2y ago
David tolnay should have filed a defensive software patent to be more precise then!
- conartist6 2y ago
  It isn't clear to me that there is any such thing as a "defensive patent." I've looked into trying to get such a thing for my work but my understanding is that it would offer me exactly 0 protection for being sued over what I have made.

nektro 2y ago

this problem also doesn't happen when you build with HEAD and manage it with lockfiles

adastra22 2y ago
This problem can arise in your dependencies regardless of your use of lockfiles.

cortesoft 2y ago

Am I misunderstanding this, or will it break dependencies that actually rely on the “rarely used api”? Seems to me like this trick is just to get the compiler to ignore semver?

aldonius 2y ago
I see it not so much as getting the compiler to ignore semver but rather giving it more granular information about the nature of the changes. As I understand it: - Anything which is breaking-changed in any way for 0.3.0 (e.g. in the article a type change from i32 to u32) is the exact same code in 0.2.1 as it was in 0.2.0. - Anything which stays the same for 0.3.0 is re-exported in 0.2.1 (from 0.3.0). The code is replaced with a `pub use ...;` - Anything which moves around in the submodule hierarchy (but is otherwise unchanged) in 0.3.0 is re-exported in 0.2.1, from where it used to be. If crate Baz depends on something in Foo 0.2.x which changes in Foo 0.3.0, then when (if!) Baz updates to Foo 0.3.0 it will need to deal with that. But Baz can upgrade to Foo 0.2.1 without concern, and any types unchanged in Foo 0.3.0 will be interoperable.
felurx 2y ago
You only re-export the unchanged parts. EVFILT_AIO is the same in 0.2.0 and 0.2.1 (which is different - in practice and to the compiler - to EVFILT_AIO in 0.3). (Only) c_void is the same in 0.2.0, 0.2.1 and 0.3 (again, in practice and to the compiler.)
- cpeterso 2y ago
  And the user can postpone making their code work with u32 EVFILT_AIO until they update from v0.2.* to v0.3.

wpollock 2y ago

Isn't this issue caused by having multiple unrelated APIs in a single package? If they were in different crates, a breaking change on one would not affect users of the others.