4 ms·

I don't know why this got downvoted; it's at the very least an interesting proposal. Would love to hear a critique arguing that it's a terrible idea. Existing

I don't know why this got downvoted; it's at the very least an interesting proposal. Would love to hear a critique arguing that it's a terrible idea. Existing package managers could even implement it in a completely backwards- compatible way: if you as a package maintainer don't care for it, you simply never add "API tests".

chacham15 3y ago
Its an idealistic view which will almost certainly fail. Test suites, as much as we like to hope that they reflect real usage, mostly dont. A simple example: if function a gets changed from o(n) to o(n^2) but otherwise behaves identically, most test suites will still pass, but if a user has that function in its own inner loop you can go from o(n^2) to o(n^2^2) which can definitely break a lot of things (simple example: transaction was holding lock for too long and so the transaction was aborted). Being able to catch the above is a high bar for a test suite which I'm fairly confident most test suites are way below that.
- samatman 3y ago
  You're allowed to fix bugs introduced by new releases, you know. It's called a patch release? Is your claim that a release which introduces bad algorithmic complexity requires a major release to fix in semver? Who thinks this?
  - pgphn 3y ago
    Right. Theres a difference between, “oh no, sorry, WE should fix that in the next patch” vs, “oh no, sorry, YOU should accommodate this incompatibility with a change on your side.”
falserum 3y ago
I would not call it terrible, but I got a “silver bullet” vibe, which it is definitely not. 1. For a library there is API and there are implementation details. What if test depended on implementation detail? 2. What if tests had undisputable bug? 3. Test refactoring requires major release now? 4. Realistically test suite will have some execution paths not covered. I like the idea of running same tests over multiple versions, to observe changes. But I disagree that it would automate semver. (Maybe in very limited subset of cases) P.S. Not an actual downvoter, but if I would have downvoted, these would have been the reasons.
- samatman 3y ago
  > For a library there is API and there are implementation details. What if test depended on implementation detail? If the test is in the API testset, it's API. If it isn't, it's an implementation detail. > What if tests had undisputable bug? If it's in the API testset, time for a major version bump. If not, fix it. > Test refactoring requires major release now? Only if you're refactoring the API, as defined by the API testset, thereby producing a breaking change. > Realistically test suite will have some execution paths not covered. Doesn't matter. If the behavior isn't in the API testset, it's not a part of the API. > But I disagree that it would automate semver. The point isn't to automate semver, I'm not even sure what that would mean. It's to define it, in a useful and objective way.
  - dxdm 3y ago
    The point of the criticism is that defining the version and what constitutes a breaking change like this will still leave people with unexpected breakage in the real world. What you've said so far has not really addressed that point straight on, which might be the reason for the comments and downvotes, I presume. I don't think your proposed scheme needs to be perfect in that regard, but acknowledging the concern and at least putting it in perspective would probably help.
    - samatman 3y ago
      I've no idea what downvotes you're referring to, I'm well into the black on that post. ¯\_(ツ)_/¯ SemVer is just a pinky-swear not to break people's code. In the real world, people's code breaks anyway, and then you get an argument about what's API, and expected behavior, and so on, and so forth. What I'm proposing is simply to replace the pinky promise with tests. From some of the other comments, I think this point may have been missed: it isn't every test in your test suite, it's the ones marked "API", only. This is a strict improvement over social-contract SemVer in two ways: one is that the package manager won't let the maintainers break the API tests without a major version bump. The other is that, if you, as a user, are unsure if some behavior is part of the stable API, you can write and submit a test to that package. If that test is accepted, great: that behavior now cannot change without a major version bump, because, again, the package manager will not bundle the package if that test breaks. Furthermore, even on a major version bump, it is instantly clear if that test is still valid, or not, you can just check before upgrading. If they don't accept the PR, you know that it isn't considered part of the API, so you add the test to your own test suite, so that at least you know quickly what broke if they change it.

chacham15 3y ago

Its an idealistic view which will almost certainly fail. Test suites, as much as we like to hope that they reflect real usage, mostly dont. A simple example: if function a gets changed from o(n) to o(n^2) but otherwise behaves identically, most test suites will still pass, but if a user has that function in its own inner loop you can go from o(n^2) to o(n^2^2) which can definitely break a lot of things (simple example: transaction was holding lock for too long and so the transaction was aborted). Being able to catch the above is a high bar for a test suite which I'm fairly confident most test suites are way below that.

samatman 3y ago
You're allowed to fix bugs introduced by new releases, you know. It's called a patch release? Is your claim that a release which introduces bad algorithmic complexity requires a major release to fix in semver? Who thinks this?
- pgphn 3y ago
  Right. Theres a difference between, “oh no, sorry, WE should fix that in the next patch” vs, “oh no, sorry, YOU should accommodate this incompatibility with a change on your side.”

falserum 3y ago

I would not call it terrible, but I got a “silver bullet” vibe, which it is definitely not. 1. For a library there is API and there are implementation details. What if test depended on implementation detail? 2. What if tests had undisputable bug? 3. Test refactoring requires major release now? 4. Realistically test suite will have some execution paths not covered. I like the idea of running same tests over multiple versions, to observe changes. But I disagree that it would automate semver. (Maybe in very limited subset of cases) P.S. Not an actual downvoter, but if I would have downvoted, these would have been the reasons.

samatman 3y ago
> For a library there is API and there are implementation details. What if test depended on implementation detail? If the test is in the API testset, it's API. If it isn't, it's an implementation detail. > What if tests had undisputable bug? If it's in the API testset, time for a major version bump. If not, fix it. > Test refactoring requires major release now? Only if you're refactoring the API, as defined by the API testset, thereby producing a breaking change. > Realistically test suite will have some execution paths not covered. Doesn't matter. If the behavior isn't in the API testset, it's not a part of the API. > But I disagree that it would automate semver. The point isn't to automate semver, I'm not even sure what that would mean. It's to define it, in a useful and objective way.
- dxdm 3y ago
  The point of the criticism is that defining the version and what constitutes a breaking change like this will still leave people with unexpected breakage in the real world. What you've said so far has not really addressed that point straight on, which might be the reason for the comments and downvotes, I presume. I don't think your proposed scheme needs to be perfect in that regard, but acknowledging the concern and at least putting it in perspective would probably help.
  - samatman 3y ago
    I've no idea what downvotes you're referring to, I'm well into the black on that post. ¯\_(ツ)_/¯ SemVer is just a pinky-swear not to break people's code. In the real world, people's code breaks anyway, and then you get an argument about what's API, and expected behavior, and so on, and so forth. What I'm proposing is simply to replace the pinky promise with tests. From some of the other comments, I think this point may have been missed: it isn't every test in your test suite, it's the ones marked "API", only. This is a strict improvement over social-contract SemVer in two ways: one is that the package manager won't let the maintainers break the API tests without a major version bump. The other is that, if you, as a user, are unsure if some behavior is part of the stable API, you can write and submit a test to that package. If that test is accepted, great: that behavior now cannot change without a major version bump, because, again, the package manager will not bundle the package if that test breaks. Furthermore, even on a major version bump, it is instantly clear if that test is still valid, or not, you can just check before upgrading. If they don't accept the PR, you know that it isn't considered part of the API, so you add the test to your own test suite, so that at least you know quickly what broke if they change it.