4 ms·

I would not call it terrible, but I got a “silver bullet” vibe, which it is definitely not. 1. For a library there is API and there are implementation details.

I would not call it terrible, but I got a “silver bullet” vibe, which it is definitely not. 1. For a library there is API and there are implementation details. What if test depended on implementation detail? 2. What if tests had undisputable bug? 3. Test refactoring requires major release now? 4. Realistically test suite will have some execution paths not covered. I like the idea of running same tests over multiple versions, to observe changes. But I disagree that it would automate semver. (Maybe in very limited subset of cases) P.S. Not an actual downvoter, but if I would have downvoted, these would have been the reasons.

samatman 3y ago
> For a library there is API and there are implementation details. What if test depended on implementation detail? If the test is in the API testset, it's API. If it isn't, it's an implementation detail. > What if tests had undisputable bug? If it's in the API testset, time for a major version bump. If not, fix it. > Test refactoring requires major release now? Only if you're refactoring the API, as defined by the API testset, thereby producing a breaking change. > Realistically test suite will have some execution paths not covered. Doesn't matter. If the behavior isn't in the API testset, it's not a part of the API. > But I disagree that it would automate semver. The point isn't to automate semver, I'm not even sure what that would mean. It's to define it, in a useful and objective way.
- dxdm 3y ago
  The point of the criticism is that defining the version and what constitutes a breaking change like this will still leave people with unexpected breakage in the real world. What you've said so far has not really addressed that point straight on, which might be the reason for the comments and downvotes, I presume. I don't think your proposed scheme needs to be perfect in that regard, but acknowledging the concern and at least putting it in perspective would probably help.
  - samatman 3y ago
    I've no idea what downvotes you're referring to, I'm well into the black on that post. ¯\_(ツ)_/¯ SemVer is just a pinky-swear not to break people's code. In the real world, people's code breaks anyway, and then you get an argument about what's API, and expected behavior, and so on, and so forth. What I'm proposing is simply to replace the pinky promise with tests. From some of the other comments, I think this point may have been missed: it isn't every test in your test suite, it's the ones marked "API", only. This is a strict improvement over social-contract SemVer in two ways: one is that the package manager won't let the maintainers break the API tests without a major version bump. The other is that, if you, as a user, are unsure if some behavior is part of the stable API, you can write and submit a test to that package. If that test is accepted, great: that behavior now cannot change without a major version bump, because, again, the package manager will not bundle the package if that test breaks. Furthermore, even on a major version bump, it is instantly clear if that test is still valid, or not, you can just check before upgrading. If they don't accept the PR, you know that it isn't considered part of the API, so you add the test to your own test suite, so that at least you know quickly what broke if they change it.
    - dxdm 3y ago
      > I've no idea what downvotes you're referring to, I'm well into the black on that post. As you should be, it's a great contribution. I was referring to the downvotes mentioned in a comment further up. I agree that what you propose is an improvement, but it can be misunderstood to claim that it can prevent _any_ real-world breakage. There will always be aspects not covered by tests and which other people still rely on. I've had this experience with API contract tests between systems. Despite covering a lot of details and preventing deployments that failed these tests, we would occasionally run into problems where passing changes would break stuff in production. There was always an area of uncodified assumptions, and for a case of tens of different clients, whereas public libraries can have millions. So, I believe this is also applicable to your proposed solution. You can argue that your solution significantly shrinks this area of uncertainty while also _defining_ it, which helps when reasoning about what you can depend on - and I agree. But it does not eliminate the gap, and this is what people were pointing out. I was just a little frustrated that the discussion even went there, because I didn't think you were even claiming what they were arguing against. That was happening because I think you left a gap by not addressing it clearly, and wanted to point it out, because people seemed to be taking past each other.

samatman 3y ago

> For a library there is API and there are implementation details. What if test depended on implementation detail? If the test is in the API testset, it's API. If it isn't, it's an implementation detail. > What if tests had undisputable bug? If it's in the API testset, time for a major version bump. If not, fix it. > Test refactoring requires major release now? Only if you're refactoring the API, as defined by the API testset, thereby producing a breaking change. > Realistically test suite will have some execution paths not covered. Doesn't matter. If the behavior isn't in the API testset, it's not a part of the API. > But I disagree that it would automate semver. The point isn't to automate semver, I'm not even sure what that would mean. It's to define it, in a useful and objective way.

dxdm 3y ago
The point of the criticism is that defining the version and what constitutes a breaking change like this will still leave people with unexpected breakage in the real world. What you've said so far has not really addressed that point straight on, which might be the reason for the comments and downvotes, I presume. I don't think your proposed scheme needs to be perfect in that regard, but acknowledging the concern and at least putting it in perspective would probably help.
- samatman 3y ago
  I've no idea what downvotes you're referring to, I'm well into the black on that post. ¯\_(ツ)_/¯ SemVer is just a pinky-swear not to break people's code. In the real world, people's code breaks anyway, and then you get an argument about what's API, and expected behavior, and so on, and so forth. What I'm proposing is simply to replace the pinky promise with tests. From some of the other comments, I think this point may have been missed: it isn't every test in your test suite, it's the ones marked "API", only. This is a strict improvement over social-contract SemVer in two ways: one is that the package manager won't let the maintainers break the API tests without a major version bump. The other is that, if you, as a user, are unsure if some behavior is part of the stable API, you can write and submit a test to that package. If that test is accepted, great: that behavior now cannot change without a major version bump, because, again, the package manager will not bundle the package if that test breaks. Furthermore, even on a major version bump, it is instantly clear if that test is still valid, or not, you can just check before upgrading. If they don't accept the PR, you know that it isn't considered part of the API, so you add the test to your own test suite, so that at least you know quickly what broke if they change it.
  - dxdm 3y ago
    > I've no idea what downvotes you're referring to, I'm well into the black on that post. As you should be, it's a great contribution. I was referring to the downvotes mentioned in a comment further up. I agree that what you propose is an improvement, but it can be misunderstood to claim that it can prevent _any_ real-world breakage. There will always be aspects not covered by tests and which other people still rely on. I've had this experience with API contract tests between systems. Despite covering a lot of details and preventing deployments that failed these tests, we would occasionally run into problems where passing changes would break stuff in production. There was always an area of uncodified assumptions, and for a case of tens of different clients, whereas public libraries can have millions. So, I believe this is also applicable to your proposed solution. You can argue that your solution significantly shrinks this area of uncertainty while also _defining_ it, which helps when reasoning about what you can depend on - and I agree. But it does not eliminate the gap, and this is what people were pointing out. I was just a little frustrated that the discussion even went there, because I didn't think you were even claiming what they were arguing against. That was happening because I think you left a gap by not addressing it clearly, and wanted to point it out, because people seemed to be taking past each other.