3 ms·

> The spec The tests are your spec. They exist precisely to document what the program is supposed to do for other humans, with the secondary benefit of also te

by 9rx 1y ago

> The spec The tests are your spec. They exist precisely to document what the program is supposed to do for other humans, with the secondary benefit of also telling a machine what the program is supposed to do, allowing implementations to automatically validate themselves against the spec. If you find yourself writing specs and tests as independent things, that's how you end up with bad, brittle tests that make development a nightmare — or you simply like pointless busywork, I suppose. But, yes, you may still have to consult a human if there is reason to believe the spec isn't accurate.

munificent 1y ago
Unfortunately, tests can never be a complete specification unless the system is simple enough to have a finite set of possible inputs. For all real-world software, a test suite tests a number of points in the space of possible inputs and we hope that those points generalize to pinning down the overall behavior of the implementation. But there's no guarantee of that generalization. An implementation that fails a test is guaranteed to not implement the spec, but an implementation that passes all of the tests is not guaranteed to implement it.
- 9rx 1y ago
  > Unfortunately, tests can never be a complete specification They are for the human, which is the intended recipient. Given infinite time the machine would also be able to validate against the complete specification, but, of course, we normally cut things short because we want to release the software in a reasonable amount of time. But, as before, that this ability exists at all is merely a secondary benefit.
godelski 1y ago
> The tests are your spec. That's not quite right, but it's almost right. Tests are an *approximation* of your spec. Tests are a description, and like all descriptions are noisy. The thing is it is very very difficult to know if your tests have complete coverage. It's very hard to know if your description is correct. How often do you figure out something you didn't realize previously? How often do you not realize something and it's instead pointed out by your peers? How often do you realize something after your peers say something that sparks an idea? Do you think that those events are over? No more things to be found? I know I'm not that smart because if I was I would have gotten it all right from the get go. There are, of course, formal proofs but even they aren't invulnerable to these issues. And these aren't commonly used in practice and at that point we're back to programming/math, so I'm not sure we should go down that route.
- 9rx 1y ago
  > Tests are a description As is a spec. "Description" is literally found in the dictionary definition. Which stands to reason as tests are merely a way to write a spec. They are the same thing. > The thing is it is very very difficult to know if your tests have complete coverage. There is no way to avoid that, though. Like you point out, not even formal proofs, the closest speccing methodology we know of to try and avoid this, is immune. > Tests are an approximation of your spec. Specs are an approximation of what you actually want, sure, but that does not change that tests are the spec. There are other ways to write a spec, of course, but if you went down that road you wouldn't also have tests. That would be not only pointless, but a nightmare due to not having a single source of truth which causes all kinds of social (and sometimes technical) problems.
  - godelski 1y ago
    > that does not change that tests are the spec. I disagree. It's, like you say, one description of your spec but that's not the spec. > not having a single source of truth Well that's the thing, there is no single source of truth. A single source of truth is for religion, not code. The point of saying this is to ensure you don't fall prey to fooling yourself. You're the easiest person for you to fool, after all. You should always carry some doubt. Not so much it is debilitating, but enough to keep you from being too arrogant. You need to constantly check that your documentation is aligned to your specs and that your specs are aligned to your goals. If you cannot see how these are different things then it's impossible to check your alignment and you've fooled yourself.
    - 9rx 1y ago
      > You need to constantly check that your documentation is aligned to your specs Documentation, tests, and specs are all ultimately different words for the same thing. You do have to check that your implementation and documentation/spec/tests are aligned, which can be a lot of work if you do so by hand, but that's why we invented automatic methods. Formal verification is theoretically best (that we know of) at this, but a huge pain in the ass for humans to write, so that is why virtually everyone has adopted tests instead. It is a reasonable tradeoff between comfort in writing documentation while still providing sufficient automatic guarantees that the documentation is true. > If you cannot see how these are different things If you see them as different things, you are either pointlessly repeating yourself over and over or inventing information that is, at best, worthless (but often actively harmful).

by 9rx 1y ago

munificent 1y ago

Unfortunately, tests can never be a complete specification unless the system is simple enough to have a finite set of possible inputs. For all real-world software, a test suite tests a number of points in the space of possible inputs and we hope that those points generalize to pinning down the overall behavior of the implementation. But there's no guarantee of that generalization. An implementation that fails a test is guaranteed to not implement the spec, but an implementation that passes all of the tests is not guaranteed to implement it.

9rx 1y ago
> Unfortunately, tests can never be a complete specification They are for the human, which is the intended recipient. Given infinite time the machine would also be able to validate against the complete specification, but, of course, we normally cut things short because we want to release the software in a reasonable amount of time. But, as before, that this ability exists at all is merely a secondary benefit.

godelski 1y ago

> The tests are your spec. That's not quite right, but it's almost right. Tests are an *approximation* of your spec. Tests are a description, and like all descriptions are noisy. The thing is it is very very difficult to know if your tests have complete coverage. It's very hard to know if your description is correct. How often do you figure out something you didn't realize previously? How often do you not realize something and it's instead pointed out by your peers? How often do you realize something after your peers say something that sparks an idea? Do you think that those events are over? No more things to be found? I know I'm not that smart because if I was I would have gotten it all right from the get go. There are, of course, formal proofs but even they aren't invulnerable to these issues. And these aren't commonly used in practice and at that point we're back to programming/math, so I'm not sure we should go down that route.

9rx 1y ago
> Tests are a description As is a spec. "Description" is literally found in the dictionary definition. Which stands to reason as tests are merely a way to write a spec. They are the same thing. > The thing is it is very very difficult to know if your tests have complete coverage. There is no way to avoid that, though. Like you point out, not even formal proofs, the closest speccing methodology we know of to try and avoid this, is immune. > Tests are an approximation of your spec. Specs are an approximation of what you actually want, sure, but that does not change that tests are the spec. There are other ways to write a spec, of course, but if you went down that road you wouldn't also have tests. That would be not only pointless, but a nightmare due to not having a single source of truth which causes all kinds of social (and sometimes technical) problems.
- godelski 1y ago
  > that does not change that tests are the spec. I disagree. It's, like you say, one description of your spec but that's not the spec. > not having a single source of truth Well that's the thing, there is no single source of truth. A single source of truth is for religion, not code. The point of saying this is to ensure you don't fall prey to fooling yourself. You're the easiest person for you to fool, after all. You should always carry some doubt. Not so much it is debilitating, but enough to keep you from being too arrogant. You need to constantly check that your documentation is aligned to your specs and that your specs are aligned to your goals. If you cannot see how these are different things then it's impossible to check your alignment and you've fooled yourself.
  - 9rx 1y ago
    > You need to constantly check that your documentation is aligned to your specs Documentation, tests, and specs are all ultimately different words for the same thing. You do have to check that your implementation and documentation/spec/tests are aligned, which can be a lot of work if you do so by hand, but that's why we invented automatic methods. Formal verification is theoretically best (that we know of) at this, but a huge pain in the ass for humans to write, so that is why virtually everyone has adopted tests instead. It is a reasonable tradeoff between comfort in writing documentation while still providing sufficient automatic guarantees that the documentation is true. > If you cannot see how these are different things If you see them as different things, you are either pointlessly repeating yourself over and over or inventing information that is, at best, worthless (but often actively harmful).