4 ms·

If your goal is "I have a blackbox of code, I must make a change, and I will assume existing behaviors are correct", this approach should be reasonable to ensur

If your goal is "I have a blackbox of code, I must make a change, and I will assume existing behaviors are correct", this approach should be reasonable to ensure that the code doesn't break after your change. You don't have an external spec to test against in this case, the existing implementation details are the spec, and so the goal is to preserve them. I'd say going from: * Find one input * Get coverage * Mutation test to ensure you have assertion density is probably a fine way to accomplish the goal. Eventually you'd want to define an external specification for the code that isn't just "the service works, therefor ensure the service does what it's been doing", at which point some of these tests may be removed, or at least enhanced.

ricardobeat 7y ago
> assume existing behaviors are correct That's part of the problem. The original test was created based on implemented behaviour and not test/functional requirements, so you don't have any confidence that what you're preserving is the right behaviour. It tells you you haven't changed the output, but once you've covered your codebase with these, and make a commit that triggers 46 snapshot failures, you have zero knowledge to decide whether that change is desirable or not.
- wool_gather 7y ago
  In the absence of a specification, documentation, or access to the people who originally implemented the program, the program itself is the spec. With no way to decide what the program "should" be doing, whatever it's doing right now is the right behavior.
- staticassertion 7y ago
  The point is that the existing behavior is de-facto correct by virtue of being the existing behavior. The goal here is not "Prove this software meets a spec" it is "This software is the spec, and we must not regress". There are many more systems that act as a defacto spec than that are built to an external one. If you choose to then move to a world where you want an external spec, as I said, at that point some of these tests will be thrown away or refactored. Most systems never make it to that stage. This is especially common when you have legacy software where the person or team that wrote the code is no longer there. It takes a monstrous amount of effort to derive a meaningful spec, and step 0 is always preserving existing behavior.

ricardobeat 7y ago

> assume existing behaviors are correct That's part of the problem. The original test was created based on implemented behaviour and not test/functional requirements, so you don't have any confidence that what you're preserving is the right behaviour. It tells you you haven't changed the output, but once you've covered your codebase with these, and make a commit that triggers 46 snapshot failures, you have zero knowledge to decide whether that change is desirable or not.

wool_gather 7y ago
In the absence of a specification, documentation, or access to the people who originally implemented the program, the program itself is the spec. With no way to decide what the program "should" be doing, whatever it's doing right now is the right behavior.
staticassertion 7y ago
The point is that the existing behavior is de-facto correct by virtue of being the existing behavior. The goal here is not "Prove this software meets a spec" it is "This software is the spec, and we must not regress". There are many more systems that act as a defacto spec than that are built to an external one. If you choose to then move to a world where you want an external spec, as I said, at that point some of these tests will be thrown away or refactored. Most systems never make it to that stage. This is especially common when you have legacy software where the person or team that wrote the code is no longer there. It takes a monstrous amount of effort to derive a meaningful spec, and step 0 is always preserving existing behavior.