3 ms·

It's more like "golden" or "snapshot" testing. These are very common for web apps, because at the end of the day you don't care about the actual html & CSS, on

It's more like "golden" or "snapshot" testing. These are very common for web apps, because at the end of the day you don't care about the actual html & CSS, only how they are rendered. > This is more of an integration test than a unit test. That's debatable. An integration test generally tests 2 or more systems. This kind of test has 2 systems, the generator and the renderer, and we care about the output of the renderer, so it kind of looks like an integration test. However in an integration test you also have control over the implementation of both systems; a regression can be in any of the systems. But that's not true in snapshot tests: the renderer is a given. If the test fails, it's very unlikely to be due to a regression in the renderer. So in that sense, you are really only testing a single component (the generator) hence it is more like a unit test.

zoover2020 4y ago
Bravo! Excellent summary
eyelidlessness 4y ago
I was also going to mention snapshot tests. But since you beat me to it, > because at the end of the day you don't care about the actual html & CSS, only how they are rendered It really depends what you’re testing. I’ve generally been skeptical of this kind of test, similarly to OP, because “nothing changed” vs “update snapshots” feels intuitively low value to me. Despite all that, I recently added a slew of snapshots (literally >1m lines, yikes) along with a custom snapshot serializer. For this use case I do care about the HTML (and XML) because those are the project’s primary responsibilities. The custom serialization slightly relaxes the snapshot value from “nothing changed” to accept known-insignificant changes: it collapses 1+N whitespace characters, sorts attributes alphabetically because their order doesn’t matter, and trims their values because no downstream users are that pedantic about leading/trailing attribute whitespace. Everything else will be treated as an API contract violation. There are some project-specific details which might result in additional custom serialization logic and creating new snapshots, where downstream users are expected to treat certain markup values as semantically equivalent to their equivalent representation in another output property. Allllllll of that is a really long way around the barn to get to: these snapshot tests are more valuable than “exactly equal” comparison specifically because there’s a known and finite set of things that can fluctuate and a lot of caution around accepting anything into that category. And adding them at all, with known flexibilities, provides value because the underlying library has very high expectations for stability. It’s very unlikely they’ll ever be updated for any change which isn’t either additive or strategic. (And the reason they were added in the first place was to allow for much needed performance improvements and refactors to proceed with high confidence that they’re safe. Since adding them, the project’s performance monitoring charts needed to run for a period of time to crop to a whole new Y axis range, and a refactor is in review which will enable it to run in client environments without need for server deployments. None of this would have been reasonable by my team’s standards or my own without a large body of evidence that it didn’t introduce regressions. And we very well may retire it after this exercise!)
- tantalor 4y ago
  Check out "characterization test" article: https://en.wikipedia.org/wiki/Characterization_test https://en.wikipedia.org/wiki/Characterization_test
  - eyelidlessness 4y ago
    This seems particularly pertinent: > Traditional tests check individual properties (whitelists them), where characterization testing checks all properties that are not removed (blacklisted). And I’d encourage anyone who finds snapshot testing appealing but problematic to consider this approach.

zoover2020 4y ago

Bravo! Excellent summary

eyelidlessness 4y ago

I was also going to mention snapshot tests. But since you beat me to it, > because at the end of the day you don't care about the actual html & CSS, only how they are rendered It really depends what you’re testing. I’ve generally been skeptical of this kind of test, similarly to OP, because “nothing changed” vs “update snapshots” feels intuitively low value to me. Despite all that, I recently added a slew of snapshots (literally >1m lines, yikes) along with a custom snapshot serializer. For this use case I do care about the HTML (and XML) because those are the project’s primary responsibilities. The custom serialization slightly relaxes the snapshot value from “nothing changed” to accept known-insignificant changes: it collapses 1+N whitespace characters, sorts attributes alphabetically because their order doesn’t matter, and trims their values because no downstream users are that pedantic about leading/trailing attribute whitespace. Everything else will be treated as an API contract violation. There are some project-specific details which might result in additional custom serialization logic and creating new snapshots, where downstream users are expected to treat certain markup values as semantically equivalent to their equivalent representation in another output property. Allllllll of that is a really long way around the barn to get to: these snapshot tests are more valuable than “exactly equal” comparison specifically because there’s a known and finite set of things that can fluctuate and a lot of caution around accepting anything into that category. And adding them at all, with known flexibilities, provides value because the underlying library has very high expectations for stability. It’s very unlikely they’ll ever be updated for any change which isn’t either additive or strategic. (And the reason they were added in the first place was to allow for much needed performance improvements and refactors to proceed with high confidence that they’re safe. Since adding them, the project’s performance monitoring charts needed to run for a period of time to crop to a whole new Y axis range, and a refactor is in review which will enable it to run in client environments without need for server deployments. None of this would have been reasonable by my team’s standards or my own without a large body of evidence that it didn’t introduce regressions. And we very well may retire it after this exercise!)

tantalor 4y ago
Check out "characterization test" article: https://en.wikipedia.org/wiki/Characterization_test https://en.wikipedia.org/wiki/Characterization_test
- eyelidlessness 4y ago
  This seems particularly pertinent: > Traditional tests check individual properties (whitelists them), where characterization testing checks all properties that are not removed (blacklisted). And I’d encourage anyone who finds snapshot testing appealing but problematic to consider this approach.