4 ms·

It's often shorthand for "this cant be unit tested" or "this isnt dependency injected" even though integration tests are perfectly capable of testing non-DI cod

It's often shorthand for "this cant be unit tested" or "this isnt dependency injected" even though integration tests are perfectly capable of testing non-DI code. The author's claims that we should isolate code under test better and rely more on snapshot testing are spot on.

diggan 1y ago
> rely more on snapshot testing are spot on Never quite liked "snapshot testing" which I think has a better name under "golden master testing" or similar anyways. Reason for the dislike, is that it's basically a codified "Trust me bro, it's correct" without actually making clear what you are asserting with that test. I haven't found any team that used snapshot testing and didn't also need to change the snapshots for every little change, which obviously defeats the purpose. The only things snapshot testing seems to be good for, is when you've written something and you know it'll never change again, for any reason. Beyond that, unit tests and functional/integration tests are much easier to structure in a way so you don't waste so much time reviewing changes.
- MoreQARespect 1y ago
  >I haven't found any team that used snapshot testing and didn't also need to change the snapshots for every little change, which obviously defeats the purpose I dont see how this even defeats the point, let alone obviously. If a UI changes I appreciate being notified. If a REST API response changes I like to see a diff. If somebody changes some CSS and it changes 50 snapshots, it isnt a huge burden to approve them all and sometimes it highlights a bug.
  - parthdesai 1y ago
    In the talk, he also talks about passing a flag which would actually update the snapshots/golden files if needed
  - necovek 1y ago
    Are you really going to be reviewing all those 50 snapshots carefully? The bound of testing on the "other side" is to test just enough not to increase the maintenance burden too much.
  - diggan 1y ago
    > I dont see how this even defeats the point, let alone obviously. You generally don't want to have to change all the tests for every change, particularly implementation details. Usually when people do snapshot testing of for example UI components, they serialize the entire component then assert the full component is the same as the snapshot, so any change requires the snapshot to be updated. > If somebody changes some CSS and it changes 50 snapshots, it isnt a huge burden to approve them all and sometimes it highlights a bug. Lets say person A initially created all these snapshots, person B did a change that shows 50 snapshots changed, who's responsibility is it to make sure the snapshots are correct? Person A doesn't have the context of the change, so less ideal. Person B doesn't know the initial conditions Person A had in mind, so also less ideal. When you have unit tests and functional tests you can read through the test and know what the person who wrote it wanted to test. With snapshots, you only know that "This was good", sometimes only with the context of the name of the test itself, but no assertions you can read and say "Ah, X still shows Y so all good".
    - MoreQARespect 1y ago
      >You generally don't want to have to change all the tests for every change You generally don't want every change to result in a lot of work. If changing a lot of tests means looking at a table of 30 images and diffs, scanning for problems and clicking an "approve button", that isn't a lot of work though. >Lets say person A initially created all these snapshots, person B did a change that shows 50 snapshots changed, who's responsibility is it to make sure the snapshots are correct? The person who made the change. >Person B doesn't know the initial conditions Person A had in mind, so also less ideal. Yes they will because the initial conditions also had a snapshot attached. If your snapshot testing is even mildly fancy it will come with a diff too. >When you have unit tests and functional tests you can read through the test and know what the person who wrote it wanted to test. With snapshots, you only know that "This was good", If you made a change and you can see the previous snapshot, current snapshot and a diff and you never know if the change was ok then you probably shouldn't be working on the project in the first place. And no, the same isn't necessarily true of unit or functional tests - I've seen hundreds of unit tests that assert things about objects and properties which are tangentially related to the end user and come with zero context attached and I have to try and figure out wtf the test writer meant by "assert xyz_obj.transitional is None". With a user facing snapshot it's obvious.
- hansvm 1y ago
  The purpose of snapshot testing is to not have observable changes if you think there shouldn't be observable changes. To that end, a pattern I like is: - Don't store/commit the snapshot and have an "update" command. Your CI/CD should run both versions of the software and diff them. That eliminates a lot of the toil. - You should have a completely trivial way to mark that a given PR intends to have observable changes. That could be a tag on GitHub, a square-bracket thing in a commit message, etc. Details don't matter a ton. The point is that the test just catches if things have changed, and a person still needs to determine if that change is appropriate, but that happens often enough that you should make that process easy. - Culturally, you should split out PRs which change golden-affecting behavior from those which don't. A bundle of bug fixes, style changes, and a couple new features is not a good thing to commit to the repo as a whole. The net effect is that: 1. Performance improvements, unrelated features, etc are tested exactly as you expect. If your perf enhancement changes behavior, that was likely wrong, and the test caught it. If it doesn't, the test gives you confidence that it really doesn't. 2. Legitimate changes to the golden behavior are easy to institute. Just toggle a flag somewhere and say that you do intend for there to be a new button or new struct field or whatever you're testing. 3. You have a historical record of the commits which actually changed that behavior, and because of the cultural shift I proposed they're all small changes. Bisecting or otherwise diagnosing tricky prod bugs becomes trivial.

diggan 1y ago

> rely more on snapshot testing are spot on Never quite liked "snapshot testing" which I think has a better name under "golden master testing" or similar anyways. Reason for the dislike, is that it's basically a codified "Trust me bro, it's correct" without actually making clear what you are asserting with that test. I haven't found any team that used snapshot testing and didn't also need to change the snapshots for every little change, which obviously defeats the purpose. The only things snapshot testing seems to be good for, is when you've written something and you know it'll never change again, for any reason. Beyond that, unit tests and functional/integration tests are much easier to structure in a way so you don't waste so much time reviewing changes.

MoreQARespect 1y ago
>I haven't found any team that used snapshot testing and didn't also need to change the snapshots for every little change, which obviously defeats the purpose I dont see how this even defeats the point, let alone obviously. If a UI changes I appreciate being notified. If a REST API response changes I like to see a diff. If somebody changes some CSS and it changes 50 snapshots, it isnt a huge burden to approve them all and sometimes it highlights a bug.
- parthdesai 1y ago
  In the talk, he also talks about passing a flag which would actually update the snapshots/golden files if needed
- necovek 1y ago
  Are you really going to be reviewing all those 50 snapshots carefully? The bound of testing on the "other side" is to test just enough not to increase the maintenance burden too much.
- diggan 1y ago
  > I dont see how this even defeats the point, let alone obviously. You generally don't want to have to change all the tests for every change, particularly implementation details. Usually when people do snapshot testing of for example UI components, they serialize the entire component then assert the full component is the same as the snapshot, so any change requires the snapshot to be updated. > If somebody changes some CSS and it changes 50 snapshots, it isnt a huge burden to approve them all and sometimes it highlights a bug. Lets say person A initially created all these snapshots, person B did a change that shows 50 snapshots changed, who's responsibility is it to make sure the snapshots are correct? Person A doesn't have the context of the change, so less ideal. Person B doesn't know the initial conditions Person A had in mind, so also less ideal. When you have unit tests and functional tests you can read through the test and know what the person who wrote it wanted to test. With snapshots, you only know that "This was good", sometimes only with the context of the name of the test itself, but no assertions you can read and say "Ah, X still shows Y so all good".
  - MoreQARespect 1y ago
    >You generally don't want to have to change all the tests for every change You generally don't want every change to result in a lot of work. If changing a lot of tests means looking at a table of 30 images and diffs, scanning for problems and clicking an "approve button", that isn't a lot of work though. >Lets say person A initially created all these snapshots, person B did a change that shows 50 snapshots changed, who's responsibility is it to make sure the snapshots are correct? The person who made the change. >Person B doesn't know the initial conditions Person A had in mind, so also less ideal. Yes they will because the initial conditions also had a snapshot attached. If your snapshot testing is even mildly fancy it will come with a diff too. >When you have unit tests and functional tests you can read through the test and know what the person who wrote it wanted to test. With snapshots, you only know that "This was good", If you made a change and you can see the previous snapshot, current snapshot and a diff and you never know if the change was ok then you probably shouldn't be working on the project in the first place. And no, the same isn't necessarily true of unit or functional tests - I've seen hundreds of unit tests that assert things about objects and properties which are tangentially related to the end user and come with zero context attached and I have to try and figure out wtf the test writer meant by "assert xyz_obj.transitional is None". With a user facing snapshot it's obvious.
hansvm 1y ago
The purpose of snapshot testing is to not have observable changes if you think there shouldn't be observable changes. To that end, a pattern I like is: - Don't store/commit the snapshot and have an "update" command. Your CI/CD should run both versions of the software and diff them. That eliminates a lot of the toil. - You should have a completely trivial way to mark that a given PR intends to have observable changes. That could be a tag on GitHub, a square-bracket thing in a commit message, etc. Details don't matter a ton. The point is that the test just catches if things have changed, and a person still needs to determine if that change is appropriate, but that happens often enough that you should make that process easy. - Culturally, you should split out PRs which change golden-affecting behavior from those which don't. A bundle of bug fixes, style changes, and a couple new features is not a good thing to commit to the repo as a whole. The net effect is that: 1. Performance improvements, unrelated features, etc are tested exactly as you expect. If your perf enhancement changes behavior, that was likely wrong, and the test caught it. If it doesn't, the test gives you confidence that it really doesn't. 2. Legitimate changes to the golden behavior are easy to institute. Just toggle a flag somewhere and say that you do intend for there to be a new button or new struct field or whatever you're testing. 3. You have a historical record of the commits which actually changed that behavior, and because of the cultural shift I proposed they're all small changes. Bisecting or otherwise diagnosing tricky prod bugs becomes trivial.