6 ms·

>I haven't found any team that used snapshot testing and didn't also need to change the snapshots for every little change, which obviously defeats the purpose

>I haven't found any team that used snapshot testing and didn't also need to change the snapshots for every little change, which obviously defeats the purpose I dont see how this even defeats the point, let alone obviously. If a UI changes I appreciate being notified. If a REST API response changes I like to see a diff. If somebody changes some CSS and it changes 50 snapshots, it isnt a huge burden to approve them all and sometimes it highlights a bug.

parthdesai 1y ago
In the talk, he also talks about passing a flag which would actually update the snapshots/golden files if needed
necovek 1y ago
Are you really going to be reviewing all those 50 snapshots carefully? The bound of testing on the "other side" is to test just enough not to increase the maintenance burden too much.
diggan 1y ago
> I dont see how this even defeats the point, let alone obviously. You generally don't want to have to change all the tests for every change, particularly implementation details. Usually when people do snapshot testing of for example UI components, they serialize the entire component then assert the full component is the same as the snapshot, so any change requires the snapshot to be updated. > If somebody changes some CSS and it changes 50 snapshots, it isnt a huge burden to approve them all and sometimes it highlights a bug. Lets say person A initially created all these snapshots, person B did a change that shows 50 snapshots changed, who's responsibility is it to make sure the snapshots are correct? Person A doesn't have the context of the change, so less ideal. Person B doesn't know the initial conditions Person A had in mind, so also less ideal. When you have unit tests and functional tests you can read through the test and know what the person who wrote it wanted to test. With snapshots, you only know that "This was good", sometimes only with the context of the name of the test itself, but no assertions you can read and say "Ah, X still shows Y so all good".
- MoreQARespect 1y ago
  >You generally don't want to have to change all the tests for every change You generally don't want every change to result in a lot of work. If changing a lot of tests means looking at a table of 30 images and diffs, scanning for problems and clicking an "approve button", that isn't a lot of work though. >Lets say person A initially created all these snapshots, person B did a change that shows 50 snapshots changed, who's responsibility is it to make sure the snapshots are correct? The person who made the change. >Person B doesn't know the initial conditions Person A had in mind, so also less ideal. Yes they will because the initial conditions also had a snapshot attached. If your snapshot testing is even mildly fancy it will come with a diff too. >When you have unit tests and functional tests you can read through the test and know what the person who wrote it wanted to test. With snapshots, you only know that "This was good", If you made a change and you can see the previous snapshot, current snapshot and a diff and you never know if the change was ok then you probably shouldn't be working on the project in the first place. And no, the same isn't necessarily true of unit or functional tests - I've seen hundreds of unit tests that assert things about objects and properties which are tangentially related to the end user and come with zero context attached and I have to try and figure out wtf the test writer meant by "assert xyz_obj.transitional is None". With a user facing snapshot it's obvious.
  - tharkun__ 1y ago
    No regular developer will carefully review 50 changed snapshots. They'll stop doing a proper job of it after the third or fourth that looks like the same trivial unimportant change and miss the bug found by snapshot 37. I do agree that a lot of people write bad tests meaning the test name does not properly describe what the test is supposed to be about so I can check the test implementation and assertions against intent. They also like you say assert on superfluous things. The problem with snapshots is that it's doing the exact same thing. It asserts on lots of completely unimportant stuff. Unlike proper unit tests however I can't make it better. In a unit test I can make an effort to educate my peers and personally do a good job of only asserting relevant things and writing individual tests where the test name explains why "transitional has to be None". Snapshots are a blunt no-effort tool for the lazy dev that then later requires constant vigilence and overcoming things humans are bad at (like carefully checking 50 snapshots) by many different humans VS a unit test that I can make good and easy to comprehend and check by putting in effort once when I write it. A good one will also be easy to adjust when needed if it comes time to actually need to change an assertion.

parthdesai 1y ago

In the talk, he also talks about passing a flag which would actually update the snapshots/golden files if needed

necovek 1y ago

Are you really going to be reviewing all those 50 snapshots carefully? The bound of testing on the "other side" is to test just enough not to increase the maintenance burden too much.

diggan 1y ago

> I dont see how this even defeats the point, let alone obviously. You generally don't want to have to change all the tests for every change, particularly implementation details. Usually when people do snapshot testing of for example UI components, they serialize the entire component then assert the full component is the same as the snapshot, so any change requires the snapshot to be updated. > If somebody changes some CSS and it changes 50 snapshots, it isnt a huge burden to approve them all and sometimes it highlights a bug. Lets say person A initially created all these snapshots, person B did a change that shows 50 snapshots changed, who's responsibility is it to make sure the snapshots are correct? Person A doesn't have the context of the change, so less ideal. Person B doesn't know the initial conditions Person A had in mind, so also less ideal. When you have unit tests and functional tests you can read through the test and know what the person who wrote it wanted to test. With snapshots, you only know that "This was good", sometimes only with the context of the name of the test itself, but no assertions you can read and say "Ah, X still shows Y so all good".

MoreQARespect 1y ago
>You generally don't want to have to change all the tests for every change You generally don't want every change to result in a lot of work. If changing a lot of tests means looking at a table of 30 images and diffs, scanning for problems and clicking an "approve button", that isn't a lot of work though. >Lets say person A initially created all these snapshots, person B did a change that shows 50 snapshots changed, who's responsibility is it to make sure the snapshots are correct? The person who made the change. >Person B doesn't know the initial conditions Person A had in mind, so also less ideal. Yes they will because the initial conditions also had a snapshot attached. If your snapshot testing is even mildly fancy it will come with a diff too. >When you have unit tests and functional tests you can read through the test and know what the person who wrote it wanted to test. With snapshots, you only know that "This was good", If you made a change and you can see the previous snapshot, current snapshot and a diff and you never know if the change was ok then you probably shouldn't be working on the project in the first place. And no, the same isn't necessarily true of unit or functional tests - I've seen hundreds of unit tests that assert things about objects and properties which are tangentially related to the end user and come with zero context attached and I have to try and figure out wtf the test writer meant by "assert xyz_obj.transitional is None". With a user facing snapshot it's obvious.
- tharkun__ 1y ago
  No regular developer will carefully review 50 changed snapshots. They'll stop doing a proper job of it after the third or fourth that looks like the same trivial unimportant change and miss the bug found by snapshot 37. I do agree that a lot of people write bad tests meaning the test name does not properly describe what the test is supposed to be about so I can check the test implementation and assertions against intent. They also like you say assert on superfluous things. The problem with snapshots is that it's doing the exact same thing. It asserts on lots of completely unimportant stuff. Unlike proper unit tests however I can't make it better. In a unit test I can make an effort to educate my peers and personally do a good job of only asserting relevant things and writing individual tests where the test name explains why "transitional has to be None". Snapshots are a blunt no-effort tool for the lazy dev that then later requires constant vigilence and overcoming things humans are bad at (like carefully checking 50 snapshots) by many different humans VS a unit test that I can make good and easy to comprehend and check by putting in effort once when I write it. A good one will also be easy to adjust when needed if it comes time to actually need to change an assertion.