3 ms·

Post in which simonw explains the "snapshot testing" approach: https://til.simonwillison.net/pytest/syrupy https://til.simonwillison.net/pytest/syrupy

p1necone 2y ago
I've found this approach super useful for anything that has a potentially complex implementation with relatively simple output (e.g. compilers, query generators, format converters). You can comprehensively cover your code with simple snapshot tests and not really anything else. It's critically important to very carefully review the snapshots before committing them for the first time though - it's far too easy to run the test, look at the output, think "yup that looks like some sql/assembly/whatever that does the thing I'm trying to do" and carry on. Only to realize days/weeks/months later that there's a bunch of bugs that you never caught because your supposed "correct" output was never actually correct.
- simonw 2y ago
  100% agree - the risk of snapshot tests is that you can get lazy, at which point you're losing out on the benefit of using tests to help protect against bugs!
- anon7000 2y ago
  This is exactly why I strongly dislike snapshot tests most of the time in big repositories with lots of collaborators. The snapshot isn’t encoding any logic, it’s just saying “are you sure you wanted to change that?” Whereas a good unit test will tell you exactly what’s broken. It’s just too easy to update the snapshot, and when you glance at changes to a large snapshot, it’s impossible to tell what the test is actually trying to check
sweetsocks21 2y ago
This reminds me of expect tests in OCaml[0]. You create a test function that prints some state and the test framework automatically handles diffing and injecting the snapshot back into the test location. It helps keep your code modular because you need to create some visual representation of it. And it's usually obvious that's wrong through the diff. [0] https://github.com/janestreet/ppx_expect https://github.com/janestreet/ppx_expect

p1necone 2y ago

I've found this approach super useful for anything that has a potentially complex implementation with relatively simple output (e.g. compilers, query generators, format converters). You can comprehensively cover your code with simple snapshot tests and not really anything else. It's critically important to very carefully review the snapshots before committing them for the first time though - it's far too easy to run the test, look at the output, think "yup that looks like some sql/assembly/whatever that does the thing I'm trying to do" and carry on. Only to realize days/weeks/months later that there's a bunch of bugs that you never caught because your supposed "correct" output was never actually correct.

simonw 2y ago
100% agree - the risk of snapshot tests is that you can get lazy, at which point you're losing out on the benefit of using tests to help protect against bugs!
anon7000 2y ago
This is exactly why I strongly dislike snapshot tests most of the time in big repositories with lots of collaborators. The snapshot isn’t encoding any logic, it’s just saying “are you sure you wanted to change that?” Whereas a good unit test will tell you exactly what’s broken. It’s just too easy to update the snapshot, and when you glance at changes to a large snapshot, it’s impossible to tell what the test is actually trying to check

sweetsocks21 2y ago

This reminds me of expect tests in OCaml[0]. You create a test function that prints some state and the test framework automatically handles diffing and injecting the snapshot back into the test location. It helps keep your code modular because you need to create some visual representation of it. And it's usually obvious that's wrong through the diff. [0] https://github.com/janestreet/ppx_expect https://github.com/janestreet/ppx_expect