3 ms·

Test speed fetishism is a bad idea. It leads people to write unrealistic tests which end up passing when there is a bug and failing when there isn't. Treating

by pydry 3y ago

Test speed fetishism is a bad idea. It leads people to write unrealistic tests which end up passing when there is a bug and failing when there isn't. Treating flakiness in integration tests as a fait accompli is also a bad idea. It's a bad idea - a bit like treating flakiness in the app itself as a fait accompli.

agentultra 3y ago
Tests that are slow to run don't get run often. People don't write tests first. They write fewer tests. They tend to test the happy paths. I'm not saying we should avoid integration testing. It's necessary. It doesn't mix well with PBT for a main test suite, in my experience. You're better off making sure your domain types and business logic are encapsulated in your types and functions and don't require a database, a network, etc. PBT is great at specifying invariants and relations between functions and sets. Use your integration testing budget for testing the integration boundaries with the database and network. Update: there's no cure for writing bad tests. you just have to write better ones.
- pydry 3y ago
  Tests get run as often as they get run. There's no rule that says that you have to run them less frequently if they are slower. It's not 1994. CPU time is cheap. Realism is much more important than speed. A test that catches 20% more bugs and takes 5 seconds instead of 0.05s is just more valuable. It's simply not worth economizing on CPU cycles in 2024. Yes, DI can make sense sometimes but definitely not all of the time. The people who have ultra fast unrealistic tests inevitably end up leaning on manual testing as they experience bugs which their unit tests could never have caught. >there's no cure for writing bad tests. you just have to write better ones. Better means more realistic, hermetic and less flaky. It doesnt necessarily mean ultrafast.

3 ms·

Test speed fetishism is a bad idea. It leads people to write unrealistic tests which end up passing when there is a bug and failing when there isn't. Treating

by pydry 3y ago

agentultra 3y ago
Tests that are slow to run don't get run often. People don't write tests first. They write fewer tests. They tend to test the happy paths. I'm not saying we should avoid integration testing. It's necessary. It doesn't mix well with PBT for a main test suite, in my experience. You're better off making sure your domain types and business logic are encapsulated in your types and functions and don't require a database, a network, etc. PBT is great at specifying invariants and relations between functions and sets. Use your integration testing budget for testing the integration boundaries with the database and network. Update: there's no cure for writing bad tests. you just have to write better ones.
- pydry 3y ago
  Tests get run as often as they get run. There's no rule that says that you have to run them less frequently if they are slower. It's not 1994. CPU time is cheap. Realism is much more important than speed. A test that catches 20% more bugs and takes 5 seconds instead of 0.05s is just more valuable. It's simply not worth economizing on CPU cycles in 2024. Yes, DI can make sense sometimes but definitely not all of the time. The people who have ultra fast unrealistic tests inevitably end up leaning on manual testing as they experience bugs which their unit tests could never have caught. >there's no cure for writing bad tests. you just have to write better ones. Better means more realistic, hermetic and less flaky. It doesnt necessarily mean ultrafast.