4 ms·

Absolutely not. It is hard enough to get developers writing meaningful tests that exercise edge cases to make code hard to break. Far too often they do it becau

Absolutely not. It is hard enough to get developers writing meaningful tests that exercise edge cases to make code hard to break. Far too often they do it because "you should" and "we have a policy". Coverage reports too easily get people focusing on numbers and policy over understanding how to identify and strengthen key functionality so it's hard to break as things evolve. What you end up with is: 1. Tests that at least invoke code that may be overlooked, but due to mocks, invoke it in a way thats meaningless. 2. Twice as much code to update when doing any feature work, refactoring, ... or even bug fixing! 3. A false sense of security because "tests passed!" 4. A lot of wasted time writing tests for code you end up either throwing out or significantly refactoring before it ever becomes a remotely active part of the project. I'm jaded. I LOVE good tests, but far, far too often they get in the way of real progress that actually matters to an end user. Tests are supposed to be worth their weight in gold. Too often they're a self-congratulating pat-on-the-back and a detriment to success. Please, prove me wrong. I know you can, because this does NOT apply to everyone.

tobilg 6y ago
Thanks for the comprehensive answer. I have similar experiences, treating the coverage numbers as a KPI will eventually trigger difficult decisions in writing tests. What I’m wondering is what would make „more sense“ if one wants to track the code quality and test coverage in a broader sense.
- mr-wendel 6y ago
  I think it really depends. * How sensitive are you operations to breakage within the repo? Are there key things you can focus on that you know will give you trouble that can be readily tested with each build? * Are you performing a lot of bug fixing as part of long term maintenance for an established system? I'd suggest adding tests for each thing just before you fix it and let tests demonstrate that is fixed. Look for any easy-pickings of related functionality while you're there, as often fixing one bug just creates another (particularly with convoluted codebases). * Is this a "leaf" or a "node" within your overall system architecture? I prefer to focus on the deeper bits first, as thats where you're more likely to cause cascading failures. Also, the flip side of testing is always monitoring. * Have systems run background jobs to self-audit themselves and report errors. Expose these as APIs, if you can, that let you probe them on-demand to help quickly troubleshoot things. * Use monitoring systems (e.g. nagios, zabbix, prometheus, etc) to audit things that are difficult (or not appropriate) to test: e.g. upstream systems, internal component health, etc. Don't underestimate the power of a quick shell script that watchings known pain points and critical interfaces. Sometimes a few hours of hacking beats the pants off waiting for QA/DevOps/etc to get clearance and a few-to-several weeks to do a "proper" solution. * Ensure you have reliable pre-production environments (e.g. at least a "staging" system thats generally prod-like). Sometimes you need multiple! Ensure health checks and monitoring is performed on all environments (albeit with toned-down escalations) to catch things before they go live.

tobilg 6y ago

Thanks for the comprehensive answer. I have similar experiences, treating the coverage numbers as a KPI will eventually trigger difficult decisions in writing tests. What I’m wondering is what would make „more sense“ if one wants to track the code quality and test coverage in a broader sense.

mr-wendel 6y ago
I think it really depends. * How sensitive are you operations to breakage within the repo? Are there key things you can focus on that you know will give you trouble that can be readily tested with each build? * Are you performing a lot of bug fixing as part of long term maintenance for an established system? I'd suggest adding tests for each thing just before you fix it and let tests demonstrate that is fixed. Look for any easy-pickings of related functionality while you're there, as often fixing one bug just creates another (particularly with convoluted codebases). * Is this a "leaf" or a "node" within your overall system architecture? I prefer to focus on the deeper bits first, as thats where you're more likely to cause cascading failures. Also, the flip side of testing is always monitoring. * Have systems run background jobs to self-audit themselves and report errors. Expose these as APIs, if you can, that let you probe them on-demand to help quickly troubleshoot things. * Use monitoring systems (e.g. nagios, zabbix, prometheus, etc) to audit things that are difficult (or not appropriate) to test: e.g. upstream systems, internal component health, etc. Don't underestimate the power of a quick shell script that watchings known pain points and critical interfaces. Sometimes a few hours of hacking beats the pants off waiting for QA/DevOps/etc to get clearance and a few-to-several weeks to do a "proper" solution. * Ensure you have reliable pre-production environments (e.g. at least a "staging" system thats generally prod-like). Sometimes you need multiple! Ensure health checks and monitoring is performed on all environments (albeit with toned-down escalations) to catch things before they go live.