3 ms·

Thanks for the comprehensive answer. I have similar experiences, treating the coverage numbers as a KPI will eventually trigger difficult decisions in writing t

by tobilg 6y ago

Thanks for the comprehensive answer. I have similar experiences, treating the coverage numbers as a KPI will eventually trigger difficult decisions in writing tests. What I’m wondering is what would make „more sense“ if one wants to track the code quality and test coverage in a broader sense.

mr-wendel 6y ago
I think it really depends. * How sensitive are you operations to breakage within the repo? Are there key things you can focus on that you know will give you trouble that can be readily tested with each build? * Are you performing a lot of bug fixing as part of long term maintenance for an established system? I'd suggest adding tests for each thing just before you fix it and let tests demonstrate that is fixed. Look for any easy-pickings of related functionality while you're there, as often fixing one bug just creates another (particularly with convoluted codebases). * Is this a "leaf" or a "node" within your overall system architecture? I prefer to focus on the deeper bits first, as thats where you're more likely to cause cascading failures. Also, the flip side of testing is always monitoring. * Have systems run background jobs to self-audit themselves and report errors. Expose these as APIs, if you can, that let you probe them on-demand to help quickly troubleshoot things. * Use monitoring systems (e.g. nagios, zabbix, prometheus, etc) to audit things that are difficult (or not appropriate) to test: e.g. upstream systems, internal component health, etc. Don't underestimate the power of a quick shell script that watchings known pain points and critical interfaces. Sometimes a few hours of hacking beats the pants off waiting for QA/DevOps/etc to get clearance and a few-to-several weeks to do a "proper" solution. * Ensure you have reliable pre-production environments (e.g. at least a "staging" system thats generally prod-like). Sometimes you need multiple! Ensure health checks and monitoring is performed on all environments (albeit with toned-down escalations) to catch things before they go live.

by tobilg 6y ago

mr-wendel 6y ago

I think it really depends. * How sensitive are you operations to breakage within the repo? Are there key things you can focus on that you know will give you trouble that can be readily tested with each build? * Are you performing a lot of bug fixing as part of long term maintenance for an established system? I'd suggest adding tests for each thing just before you fix it and let tests demonstrate that is fixed. Look for any easy-pickings of related functionality while you're there, as often fixing one bug just creates another (particularly with convoluted codebases). * Is this a "leaf" or a "node" within your overall system architecture? I prefer to focus on the deeper bits first, as thats where you're more likely to cause cascading failures. Also, the flip side of testing is always monitoring. * Have systems run background jobs to self-audit themselves and report errors. Expose these as APIs, if you can, that let you probe them on-demand to help quickly troubleshoot things. * Use monitoring systems (e.g. nagios, zabbix, prometheus, etc) to audit things that are difficult (or not appropriate) to test: e.g. upstream systems, internal component health, etc. Don't underestimate the power of a quick shell script that watchings known pain points and critical interfaces. Sometimes a few hours of hacking beats the pants off waiting for QA/DevOps/etc to get clearance and a few-to-several weeks to do a "proper" solution. * Ensure you have reliable pre-production environments (e.g. at least a "staging" system thats generally prod-like). Sometimes you need multiple! Ensure health checks and monitoring is performed on all environments (albeit with toned-down escalations) to catch things before they go live.