4 ms·

I 100% agree with the central thesis that tests are about evidence-based correctness. I think keeping this in mind is important as a product and codebase scale

by omeze 5y ago

I 100% agree with the central thesis that tests are about evidence-based correctness. I think keeping this in mind is important as a product and codebase scales since a lot of testing approaches shift in importance over time. The articles sidebar about TDD is a micro-example of this, but there are ones on the order of months/quarters/years that Ive come across as well. One example is how you likely only want end to end and integration tests early on in a development lifecycle, rather than excessive subsystem tests and mocks. This is because your end to end product experience is likely to change less than the subsystems that implement specific features (eg user auth). Over time, as your subsystems solidify you’ll likely want to reprioritize and keep end to end tests lower in proportion since they take longer to execute. In other words, your test suite should be proportional to the expected requirement changes over time. Another more out-there example is the disabling of tests over time as test coverage starts to overlap. A lot of tests in a large codebase exercise the same codepaths, and this is usually not a net benefit (it sometimes can be, eg if one test runs really fast vs another more comprehensive one thats slow). So having a way to evaluate whether a test is actionable or useful is something that emerges organically as large codebases begin to change.

rhdunn 5y ago
I also personally like to have hierarchies of tests, where higher-level tests don't need to test specific details covered by the lower-level tests like edge cases. Take implementing a language (lexer, parser, static analysis, etc.) for an IDE or compiler. For the lexer, I have the tests cover the different EBNF fixed string tokens (keywords and symbols) in a given token, along with the more complex tokens for numbers, etc. For the more complex tokens I have lexer tests covering the different valid number forms, identifier characters, etc. This leads to overlap in things like keywords, but not an overlap in implementing different specifications (e.g. CSS modules). For the parser tests, because I have the lexical coverage, I can just use a representative example of a number, etc. That allows the parser tests to focus on the different paths (optional parts, loops, etc.) of the EBNF and things like error recovery when different symbols are missing. With these, I validate the AST is correct, including the AST class that implements that node in the tree. For the AST tests, because I have the lexer and parser coverage, I can focus on the data model and how the information in the parse tree is exposed to the AST data model. For the higher-level tests (resolving variables to their declarations, etc.) I can focus on the cases relevant to that level instead of having to also test combinations of the lower-level tests.
wallacoloo 5y ago
> Another more out-there example is the disabling of tests over time as test coverage starts to overlap. A lot of tests in a large codebase exercise the same codepaths, and this is usually not a net benefit (it sometimes can be, eg if one test runs really fast vs another more comprehensive one thats slow). So having a way to evaluate whether a test is actionable or useful is something that emerges organically as large codebases begin to change. If the cost of maintaining these tests exceeds the value they provide, don’t just disable them: delete them. But if the maintenance isn’t an issue, and it’s just the compute resources that are an issue, keep them enabled and schedule your tests more effectively! For example, at my company we have tests which are run the moment you want to commit, and will block your commit if they fail, and then we have a separate service which continually runs tests against tip 24/7. In order to keep low friction in the dev process, we only run a subset of tests as part of the commit process, and assume that a few failures will make it into tip but will be caught within a couple days (we ship every two weeks so that timing’s been pretty safe).

by omeze 5y ago

rhdunn 5y ago

I also personally like to have hierarchies of tests, where higher-level tests don't need to test specific details covered by the lower-level tests like edge cases. Take implementing a language (lexer, parser, static analysis, etc.) for an IDE or compiler. For the lexer, I have the tests cover the different EBNF fixed string tokens (keywords and symbols) in a given token, along with the more complex tokens for numbers, etc. For the more complex tokens I have lexer tests covering the different valid number forms, identifier characters, etc. This leads to overlap in things like keywords, but not an overlap in implementing different specifications (e.g. CSS modules). For the parser tests, because I have the lexical coverage, I can just use a representative example of a number, etc. That allows the parser tests to focus on the different paths (optional parts, loops, etc.) of the EBNF and things like error recovery when different symbols are missing. With these, I validate the AST is correct, including the AST class that implements that node in the tree. For the AST tests, because I have the lexer and parser coverage, I can focus on the data model and how the information in the parse tree is exposed to the AST data model. For the higher-level tests (resolving variables to their declarations, etc.) I can focus on the cases relevant to that level instead of having to also test combinations of the lower-level tests.

wallacoloo 5y ago

> Another more out-there example is the disabling of tests over time as test coverage starts to overlap. A lot of tests in a large codebase exercise the same codepaths, and this is usually not a net benefit (it sometimes can be, eg if one test runs really fast vs another more comprehensive one thats slow). So having a way to evaluate whether a test is actionable or useful is something that emerges organically as large codebases begin to change. If the cost of maintaining these tests exceeds the value they provide, don’t just disable them: delete them. But if the maintenance isn’t an issue, and it’s just the compute resources that are an issue, keep them enabled and schedule your tests more effectively! For example, at my company we have tests which are run the moment you want to commit, and will block your commit if they fail, and then we have a separate service which continually runs tests against tip 24/7. In order to keep low friction in the dev process, we only run a subset of tests as part of the commit process, and assume that a few failures will make it into tip but will be caught within a couple days (we ship every two weeks so that timing’s been pretty safe).