3 ms·
Why would you think there are "no tests"?
by mvdtnz 2mo ago
Why would you think there are "no tests"?
- 27183 2mo agoWe have a pretty strong existence proof... the thing happened in production. Unless they have some means to override a failing test and scp broken shit to prod, there wasn't a test.
- nullorempty 2mo agoTechnically, there could be a test. It could just be wrong!
- 27183 2mo agoIf a tree falls in the forest and nobody hears it... [edit] Testing your tests, like testing your backups, is a good idea
- fragmede 2mo agoYes, test the negative case as well. eg if you get the system setup so you can log in, also make sure you get permission denied for bad login info.
- 27183 2mo agoYeah, negative tests are more important than the "test your tests" thing. Negative tests are themselves regression tests. Testing that a test works is something you do at dev time, it doesn't really live in CI. Negative tests, as well as positive tests, establish invariants on the code under test. They're effectively permanent, immortalized in the CI suite. Testing that the test actually works is a one time thing at the time the test is written. Permute the code under test, check that the test fails in the expected way, done. You just have to trust your colleagues won't edit the tests in such a way that vandalizes the invariant. That's why you can never trust an LLM to edit a test. They're notorious for tweaking tests such that they pass.
- chasd00 2mo agowhy would a test setting unit to Bytes fail and not MB, KB, or GB, and so on? That's like trying create a unit test for email opt-in, both true and false are valid values. It's up to the user to select the right one.
- 27183 2mo agoI'm not quite following your objection.. I'd expect a test that checks the multiplier is correct would detect orders of magnitude discrepancy. So if you're billing $x/byte you'd write a test for the billing thing that checks that, given y bytes, the bill is x*y. [edit] This may need to be an integration test to be effective, there is a certain peril to mocking that could bite you here. But that's fine, we have the technology.
- soniclettuce 2mo agoI think they're saying that their understanding of the hypothetical error is that somebody more or less configured a price as 5¢/byte instead of 5¢/GB. And if that's the case, that's not really a thing testing will catch. 5¢/byte is as "valid" a price as 5¢/GB. There's a lot of hypotheticals in there, so who knows if testing could have caught this specific problem. But I see what they're saying in principle.
- hsbauauvhabzb 2mo agoWhy have a default at all? Explicit is better than implicit.
- 8note 2mo agomissing canaries more likely? insufficient tests that dont assert on the right things? the existence of a test doesnt mean it catches the right thing based on the description, id bet the COE action item will be to do a migration that enforces units are passed at the billing service level theres no good reason for the billing service to make up its own units.
- 27183 2mo agoYeah types and APIs that are difficult to misuse are way better than any regression test, canary, or what have you. Systems that are correct by construction always beat "you're holding it wrong".