5 ms·
My preference is always to actually test parts in as close as possible to the real state, avoiding mocks unless they're necessary. The more "fake" the system is
by tialaramex 6y ago
My preference is always to actually test parts in as close as possible to the real state, avoiding mocks unless they're necessary. The more "fake" the system is, the less it tells you about how your software actually works.
So since I could generate collisions for the real hash, that's what I did. If it had been infeasible, I would of course have found a different way to test.
This particular software was written mostly in C (today I would not write it in C) which makes it even more likely that introducing never-used-in-production behaviour unexpectedly perturbs the system. This same system for example ran into an obscure (and since fixed) libc allocator bug, and a defect in the Linux kernel, because we were really kicking the shit out of the memory mapped file I/O with what we were doing.
- seanmcdirmid 6y agoThe only high fidelity mantra in testing is a bit misguided. If you only test your system in as close to production state as possible, you are never going to encounter stress states that would only very rarely appear in production. Seeing how your components independently deal with less ideal components in the system can tell you a lot about how they might handle rare but bad cases in production, and allow you to over engineer them a bit more to avoid that. > which makes it even more likely that introducing never-used-in-production behaviour unexpectedly perturbs the system. Isn’t that a good thing? Isn’t the goal to deal with the unexpected in the first case?
- jniedrauer 6y agoTests come with a non-zero cost. They're code that needs to be maintained. Often complex code, with poor documentation. I don't think the value provided by testing states that you will never actually encounter in production outweighs the cost of maintaining the tests.
- seanmcdirmid 6y agoTo achieve “defense in depth,” testing components in degenerate non-production situations (and then over engineering them to gracefully deal with those situations) is the only way to get there. Of course, we aren’t building bridges, so maybe it’s less important.
- TickleSteve 6y agoThats not really a "defense in depth" situation. Defense in depth is regarding layering different types of security, not targeted testing. https://en.wikipedia.org/wiki/Defense_in_depth_(computing) https://en.wikipedia.org/wiki/Defense_in_depth_(computing)
- nitrogen 6y agoIt's an analogy. Multiple layers of defense against bugs.
- tikhonj 6y agoThis approach works well until you do encounter a situation like that in production :). It's worth testing conservatively because our understanding of our systems is imperfect and because assumptions that may be valid now are likely to change in the future. It's especially true for code that might operate at scale or has security considerations, because both of those can magnify the effects of conditions that occur with arbitrarily low probabilities. Personally, I see tests as not just a way to prevent bugs, but as a way to learn about the system being tested—both how it behaves now and, continually, how it behaves in future iterations. In this specific case, understanding what happens if two URIs hash to the same value even if they don't "really" do that is an important bit of information: it tells us not only what would happen with a real hash collision but also how our system would behave if there were a bug either in the hashing algorithm itself or in the code that connects the hashing algorithm to the component that uses the hashes. What happens if we introduce a caching layer between where the hash is generated and where it's used, and there's a cache invalidation bug? I don't know how realistic that is in this specific hypothetical, but I'm sure there are other performance optimizations with the same bug potential that I'm not even thinking of. Changes like that can lead to absolute debugging nightmares! Testing even "impossible" edge cases helps catch this in a systematic way, without needing full knowledge of which conditions might matter. Of course, none of this talks to the trade-off you pointed out: test code does have a real cost. That is absolutely something to consider. I just don't think the answer is to draw a hard line at "don't test situations that can't come up in production"—and, perhaps naively, I think the "real" solution is less about deciding what is and isn't worth testing and more about reducing the costs of testing more. Write application code in a way that's easy to test and treat your test code as code—keep it clean, include comments, and invest time in setup and tooling to make your tests as easy to write and maintain as possible.
- kortilla 6y ago> The only high fidelity mantra in testing is a bit misguided. That’s understating it. It’s incompetence because it’s an excuse to not test a huge surface area of edge cases.
- deleted 6y ago[deleted]
- kqr 6y ago> My preference is always to actually test parts in as close as possible to the real state, avoiding mocks unless they're necessary. The more "fake" the system is, the less it tells you about how your software actually works. I've heard this before, but never actually had that problem with fakes in practise. Where can I read more about it?
- brianwawok 6y agoHave a production outage. See the code in question is 100% tested. Scratch head a bit. Vow to fake less next time.
- nitrogen 6y agoSupposedly the software problems with the Starliner spacecraft were also caused by extensive unit testing under test harnesses without integration testing, as a visible example that reached the news.
- tshaddox 6y ago“Test as close as possible to the real state” makes sense as a rule of thumb, but I’m having trouble coming up with a reason in this specific case how mocking the hash function could go wrong. Assuming your testing/mocking library is reliable, this should be testing precisely what you intend to test. I guess it’s possible that your “more real” test relies on some hidden behavior, like how your real hash function utilizes system resources, which could affect production, but I don’t know that tests which are testing things you’re not even aware of is a great thing to advocate for.
- naniwaduni 6y agoThe point of the principle is that you don't know how the mocking can go wrong. Assuming your testing/mocking library is reliable gives you one more thing that can unexpectedly go wrong. Of course, sometimes getting verisimilar test cases is difficult enough that you judge it not worth the effort over mocking it up and dealing with any issues with the mockup, foreseen and unforeseen.
- lisper 6y agoWhy not just use a hash function that special-cases two test URLs (or a whole class of test URLs) and produces collisions for them? You can use that same hash function in production.
- kortilla 6y agoThat’s a stupid principle. If you don’t have your interface well-defined enough that switching out the hashing implementation is difficult, your code is too tightly coupled. A test handing a hash collision is not a test of the hashing algorithm so you should be able to switch out the algorithm to something super predictable. This isn’t a discussion about mocking frameworks - you should be able to do with by just plugging in a different hash implementation when you construct the test.
- kuiper0x2 6y agoThese types of blanket rules are useful to ensure a 100% accurate test case. If an error in production costs me $15 Million I don't really care if the engineer wastes a few days to follow a dumb rule. Also I can employ a $75k - $100k/year engineer and say follow this protocol for testing. If I want someone who can optimize test case 100% of the time with zero mistakes ever then I might have to pay $150k - $200k / year.