4 ms·
Couldn’t you also test the recovery logic with a hasher that yielded collision more often? It’s like take your system, make it worse (use very dumb hashing), an
by seanmcdirmid 6y ago
Couldn’t you also test the recovery logic with a hasher that yielded collision more often? It’s like take your system, make it worse (use very dumb hashing), and see how it degrades and/or breaks?
- xxxtentachyon 6y agoOr, even simpler, just stub out your hash function and to something that always generates the same value and test with that.
- rocqua 6y agoSeems like that would make it less of an integration test
- jmholla 6y agoThey said it was for unit tests.
- Heliosmaster 6y agothey mention this was a unit test, for which stubbing the hashing function would have been totally fine
- treeman79 6y agoThis! Take over a project with thousands of integration tests. I trust none of them. Why? Because anything hard to test has been mocked to oblivion. I have api endpoints that have massive complicated integration tests. I eventually just got source for every project in company and checked to see what they actually did with API.
- thaumasiotes 6y agoIn this case, I think mocking is the right thing to do. You want to know that you can handle the case where two hashes collide. But for this purpose, you don't care at all how the hashes were generated. If the website changes its URI hash algorithm, the "honest" test that just supplies two URIs that happened to collide under the earlier algorithm will suddenly become worthless. The mocked test will continue doing exactly what you always wanted it to do -- exercise the "what if the hashes collide?" case in your code.
- cle 6y agoYou still have to reason through “what if it collides with the real hash function?” How do you know it will behave the same if you don’t test it? (I can sympathize with both approaches and don’t think either one is better.)
- jskdvsksnb 6y agoI have the inverse - I started at a place that has hundreds integration tests that require a database, k8s, the whole shebang. No mocking anywhere. Tests take an hour to run, are extremely fragile and flakey. Getting your local dev machine set up to run even a subset of the tests is a half day ordeal, and it can change periodically. If you have external dependencies that don't version their APIs well it may be inevitable that you need to run tests against them. But for developer productivity you should have a fast/non-flakey path to run local unit tests with no network dependencies. It's also great for working on a plane.
- 978e4721a 6y agoCause everyone works mostly on a planes, right...
- njharman 6y ago> Because anything hard to test has been mocked to oblivion. Agree. Not an integration test. If it has mocks in it, it is not testing the INTEGRATION of/between components. Mocks are for Unit tests. Where you want to isolate testing one single UNIT from all the other crap it integrates with (which have their own unit tests) After all these unit tests run, you test the whole shebang together with integration tests.
- tialaramex 6y agoMy preference is always to actually test parts in as close as possible to the real state, avoiding mocks unless they're necessary. The more "fake" the system is, the less it tells you about how your software actually works. So since I could generate collisions for the real hash, that's what I did. If it had been infeasible, I would of course have found a different way to test. This particular software was written mostly in C (today I would not write it in C) which makes it even more likely that introducing never-used-in-production behaviour unexpectedly perturbs the system. This same system for example ran into an obscure (and since fixed) libc allocator bug, and a defect in the Linux kernel, because we were really kicking the shit out of the memory mapped file I/O with what we were doing.
- seanmcdirmid 6y agoThe only high fidelity mantra in testing is a bit misguided. If you only test your system in as close to production state as possible, you are never going to encounter stress states that would only very rarely appear in production. Seeing how your components independently deal with less ideal components in the system can tell you a lot about how they might handle rare but bad cases in production, and allow you to over engineer them a bit more to avoid that. > which makes it even more likely that introducing never-used-in-production behaviour unexpectedly perturbs the system. Isn’t that a good thing? Isn’t the goal to deal with the unexpected in the first case?
- jniedrauer 6y agoTests come with a non-zero cost. They're code that needs to be maintained. Often complex code, with poor documentation. I don't think the value provided by testing states that you will never actually encounter in production outweighs the cost of maintaining the tests.
- seanmcdirmid 6y agoTo achieve “defense in depth,” testing components in degenerate non-production situations (and then over engineering them to gracefully deal with those situations) is the only way to get there. Of course, we aren’t building bridges, so maybe it’s less important.
- harpiaharpyja 6y agoI was wondering why GP couldn't have swapped out the hash function for one that just returns a fixed value, making everything a collision. Then it would just be a matter of verifying the system still works under that condition (ignoring the performance hit). You could even measure the performance hit from that degraded condition and create a heuristic to warn you if something is wrong with the fast path.