3 ms·
> Running tests against different infra than your customers have to deal with is asking for trouble. What bugs will exist in the real production infra that won'
by lamontcg 5y ago
> Running tests against different infra than your customers have to deal with is asking for trouble. What bugs will exist in the real production infra that won't in your fake infra?
We actually do pretty well testing against fake infra.
We have a large test suite that enforces the contract on our REST server API. That is implement both in one heavy server written in erlang which is the production code and one lightweight server written in ruby which would never scale but is the same API. When the test suite is updated both the implementations need to be fixed. When the client code runs integration tests we can test against the lightweight ruby code and when it passes we actually have pretty high confidence that it runs against the production code. We have hundreds of those tests and they can be run as fast as spinning up a ruby process with an in-memory datastore which is trashed on every test. Compare that to end to end tests that might fire up a set of images, terraform them into production servers and clients and run a scripted interaction or set of interactions and then throws that away and does it again.
At some point there's a tradeoff between the realism of your tests and the cost of them and how many of them you can do. The right strategy is that you want to have enough of the most realistic tests to give you a high level of confidence that your faster, slightly less realistic tests are useful, and by having those faster tests you increase your amount of coverage, and on down the stack iteratively until you may get to unit tests of individual objects.
- mabbo 5y ago> At some point there's a tradeoff between the realism of your tests and the cost of them and how many of them you can do. Yeah, you're not wrong. I'm just griping because of the previous list of complaints. There are some benefits to running your tests against a mirror infra to reality. But they are limited. The bugs they catch that you won't catch running fake infra are very small in number, but terrifying in difficulty to solve.
- lamontcg 5y agoThe article did make its points fairly poorly. I think in the end they more or less did what you suggested as well they just didn't call it end to end testing.