3 ms·

This is a very bad list of complaints and it actually makes me angry to read it. > Engineers had to wait more and more to get feedback from this long-running s

by mabbo 5y ago

This is a very bad list of complaints and it actually makes me angry to read it. > Engineers had to wait more and more to get feedback from this long-running suite So speed up your tests. Run them in parallel. Find better frameworks for running tests. > Flaky tests meant that we had to re-run the suite frequently to see if something was really wrong or just a false negative; Fix your flaky tests! Why anyone just accepts that "Oh, sometimes that test fails and we have to restart everything" is beyond me. Root cause the problem and FIX IT. > Manual changes in our staging environment corrupted test data fixtures and maintaining the environment “clean” was a challenge; Tests should not rely on pre-existing state. Have a setup phase for each test that creates new data in the state you want it to be in. As it makes this data, also note down a reference to it with a Time To Live so that a follow-up process can clean up this unneeded data. > Test failures were very hard to debug That's not the fault of the tests, that's the fault of a complex system that is hard to debug. Improve your tracing between services. > Queueing of commits in the End-to-End suite resulted in less frequent deployments There are well known solutions to this problem. Lots of companies have overcome this already. > Few bugs caught in this stage. One experiment suggested that, for every 1000 runs, we had 42 failures, only 1 bug If your tests have false-positives, you need to adjust your tests. Accepting that the test failed but there isn't a problem, and then not fixing the reason the test failed means that you don't have reliable tests. > Bugs were still being found in production Bugs will always make it to production. But after you fix a bug, you write a test so that this bug cannot happen again. Over time, the number of possible bugs that can make it to production shrinks. And lastly: > The main difference to the old E2E is that they encompass only a subset of services and don’t require spinning a production-like environment (the services run in memory on a single JVM and HTTP/Kafka communication is replaced by in-process communication). They are used in specific flows that we find too critical to only rely on Contract Tests. Running tests against different infra than your customers have to deal with is asking for trouble. What bugs will exist in the real production infra that won't in your fake infra?

lamontcg 5y ago
> Running tests against different infra than your customers have to deal with is asking for trouble. What bugs will exist in the real production infra that won't in your fake infra? We actually do pretty well testing against fake infra. We have a large test suite that enforces the contract on our REST server API. That is implement both in one heavy server written in erlang which is the production code and one lightweight server written in ruby which would never scale but is the same API. When the test suite is updated both the implementations need to be fixed. When the client code runs integration tests we can test against the lightweight ruby code and when it passes we actually have pretty high confidence that it runs against the production code. We have hundreds of those tests and they can be run as fast as spinning up a ruby process with an in-memory datastore which is trashed on every test. Compare that to end to end tests that might fire up a set of images, terraform them into production servers and clients and run a scripted interaction or set of interactions and then throws that away and does it again. At some point there's a tradeoff between the realism of your tests and the cost of them and how many of them you can do. The right strategy is that you want to have enough of the most realistic tests to give you a high level of confidence that your faster, slightly less realistic tests are useful, and by having those faster tests you increase your amount of coverage, and on down the stack iteratively until you may get to unit tests of individual objects.
- mabbo 5y ago
  > At some point there's a tradeoff between the realism of your tests and the cost of them and how many of them you can do. Yeah, you're not wrong. I'm just griping because of the previous list of complaints. There are some benefits to running your tests against a mirror infra to reality. But they are limited. The bugs they catch that you won't catch running fake infra are very small in number, but terrifying in difficulty to solve.
  - lamontcg 5y ago
    The article did make its points fairly poorly. I think in the end they more or less did what you suggested as well they just didn't call it end to end testing.
mbrodersen 5y ago
Agree. They clearly are lacking in test engineering skills. So the right solution would have been to hire somebody who knows how to do this well. Bad tests are simply bad code. So fix it!

by mabbo 5y ago

lamontcg 5y ago

> Running tests against different infra than your customers have to deal with is asking for trouble. What bugs will exist in the real production infra that won't in your fake infra? We actually do pretty well testing against fake infra. We have a large test suite that enforces the contract on our REST server API. That is implement both in one heavy server written in erlang which is the production code and one lightweight server written in ruby which would never scale but is the same API. When the test suite is updated both the implementations need to be fixed. When the client code runs integration tests we can test against the lightweight ruby code and when it passes we actually have pretty high confidence that it runs against the production code. We have hundreds of those tests and they can be run as fast as spinning up a ruby process with an in-memory datastore which is trashed on every test. Compare that to end to end tests that might fire up a set of images, terraform them into production servers and clients and run a scripted interaction or set of interactions and then throws that away and does it again. At some point there's a tradeoff between the realism of your tests and the cost of them and how many of them you can do. The right strategy is that you want to have enough of the most realistic tests to give you a high level of confidence that your faster, slightly less realistic tests are useful, and by having those faster tests you increase your amount of coverage, and on down the stack iteratively until you may get to unit tests of individual objects.

mabbo 5y ago
> At some point there's a tradeoff between the realism of your tests and the cost of them and how many of them you can do. Yeah, you're not wrong. I'm just griping because of the previous list of complaints. There are some benefits to running your tests against a mirror infra to reality. But they are limited. The bugs they catch that you won't catch running fake infra are very small in number, but terrifying in difficulty to solve.
- lamontcg 5y ago
  The article did make its points fairly poorly. I think in the end they more or less did what you suggested as well they just didn't call it end to end testing.

mbrodersen 5y ago

Agree. They clearly are lacking in test engineering skills. So the right solution would have been to hire somebody who knows how to do this well. Bad tests are simply bad code. So fix it!