2 ms·

E2E tests are required because no matter how well-defined your other tests are or how completely they've tested everything . . . you can't prove that they'd abs

E2E tests are required because no matter how well-defined your other tests are or how completely they've tested everything . . . you can't prove that they'd absolutely catch all the bugs. https://en.wikipedia.org/wiki/Argument_from_ignorance https://en.wikipedia.org/wiki/Argument_from_ignorance Kudos to Nubank for whatever combination of logic and bravery led them to their decision.

deleted 5y ago
[deleted]
bryan_w 5y ago
Catching all the bugs is generous. You won't be able to prove...basic functionally when all the components are deployed in production
coryrc 5y ago
E2E can't catch all bugs either. This team decided the number of bugs their test suite caught was not enough to be worth keeping it. With a robust canary deployment, they will quickly find and rollback breakages whether or not the e2e suite would have caught it.
jniedrauer 5y ago
E2E tests are still extremely limited and let bugs through, unless they also fuzz somehow. But that will make them even more flaky and difficult to debug, costing more time. It's a tradeoff. Example: You run some number of operations and then batch them. If you run the exact same operations each time you test, you may not catch conflicts between them. Instead, you'd have to run a random number of, and type of, operations. But then test failures would become extraordinarily difficult to reproduce. You'd have to hope that your tests log exactly what the inputs were, and have a semi-efficient way to recreate those inputs locally.
dgb23 5y ago
Reading the article beyond the title reveals that this decision was engineering driven, measured and they ended up with a simpler, disentangled solution, by separating contracts that verify compatible schemas on one side and acceptance tests on the other side.
NewEntryHN 5y ago
Which can't be proven by E2E either, and which you don't need to prove.
mbrodersen 5y ago
I don’t think it was bravery. I think it was lack of end-to-end test engineering skills. If your tests are flaky then do what you would do with any other software: fix it! If your end-to-end tests are slow then do what you would do with any other software: optimise it! Not having proper end-to-end tests is a massive red flag.

deleted 5y ago

[deleted]

bryan_w 5y ago

Catching all the bugs is generous. You won't be able to prove...basic functionally when all the components are deployed in production

coryrc 5y ago

E2E can't catch all bugs either. This team decided the number of bugs their test suite caught was not enough to be worth keeping it. With a robust canary deployment, they will quickly find and rollback breakages whether or not the e2e suite would have caught it.

jniedrauer 5y ago

E2E tests are still extremely limited and let bugs through, unless they also fuzz somehow. But that will make them even more flaky and difficult to debug, costing more time. It's a tradeoff. Example: You run some number of operations and then batch them. If you run the exact same operations each time you test, you may not catch conflicts between them. Instead, you'd have to run a random number of, and type of, operations. But then test failures would become extraordinarily difficult to reproduce. You'd have to hope that your tests log exactly what the inputs were, and have a semi-efficient way to recreate those inputs locally.

dgb23 5y ago

Reading the article beyond the title reveals that this decision was engineering driven, measured and they ended up with a simpler, disentangled solution, by separating contracts that verify compatible schemas on one side and acceptance tests on the other side.

NewEntryHN 5y ago

Which can't be proven by E2E either, and which you don't need to prove.

mbrodersen 5y ago

I don’t think it was bravery. I think it was lack of end-to-end test engineering skills. If your tests are flaky then do what you would do with any other software: fix it! If your end-to-end tests are slow then do what you would do with any other software: optimise it! Not having proper end-to-end tests is a massive red flag.