3 ms·
Would you fly an airline that did 24 hours of verification before every flight thus making flights cost 10x as much? (and same argument for all your other examp
by puredanger 13y ago
Would you fly an airline that did 24 hours of verification before every flight thus making flights cost 10x as much? (and same argument for all your other examples...)
Our job as professionals is to verify that the software we write does what we expect. Our job is also to manage costs appropriate to the application (not just up-front but over the long haul). For me, that means that we want our tests to maximize value per effort.
I am not saying a) write fewer tests b) tests are a waste of time or c) tests are too expensive. I'm saying: a) write better tests, b) some tests are a waste of time and c) let's write tests that do more work for us.
- pbiggar 13y ago[disclaimer: I work at https://circleci.com https://circleci.com, so I do have a vested interest in disagreeing with you ;) ] I think is a strawman. If running a huge suite of automated verification was really cheap, then airlines would run it before each flight.
- puredanger 13y agoHey Paul, I totally agree that the cost of running big test suites can be mitigated and CircleCI is doing awesome stuff there. That does nothing to address the problem of maintaining a large suite of tests tightly coupled to your code. That is, the cost of updating many example-based tests when you change the code.
- pbiggar 13y agoI guess that depends on how much of your suite is in flux. We have largely example-driven tests, and our suite is pretty big: about 16kLOC. I can't think of a case where a significant amount of time was spent fixing the test suite (and in those cases, it was generally useful to think about why). Also, I haven't seen anything that works better. Generative tests are nice in theory, but I haven't seem the be that useful for anything other than short, pure, functions.
- jamii 13y agoI've built generative tests for an entire betting exchange. We run the database in-memory, throw generated events in one end and make assertions about the state when processing is finished. The tests range from very specific, narrow pseudo-unit-tests to coarse properties like 'if we process 1000 random orders from 80 different customers in 3 different currencies, the total amount of money in the system remains constant'. I can't imagine reaching the same level of stability in the core exchange if we had to write all the test cases by hand.