5 ms·
Not all code is as simple as an e-mail validator. I don't think anyone would argue that unit testing pure functions like this is so trivial that there is no arg
by gfodor 16y ago
Not all code is as simple as an e-mail validator. I don't think anyone would argue that unit testing pure functions like this is so trivial that there is no argument against doing TDD.
However, there is more to the world than pure functions, and there is more to the world than writing simple CRUD applications.
First, code that is meant as "glue" between disparate systems. For example, a piece of code that pops off a live queue, handles gracefully when the queue is slow or lagged, and then performs some action that has an additional side effect.
The retort here is that you should mock out the queue and whatever other systems. This can only get you so far though. At the end of the day you need integration tests that simulate the entire environment. Often it's not possible to build up and tear down an environment that simulates reality, and additionally it is not always possible to effectively simulate failures or load. In this case, I prefer to spin up EC2 instances with a replica of the real environment and manually test the code with real services under real load. This can be automated fairly well nowadays.
Second, there are classes of algorithms that are highly data-driven, require large amounts of data, and are qualitative in nature. Machine learning algorithms, search algorithms, etc. Building these algorithms usually requires a) a large dataset and b) subjective relevance feedback from a human. TDD is not going to help you here. Sure, you can unit test the pure functions within your algorithm (for example, it's common to compute tf*idf in search algorithms, so test your math there) but the "full stack" test and iteration process for this code generally requires you to manually look at results and make judgements. This isn't something you can just automate in a unit test, because regressions against the input data will be false positives or false negatives, depending on which approach you take to compute a diff :) You can get some headway by looking for large differences in RMSE or other metrics between commits, but at the end of the day your tests will still remain brittle under big improvements to the core algorithms.
Finally, another class of algorithms that aren't amicable to TDD are computer graphics algorithms. The reason is the same: judging correctness generally requires a human. Again, unit test your vector math and so on. But don't expect TDD and unit testing to provide an exhaustive safety net to protect you from introducing subtle bugs in your rendering code. Looking at output is the best way to do that.
- Lewisham 16y agoAbsolutely. Unit testing is just one gun in your automated testing arsenal. When I was working as a web dev and I saw Selenium for the first time, my jaw dropped. I totted up all the hours I'd wasted of my life hitting refresh buttons, because "you can't run automated tests for a browser". How wrong I was! Once you get into those subjective bits, you're in new waters. I think/hope there is some research going on here, using AI-based "critics". A lot of AI uses a generate/test paradigm, where a result is generated and then tested for fitness in some manner. One could imagine that testing in some of these fuzzier fields, like graphics, would be improved by independent AI critics. I had the opportunity to head into AI, and the inability to automatically test my algorithms and say "this really works" versus "oh no, it's broken" drove me crazy. However, this uncertainty is what the AI guys loved :)
- tel 16y agoI disagree (greatly) about testing machine learning code. I test ruthlessly my pure functions and even full stack stuff, but of course my full-data experiments require interpretation. The difference is huge! Without very solid separation between those two phases, and then without very solid actual testing, you cannot achieve trust or repeatability in your experiments. It's actually turning into a pretty interesting problem (that I've patched together solutions using Fabric and Org-mode) to track experiments to versions of code while maintaining good testing practices.
- IgorPartola 16y agoAgreed. Code that retains state does not lend itself well to unit testing. One of my latest projects at work was a MySQL node monitor with automatic failover. I did add unit tests to it, but I can tell you that I found more bugs, or rather quirks of MySQL's replication behavior and the MySQLdb driver through doing my manual testing. I also spent quite a bit of time fixing the test rather than the code, since the quirks meant I needed different behavior. As this piece of code is quite important to us, the ROI on it will be high. Then again the investment was sizable as well. I think unit testing should be viewed in terms of ROI because, just like premature optimization, often times a lot of effort goes into it where the return is negligible. Pure functions can be easily tested, so test your critical calculations every time. On the other end of the spectrum you have things that interact with external systems. Do you unit test an email notification function? How about a CSS layout? Can you easily test for an out of memory condition or a failed hard disk? Sometimes it is just cheaper to use human judgement. Lastly, as many have pointed out, unit testing is not a silver bullet. Just because you did not break the unit tests does not meant the code still functions properly.