5 ms·

I think your argument is missing the "cost" of feature tests, which is that they necessarily must run much slower than tests that are only testing a small unit

I think your argument is missing the "cost" of feature tests, which is that they necessarily must run much slower than tests that are only testing a small unit of code. For example, I worked on payroll software at some point. After finding a bug, we'd want to ensure that could never happen again, and would want to add a test for it going forward. So for example, I may have needed to write a test around someone who worked in one city in Ohio, lived in another, previous income above some amount, and with certain tax advantaged benefits. Setting up this test via feature testing is certainly possible, but it's likely the test itself will take a significant amount of wall time to execute. It's way faster to just test the payroll calculation code, which means you can run the tests more often, and with less developer inconvenience.

t0mbstone 8y ago
This is why you actually need a mix of feature tests and unit tests. Have automated feature tests that test most of the "happy path" run-throughs of features that your users do on the front end, and then unit tests that test the minutiae. My favorite project that I ever worked on was one that was set up this way. We had literally hundreds of feature tests, and thousands of unit tests. When running in a single thread on a local machine, it would take two hours, but when running on Circle CI with parallelization, it would run the entire suite in 6 minutes. This enabled us to release features to production with a high amount of confidence, any time we wanted. It was not uncommon for us to release bug fixes to production, while still on the phone with the complaining customer. We earned a lot of customer loyalty points any time we pulled that off. And the best part was that because we had the massive feature test suite running as part of of CI/CD, we were able to do that with the same amount of confidence that we would have compared to having a massive QA team with 50 people testing our complete app before every deployment. It was awesome.
- GordonS 8y ago
  This is my preference too - mainly feature tests, and unit and integration tests where they provide enough value to be worthwhile. I generally end up with 70-75% test coverage, but I don't pay too much attention to the actual number. We just released a fairly large system that took this testing approach - despite a large number of tests hitting a database, file system and a blob storage emulator, they still completed in only 5 minutes or so on our CI machine (or 1 minute locally on very beefy laptops).
austincheney 8y ago
> but it's likely the test itself will take a significant amount of wall time to execute. Why? What about your application changed so that it is slower when testing compared to real world use? If anything it should be dramatically faster because people don't provide microsecond accurate automated responses. If the application naturally executes very quickly I would imagine it would take far long to set up the test scenario than to execute against it. In my own applications if they take more than two seconds to deliver a response (even for 5mb input) then at the very least I have a critical performance defect. There are not many administrative tasks I can complete from start to finish in that frame of time.
- rkangel 8y ago
  I'll give you one real example: because my application is some embedded software that has to be built, downloaded onto the target and then run. That is the only environment the real application runs in. Some tests should be run at this level, but the cost to automate and the benefit means that you usually don't try and get full coverage. You can build a desktop version of the software with hardware mocked out, which is very useful, but then you're doing something much more akin to an integration test.
  - austincheney 8y ago
    Unless the software is firmware or an OS it rarely directly touches the hardware. It only knows of the hardware available because the firmware or OS provide that information. I have seen a lot of organizations fake this with an array of VMs or when that is not enough they have a bunch of spare hardware to test against, like a whole bunch of different models of Android phones.
- Macha 8y ago
  We have 1000s of features. Real users don't use every feature every time they perform an action. A test case at the feature level will likely require several requests if its starting from clean state.
  - 8y ago

t0mbstone 8y ago

This is why you actually need a mix of feature tests and unit tests. Have automated feature tests that test most of the "happy path" run-throughs of features that your users do on the front end, and then unit tests that test the minutiae. My favorite project that I ever worked on was one that was set up this way. We had literally hundreds of feature tests, and thousands of unit tests. When running in a single thread on a local machine, it would take two hours, but when running on Circle CI with parallelization, it would run the entire suite in 6 minutes. This enabled us to release features to production with a high amount of confidence, any time we wanted. It was not uncommon for us to release bug fixes to production, while still on the phone with the complaining customer. We earned a lot of customer loyalty points any time we pulled that off. And the best part was that because we had the massive feature test suite running as part of of CI/CD, we were able to do that with the same amount of confidence that we would have compared to having a massive QA team with 50 people testing our complete app before every deployment. It was awesome.

GordonS 8y ago
This is my preference too - mainly feature tests, and unit and integration tests where they provide enough value to be worthwhile. I generally end up with 70-75% test coverage, but I don't pay too much attention to the actual number. We just released a fairly large system that took this testing approach - despite a large number of tests hitting a database, file system and a blob storage emulator, they still completed in only 5 minutes or so on our CI machine (or 1 minute locally on very beefy laptops).

austincheney 8y ago

> but it's likely the test itself will take a significant amount of wall time to execute. Why? What about your application changed so that it is slower when testing compared to real world use? If anything it should be dramatically faster because people don't provide microsecond accurate automated responses. If the application naturally executes very quickly I would imagine it would take far long to set up the test scenario than to execute against it. In my own applications if they take more than two seconds to deliver a response (even for 5mb input) then at the very least I have a critical performance defect. There are not many administrative tasks I can complete from start to finish in that frame of time.

rkangel 8y ago
I'll give you one real example: because my application is some embedded software that has to be built, downloaded onto the target and then run. That is the only environment the real application runs in. Some tests should be run at this level, but the cost to automate and the benefit means that you usually don't try and get full coverage. You can build a desktop version of the software with hardware mocked out, which is very useful, but then you're doing something much more akin to an integration test.
- austincheney 8y ago
  Unless the software is firmware or an OS it rarely directly touches the hardware. It only knows of the hardware available because the firmware or OS provide that information. I have seen a lot of organizations fake this with an array of VMs or when that is not enough they have a bunch of spare hardware to test against, like a whole bunch of different models of Android phones.
Macha 8y ago
We have 1000s of features. Real users don't use every feature every time they perform an action. A test case at the feature level will likely require several requests if its starting from clean state.
- 8y ago