3 ms·

It still seems like the core of your argument is "Some things are hard to test, so you might as well not test anything at all" - which I really don't buy. > Ho

It still seems like the core of your argument is "Some things are hard to test, so you might as well not test anything at all" - which I really don't buy. > How about you just answer me this question: Did you still see significant bug volumes after implementing the unit tests for the FileProcessor ? Kinda tricky to answer this, since there were test for this class from the start. But, overall the answer is "no" - we did not see significant bugs in that class. Occasionally we'd get a bug because the SFTP connection failed, or execution took to long and the connection closed - the type of bug that to you seems to negate the value of uniting testing the FileProcessor. But, without the unit tests for the FileProcessor, I'd have those bugs plus more bugs/bad behavior in the business logic. How is that better, exactly? The idea that tests reduce performance is ridiculous. Improving performance requires making changes to a system while ensuring it's behavior hasn't changed. This is exactly what testing provides. Without tests, you can't optimize the code at all without fear of introducing regressions.

candiodari 8y ago
> It still seems like the core of your argument is "Some things are hard to test, so you might as well not test anything at all" Nope. My argument is twofold: 1) unit tests don't actually provide the guarantees that people keep putting forward as reasons to write them 2) this makes them a dangerous, as they provide "reassurance" that isn't actually backed by reality > But, without the unit tests for the FileProcessor, I'd have those bugs plus more bugs/bad behavior in the business logic. How is that better, exactly? So the tests failed to catch problems with the program's behavior. Maybe it's just me, but I call that a problem. Testing the business logic as a whole is not a unit test, except in the marginal cases where all your business logic fits neatly in a single function, or at least a single class. If it's actually a few algorithms, a few files, a bunch of functions and you test everything together, that's an integration test and not a functional test. If you use actual (or slightly changed) data to test that business logic, as opposed to artificially crafted data, that again makes it not a unit test. > The idea that tests reduce performance is ridiculous If you use tests to optimize code you're optimizing a piece of code in isolation, without taking the rest of the system into account. That works for trivially simple initial optimization, but falls completely on it's face when you're actually writing programs that stress the system. > Improving performance requires making changes to a system while ensuring it's behavior hasn't changed. The system as a whole, sort of. Individual units ? Absolutely not. Besides, tests merely provide FALSE assurance behavior hasn't changed. Time and time again I've had to fix "I've optimized it" bugs. TLDR is always the same. Unit tests pass so the code "must be right" (meaning they don't run the code outside of unit tests, and the unit tests only directly test the code, not blackbox). Then in the actual run edge cases were hit. And "edge cases" makes it sound like you hardly ever hit this. Just yesterday we had a big issue. What happened ? Well we had a method that disables an inbound phone line (e.g. for maintenance, or changes). All unit tests passed. Unfortunately we really should have tested that it does NOT disable anything else (method was essentially "go through list, if it matches, disable". Needless to say, it disabled every line). Regression testing added. We had someone optimize the dial plan handling. He didn't realize that his "cache" was in fact recreated at the very beginning of every request, when the code evaluated a dial plan, in other words, it was a serious performance regression rather than an improvement. Really looked like an improvement though. Unit tests ... passed. Of course, "the behavior hadn't changed". About 2 calls got through for ~6 minutes (normally thousands). Now we have a test that turns up the whole system, with the actual production dial plan, and then goes through 10000 calls. If it takes more than 1 minute, test fails. Developers hate it, but it's non-negotiable at this point. I can go on for quite a while enumerating problems like this. Billing files erased (forgot to append). Entire dialplan erased on startup (essentially same problem). Lots of ways to trigger cpu starvation. Memory exhaustion. Memory leaks (the system is mostly Java). Connecting unrelated calls together (that was pretty fun for the first 2-3 minutes). Ignoring manager signals (one server having to do all calls ... which is not happening) This is a repeating pattern in our failures. Unit tests ... essentially always pass. And time and time again someone tries to make the point that this must mean the code is correct. And that's bullshit. That's the problem.
- fredifrum 8y ago
  OK, I get that you don't like unit tests. You also seem to have a very strict and unhelpful definition of what a unit test is. I don't really care for the academia of it, I just want to know that my code works. So, if you call it a unit test or an integration test or a functional test or a behavioral test - whatever. The important thing that it allows me get some idea of whether or not the code is doing what it's supposed to do. What do you propose instead of testing? Manually QA every single piece of the system for every change? Not a lot of companies have the headcount for a the fleet of QA people that would require.
  - mercer 8y ago
    For what it's worth I think you both make great points. Whomever one agrees with perhaps mostly hinges on whether unit tests can get in the way of integration tests. I'm inclined to believe that some of the exhaustive unit testing culture can provide a false sense of security, and the work involved to write them can get in the way of writing proper integration tests (or 'degrees of' if you see it as a spectrum). Provided that proper test coverage includes both integration tests and unit tests, it probably won't hurt to do both. I like unit tests as part of my process (the red-green TDD process can be quite addictive and useful), but under pressure I'd prioritize integration tests, knowing that the more time I spend unit testing, the less I'll spend integration testing.

candiodari 8y ago

> It still seems like the core of your argument is "Some things are hard to test, so you might as well not test anything at all" Nope. My argument is twofold: 1) unit tests don't actually provide the guarantees that people keep putting forward as reasons to write them 2) this makes them a dangerous, as they provide "reassurance" that isn't actually backed by reality > But, without the unit tests for the FileProcessor, I'd have those bugs plus more bugs/bad behavior in the business logic. How is that better, exactly? So the tests failed to catch problems with the program's behavior. Maybe it's just me, but I call that a problem. Testing the business logic as a whole is not a unit test, except in the marginal cases where all your business logic fits neatly in a single function, or at least a single class. If it's actually a few algorithms, a few files, a bunch of functions and you test everything together, that's an integration test and not a functional test. If you use actual (or slightly changed) data to test that business logic, as opposed to artificially crafted data, that again makes it not a unit test. > The idea that tests reduce performance is ridiculous If you use tests to optimize code you're optimizing a piece of code in isolation, without taking the rest of the system into account. That works for trivially simple initial optimization, but falls completely on it's face when you're actually writing programs that stress the system. > Improving performance requires making changes to a system while ensuring it's behavior hasn't changed. The system as a whole, sort of. Individual units ? Absolutely not. Besides, tests merely provide FALSE assurance behavior hasn't changed. Time and time again I've had to fix "I've optimized it" bugs. TLDR is always the same. Unit tests pass so the code "must be right" (meaning they don't run the code outside of unit tests, and the unit tests only directly test the code, not blackbox). Then in the actual run edge cases were hit. And "edge cases" makes it sound like you hardly ever hit this. Just yesterday we had a big issue. What happened ? Well we had a method that disables an inbound phone line (e.g. for maintenance, or changes). All unit tests passed. Unfortunately we really should have tested that it does NOT disable anything else (method was essentially "go through list, if it matches, disable". Needless to say, it disabled every line). Regression testing added. We had someone optimize the dial plan handling. He didn't realize that his "cache" was in fact recreated at the very beginning of every request, when the code evaluated a dial plan, in other words, it was a serious performance regression rather than an improvement. Really looked like an improvement though. Unit tests ... passed. Of course, "the behavior hadn't changed". About 2 calls got through for ~6 minutes (normally thousands). Now we have a test that turns up the whole system, with the actual production dial plan, and then goes through 10000 calls. If it takes more than 1 minute, test fails. Developers hate it, but it's non-negotiable at this point. I can go on for quite a while enumerating problems like this. Billing files erased (forgot to append). Entire dialplan erased on startup (essentially same problem). Lots of ways to trigger cpu starvation. Memory exhaustion. Memory leaks (the system is mostly Java). Connecting unrelated calls together (that was pretty fun for the first 2-3 minutes). Ignoring manager signals (one server having to do all calls ... which is not happening) This is a repeating pattern in our failures. Unit tests ... essentially always pass. And time and time again someone tries to make the point that this must mean the code is correct. And that's bullshit. That's the problem.

fredifrum 8y ago
OK, I get that you don't like unit tests. You also seem to have a very strict and unhelpful definition of what a unit test is. I don't really care for the academia of it, I just want to know that my code works. So, if you call it a unit test or an integration test or a functional test or a behavioral test - whatever. The important thing that it allows me get some idea of whether or not the code is doing what it's supposed to do. What do you propose instead of testing? Manually QA every single piece of the system for every change? Not a lot of companies have the headcount for a the fleet of QA people that would require.
- mercer 8y ago
  For what it's worth I think you both make great points. Whomever one agrees with perhaps mostly hinges on whether unit tests can get in the way of integration tests. I'm inclined to believe that some of the exhaustive unit testing culture can provide a false sense of security, and the work involved to write them can get in the way of writing proper integration tests (or 'degrees of' if you see it as a spectrum). Provided that proper test coverage includes both integration tests and unit tests, it probably won't hurt to do both. I like unit tests as part of my process (the red-green TDD process can be quite addictive and useful), but under pressure I'd prioritize integration tests, knowing that the more time I spend unit testing, the less I'll spend integration testing.