3 ms·

Now, I've never worked at a big company with lots of developers, but testing seems dramatically overvalued. My company currently employs zero testing (meaning z

Now, I've never worked at a big company with lots of developers, but testing seems dramatically overvalued. My company currently employs zero testing (meaning zero automated tests). Anything that could negatively impact the company if it blew up is examined pretty closely and then set loose. Occasionally things break. Our company serves tens of millions of users monthly. We have bugs, we fix them as needed. We're not writing banking software so the risk profile is pretty small, but I think more companies fall into this category than test engineers would have you believe. I'd give myself 2 more years in the industry if I had to do TDD, or anything else that strives for even moderate coverage.

jeffbee 5y ago
I am a huge fan of unit testing and the best thing about TDD is getting your function signatures and other APIs sorted out before writing your implementation. Having to write the calling function first helps ensure that the called function has a signature that will be useful to other callers, instead of something awkward that seemed OK when you started writing the implementation but later turned out to be imperfect. Also, TDD wards off the problem of people who write untestable code. A lot of functions simply cannot be tested. TDD eliminates that problem.
- lwansbrough 5y ago
  The notion that you can get software right the first time seems... a bit naive to me, regardless of the methodology, respectfully. We solved the internal API problems in our app simply by either versioning internal services/APIs (ie. MyServiceClassV2), adding a new method signature and deprecating the old one, or updating the existing method signature, which is pretty safe in strictly typed languages. (And if you write code that uses reflection, you get to be on call when it breaks.)
  - jeffbee 5y ago
    I find your comments quite interesting. How do you validate MyServiceClassV2 if you didn't bother writing any tests for MyServiceClass? One of the benefits of testing is it enables refactoring. Without tests, code bases just become increasingly haunted graveyards where nobody is willing to change anything.
    - lwansbrough 5y ago
      It's not really about validation. Either MyServiceClass works or it doesn't (whether that's DX, or some other problem), and if it doesn't you make a new one or fix it. Most code isn't so complicated that coming up with the correct method signature is prohibitively difficult without writing tests. If down the line the way it was written isn't working, maybe you need a V2. Anything important that needs updating to V2 gets updated, anything not worth the effort stays on V1. Usually that's not an issue. Sometimes you have to drop V1 for one reason or another, and that can result in some work, but I reckon it's time saved vs. writing test suites, and it always feels worthwhile to do, vs. writing insurance code. > Without tests, code bases just become increasingly haunted graveyards where nobody is willing to change anything. People will be fine with changing things if the cultural standard is that bugs can happen but just try to avoid them. As long as you have good logging it's largely a non-issue. I'd be more hesitant to change code if I had to update 10 tests along the way. Maybe that's laziness but it also feels wildly unproductive.
      - exdsq 5y ago
        It really depends on what your building, how quickly you need to build it, and what the client cares about. It could be no-one cares if MyServiceClass has a small bug that causes it to crash and restart from time to time - I worked in a startup that had something like that and it was fine within the needs of the company. Nowadays I work on financial stuff and a crash could cost ££££ so it's unacceptable, meaning we need more testing to check stuff. Also with interfaces, they matter more if your working in a large distributed team. Sometimes you need to have the method signatures designed upfront so other teams can work on it, meaning it's less trivial than say an internal subroutine used in a private class.
kstrauser 5y ago
I don't like TDD. A lot of times I don't know what the code will look like until I write it, and whatever's on the screen is just a scratchpad for my thoughts until the final form becomes clear to me. That said, I love having a good test suite. I updated a major internal project from Python 2 to 3 and my metric for being done was whether all the tests passed. When they did, we launched it in production, and it worked without serious bugs from the very beginning. I can't even imagine what that process would've looked like without some reassurance that I was catching at least most of the corner cases we'd previously identified. Also, I love tests as a way of asserting that a bug or security issue stays fixed. In fact, I doubt it's possible to pass something like a SOC 2 audit without it.
jrockway 5y ago
The problem is that you're missing guard rails. People know they have to be paranoid, so they move slowly; just like people drive more carefully without seatbelts and ride bicycles more carefully without helmets. Care is good, but does not come without cost. I think it's hard to measure the impact of not testing, because in order to compare A vs B, you'll have to write tests (and it's a large job to write a test suite for existing code), but that's why it's insidious -- you know writing tests is going to slow you down, and you don't know how much time it's going to save in the long run. You end up with a heavy bias for the status quo. And, if f the tests aren't any good, you'll spend a lot of time writing them and you'll just get friction on future changes, slowing you down even more. But if the tests are good, then people that are new to the codebase can arrive with confidence. Manual testing is good, but it scales O(n) with the number of features. What works for the simple prototype with two big features quickly becomes a drag when every engineer has to test all 100 features against every change they make. (That's why organizations make some other team do the tests, and then the bug comes back to you 3 weeks later, and then you stop what you're working on to fix the bug you caused -- context switching off of new feature work, and delaying that feature. Slow, and annoying!) The solution to this slowdown is usually more process (if we just write it down, it feels like it's not work) and "we need more engineers". The associated O(1) overhead of new proces plus the O(n!) communication slowdown means that the n in n features to test for every change grows much more slowly -- it's feels under control, but what you're really doing is less work with more resources. I look at the tricky edge cases in my own work, and look at where the bugs creep in (we do a quarterly review of these), and it's always in the "that's too hard to test" code. Some examples: assuming that the test suite's view of the database is the same as a database without the new migrations applied, simple refactoring leading to null pointer dereferences in an edge case, third party applications that call into the API that "has no users" (according to grep), and things like that. These are the things that burn new team members, and make them overly cautious forever. Caution and velocity are incompatible, so to me, it's crucial that these dark corners get addressed, so a passing test suite means a codebase with only bugs we haven't seen before. (There are always going to be bugs, but you shouldn't fix the same bug twice.) (Oh, and there are definitely bugs in code with 100% coverage. 100% coverage just means you found the bugs you already thought of, but it doesn't mean you found every bug. A test suite will never ensure your code is bug free.) There are a lot of open source projects that lose their primary maintainer, and they never get another feature again because of this. Someone wants to add one, but they can't figure out how, and just rewrite it, or give up completely. Be on the lookout!

jeffbee 5y ago

I am a huge fan of unit testing and the best thing about TDD is getting your function signatures and other APIs sorted out before writing your implementation. Having to write the calling function first helps ensure that the called function has a signature that will be useful to other callers, instead of something awkward that seemed OK when you started writing the implementation but later turned out to be imperfect. Also, TDD wards off the problem of people who write untestable code. A lot of functions simply cannot be tested. TDD eliminates that problem.

lwansbrough 5y ago
The notion that you can get software right the first time seems... a bit naive to me, regardless of the methodology, respectfully. We solved the internal API problems in our app simply by either versioning internal services/APIs (ie. MyServiceClassV2), adding a new method signature and deprecating the old one, or updating the existing method signature, which is pretty safe in strictly typed languages. (And if you write code that uses reflection, you get to be on call when it breaks.)
- jeffbee 5y ago
  I find your comments quite interesting. How do you validate MyServiceClassV2 if you didn't bother writing any tests for MyServiceClass? One of the benefits of testing is it enables refactoring. Without tests, code bases just become increasingly haunted graveyards where nobody is willing to change anything.
  - lwansbrough 5y ago
    It's not really about validation. Either MyServiceClass works or it doesn't (whether that's DX, or some other problem), and if it doesn't you make a new one or fix it. Most code isn't so complicated that coming up with the correct method signature is prohibitively difficult without writing tests. If down the line the way it was written isn't working, maybe you need a V2. Anything important that needs updating to V2 gets updated, anything not worth the effort stays on V1. Usually that's not an issue. Sometimes you have to drop V1 for one reason or another, and that can result in some work, but I reckon it's time saved vs. writing test suites, and it always feels worthwhile to do, vs. writing insurance code. > Without tests, code bases just become increasingly haunted graveyards where nobody is willing to change anything. People will be fine with changing things if the cultural standard is that bugs can happen but just try to avoid them. As long as you have good logging it's largely a non-issue. I'd be more hesitant to change code if I had to update 10 tests along the way. Maybe that's laziness but it also feels wildly unproductive.
    - exdsq 5y ago
      It really depends on what your building, how quickly you need to build it, and what the client cares about. It could be no-one cares if MyServiceClass has a small bug that causes it to crash and restart from time to time - I worked in a startup that had something like that and it was fine within the needs of the company. Nowadays I work on financial stuff and a crash could cost ££££ so it's unacceptable, meaning we need more testing to check stuff. Also with interfaces, they matter more if your working in a large distributed team. Sometimes you need to have the method signatures designed upfront so other teams can work on it, meaning it's less trivial than say an internal subroutine used in a private class.

kstrauser 5y ago

I don't like TDD. A lot of times I don't know what the code will look like until I write it, and whatever's on the screen is just a scratchpad for my thoughts until the final form becomes clear to me. That said, I love having a good test suite. I updated a major internal project from Python 2 to 3 and my metric for being done was whether all the tests passed. When they did, we launched it in production, and it worked without serious bugs from the very beginning. I can't even imagine what that process would've looked like without some reassurance that I was catching at least most of the corner cases we'd previously identified. Also, I love tests as a way of asserting that a bug or security issue stays fixed. In fact, I doubt it's possible to pass something like a SOC 2 audit without it.

jrockway 5y ago

The problem is that you're missing guard rails. People know they have to be paranoid, so they move slowly; just like people drive more carefully without seatbelts and ride bicycles more carefully without helmets. Care is good, but does not come without cost. I think it's hard to measure the impact of not testing, because in order to compare A vs B, you'll have to write tests (and it's a large job to write a test suite for existing code), but that's why it's insidious -- you know writing tests is going to slow you down, and you don't know how much time it's going to save in the long run. You end up with a heavy bias for the status quo. And, if f the tests aren't any good, you'll spend a lot of time writing them and you'll just get friction on future changes, slowing you down even more. But if the tests are good, then people that are new to the codebase can arrive with confidence. Manual testing is good, but it scales O(n) with the number of features. What works for the simple prototype with two big features quickly becomes a drag when every engineer has to test all 100 features against every change they make. (That's why organizations make some other team do the tests, and then the bug comes back to you 3 weeks later, and then you stop what you're working on to fix the bug you caused -- context switching off of new feature work, and delaying that feature. Slow, and annoying!) The solution to this slowdown is usually more process (if we just write it down, it feels like it's not work) and "we need more engineers". The associated O(1) overhead of new proces plus the O(n!) communication slowdown means that the n in n features to test for every change grows much more slowly -- it's feels under control, but what you're really doing is less work with more resources. I look at the tricky edge cases in my own work, and look at where the bugs creep in (we do a quarterly review of these), and it's always in the "that's too hard to test" code. Some examples: assuming that the test suite's view of the database is the same as a database without the new migrations applied, simple refactoring leading to null pointer dereferences in an edge case, third party applications that call into the API that "has no users" (according to grep), and things like that. These are the things that burn new team members, and make them overly cautious forever. Caution and velocity are incompatible, so to me, it's crucial that these dark corners get addressed, so a passing test suite means a codebase with only bugs we haven't seen before. (There are always going to be bugs, but you shouldn't fix the same bug twice.) (Oh, and there are definitely bugs in code with 100% coverage. 100% coverage just means you found the bugs you already thought of, but it doesn't mean you found every bug. A test suite will never ensure your code is bug free.) There are a lot of open source projects that lose their primary maintainer, and they never get another feature again because of this. Someone wants to add one, but they can't figure out how, and just rewrite it, or give up completely. Be on the lookout!