5 ms·

Yes, but when have you ever encountered a large company where unit tests are judged by what needs testing as opposed to "coverage" ? There is almost no use for

by iofj 11y ago

Yes, but when have you ever encountered a large company where unit tests are judged by what needs testing as opposed to "coverage" ? There is almost no use for unit tests, as they lock down function implementations without verifying functionality. They have some amount of use for a programmer to verify if what they wrote really is what they wrote, and maybe for data structure methods (not in Go of course), but that means maybe 1 in 50 methods justifies a unit test. Everything else should be system tested, to see if the components fit together and if the interactions between various parts of the application really are what you think they are. It's like javadoc back in the day. "Document all your function variables, you've only got them for x%", ... you write a parser that parses the app and simply adds javadoc everywhere, giving the obvious descriptions to obvious names. 2 hours of work for the generator, but actually interesting work, half an hour to read through it and change a few things, and boom, 30000 lines added in a day. And the worst part is, you've just made everything harder to read, but everybody's happy with you.

winstonewert 11y ago
It sounds to me like you need to stop abusing mocks in your tests. I usually find such tests result from using mocks poorly.
eropple 11y ago
> There is almost no use for unit tests, as they lock down function implementations without verifying functionality. This is...curious, given that every place I've ever been with any sort of testing culture tested contracts, not function implementations. They test the range and domain of the function, not the behavior of it. Unit tests do not replace system-level/integration testing, but to dismiss them out of hand is manifestly unwise.
- ori_b 11y ago
  Generally, I see a lot of testing of internal functions, which often should change their contract as refactoring happens. All too rarely do I see people focusing testing on public APIs.
  - eropple 11y ago
    It depends on your definition of "internal". I write unit tests at whatever boundary that makes sense, and I certainly will write unit tests to test contracts between related pieces that others might call "internal." But then I can refactor trivially at that level, which I may (read: usually do) want to have available later. I don't generally write, nor really see, see a lot of internal contractual testing in unit testing (not least because it becomes very hard to do so if you actually use visibility in OOP).
andrewguenther 11y ago
> There is almost no use for unit tests, as they lock down function implementations without verifying functionality. I would disagree. You can change the implementation all you want, but you cannot change the functionality (contract). If your unit test verifies implementation rather than functionality, then it is a poor test.
bluejekyll 11y ago
I'm sorry, but what?! Unit testing is not an exercise in the mundane by making your manager happy, it is about validating that the thing you just wrote actually works, so that you can move on to the next thing, eventually aggregating all of that work together and constructing code that works! As others have said, you test inputs and outputs (and maybe you need to test some failure modes depending on the complexity of the situation). When you're working on a parser, or lexical analyzer of any sort, wouldn't it be nice to know that it is capable of parsing what you thought it should? System/BlackBox/Integration tests are too high-level and the more you rely on those as your sole testing, the more brittle and difficult you make it to track down the cause of test failures. It also means that you have to spend more time constructing test scenarios to testing the inner-workings of the system, things that are easier to do in unit tests. Bloated tests at that layer actually make it MORE difficult to refactor. Decent unit tests help you define the inputs and outputs of your functions/classes/design. If it's hard to write a unit test on it, then you probably have side-effects in the code that you can't properly account for (bad code design). When it comes to a point that you need to refactor and the unit test is in your way, then guess what? refactor/delete the test...
- twic 11y ago
  There's an old and frustrating terminology problem here. You mention testing a parser; assuming the parser contains more than one class, or group of functions, i would consider a test of the whole parser to be an integration test, not a unit test.
  - bluejekyll 11y ago
    By parser I mean any small thing that might parse text. Since I tend to deal with distributed systems, I generally think in terms of blackbox tests as dealing with application APIs rather than function APIs, and integration tests as multiple system components being tested together.
- iofj 11y ago
  > As others have said, you test inputs and outputs (and maybe you need to test some failure modes depending on the complexity of the situation). When you're working on a parser, or lexical analyzer of any sort, wouldn't it be nice to know that it is capable of parsing what you thought it should? That would be a system test. Nobody would write a parser in one function, so it's full functionality can't be tested in a unit test. This is also exactly the sort of test I'm advocating. The thing I'll guarantee you though : it won't provide 100% code coverage. Unit tests are this kind of crap, testing trivial things 100 times. You immediately see why they provide 100% code coverage : https://en.wikipedia.org/wiki/Unit_testing#Design https://en.wikipedia.org/wiki/Unit_testing#Design > the more brittle and difficult you make it to track down the cause of test failures Yes ! This of course happens because they find ever-more subtle bugs, not because there's something wrong with the test.
NathanKP 11y ago
You should consider testing to be a pyramid: 1) A smallish number of full end to end integration tests which actually make TCP requests to your app, and which actually have service talk to each other, etc. 2) A medium number of black box integration tests that just make HTTP requests to a single isolated service or component and verify that the response comes back out as expected (with no expectations on what happens in between input and output). 3) A much larger number of unit tests that are direct execution of business logic (just calling the code directly instead of calling the server externally) to verify various edge conditions in the inner underlying code. To use a real world example you might have full end to end integration tests to ensure that your service allows the creation of a new account, and that when the account is created an email is dispatched, and that an authentication cookie is set which gives this new user the ability to make requests to the service. This is just a high level test though. To back this test up you should have at least a couple hundred unit tests verifying that various edge cases with email formats, name formats, unicode characters in input, etc are handled correctly. To speed up test suite execution time it doesn't make sense to run each of these hundreds of edge cases as full end to end integration tests, or you end up with a horrific test suite that will take minutes to run. So instead you should use unit tests to cover the underlying edge conditions. You can usually run a hundred unit tests in the amount of time that a single integration test will run.
- twic 11y ago
  > You should consider testing to be a pyramid: Why? I've heard this story a million times. Nobody's ever been able to robustly justify it to me. I don't believe it's true.
  - olefoo 11y ago
    If you don't have a test for a given item of functionality; how do you know it works? If you don't have a test for your subsystem that covers the more common use cases; how will you tell if it broke when you updated one of the functions/methods it uses? If you don't have a test for individual flows at the level of the entire system how will you know when you did something that broke the build outright? If you do none of that testing; you're handing off that function of your development team to the users of the system you're building. Testing is an engineering practice that lets you make changes to complex systems with confidence that the system is still doing it's job after the changes you make. It's not magic; it's engineering. As such, it's aim is to be boringly predictable. And yeah, it's annoying; but it's a lot less annoying than facing an angry customer who is screaming about lost data, lost money and how they're going to sue you into a smoking hole in the ground for deliberately deceiving them about the issues with your software.

by iofj 11y ago

winstonewert 11y ago

It sounds to me like you need to stop abusing mocks in your tests. I usually find such tests result from using mocks poorly.

eropple 11y ago

> There is almost no use for unit tests, as they lock down function implementations without verifying functionality. This is...curious, given that every place I've ever been with any sort of testing culture tested contracts, not function implementations. They test the range and domain of the function, not the behavior of it. Unit tests do not replace system-level/integration testing, but to dismiss them out of hand is manifestly unwise.

ori_b 11y ago
Generally, I see a lot of testing of internal functions, which often should change their contract as refactoring happens. All too rarely do I see people focusing testing on public APIs.
- eropple 11y ago
  It depends on your definition of "internal". I write unit tests at whatever boundary that makes sense, and I certainly will write unit tests to test contracts between related pieces that others might call "internal." But then I can refactor trivially at that level, which I may (read: usually do) want to have available later. I don't generally write, nor really see, see a lot of internal contractual testing in unit testing (not least because it becomes very hard to do so if you actually use visibility in OOP).

andrewguenther 11y ago

> There is almost no use for unit tests, as they lock down function implementations without verifying functionality. I would disagree. You can change the implementation all you want, but you cannot change the functionality (contract). If your unit test verifies implementation rather than functionality, then it is a poor test.

bluejekyll 11y ago

I'm sorry, but what?! Unit testing is not an exercise in the mundane by making your manager happy, it is about validating that the thing you just wrote actually works, so that you can move on to the next thing, eventually aggregating all of that work together and constructing code that works! As others have said, you test inputs and outputs (and maybe you need to test some failure modes depending on the complexity of the situation). When you're working on a parser, or lexical analyzer of any sort, wouldn't it be nice to know that it is capable of parsing what you thought it should? System/BlackBox/Integration tests are too high-level and the more you rely on those as your sole testing, the more brittle and difficult you make it to track down the cause of test failures. It also means that you have to spend more time constructing test scenarios to testing the inner-workings of the system, things that are easier to do in unit tests. Bloated tests at that layer actually make it MORE difficult to refactor. Decent unit tests help you define the inputs and outputs of your functions/classes/design. If it's hard to write a unit test on it, then you probably have side-effects in the code that you can't properly account for (bad code design). When it comes to a point that you need to refactor and the unit test is in your way, then guess what? refactor/delete the test...

twic 11y ago
There's an old and frustrating terminology problem here. You mention testing a parser; assuming the parser contains more than one class, or group of functions, i would consider a test of the whole parser to be an integration test, not a unit test.
- bluejekyll 11y ago
  By parser I mean any small thing that might parse text. Since I tend to deal with distributed systems, I generally think in terms of blackbox tests as dealing with application APIs rather than function APIs, and integration tests as multiple system components being tested together.
iofj 11y ago
> As others have said, you test inputs and outputs (and maybe you need to test some failure modes depending on the complexity of the situation). When you're working on a parser, or lexical analyzer of any sort, wouldn't it be nice to know that it is capable of parsing what you thought it should? That would be a system test. Nobody would write a parser in one function, so it's full functionality can't be tested in a unit test. This is also exactly the sort of test I'm advocating. The thing I'll guarantee you though : it won't provide 100% code coverage. Unit tests are this kind of crap, testing trivial things 100 times. You immediately see why they provide 100% code coverage : https://en.wikipedia.org/wiki/Unit_testing#Design https://en.wikipedia.org/wiki/Unit_testing#Design > the more brittle and difficult you make it to track down the cause of test failures Yes ! This of course happens because they find ever-more subtle bugs, not because there's something wrong with the test.

NathanKP 11y ago

You should consider testing to be a pyramid: 1) A smallish number of full end to end integration tests which actually make TCP requests to your app, and which actually have service talk to each other, etc. 2) A medium number of black box integration tests that just make HTTP requests to a single isolated service or component and verify that the response comes back out as expected (with no expectations on what happens in between input and output). 3) A much larger number of unit tests that are direct execution of business logic (just calling the code directly instead of calling the server externally) to verify various edge conditions in the inner underlying code. To use a real world example you might have full end to end integration tests to ensure that your service allows the creation of a new account, and that when the account is created an email is dispatched, and that an authentication cookie is set which gives this new user the ability to make requests to the service. This is just a high level test though. To back this test up you should have at least a couple hundred unit tests verifying that various edge cases with email formats, name formats, unicode characters in input, etc are handled correctly. To speed up test suite execution time it doesn't make sense to run each of these hundreds of edge cases as full end to end integration tests, or you end up with a horrific test suite that will take minutes to run. So instead you should use unit tests to cover the underlying edge conditions. You can usually run a hundred unit tests in the amount of time that a single integration test will run.

twic 11y ago
> You should consider testing to be a pyramid: Why? I've heard this story a million times. Nobody's ever been able to robustly justify it to me. I don't believe it's true.
- olefoo 11y ago
  If you don't have a test for a given item of functionality; how do you know it works? If you don't have a test for your subsystem that covers the more common use cases; how will you tell if it broke when you updated one of the functions/methods it uses? If you don't have a test for individual flows at the level of the entire system how will you know when you did something that broke the build outright? If you do none of that testing; you're handing off that function of your development team to the users of the system you're building. Testing is an engineering practice that lets you make changes to complex systems with confidence that the system is still doing it's job after the changes you make. It's not magic; it's engineering. As such, it's aim is to be boringly predictable. And yeah, it's annoying; but it's a lot less annoying than facing an angry customer who is screaming about lost data, lost money and how they're going to sue you into a smoking hole in the ground for deliberately deceiving them about the issues with your software.