5 ms·

Being a gatekeeper from unit testing is the last thing I aspire to be, but past me (and that’s my imaginary audience) would’ve loved this explanation before he

by hynek 4y ago

Being a gatekeeper from unit testing is the last thing I aspire to be, but past me (and that’s my imaginary audience) would’ve loved this explanation before he got some projects into mocking hell. I’m the first to say that if you have no tests and no idea how to get start, go with end-to-end tests first and take it from there. That doesn’t mean that guidelines that help you to have less brittle and more idiomatic unit tests are bad per se IMHO.

hinkley 4y ago
> go with end-to-end tests I've found that E2E tests make for a pretty bad case of hill-climbing where tests are concerned. To the point where I think that's the pathology of the testing icecream cone. It's easier conceptually go to 'up' the testing tree from unit tests to E2E tests, than it is to come down. People do what they are most familiar with, and if the E2E tests are first, then they are the most familiar. They are also glitchy as hell, and I see disdain for that get applied to all tests, not just the human tragedy that Selenium turned out to be. Also the ways in which unit test habits are considered 'bad' in functional tests are either less consequential or easier to walk away from. I'm not entirely sure which it is, or if it's both, but it's definitely less painful to relearn. We also overvalue the E2E test versus human testing. I can't count how many times someone has come to tell me a login button is broken, 20 minutes before the E2E tests fail and tell me the same thing. Computers arbitrating code regressions are half the point of CI, so if your tests aren't arbitrating, they aren't doing their job, and should be fired.
- hynek 4y ago
  I’m aware of all the problems that e2e tests have, and yet I find it more useful to have 1 test that adds something to a basket and 1 test that removes it than 100 unit tests without contract tests (which I never got the hang of TBH). (To be fair, my projects are low on JavaScript which somewhat changes the trade-off calculations, but I haven’t seen any assertions about project types above.)
- BeetleB 4y ago
  > It's easier conceptually go to 'up' the testing tree from unit tests to E2E tests, than it is to come down. It all depends on the language and the project, but my experience is usually the opposite. It's often much easier to write an end to end test. To be able to write a unit test you usually have to architect the code to make it easy to mock, etc - and that takes significant skill for most real world projects. E2E tests have a lot of downsides, but being challenging to write is probably not one of them.
  - quickthrower2 4y ago
    I took it as: if you only have E2E tests then retrofitting unit tests is hard. My experience agrees with this. Adding unit tests to untestable code (think: class has 10 dependencies that it news up being the first branching of a depdendecy graph to be untangled) requires refactors that just don’t get done with the busyness of other priorities.
    - BeetleB 4y ago
      > I took it as: if you only have E2E tests then retrofitting unit tests is hard. That's been my experience. My point is that the reason the project has E2E tests and not unit tests is because it was easier to add E2E tests.
- the_gipsy 4y ago
  Yea, you definitely don't want people writing e2e tests because it's easier than writing other tests or because there aren't any other tests.
- pydry 4y ago
  E2E have higher capex and lower opex - hard to set up well, cheap to maintain if you do. Low level tests are the reverse - easy to get started, hard to maintain. If youve got a really well oiled framework e2e tests are spectacular but getting to that point can vary between easy and "harder than the actual project itself" and if you half ass it (which most people do) it ends up a useless flaky mess.
  - hinkley 4y ago
    > cheap to maintain if you do. I've only seen this with projects in maintenance mode. Any major functionality changes tend to land teeth-first in the E2E tests, and after a few cycles of that you start dragging your feet a bit on major changes. E2E tests tend to lock in assumptions about the structure of your application, not just the minutia (which is not without its own set of problems). It is more difficult to get people to write maintainable E2E tests than it is getting them to write maintainable application code, so I've retreated and retrenched.
    - pydry 4y ago
      Weird Ive always found the opposite - E2E tests tend to make the fewest assumptions about the way the application is built. In most cases we could literally rewrite the whole app in a different language and barely change the tests. E2E tests do require a solid infrastructure though. If you have tooling problems you cant fix (e.g. you hit selenium's dark corners way too frequently because your web app is even a bit quirky) then maintainence cost can quite quickly exceed any benefit you derive. Ive also hit this problem of "Id have to create my own equivalent of selenium for interacting with this app's interface" a few times and yeah, the required engineering effort to do it well explodes to the point where its just not worth it.
luigi23 4y ago
I have an idea and I know how to get started. Not doing end-to-end because I also do unit tests and they improve my API. It's just that they're not perfect, especially duration-wise. My point is that I don't want get discouraged by being confused about best practices. Most of the software suffers from any testing, and many juniors read articles like these. Seniors do YOLO, correctly or their own way. Bit offtop, not debunking the content of this post though!
- yojo 4y ago
  I have personally worked on a team where velocity collapsed under the weight of the test suite. There were lots of brittle mocks. I think we even did mock out the Ruby equivalent of the HTTP client. Every change would inevitably break tests (not functionality!) and developers were spending more than half their time wrestling with the test suite. All of us could have used this article. Please always strive to improve your practice. Commit things that aren’t perfect, but try to internalize best practices over time. Gradually raise your own code quality bar as you get better. Learning from posts like the OP is super important if you don’t have a good senior Eng on your team. A group of smart juniors can easily code themselves into a hole of left to their own devices. I know this, because I did this.
  - pydry 4y ago
    >Every change would inevitably break tests (not functionality!) and developers were spending more than half their time wrestling with the test suite. I find that this invariably happens because the team tries to test implementation at a very low level rather than behavior at a high level. I blame an overemphasis on unit tests over integration testing (e.g. J B Rainsberger's inane rant) and tutorials that imply that, for example, if you build a class you have to have some tests for that class.
    - icedchai 4y ago
      At a previous company, someone established a "rule" that every "new function or method" needed its own test. Unless you lucked out with a lax reviewer (like me!) , your code would not pass review if this wasn't followed. Meanwhile, there were basically zero integration or e2e tests. People spent more time adding and maintaining useless tests...

by hynek 4y ago

hinkley 4y ago

> go with end-to-end tests I've found that E2E tests make for a pretty bad case of hill-climbing where tests are concerned. To the point where I think that's the pathology of the testing icecream cone. It's easier conceptually go to 'up' the testing tree from unit tests to E2E tests, than it is to come down. People do what they are most familiar with, and if the E2E tests are first, then they are the most familiar. They are also glitchy as hell, and I see disdain for that get applied to all tests, not just the human tragedy that Selenium turned out to be. Also the ways in which unit test habits are considered 'bad' in functional tests are either less consequential or easier to walk away from. I'm not entirely sure which it is, or if it's both, but it's definitely less painful to relearn. We also overvalue the E2E test versus human testing. I can't count how many times someone has come to tell me a login button is broken, 20 minutes before the E2E tests fail and tell me the same thing. Computers arbitrating code regressions are half the point of CI, so if your tests aren't arbitrating, they aren't doing their job, and should be fired.

hynek 4y ago
I’m aware of all the problems that e2e tests have, and yet I find it more useful to have 1 test that adds something to a basket and 1 test that removes it than 100 unit tests without contract tests (which I never got the hang of TBH). (To be fair, my projects are low on JavaScript which somewhat changes the trade-off calculations, but I haven’t seen any assertions about project types above.)
BeetleB 4y ago
> It's easier conceptually go to 'up' the testing tree from unit tests to E2E tests, than it is to come down. It all depends on the language and the project, but my experience is usually the opposite. It's often much easier to write an end to end test. To be able to write a unit test you usually have to architect the code to make it easy to mock, etc - and that takes significant skill for most real world projects. E2E tests have a lot of downsides, but being challenging to write is probably not one of them.
- quickthrower2 4y ago
  I took it as: if you only have E2E tests then retrofitting unit tests is hard. My experience agrees with this. Adding unit tests to untestable code (think: class has 10 dependencies that it news up being the first branching of a depdendecy graph to be untangled) requires refactors that just don’t get done with the busyness of other priorities.
  - BeetleB 4y ago
    > I took it as: if you only have E2E tests then retrofitting unit tests is hard. That's been my experience. My point is that the reason the project has E2E tests and not unit tests is because it was easier to add E2E tests.
the_gipsy 4y ago
Yea, you definitely don't want people writing e2e tests because it's easier than writing other tests or because there aren't any other tests.
pydry 4y ago
E2E have higher capex and lower opex - hard to set up well, cheap to maintain if you do. Low level tests are the reverse - easy to get started, hard to maintain. If youve got a really well oiled framework e2e tests are spectacular but getting to that point can vary between easy and "harder than the actual project itself" and if you half ass it (which most people do) it ends up a useless flaky mess.
- hinkley 4y ago
  > cheap to maintain if you do. I've only seen this with projects in maintenance mode. Any major functionality changes tend to land teeth-first in the E2E tests, and after a few cycles of that you start dragging your feet a bit on major changes. E2E tests tend to lock in assumptions about the structure of your application, not just the minutia (which is not without its own set of problems). It is more difficult to get people to write maintainable E2E tests than it is getting them to write maintainable application code, so I've retreated and retrenched.
  - pydry 4y ago
    Weird Ive always found the opposite - E2E tests tend to make the fewest assumptions about the way the application is built. In most cases we could literally rewrite the whole app in a different language and barely change the tests. E2E tests do require a solid infrastructure though. If you have tooling problems you cant fix (e.g. you hit selenium's dark corners way too frequently because your web app is even a bit quirky) then maintainence cost can quite quickly exceed any benefit you derive. Ive also hit this problem of "Id have to create my own equivalent of selenium for interacting with this app's interface" a few times and yeah, the required engineering effort to do it well explodes to the point where its just not worth it.

luigi23 4y ago

I have an idea and I know how to get started. Not doing end-to-end because I also do unit tests and they improve my API. It's just that they're not perfect, especially duration-wise. My point is that I don't want get discouraged by being confused about best practices. Most of the software suffers from any testing, and many juniors read articles like these. Seniors do YOLO, correctly or their own way. Bit offtop, not debunking the content of this post though!

yojo 4y ago
I have personally worked on a team where velocity collapsed under the weight of the test suite. There were lots of brittle mocks. I think we even did mock out the Ruby equivalent of the HTTP client. Every change would inevitably break tests (not functionality!) and developers were spending more than half their time wrestling with the test suite. All of us could have used this article. Please always strive to improve your practice. Commit things that aren’t perfect, but try to internalize best practices over time. Gradually raise your own code quality bar as you get better. Learning from posts like the OP is super important if you don’t have a good senior Eng on your team. A group of smart juniors can easily code themselves into a hole of left to their own devices. I know this, because I did this.
- pydry 4y ago
  >Every change would inevitably break tests (not functionality!) and developers were spending more than half their time wrestling with the test suite. I find that this invariably happens because the team tries to test implementation at a very low level rather than behavior at a high level. I blame an overemphasis on unit tests over integration testing (e.g. J B Rainsberger's inane rant) and tutorials that imply that, for example, if you build a class you have to have some tests for that class.
  - icedchai 4y ago
    At a previous company, someone established a "rule" that every "new function or method" needed its own test. Unless you lucked out with a lax reviewer (like me!) , your code would not pass review if this wasn't followed. Meanwhile, there were basically zero integration or e2e tests. People spent more time adding and maintaining useless tests...