11 ms·

"Don't do X, don't break the rule, you're doing it wrong"... that's why people don't write tests (well, maybe not the main reason, but that was my case at least

"Don't do X, don't break the rule, you're doing it wrong"... that's why people don't write tests (well, maybe not the main reason, but that was my case at least). Do whatever that will increase confidence in code and testability. If I want to make it easier for me to develop, I'm gonna do anything that will enable me to get shit done. Test takes 0.1 second instead of 0.001? Whatever. I can improve it if needed, when I get more knowledge or I can discard it completely. When I stopped caring about 'should I use mock, or inject closure, or interface...' I started writing more tests and producing better software. Because often the end goal is about writing clearer code even with shitty tests.

Groxx 4y ago
Full agree, with one additional detail: Tests are code too. Like the code being tested, they can be shitty at times, but they are still code and should be treated with the same level of care. Which is not to say "your tests should be as DRY as a desert". That's frequently a poor fit for test code's lifecycle. But it does mean you shouldn't do stuff that wouldn't pass code review elsewhere, just because "it's just a test". Someone later has to read, understand, and maintain that code - treat it as such.
oweiler 4y ago
I get paid for code that works, not for tests Kent Beck
- deterministic 4y ago
  … and if tests helps me making the code work then it is worth doing.
arubania2 4y ago
> "Don't do X, don't break the rule, you're doing it wrong"... that's why people don't write tests I think that's why people don't read blog posts like this - such wording is rude and patronizing.
lifefeed 4y ago
The implication with these types of articles is always "any tests are better than no tests, but here's how to do better tests."
- fleventynine 4y ago
  No tests can be better than a bunch of brittle mock object tests that don't test anything important. Thankfully it's very easy to delete them. If changing one line of code regularly requires me to update mocks in 100 test cases, those test cases may have negative value.
  - scaryclam 4y ago
    I've seen plenty of scenarios when a test has also given a false sense of security. It's not actually testing anything important, but looks like it does and should be, but doesn't actually catch defects. This is also a case of no tests being better as at least everyone knows something's untested :/
BeetleB 4y ago
> Do whatever that will increase confidence in code and testability. Heh. I've long felt a need to write a blog post touching this very idea, but have been too lazy to. I think a lot of people (including younger me) would dive right in to things like unit tests without too much thought. Now I focus on: - What should we test? (Think about the feature you are adding and why - what aspects need to work?) - Why? should we test it? - Should we test it? (really the same as above). Common to see junior folks end up testing 3rd party libraries. - How should we test it? Mocks? Unit tests? Integration tests? Something else? - Is it even testable in an automated fashion? If not, ponder over what makes it hard to test. Does that need to be rectified? - How can we be sure we're testing what we think we are? Extremely common to see people write tests that appear to pass, but are testing the wrong thing (buggy test). It's very rare that I see a colleague intentionally introduce a bug in the feature to see if the expected test will fail. - What is the cost of this feature having a bug? If it's very costly, you may want to have multiple types of tests, and possibly even a human testing it each release. If the cost is almost zero, and you won't need to fix it quickly if a customer reports it, and it's hard to write an automated test for it - just skip it. If this mentality bothers you, know that this is recommended by the ISO standard for SW in cars - even if a bug can end up in loss of life, if you can show it's extremely unlikely, you are conformant if you don't test it. You should first answer these questions, and they will then guide you into whether to write unit tests, or E2E tests, or something in between. I generally find that people who dive into testing without answering the above write crappy tests (buggy, pointless, etc). Answering these can lead to better mocks (or a decision not to rely on mocks). In summary, step back and forget everything you've learned about testing before answering these questions. Pretend you don't know the difference between unit tests, integration tests, etc. You've written (or plan to write) a feature.[1] What would you like to test in an automated fashion, and how can you do it? Feel free to use any tool that works for your project. [1] You're not writing code. You're writing a feature.
Akronymus 4y ago
Part of the reason why I like FP so much, is that you DON'T need to mock anything most of the time. Just call the damn function and compare the expected with the actual output. Useful if you have a reference implementation that is obviously correct but slow, and a fast one where it may not be correct.
hynek 4y ago
Being a gatekeeper from unit testing is the last thing I aspire to be, but past me (and that’s my imaginary audience) would’ve loved this explanation before he got some projects into mocking hell. I’m the first to say that if you have no tests and no idea how to get start, go with end-to-end tests first and take it from there. That doesn’t mean that guidelines that help you to have less brittle and more idiomatic unit tests are bad per se IMHO.
- hinkley 4y ago
  > go with end-to-end tests I've found that E2E tests make for a pretty bad case of hill-climbing where tests are concerned. To the point where I think that's the pathology of the testing icecream cone. It's easier conceptually go to 'up' the testing tree from unit tests to E2E tests, than it is to come down. People do what they are most familiar with, and if the E2E tests are first, then they are the most familiar. They are also glitchy as hell, and I see disdain for that get applied to all tests, not just the human tragedy that Selenium turned out to be. Also the ways in which unit test habits are considered 'bad' in functional tests are either less consequential or easier to walk away from. I'm not entirely sure which it is, or if it's both, but it's definitely less painful to relearn. We also overvalue the E2E test versus human testing. I can't count how many times someone has come to tell me a login button is broken, 20 minutes before the E2E tests fail and tell me the same thing. Computers arbitrating code regressions are half the point of CI, so if your tests aren't arbitrating, they aren't doing their job, and should be fired.
  - hynek 4y ago
    I’m aware of all the problems that e2e tests have, and yet I find it more useful to have 1 test that adds something to a basket and 1 test that removes it than 100 unit tests without contract tests (which I never got the hang of TBH). (To be fair, my projects are low on JavaScript which somewhat changes the trade-off calculations, but I haven’t seen any assertions about project types above.)
  - BeetleB 4y ago
    > It's easier conceptually go to 'up' the testing tree from unit tests to E2E tests, than it is to come down. It all depends on the language and the project, but my experience is usually the opposite. It's often much easier to write an end to end test. To be able to write a unit test you usually have to architect the code to make it easy to mock, etc - and that takes significant skill for most real world projects. E2E tests have a lot of downsides, but being challenging to write is probably not one of them.
chickenpotpie 4y ago
As the author notes, you can break this rule. As with all rules in software, you learn the rule so you know that you can break the rule. > "Because often the end goal is about writing clearer code even with shitty tests." If you're worried about people being afraid to write tests, this really scares people away. Nothing makes me want to write a test for my changes less than a 2000 line test file with 50 mocks.
Dangeranger 4y ago
If you ignore the past, your destination will be the same, and your reward will be to suffer the same pain.
throwaway894345 4y ago
> Do whatever that will increase confidence in code and testability. Maybe for some strict definition of "testability", but a lot of people in the Python world use patching to circumvent anything that might reasonably be called "testability" in the pursuit of "confidence" and some puritanical desire to avoid changing the code base to make it testable. This makes everything a complete mess for anyone who comes along later to extend the code or increase test coverage, as the tests now depend on the private internals of the code and the patching logic is magical and unintuitive. If your "testability" remark precludes patching, then I'm more bought in, but I do worry that some will read this and use it to argue for patching. That said, experience tells me that mocking delivers a lot less confidence than using the real deal, although there are some cases (e.g., network errors) that effectively require mocks _somewhere_.
- icedchai 4y ago
  Definitely this. I ran into this at a previous company (Python environment) where every test basically relied on mucking around with the internals through patches. Any sort of integration test that relied on calling an internal service was difficult or impossible, so it had to be mocked. External services were always mocked. The code base was a pain to work with. At at an earlier company, we had very little patching or mocking. We spun up all the internal services when running the tests. Internal calls were never mocked. External calls were sometimes mocked (if test accounts / API keys were not convenient.) We preferred testing the "real thing." Tests ran slower but in general I had much more confidence tests actually did something useful.
  - throwaway894345 4y ago
    I like to distinguish between mocks and fakes. Mocks just return whatever you tell them to return for that test, but a fake is a simplified in-memory implementation (fakes aren't useful for testing error conditions, however, so you may need a few tests which actually do mock if you want to get more complete coverage). Most of the time for third-party services, I'll just fake the service client for unit tests. For example, if I'm writing a BookStoreApp that has a BookStore dependency (where BookStore is an interface), the concrete implementation for which is a PostgresBookStore, then I'll write my PostgresBookStore client library with unit tests that include a real postgres connection but then my BookStoreApp tests will use a FakeBookStore (which is probably a thin wrapper around a dict, but which behaves like a BookStore should modulo persistence). Finally, I'll have considerably fewer API tests which actually do stand up the complete backend service (including a PostgresBookStore). This way I get a high degree of confidence and also speed of test execution--it's also nice not to have to stand up the entire world just to run one test (in particular, it's nice not to have to debug unrelated connection/auth issues).
samstave 4y ago
I'm curious, as I have never asked the question prior ; What is the canonical "unit test" whatever [method|practice|technique|whatever] that one should know? Is there a place where tests/unit-test-methodologies are discussed, or codified? How learn?
- aidenn0 4y ago
  Ultimately the two goals are that changes that don't break the product to pass all the tests and changes that do break the product to fail at least one test. If you remember those two goals, you will go far. All rules should be in service of the above. I don't know that there is a "most people agree with this" set of rules, but if you read any source on how to test, think about how each rule affects the two goals. Senior developers tend to have developed an instinct for "things like this have failed one of the two rules" but it is hard to codify that instinct into rules.
- btilly 4y ago
  As soon as you say "canonical", you invite dogma from those with rigid opinions. It is better to integrate advice you hear with deliberate practice. If you do not see, from you own experience, why it makes sense, then discount it. But keep an open mind to changing your opinion later. Your own opinion that you understand why you believe is a million times better than regurgitating slogans that you don't. The PRACTICAL rule is always have unit tests. Start with writing tests for bugs you actually encountered. Use whatever testing framework is popular in your current language, particularly if it ships with the language. Always try to test things in the most straightforward way possible. Add automation to run all tests regularly. BUT keep in mind, test code is code. You're now writing code twice, once for the code, once for the test. Either or both can be wrong. In the future, both may have to change. Tests improve reliability, but are not free.
  - samstave 4y ago
    Thank you (albeit condescending) for the answer... --- What is the PRACTICAL learnings of CREATING tests? Meaning, how does one succeed at learning to employ the same sentiment above usch that one does well from ground zero? (integrate tests into your design, but what does one need to know in order to do such?) [ELINSW: Explain like im a newbie software engineer] ---- I understand the philos... I am talking about IS THERE A LEARNING CHANNEL TO BECOME AN EXPERT AT DEVELOPING UNIT TESTS, or is it something that is lit. learned on job?
treeman79 4y ago
Few years ago I inherited a large code base. Tens of thousands of tests. Company was very proud. Basically all the test did was prove that the mocking library worked well. Almost no actual project code or business logic was tested. Company was not happy to hear.
- hynek 4y ago
  Cue Mock Hell https://www.youtube.com/watch?v=CdKaZ7boiZ4 https://www.youtube.com/watch?v=CdKaZ7boiZ4 :)
- kinetronic 4y ago
  James Coplien wrote in 2014 I think a paper titled "Why most unit testing is waste" which stirred up a storm among developers. I've seen many rebuttals, and they keep cropping up every year, but most seem to agree with his core premise: unit testing is valuable for testing algorithms -- stateless functions performing a computation -- and not much else.

Groxx 4y ago

Full agree, with one additional detail: Tests are code too. Like the code being tested, they can be shitty at times, but they are still code and should be treated with the same level of care. Which is not to say "your tests should be as DRY as a desert". That's frequently a poor fit for test code's lifecycle. But it does mean you shouldn't do stuff that wouldn't pass code review elsewhere, just because "it's just a test". Someone later has to read, understand, and maintain that code - treat it as such.

oweiler 4y ago

I get paid for code that works, not for tests Kent Beck

deterministic 4y ago
… and if tests helps me making the code work then it is worth doing.

arubania2 4y ago

> "Don't do X, don't break the rule, you're doing it wrong"... that's why people don't write tests I think that's why people don't read blog posts like this - such wording is rude and patronizing.

lifefeed 4y ago

The implication with these types of articles is always "any tests are better than no tests, but here's how to do better tests."

fleventynine 4y ago
No tests can be better than a bunch of brittle mock object tests that don't test anything important. Thankfully it's very easy to delete them. If changing one line of code regularly requires me to update mocks in 100 test cases, those test cases may have negative value.
- scaryclam 4y ago
  I've seen plenty of scenarios when a test has also given a false sense of security. It's not actually testing anything important, but looks like it does and should be, but doesn't actually catch defects. This is also a case of no tests being better as at least everyone knows something's untested :/

BeetleB 4y ago

> Do whatever that will increase confidence in code and testability. Heh. I've long felt a need to write a blog post touching this very idea, but have been too lazy to. I think a lot of people (including younger me) would dive right in to things like unit tests without too much thought. Now I focus on: - What should we test? (Think about the feature you are adding and why - what aspects need to work?) - Why? should we test it? - Should we test it? (really the same as above). Common to see junior folks end up testing 3rd party libraries. - How should we test it? Mocks? Unit tests? Integration tests? Something else? - Is it even testable in an automated fashion? If not, ponder over what makes it hard to test. Does that need to be rectified? - How can we be sure we're testing what we think we are? Extremely common to see people write tests that appear to pass, but are testing the wrong thing (buggy test). It's very rare that I see a colleague intentionally introduce a bug in the feature to see if the expected test will fail. - What is the cost of this feature having a bug? If it's very costly, you may want to have multiple types of tests, and possibly even a human testing it each release. If the cost is almost zero, and you won't need to fix it quickly if a customer reports it, and it's hard to write an automated test for it - just skip it. If this mentality bothers you, know that this is recommended by the ISO standard for SW in cars - even if a bug can end up in loss of life, if you can show it's extremely unlikely, you are conformant if you don't test it. You should first answer these questions, and they will then guide you into whether to write unit tests, or E2E tests, or something in between. I generally find that people who dive into testing without answering the above write crappy tests (buggy, pointless, etc). Answering these can lead to better mocks (or a decision not to rely on mocks). In summary, step back and forget everything you've learned about testing before answering these questions. Pretend you don't know the difference between unit tests, integration tests, etc. You've written (or plan to write) a feature.[1] What would you like to test in an automated fashion, and how can you do it? Feel free to use any tool that works for your project. [1] You're not writing code. You're writing a feature.

Akronymus 4y ago

Part of the reason why I like FP so much, is that you DON'T need to mock anything most of the time. Just call the damn function and compare the expected with the actual output. Useful if you have a reference implementation that is obviously correct but slow, and a fast one where it may not be correct.

hynek 4y ago

Being a gatekeeper from unit testing is the last thing I aspire to be, but past me (and that’s my imaginary audience) would’ve loved this explanation before he got some projects into mocking hell. I’m the first to say that if you have no tests and no idea how to get start, go with end-to-end tests first and take it from there. That doesn’t mean that guidelines that help you to have less brittle and more idiomatic unit tests are bad per se IMHO.

hinkley 4y ago
> go with end-to-end tests I've found that E2E tests make for a pretty bad case of hill-climbing where tests are concerned. To the point where I think that's the pathology of the testing icecream cone. It's easier conceptually go to 'up' the testing tree from unit tests to E2E tests, than it is to come down. People do what they are most familiar with, and if the E2E tests are first, then they are the most familiar. They are also glitchy as hell, and I see disdain for that get applied to all tests, not just the human tragedy that Selenium turned out to be. Also the ways in which unit test habits are considered 'bad' in functional tests are either less consequential or easier to walk away from. I'm not entirely sure which it is, or if it's both, but it's definitely less painful to relearn. We also overvalue the E2E test versus human testing. I can't count how many times someone has come to tell me a login button is broken, 20 minutes before the E2E tests fail and tell me the same thing. Computers arbitrating code regressions are half the point of CI, so if your tests aren't arbitrating, they aren't doing their job, and should be fired.
- hynek 4y ago
  I’m aware of all the problems that e2e tests have, and yet I find it more useful to have 1 test that adds something to a basket and 1 test that removes it than 100 unit tests without contract tests (which I never got the hang of TBH). (To be fair, my projects are low on JavaScript which somewhat changes the trade-off calculations, but I haven’t seen any assertions about project types above.)
- BeetleB 4y ago
  > It's easier conceptually go to 'up' the testing tree from unit tests to E2E tests, than it is to come down. It all depends on the language and the project, but my experience is usually the opposite. It's often much easier to write an end to end test. To be able to write a unit test you usually have to architect the code to make it easy to mock, etc - and that takes significant skill for most real world projects. E2E tests have a lot of downsides, but being challenging to write is probably not one of them.

chickenpotpie 4y ago

As the author notes, you can break this rule. As with all rules in software, you learn the rule so you know that you can break the rule. > "Because often the end goal is about writing clearer code even with shitty tests." If you're worried about people being afraid to write tests, this really scares people away. Nothing makes me want to write a test for my changes less than a 2000 line test file with 50 mocks.

Dangeranger 4y ago

If you ignore the past, your destination will be the same, and your reward will be to suffer the same pain.

throwaway894345 4y ago

> Do whatever that will increase confidence in code and testability. Maybe for some strict definition of "testability", but a lot of people in the Python world use patching to circumvent anything that might reasonably be called "testability" in the pursuit of "confidence" and some puritanical desire to avoid changing the code base to make it testable. This makes everything a complete mess for anyone who comes along later to extend the code or increase test coverage, as the tests now depend on the private internals of the code and the patching logic is magical and unintuitive. If your "testability" remark precludes patching, then I'm more bought in, but I do worry that some will read this and use it to argue for patching. That said, experience tells me that mocking delivers a lot less confidence than using the real deal, although there are some cases (e.g., network errors) that effectively require mocks _somewhere_.

icedchai 4y ago
Definitely this. I ran into this at a previous company (Python environment) where every test basically relied on mucking around with the internals through patches. Any sort of integration test that relied on calling an internal service was difficult or impossible, so it had to be mocked. External services were always mocked. The code base was a pain to work with. At at an earlier company, we had very little patching or mocking. We spun up all the internal services when running the tests. Internal calls were never mocked. External calls were sometimes mocked (if test accounts / API keys were not convenient.) We preferred testing the "real thing." Tests ran slower but in general I had much more confidence tests actually did something useful.
- throwaway894345 4y ago
  I like to distinguish between mocks and fakes. Mocks just return whatever you tell them to return for that test, but a fake is a simplified in-memory implementation (fakes aren't useful for testing error conditions, however, so you may need a few tests which actually do mock if you want to get more complete coverage). Most of the time for third-party services, I'll just fake the service client for unit tests. For example, if I'm writing a BookStoreApp that has a BookStore dependency (where BookStore is an interface), the concrete implementation for which is a PostgresBookStore, then I'll write my PostgresBookStore client library with unit tests that include a real postgres connection but then my BookStoreApp tests will use a FakeBookStore (which is probably a thin wrapper around a dict, but which behaves like a BookStore should modulo persistence). Finally, I'll have considerably fewer API tests which actually do stand up the complete backend service (including a PostgresBookStore). This way I get a high degree of confidence and also speed of test execution--it's also nice not to have to stand up the entire world just to run one test (in particular, it's nice not to have to debug unrelated connection/auth issues).

samstave 4y ago

I'm curious, as I have never asked the question prior ; What is the canonical "unit test" whatever [method|practice|technique|whatever] that one should know? Is there a place where tests/unit-test-methodologies are discussed, or codified? How learn?

aidenn0 4y ago
Ultimately the two goals are that changes that don't break the product to pass all the tests and changes that do break the product to fail at least one test. If you remember those two goals, you will go far. All rules should be in service of the above. I don't know that there is a "most people agree with this" set of rules, but if you read any source on how to test, think about how each rule affects the two goals. Senior developers tend to have developed an instinct for "things like this have failed one of the two rules" but it is hard to codify that instinct into rules.
btilly 4y ago
As soon as you say "canonical", you invite dogma from those with rigid opinions. It is better to integrate advice you hear with deliberate practice. If you do not see, from you own experience, why it makes sense, then discount it. But keep an open mind to changing your opinion later. Your own opinion that you understand why you believe is a million times better than regurgitating slogans that you don't. The PRACTICAL rule is always have unit tests. Start with writing tests for bugs you actually encountered. Use whatever testing framework is popular in your current language, particularly if it ships with the language. Always try to test things in the most straightforward way possible. Add automation to run all tests regularly. BUT keep in mind, test code is code. You're now writing code twice, once for the code, once for the test. Either or both can be wrong. In the future, both may have to change. Tests improve reliability, but are not free.
- samstave 4y ago
  Thank you (albeit condescending) for the answer... --- What is the PRACTICAL learnings of CREATING tests? Meaning, how does one succeed at learning to employ the same sentiment above usch that one does well from ground zero? (integrate tests into your design, but what does one need to know in order to do such?) [ELINSW: Explain like im a newbie software engineer] ---- I understand the philos... I am talking about IS THERE A LEARNING CHANNEL TO BECOME AN EXPERT AT DEVELOPING UNIT TESTS, or is it something that is lit. learned on job?

treeman79 4y ago

Few years ago I inherited a large code base. Tens of thousands of tests. Company was very proud. Basically all the test did was prove that the mocking library worked well. Almost no actual project code or business logic was tested. Company was not happy to hear.

hynek 4y ago
Cue Mock Hell https://www.youtube.com/watch?v=CdKaZ7boiZ4 https://www.youtube.com/watch?v=CdKaZ7boiZ4 :)
kinetronic 4y ago
James Coplien wrote in 2014 I think a paper titled "Why most unit testing is waste" which stirred up a storm among developers. I've seen many rebuttals, and they keep cropping up every year, but most seem to agree with his core premise: unit testing is valuable for testing algorithms -- stateless functions performing a computation -- and not much else.