13 ms·

The big TDD misunderstanding (2022)

125 points by WolfOliver 3y ago

jgalt212 3y ago
> Write the tests from outside in. With this, I mean you should write your tests from a realistic user perspective. To have the best quality assurance and refactor resistance, you would write e2e or integration tests. Yah, yah. But good look trying to figure out what went wrong when only the failing test you have is an e2e or integration test.
- MoreQARespect 3y ago
  My e2e tests automatically dump a multitude of debugging information and throw open a console from which I can quickly fire up code debuggers, logs, screenshots, network traces and browser traces in a few seconds. Building test tooling is hard I'll grant you. It requires engineering chops and being up to date on the latest tooling. But not luck.
iamwil 3y ago
One way I think about it is that unit tests help me maintain invariances that the compiler for the current language can't do for me. That saves me from having to test every combination of inputs.
locallost 3y ago
Michael Feathers in "working with legacy code" tackled the unit test definition, and in the end defined a unit test as a test that runs fast. Also a very confusing topic for me early on as I tried to grasp how to actually do it. So I read Kent Beck's book on TDD and was even more confused because he did not write tests in isolation there, and I was told to write unit tests like that. But then it hit me: most people are just cargo cultists and repeat what somebody told them.
- malkia 3y ago
  One of the best things in bazel is that it's very straightforward when comes to tests - https://bazel.build/reference/test-encyclopedia#role-test-runner https://bazel.build/reference/test-encyclopedia#role-test-ru... - you need to classify your test as short, moderate, long, etc., but also how much memory it takes. In addition if you tests requires isolation, put it too. So potentionally one can combine multiple independent code bases, that share the same testing framework (say gtest), into single executable, and distributevly execute them using sharding on multiple machines (e.g. depending on what changes were detected in the presbumit/preflight/whatever you call phase). And because the tests were marked as such and such, it'll use that knowledge to group them better, and if they no longer qualify, an error would be printed out, and how to fix this (if it's worth fixing).
voiceofunreason 3y ago
<< Originally, the term “unit” in “unit test” referred not to the system under test but to the test itself. >> Retroactive continuity - how does it work? For today's lucky 10,000: "unit test", as a label, was in wide use prior to the arrival of the Extreme Programming movement in the late 1990s. And the "unit" in question was the test subject. But, as far as I can tell, _Smalltalk_ lacked a testing culture (Beck, 1994), so perhaps the testing community's definitions weren't well represented in Smalltalk spaces. "The past was alterable. The past never had been altered." (Not particularly fair to single out this one author - this origin myth has been common during the last 10 years or so.)
- jdougan 3y ago
  At my job we wrote tests in Smalltalk and that was before Kent's work. It wasn't the later tests-first discipline. If I recall correctly write tests later was fairly common then in ST shops. SUnit also had the virtue of standardizing the terminology and testing frameworks.
emadb 3y ago
The big TDD misunderstanding is that most people consider TDD a testing practice. The article doesn’t talk about TDD, it gives the reader some tips on how to write tests. That’s not TDD.
- shimst3r 3y ago
  Instead of Test-Driven Design, it should’ve been called Design-By-Testing.
  - paulluuk 3y ago
    Did you mean Test Driven Development, or is Test-Driven Design a whole other thing?
- skrebbel 3y ago
  Yeah it’s kind of unfortunate because they make a very good argument about defining a thing better, and in the title use a wrong definition of an adjacent term.
- WolfOliver 3y ago
  Maybe the term TDD in the title can be replaced with "unit testing". But unit testing is an major part of TDD.
- MoreQARespect 3y ago
  I'm fully aware of the idea that TDD is a "design practice" but I find it to be completely wrongheaded. The principle that tests that couple to low level code give you feedback about tightly coupled code is true but it does that because low level/unit tests couple too tightly to your code - I.e. because they too are bad code! Have you ever refactored working code into working code and had a slew of tests fail anyway? That's the child of test driven design. High level/integration TDD doesnt give "feedback" on your design it just tells you if your code matches the spec. This is actually more useful. It then lets you refactor bad code with a safety harness and give failures that actually mean failure and not "changed code". I keep wishing for the idea of test driven design to die. Writing tests which break on working code is inordinately uneconomic way to detect design issues as compared to developing an eye for it and fixing it under a test harness with no opinion on your design. So, yes this - high level test driven development - is TDD and moreover it's got a better cost/benefit trade off than test driven design.
michalc 3y ago
> Now, you change a little thing in your code base, and the only thing the testing suite tells you is that you will be busy the rest of the day rewriting false positive test cases. If there is anything that makes me cry, it’s hearing “it’s done, now I need to fix the tests”
- WolfOliver 3y ago
  Agree, this is usually a sign the team writes tests for the sake of writing tests.
- scaramanga 3y ago
  If changing the implementation but not the behaviour breaks a test, I just delete the test.
- tetha 3y ago
  It's something we've changed when we switched our configuration management. The old config management had very, very meticulous tests of everything. This resulted in great "code" coverage, but whenever you changed a default value, at least 6 tests would fail now. Now, we much rather go ahead and test much more coarsely. If the config management can take 3 VMs and setup a RabbitMQ cluster that clusters and accepts messages, how wrong can it be? And this has also bled into my development and strengthened my support of bug-driven testing. For a lot of pretty simple business logic, do a few high level e2e tests for the important behaviors. And then when it breaks, add more tests for those parts. But note, this may be different for very fiddly parts of the code base - complex algorithms, math-heavy and such. But that's when you'd rather start table based testing and such. At a past gamedev job, we had several issues with some complex cost balancing math, so I eventually setup a test that allows the game balancing team to supply CSV files with expected results. That cleared up these issues within 2 days or so.
  - tremon 3y ago
    whenever you changed a default value, at least 6 tests would fail now Testing default values makes a lot of sense. Both non-set configuration values and non-supplied function parameters become part of your API. Your consumers will rely on those default values, and if you alter them, your consumers will see different behaviour.
gardenhedge 3y ago
> Never change your code without having a red test I'll never understand why people insist on this. If you want to write your tests first, that is fine. Noone is going to stop you. But why must you insist everyone does it this way?
- shimst3r 3y ago
  If you do TDD, you write your tests first. If you don’t follow TDD, you don’t have to. The author insists on this because they assume a TDD approach.
  - gardenhedge 3y ago
    I suppose that is a fair reading. If TDD is the goal then the "never" instruction makes sense.
- stepbeek 3y ago
  In the context of the article, I read this as "modify the test first for an existing unit test and a bit of production code under test". I don't think this is equivalent to "always write the test first".
- tarcio 3y ago
  It helps you think about the problem better. A particular test has assertions that must be implemented correctly in order for them to pass. By starting in red you gradually make your way towards green usually paving your thinking and potentially unlocking other tests. I don't believe you must do this every time though.
  - gardenhedge 3y ago
    > I don't believe you must do this every time though. So I think we agree. If you find it beneficial, please carry on doing it. For anyone who doesn't find it beneficial, they can write tests in whatever way they want.
- resonious 3y ago
resonious 3y ago
Huh, I found this more interesting than I thought I would. I hadn't heard before that the "unit" in "unit test" just meant "can run independently". I once failed an interview partly because of only writing "feature tests" and not "unit tests" in the project I showed. But actually those tests didn't depend on each other, so... looks like they really were unit tests! Anyway, I'm still not totally sure about TDD itself - the "don't write any code without a red test" part. I get the idea, but it doesn't feel very productive when I try it. Of course maybe I'm just bad at it, but I also haven't seen any compelling arguments for it other than it makes the tests stronger (against what? someone undoing my commits?). I think even Uncle Bob's underlying backing argument was that TDD is more "professional", leading me to believe it's just a song-and-dance secret handshake that helps you get into a certain kind of company. OR, it's a technique to try and combat against lazy devs, to try and make it impossible for them to write bad tests. And maybe it is actually good but only for some kinds of projects... I wish we had a way to actually research this stuff rather than endlessly share opinions and anecdotes.
- viraptor 3y ago
  > arguments for it other than it makes the tests stronger It's supposed to lead to a better design. It's easy to write some code that maybe works and you can't actually test (lots of interdependencies, weird state, etc.), or you only think you're trying correctly. But making the test first forces you to write something that 1. You can test (by definition) 2. Is decoupled to the level where you can check mainly for the behaviour you're interested in. 3. You won't bypass accidentally. It's not even someone undoing your commits, but some value in the call chain changing in a way that accidentally makes the feature not run at all. I've seen it many times in practice and will bet that any large project where the tests were written after the code, has some tests that don't actually do anything. They were already passing before the thing they're supposedly testing was implemented.
- MoreQARespect 3y ago
  If I have the tooling all set up (e.g. playwright, database fixtures, mitmproxy) and the integration test closely resembles the requirement then I'm about as productive doing TDD as not doing TDD except I get tests as a side effect. If I do snapshot test driven development (e.g. actual rest API responses are written into the "expected" portion of the test by the test) then I'm sometimes a little bit more productive. There's a definite benefit to fixing the requirement rather than letting it evaporate into the ether. Uncle bob style unit test driven development, on the other hand, is something more akin to a ritual from a cult. Unit test driven development on integration code (e.g. code that handles APIs, databases, UIs) is singularly useless. It only really works well on algorithmic or logical code - parsers, pricing engines, etc. where the requirement can be well represented.
philippta 3y ago
In my experience a lot of engineers are stuck thinking in MVC terms an fail to write modular code. As a result most business logic is part of a request / response flow. This makes it infeasible to even attempt to write tests first, thus leaving integration or e2e tests as the only remaining options.
- trwrto 3y ago
  Where should the business logic rather be? My tests are typically calling APIs to test the business logic. Trying to improve myself here.
  - philippta 3y ago
    There aren‘t any hard rules here, but you can try to build your business logic as if it was a library, where your HTTP API is merely a interface to it.
- laurencerowe 3y ago
  I’m not a TDD purist but I’ve found that so long as the request / response flow is a JSON api or similar (as opposed to old style forms and html rendering) then writing integration tests first is quite easy so long as you make sure your test fixtures are fairly fast.
  - _heimdall 3y ago
    With this approach do you stop at testing the JSON API or do you still make it to testing the rendered HTML actually shown to the user? I've always actually likes the simplicity of testing HTML APIs in a frontend project. For me, tests get a lot more simple when I can verify the final HTML directly from the response and don't need to parse the JSON and run it through client-side rendering logic first.
    - laurencerowe 3y ago
      While we had a few end to end browser tests they tended to be slow so the pattern we ended up at was to generate output from the API to use as mock data in the UI tests. I think it usually make sense to test the business logic at the JSON API level (except for particularly complicated parts where unit testing of that complex logic may make sense) and then use snapshot tests for the display logic.
almostnormal 3y ago
Part of the problem is caused by all sides using the same terms but with a different meaning. > You just don’t know if your system works as a whole, even though each line is tested. ... even though each line has been executed. One test per line is strongly supported by tools calculating coverage and calling that "tested". A test for one specific line is rarely possible. It may be missing some required behavior that hasn't been challanged by any test, or it may be inconsistent with other parts of the code. A good start would be to stop calling something just executed "tested".
- usrusr 3y ago
  I like the term "exercised" for coverage of questionable assertative value. It's rather pointless in environments with a strict compiler and even some linters get there, but there are still others where you barely know more than that the brackets are balanced before trying to execute. That form of coverage is depressingly valuable there. Makes me wonder if there is a school of testing that deliberately tries to restrict meaningful assertions to higher level tests, so that their "exercise" part is cheaper to maintain? About the terms thing: Semantics drifting away over time from whatever a sequence of letters were originally used for isn't an exception, it's standard practice in human communication. The winning strategy is adjusting expectations while receiving and being aware of possible ambiguities while sending, not sweeping together a proud little hill of pedantry to die on. The good news is that neither the article nor your little jab at the term "tested" really do that, the pedantry is really just a front to make the text a more interesting read. But it also invites the kind of sallow attack that is made very visible by discussions on the internet, but will also play out in the heads of readers elsewhere.
  - almostnormal 3y ago
    > Makes me wonder if there is a school of testing that deliberately tries to restrict meaningful assertions to higher level tests, so that their "exercise" part is cheaper to maintain? That's a nice summary of the effect I observe in the bubble I work in, but I'm sure it is not deliberate but Hanlon's razor applies. With sufficient freedom for interpretation it is what the maturity model of choice requires.
thom 3y ago
I think this article misunderstands the motivation for unit tests being isolated and ends up muddying the definition unnecessarily. The unit of unit testing means we want to assign blame to a single thing when a single test fails. That’s why order dependencies have to be eliminated, but that’s an implementation detail.
imiric 3y ago
I also wasn't aware that "unit" referred to an isolated test, not to the SUT. I usually distinguish tests by their relative level, since "unit" can be arbitrary and bring up endless discussions about what it actually means. So low-level tests are those that test a single method or class, and integration and E2E tests confirm the functionality at a higher level. I disagree with the premise that "unit", or low-level tests, are not useful because they test the implementation. These are the tests that check every single branch in the code, every possible happy and sad path, use invalid inputs, etc. The reason they're so useful is because they should a) run very quickly, and b) not require any external state or setup, i.e. the traditional "unit". This does lead to a lot of work maintaining them whenever the implementation changes, but this is a necessary chore because of the value they provide. If I'm only relying on high-level integration and E2E tests, because there's much fewer of them and they are slower and more expensive to run, I might miss a low-level bug that is only manifested under very specific conditions. This is why I still think that the traditional test pyramid is the best model to follow. Every new school of thought since then is a reaction towards the chore of maintaining "unit" tests. Yet I think we can all agree that projects like SQLite are much better for having very high testing standards[1]. I'm not saying that every project needs to do the same, but we can certainly follow their lead and aspire to that goal. [1]: https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html
- janosdebugs 3y ago
  Genuine question: can somebody please explain why there needs to be a distinction between "true" unit tests and tests that work on several layers at once as long as said tests are runnable <1min on a consumer-grade laptop without any prior setup apart from a standard language + container env setup? Over the years I had several discussion to that effect and I truly, genuinely don't understand. I have test cases that test a connector to, say, Minio, so I spin up a Minio container dynamically for each test case. I need to test an algorithm, so I isolate its dependencies amd test it. Shouldn't the point be that the thing is tested with the best tool available for the job that ensures robustness in the face of change rather than riding on semantics?
  - imiric 3y ago
JonChesterfield 3y ago
This article is internally inconsistent. It leads with considering "unit" to be "the whole system" being bad, and then tip #1 is to test from the outside in, at whole system granularity. On the other hand, it does point out that "design for test" is a nonsense, so that meets my priors. By far the worst part of TDD was the proposed resolution to the tension with encapsulation. The parts one wants to unit test are the small, isolated parts, aka "the implementation", which are also the parts one generally wants an abstraction boundary over. Two schools of thought on that: - one is to test through the API, which means a lot of tests trying to thread the needle to hit parts of the implementation. The tests will be robust to changes in the implementation, but the grey box coverage approach won't be, and you'll have a lot of tests - two is to change the API to expose the internals, market that as "good testable design" and then test the new API, much of which is only used from test code in the immediate future. Talk about how one doesn't test the implementation and don't mention the moving of goal posts Related to that is enthusiasm for putting test code somewhere separate to production code so it gets hit by the usual language isolation constraints that come from cross-module boundaries. Both of those are insane nonsense. Don't mess up your API to make testing it easier, the API was literally the point of what you're building. Write the tests in the same module as the implementation and most of the API challenge evaporates. E.g. in C++, write the tests in an anonymous namespace in the source file. Have more tests that go through the interface from outside if you like, but don't only have those, as you need way more to establish whether the implementation is still working. Much like having end to end tests helps but having only end to end tests is not helpful. I like test driven development. It's pretty hard to persuade colleagues to do it so multi-developer stuff is all end to end tested. Everything I write for myself has unit tests that look a lot like the cases I checked in the repl while thinking about the problem. It's an automated recheck-prior-reasoning system, wouldn't want to be without that.
- WolfOliver 3y ago
  > It leads with considering "unit" to be "the whole system" being bad I do not understand this statement. Could you point out which part of the article you mean.
  - 3y ago
nu11ptr 3y ago
I am not going to say that some of these testing religions don't have a place, but mostly they miss the point. By focusing on TDD or "code coverage" the essential questions are missed. Instead of focusing on methodology instead I recommend asking yourself simple questions starting with: 1. How do I meet my quality goals with this project (or module of a project)? This is the root question and it will lead to other questions: 2. What design and testing practice is most likely to lead to this outcome? 3. What is the pay off value for this module for a given type of testing? 4. How can I be confident this project/module will continue to work after refactoring? etc. etc. I have used TDD style unit testing for certain types of modules that were very algorithmic centric. I have also used integration testing only for other modules that were I/O centric without much logic. I personally think choosing a testing strategy as the "one right way" and then trying to come up with different rules to justify it is exactly inverse of how one should be thinking about it (top down vs bottom up design of sorts).
danielovichdk 3y ago
Read https://www.manning.com/books/unit-testing https://www.manning.com/books/unit-testing it's the best book on the subject and is presenting the matter with good evidence. "Tip #4: TDD says the process of writing tests first will/should drive the design of your software. " Yes and if that does not happen during TDD i would argue you are not doing TDD. Sure you always have some sort of boundaries but design up front is a poor choice when you try to iterate towards the best possible solution.
projektfu 3y ago
I think it's both attention-getting and distracting to start with a definition of unit testing that hardly anybody uses. Now I'm not interested in the article because I have to see what your sources are and whether you're gaslighting me. The reason people use the term unit test to mean the size of the system under test is because that's what it's generally meant. Before OO, it would mean module. Now it means class. The original approach would be to have smaller, testable functions that made up the functionality of the module and test them individually. Decoupling was done so that you didn't need to mock the database or the filesystem, just the logic that you're writing. Some people disagree with unit testing and focus on functional testing. For example, the programming style developed by Harlan Mills at IBM was to specify the units very carefully using formal methods and write to the specification. Then, black-box testing was used to gain confidence in the system as a whole. I feel that a refactor shouldn't break unit tests, at least not if the tools are smart enough. If you rename a method or class, its uses should have been renamed in the unit tests. If you push a method down or up in a hierarchy, a failing test tells you that the test is assuming the wrong class. But most cases of failing tests should be places where you made a mistake. However, I agree that functional tests are the hurdle you should have crossed before shipping code. Use unit testing to get 100ms results as you work, functional tests to verify that everything is working correctly. Write them so that you could confidently push to production whenever they're green.
deleted 3y ago
[deleted]
gombosg 3y ago
I think that unit tests are super valuable because when used properly, they serve as micro-specifications for each component involved. These would be super hard to backfill later, because usually only the developer who implements them knows everything about the units (services, methods, classes etc.) in question. With a strongly typed language, a suite of fast unit tests can already be in feature parity with a much slower integration test, because even if mocked out, they essentially test the whole call chain. They can offer even more, because unit tests are supposed to test edge cases, all error cases, wrong/malformed/null inputs etc. By using integration tests only, as the call chain increases on the inside, it would take an exponentially higher amount of integration tests to cover all cases. (E.g. if a call chain contains 3 services, with 3 outcomes each, theoretically it could take up to 27 integration test cases to cover them all.) Also, ballooning unit test sizes or resorting to unit testing private methods give the developer feedback that the service is probably not "single responsibility" enough, providing incentive to split and refactor it. This leads to a more maintainable service architecture, that integration tests don't help with. (Of course, let's not forget that this kind of unit testing is probably only reasonable on the backend. On the frontend, component tests from a functional/user perspective probably bring better results - hence the popularity of frameworks like Storybook and Testing Library. I consider these as integration rather than unit tests.)
skohan 3y ago
I think it also heavily depends on the language you are working with. For instance, unit tests are much more important in a duck-typed language than a strongly typed language, since the compiler is less capable of catching a number of issues.
seanwilson 3y ago
Which companies or large projects use TDD at the moment? There's always such intense discussion about what it is and its benefits, yet I don't see anyone actually doing TDD.
- JackMorgan 3y ago
  I've been in several multi-million line codebases that all were built with TDD. It's possible. The default way of organizing code with DI makes unit tests extremely expensive to write and maintain. Mocks should be banned if you want to add unit tests or practice TDD. Instead the tested code should be pure. Pure code is easy to test, even if it's calling a dozen helper functions.
- hmeh 3y ago
  Dozen-ish person team, 3 year project so far, billion dollar revenue company (not a software company), >500k LOC, TDD since the beginning. Have been doing TDD for 18 years or so. Still getting better at it.
andsmedeiros 3y ago
TDD can be valuable but sometimes hindering. I find myself often with an incomplete idea of what I want and, thus, no clear API to start testing. Writing a quick prototype -- sometimes on godbolt or replit -- and then writing tests and production code will actually yield me a better productivity. I usually test all of the public API of something and only it. Exported functions, classes, constants and whatever should be tested and properly documented. If writing tests for the public surface is not enough, most likely the underlying code is poorly written, probably lacking proper abstractions to expose the adequate state associated with a determined behaviour (e.g.: a class that does too much).
- tsimionescu 3y ago
  I think this is only true upto some point. Ultimately the API of a unit of code is not fully defined by the public VS private language features, it is defined by the conventions for its use. If a field/method is public but it is documented to not be used by anything except, say, some tests, then it shouldn't be considered part of the actual API. Even in languages which have a very powerful type system, there are assumptions that have to be left to documentation (e.g. that the Monad laws are respected by types which match the Monad typeclasses in Haskell). Testing parts which are documented to not be relevant is often actively harmful, since it causes problems with later changes.
dmos62 3y ago
This resonates. I learned the hard way that you want your main tests to integrate all layers of your system: if the system is an HTTP API, the principal tests should be about using that API. All other tests are secondary and optional: can be used if they seem useful during implementation or maintenance, but should never be relied upon to test correctness. Sometimes you have to compromise, because testing the full stack is too expensive, but that's the only reason to compromise. This is largely because if you try to test parts of your system separately, you have to perfectly simulate how they integrate with other parts, otherwise you'll get there worst case scenario: false test passes. That's too hard to do in practice. I suspect that heavy formalization of the parts' interfaces would go a long way here, but I've not yet seen that done.
- troupo 3y ago
  > if the system is an HTTP API, the principal tests should be about using that API So many times yes! Funnily enough it's also the quickest way to get high code coverage numbers that is still used as a metric everywhere
  - shoo 3y ago
    > the quickest way to get high code coverage numbers apochryphal tale of test coverage + Goodhart's law: Team is responsible for a software component that's two-thirds gnarly driver code that's impossible to unit test, and one third trivial stuff that's easy to unit test. Team has 33% unit test coverage. Test coverage becomes an organisational KPI. Teams must hit at least 80% statement coverage. Oh no! Team re-architects their component & adds several abstraction layers that wrap the gnarly driver code, that don't serve any functional purpose. Abstraction layers involve many lines of code, but are elegantly designed so they are easy to test. Now the codebase is 20% gnarly driver code that's impossible to unit test for, 10% trivial stuff that's easy to unit test, and 70% layers of unnecessary nonsense that's easy to unit test. 80% statement coverage target achieved! Raises and promos for everyone!
    - rexpop 3y ago
      I find it hard to imagine anyone stupid and scrupulous enough to go that route when the honorable and beneficial option is to a) push back on arbitrary metrics and b) strive to extract maximal value out of testing practices in earnest.
sebtron 3y ago
When I first leant about unit tests / TDD, I was confused because everyone assumes you are doing OOP. What am I supposed to do with my C code? I can just test a function, right? Or do I have to forcefully turn my program into some OO-syle architecture? But then I realized it does not matter, there is only important thing about unit tests: that they exists. All the rest is implementation detail. Mocking or not, isolated "unit" or full workflow, it does not matter. All I care about is that I can press a button (or type "make test" or whatever) and my tests run and I know if I broke something. Sure, your tests need to be maintainable, you should not need to rewrite them when you make internal changes, and so on. You'll learn as you go. Just write them and make them easy to run.
- okl 3y ago
  For C code you can use link-time substitution and a mock generator like CMock (http://www.throwtheswitch.org/cmock http://www.throwtheswitch.org/cmock). Link-time substitution means that you swap out certain objects with others when you build your test binaries. For example, let's say your production software binary consists of a main function and objects A, B and C. For a unit test you could use a different main (the test), object B and a mock for object C - leaving out A.
tbrownaw 3y ago
> Tip #1: Write the tests from outside in. > Tip #2: Do not isolate code when you test it. > Tip #3: Never change your code without having a red test. > Tip #4: TDD says the process of writing tests first will/should drive the design of your software. I never understood this. Maybe this works for other people but it does not work for me. It is software architecture 101 — Non-functional requirements (NFR) define your architecture. NFR usually do not play a role when writing unit tests. The one time I ever did "proper" red/green cycle TDD, it worked because I was writing a client library for an existing wire protocol, and knew in adance exactly what it needed to do and how it needed to do it. Item 2 is right, but this also means that #1 is wrong. And knowing what order #2 requires, means knowing how the code is designed (#4).
- sheepshear 3y ago
  The tips are not contradictory if you follow the advice to start at a higher level. Let's say you had to invent that wire protocol. You would write a test for a client that doesn't care which wire protocol is used.
  - HumblyTossed 3y ago
    TDD works great for this. Usually before I am sent a new piece of equipment (has to go through the approval/purchase process) I’m given the docs. I’ll write unit tests using the examples in the docs (or made up examples based on the docs). I’ll write my software controller against that. By the time I get the actual device I’m just confirming my code works.
- randomdata 3y ago
  TDD was later given the name Behavior Driven Development (before being usurped by the likes of Cucumber, Gerkhin) in an attempt to avoid this confusion. TDD advocates that you test that the client library does what its public interface claims it does – its behavior, not how it is implemented under the hood. The wire protocol is almost irrelevant. The tests should hold true even when the wire protocol is replaced with another protocol.
osigurdson 3y ago
My view on unit testing is if there are no dependencies, there is no real reason not to write tests for all behaviours. While you may have a wonderful integration testing suite, it is still great to know that building blocks work as intended. The problems arise with dependencies as now you need to decide to mock them or use concrete implementations. The concrete implementation might be hard to set up , slow to run in a test - or both. Using a mock, on the other hand, is essentially an alternate implementation. So now your code has the real implementation + one implementation per test (in the limit) which is plainly absurd. My current thinking (after writing a lot of mocks) is to try to shape code so that more of it can be tested without hard to setup dependencies. When this can't be done, think hard about the right approach. Try to put yourself in the shoes of a future maintainer. For example, instead of creating a bespoke mock for just your particular test, consider creating a common test utility that mocks a commonly used dependency in accordance with common testing patterns. This is just one example. Annoyingly, a lot of creativity is required once dependencies of this nature are involved which is why it is great to shape code to avoid it where possible.
- patrick451 3y ago
  The tests in the codebase I currently work in is a mocking nightmare. It feels like somebody learned about c++ interface classes and gmock for the first time when the codebase was first being put together and went completely bananas. There are almost no classes which don't inherit from a pure interface. Two of the main drawbacks to this are - Classes which have only a single implementation inherit from an interface just so they can be mocked. We often only need polymorphism for testing, but not at runtime. This not only makes the code slower (minor concern, often) but more importantly much more difficult to follow. - The tests rely heavily on implementation details. The typically assertion is NOT on the input/output behavior of some method. Rather, it's on asserting that various mocked dependencies got called at certain times within the method under test. This heavily couples the tests to the implementation and makes refactoring a pain. - We have no tests which tests multiple classes together that aren't at the scale of of system wide, end-to-end tests. So when we DI class Bar into class Foo, we use a mock and don't actually test that Bar and Foo work well together. Personally, I think the code base would be in much better shape if we completely banned gmock.
mannykannot 3y ago
The article highlights this claim: "Now, you change a little thing in your code base, and the only thing the testing suite tells you is that you will be busy the rest of the day rewriting false positive test cases." Whenever this is the case, it would seem at least one of the following is true: 1) There are many ways the 'little change' could break the system. 2) Many of the existing tests are testing for accidental properties which are not relevant to the correct functioning of the system. If only the second proposition describes the situation, then, in my experience, it is usually a consequence of tests written to help get the implementation correct being retained in the test suite. That is not necessarily a bad thing: with slight modification, they might save time in writing tests that are useful in getting the new implementation correct. I should make it clear that I don't think this observation invalidates any of the points the author is making; in fact, I think it supports them.
acidburnNSA 3y ago
Was 'unit' originally intended to be a test you could run in isolation? I don't think so. I'm not an expert in testing history, but this Dec 2000 Software Quality Assurance guide from the Nuclear Regulatory Commission defines Unit Testing as: > Unit Testing - It is defined as testing of a unit of software such as a subroutine that can be compiled or assembled. The unit is relatively small; e.g., on the order of 100 lines. A separate driver is designed and implemented in order to test the unit in the range of its applicability. NUREG-1737 https://www.nrc.gov/docs/ML0101/ML010170081.pdf https://www.nrc.gov/docs/ML0101/ML010170081.pdf Going back, this 1993 nuclear guidance has simililar language: > A unit of software is an element of the software design that can be compiled or assembled and is relatively small (e.g., 100 lines of high-order language code). Require that each software unit be separately tested. NUREG/BR-0167 https://www.nrc.gov/docs/ML0127/ML012750471.pdf https://www.nrc.gov/docs/ML0127/ML012750471.pdf
- voiceofunreason 3y ago
  See also: ANSI/IEEE 1008-1987
euos 3y ago
I took a break from a big tech and joined a startup. It was infuriating how people were actively opposing my TDD approach. It was redeeming when I was shipping one of my projects (service for integration with 3rd party) - and product managers and others were expecting we will need weeks to test and fix the bugs - but instead the other party just said "it's perfect, no comments". All because I was "wasting my time" on writing "useless tests" that helped me identify and discuss edge cases early in the process. Also, I could continue working on parts even while waiting for a response from product managers or while DevOps were still struggling to provision the resources.
dave333 3y ago
For some classic wisdom about writing tests see the classic "Art of Software Testing" by Glenford Myers. It's $149+++ on Amazon, but only $5 on ebay: https://www.ebay.com/sch/i.html?_from=R40&_trksid=p3671980.m570.l1313&_nkw=art+of+software+testing+myers&_sacat=0 https://www.ebay.com/sch/i.html?_from=R40&_trksid=p3671980.m... This was originally published before TDD was a thing, but is highly applicable.

125 points by WolfOliver 3y ago

> Write the tests from outside in. With this, I mean you should write your tests from a realistic user perspective. To have the best quality assurance and refactor resistance, you would write e2e or integration tests. Yah, yah. But good look trying to figure out what went wrong when only the failing test you have is an e2e or integration test.

MoreQARespect 3y ago
My e2e tests automatically dump a multitude of debugging information and throw open a console from which I can quickly fire up code debuggers, logs, screenshots, network traces and browser traces in a few seconds. Building test tooling is hard I'll grant you. It requires engineering chops and being up to date on the latest tooling. But not luck.

iamwil 3y ago

One way I think about it is that unit tests help me maintain invariances that the compiler for the current language can't do for me. That saves me from having to test every combination of inputs.

locallost 3y ago

Michael Feathers in "working with legacy code" tackled the unit test definition, and in the end defined a unit test as a test that runs fast. Also a very confusing topic for me early on as I tried to grasp how to actually do it. So I read Kent Beck's book on TDD and was even more confused because he did not write tests in isolation there, and I was told to write unit tests like that. But then it hit me: most people are just cargo cultists and repeat what somebody told them.

malkia 3y ago
One of the best things in bazel is that it's very straightforward when comes to tests - https://bazel.build/reference/test-encyclopedia#role-test-runner https://bazel.build/reference/test-encyclopedia#role-test-ru... - you need to classify your test as short, moderate, long, etc., but also how much memory it takes. In addition if you tests requires isolation, put it too. So potentionally one can combine multiple independent code bases, that share the same testing framework (say gtest), into single executable, and distributevly execute them using sharding on multiple machines (e.g. depending on what changes were detected in the presbumit/preflight/whatever you call phase). And because the tests were marked as such and such, it'll use that knowledge to group them better, and if they no longer qualify, an error would be printed out, and how to fix this (if it's worth fixing).

voiceofunreason 3y ago

<< Originally, the term “unit” in “unit test” referred not to the system under test but to the test itself. >> Retroactive continuity - how does it work? For today's lucky 10,000: "unit test", as a label, was in wide use prior to the arrival of the Extreme Programming movement in the late 1990s. And the "unit" in question was the test subject. But, as far as I can tell, _Smalltalk_ lacked a testing culture (Beck, 1994), so perhaps the testing community's definitions weren't well represented in Smalltalk spaces. "The past was alterable. The past never had been altered." (Not particularly fair to single out this one author - this origin myth has been common during the last 10 years or so.)

jdougan 3y ago
At my job we wrote tests in Smalltalk and that was before Kent's work. It wasn't the later tests-first discipline. If I recall correctly write tests later was fairly common then in ST shops. SUnit also had the virtue of standardizing the terminology and testing frameworks.

emadb 3y ago

The big TDD misunderstanding is that most people consider TDD a testing practice. The article doesn’t talk about TDD, it gives the reader some tips on how to write tests. That’s not TDD.

shimst3r 3y ago
Instead of Test-Driven Design, it should’ve been called Design-By-Testing.
- paulluuk 3y ago
  Did you mean Test Driven Development, or is Test-Driven Design a whole other thing?
skrebbel 3y ago
Yeah it’s kind of unfortunate because they make a very good argument about defining a thing better, and in the title use a wrong definition of an adjacent term.
WolfOliver 3y ago
Maybe the term TDD in the title can be replaced with "unit testing". But unit testing is an major part of TDD.
MoreQARespect 3y ago
I'm fully aware of the idea that TDD is a "design practice" but I find it to be completely wrongheaded. The principle that tests that couple to low level code give you feedback about tightly coupled code is true but it does that because low level/unit tests couple too tightly to your code - I.e. because they too are bad code! Have you ever refactored working code into working code and had a slew of tests fail anyway? That's the child of test driven design. High level/integration TDD doesnt give "feedback" on your design it just tells you if your code matches the spec. This is actually more useful. It then lets you refactor bad code with a safety harness and give failures that actually mean failure and not "changed code". I keep wishing for the idea of test driven design to die. Writing tests which break on working code is inordinately uneconomic way to detect design issues as compared to developing an eye for it and fixing it under a test harness with no opinion on your design. So, yes this - high level test driven development - is TDD and moreover it's got a better cost/benefit trade off than test driven design.

michalc 3y ago

> Now, you change a little thing in your code base, and the only thing the testing suite tells you is that you will be busy the rest of the day rewriting false positive test cases. If there is anything that makes me cry, it’s hearing “it’s done, now I need to fix the tests”

WolfOliver 3y ago
Agree, this is usually a sign the team writes tests for the sake of writing tests.
scaramanga 3y ago
If changing the implementation but not the behaviour breaks a test, I just delete the test.
tetha 3y ago
It's something we've changed when we switched our configuration management. The old config management had very, very meticulous tests of everything. This resulted in great "code" coverage, but whenever you changed a default value, at least 6 tests would fail now. Now, we much rather go ahead and test much more coarsely. If the config management can take 3 VMs and setup a RabbitMQ cluster that clusters and accepts messages, how wrong can it be? And this has also bled into my development and strengthened my support of bug-driven testing. For a lot of pretty simple business logic, do a few high level e2e tests for the important behaviors. And then when it breaks, add more tests for those parts. But note, this may be different for very fiddly parts of the code base - complex algorithms, math-heavy and such. But that's when you'd rather start table based testing and such. At a past gamedev job, we had several issues with some complex cost balancing math, so I eventually setup a test that allows the game balancing team to supply CSV files with expected results. That cleared up these issues within 2 days or so.
- tremon 3y ago
  whenever you changed a default value, at least 6 tests would fail now Testing default values makes a lot of sense. Both non-set configuration values and non-supplied function parameters become part of your API. Your consumers will rely on those default values, and if you alter them, your consumers will see different behaviour.

gardenhedge 3y ago

> Never change your code without having a red test I'll never understand why people insist on this. If you want to write your tests first, that is fine. Noone is going to stop you. But why must you insist everyone does it this way?

shimst3r 3y ago
If you do TDD, you write your tests first. If you don’t follow TDD, you don’t have to. The author insists on this because they assume a TDD approach.
- gardenhedge 3y ago
  I suppose that is a fair reading. If TDD is the goal then the "never" instruction makes sense.
stepbeek 3y ago
In the context of the article, I read this as "modify the test first for an existing unit test and a bit of production code under test". I don't think this is equivalent to "always write the test first".
tarcio 3y ago
It helps you think about the problem better. A particular test has assertions that must be implemented correctly in order for them to pass. By starting in red you gradually make your way towards green usually paving your thinking and potentially unlocking other tests. I don't believe you must do this every time though.
- gardenhedge 3y ago
  > I don't believe you must do this every time though. So I think we agree. If you find it beneficial, please carry on doing it. For anyone who doesn't find it beneficial, they can write tests in whatever way they want.
resonious 3y ago

resonious 3y ago

Huh, I found this more interesting than I thought I would. I hadn't heard before that the "unit" in "unit test" just meant "can run independently". I once failed an interview partly because of only writing "feature tests" and not "unit tests" in the project I showed. But actually those tests didn't depend on each other, so... looks like they really were unit tests! Anyway, I'm still not totally sure about TDD itself - the "don't write any code without a red test" part. I get the idea, but it doesn't feel very productive when I try it. Of course maybe I'm just bad at it, but I also haven't seen any compelling arguments for it other than it makes the tests stronger (against what? someone undoing my commits?). I think even Uncle Bob's underlying backing argument was that TDD is more "professional", leading me to believe it's just a song-and-dance secret handshake that helps you get into a certain kind of company. OR, it's a technique to try and combat against lazy devs, to try and make it impossible for them to write bad tests. And maybe it is actually good but only for some kinds of projects... I wish we had a way to actually research this stuff rather than endlessly share opinions and anecdotes.

viraptor 3y ago
> arguments for it other than it makes the tests stronger It's supposed to lead to a better design. It's easy to write some code that maybe works and you can't actually test (lots of interdependencies, weird state, etc.), or you only think you're trying correctly. But making the test first forces you to write something that 1. You can test (by definition) 2. Is decoupled to the level where you can check mainly for the behaviour you're interested in. 3. You won't bypass accidentally. It's not even someone undoing your commits, but some value in the call chain changing in a way that accidentally makes the feature not run at all. I've seen it many times in practice and will bet that any large project where the tests were written after the code, has some tests that don't actually do anything. They were already passing before the thing they're supposedly testing was implemented.
MoreQARespect 3y ago
If I have the tooling all set up (e.g. playwright, database fixtures, mitmproxy) and the integration test closely resembles the requirement then I'm about as productive doing TDD as not doing TDD except I get tests as a side effect. If I do snapshot test driven development (e.g. actual rest API responses are written into the "expected" portion of the test by the test) then I'm sometimes a little bit more productive. There's a definite benefit to fixing the requirement rather than letting it evaporate into the ether. Uncle bob style unit test driven development, on the other hand, is something more akin to a ritual from a cult. Unit test driven development on integration code (e.g. code that handles APIs, databases, UIs) is singularly useless. It only really works well on algorithmic or logical code - parsers, pricing engines, etc. where the requirement can be well represented.

philippta 3y ago

In my experience a lot of engineers are stuck thinking in MVC terms an fail to write modular code. As a result most business logic is part of a request / response flow. This makes it infeasible to even attempt to write tests first, thus leaving integration or e2e tests as the only remaining options.

trwrto 3y ago
Where should the business logic rather be? My tests are typically calling APIs to test the business logic. Trying to improve myself here.
- philippta 3y ago
  There aren‘t any hard rules here, but you can try to build your business logic as if it was a library, where your HTTP API is merely a interface to it.
laurencerowe 3y ago
I’m not a TDD purist but I’ve found that so long as the request / response flow is a JSON api or similar (as opposed to old style forms and html rendering) then writing integration tests first is quite easy so long as you make sure your test fixtures are fairly fast.
- _heimdall 3y ago
  With this approach do you stop at testing the JSON API or do you still make it to testing the rendered HTML actually shown to the user? I've always actually likes the simplicity of testing HTML APIs in a frontend project. For me, tests get a lot more simple when I can verify the final HTML directly from the response and don't need to parse the JSON and run it through client-side rendering logic first.
  - laurencerowe 3y ago
    While we had a few end to end browser tests they tended to be slow so the pattern we ended up at was to generate output from the API to use as mock data in the UI tests. I think it usually make sense to test the business logic at the JSON API level (except for particularly complicated parts where unit testing of that complex logic may make sense) and then use snapshot tests for the display logic.

almostnormal 3y ago

Part of the problem is caused by all sides using the same terms but with a different meaning. > You just don’t know if your system works as a whole, even though each line is tested. ... even though each line has been executed. One test per line is strongly supported by tools calculating coverage and calling that "tested". A test for one specific line is rarely possible. It may be missing some required behavior that hasn't been challanged by any test, or it may be inconsistent with other parts of the code. A good start would be to stop calling something just executed "tested".

usrusr 3y ago
I like the term "exercised" for coverage of questionable assertative value. It's rather pointless in environments with a strict compiler and even some linters get there, but there are still others where you barely know more than that the brackets are balanced before trying to execute. That form of coverage is depressingly valuable there. Makes me wonder if there is a school of testing that deliberately tries to restrict meaningful assertions to higher level tests, so that their "exercise" part is cheaper to maintain? About the terms thing: Semantics drifting away over time from whatever a sequence of letters were originally used for isn't an exception, it's standard practice in human communication. The winning strategy is adjusting expectations while receiving and being aware of possible ambiguities while sending, not sweeping together a proud little hill of pedantry to die on. The good news is that neither the article nor your little jab at the term "tested" really do that, the pedantry is really just a front to make the text a more interesting read. But it also invites the kind of sallow attack that is made very visible by discussions on the internet, but will also play out in the heads of readers elsewhere.
- almostnormal 3y ago
  > Makes me wonder if there is a school of testing that deliberately tries to restrict meaningful assertions to higher level tests, so that their "exercise" part is cheaper to maintain? That's a nice summary of the effect I observe in the bubble I work in, but I'm sure it is not deliberate but Hanlon's razor applies. With sufficient freedom for interpretation it is what the maturity model of choice requires.

thom 3y ago

I think this article misunderstands the motivation for unit tests being isolated and ends up muddying the definition unnecessarily. The unit of unit testing means we want to assign blame to a single thing when a single test fails. That’s why order dependencies have to be eliminated, but that’s an implementation detail.

imiric 3y ago

I also wasn't aware that "unit" referred to an isolated test, not to the SUT. I usually distinguish tests by their relative level, since "unit" can be arbitrary and bring up endless discussions about what it actually means. So low-level tests are those that test a single method or class, and integration and E2E tests confirm the functionality at a higher level. I disagree with the premise that "unit", or low-level tests, are not useful because they test the implementation. These are the tests that check every single branch in the code, every possible happy and sad path, use invalid inputs, etc. The reason they're so useful is because they should a) run very quickly, and b) not require any external state or setup, i.e. the traditional "unit". This does lead to a lot of work maintaining them whenever the implementation changes, but this is a necessary chore because of the value they provide. If I'm only relying on high-level integration and E2E tests, because there's much fewer of them and they are slower and more expensive to run, I might miss a low-level bug that is only manifested under very specific conditions. This is why I still think that the traditional test pyramid is the best model to follow. Every new school of thought since then is a reaction towards the chore of maintaining "unit" tests. Yet I think we can all agree that projects like SQLite are much better for having very high testing standards[1]. I'm not saying that every project needs to do the same, but we can certainly follow their lead and aspire to that goal. [1]: https://www.sqlite.org/testing.html https://www.sqlite.org/testing.html

janosdebugs 3y ago
Genuine question: can somebody please explain why there needs to be a distinction between "true" unit tests and tests that work on several layers at once as long as said tests are runnable <1min on a consumer-grade laptop without any prior setup apart from a standard language + container env setup? Over the years I had several discussion to that effect and I truly, genuinely don't understand. I have test cases that test a connector to, say, Minio, so I spin up a Minio container dynamically for each test case. I need to test an algorithm, so I isolate its dependencies amd test it. Shouldn't the point be that the thing is tested with the best tool available for the job that ensures robustness in the face of change rather than riding on semantics?
- imiric 3y ago

JonChesterfield 3y ago

This article is internally inconsistent. It leads with considering "unit" to be "the whole system" being bad, and then tip #1 is to test from the outside in, at whole system granularity. On the other hand, it does point out that "design for test" is a nonsense, so that meets my priors. By far the worst part of TDD was the proposed resolution to the tension with encapsulation. The parts one wants to unit test are the small, isolated parts, aka "the implementation", which are also the parts one generally wants an abstraction boundary over. Two schools of thought on that: - one is to test through the API, which means a lot of tests trying to thread the needle to hit parts of the implementation. The tests will be robust to changes in the implementation, but the grey box coverage approach won't be, and you'll have a lot of tests - two is to change the API to expose the internals, market that as "good testable design" and then test the new API, much of which is only used from test code in the immediate future. Talk about how one doesn't test the implementation and don't mention the moving of goal posts Related to that is enthusiasm for putting test code somewhere separate to production code so it gets hit by the usual language isolation constraints that come from cross-module boundaries. Both of those are insane nonsense. Don't mess up your API to make testing it easier, the API was literally the point of what you're building. Write the tests in the same module as the implementation and most of the API challenge evaporates. E.g. in C++, write the tests in an anonymous namespace in the source file. Have more tests that go through the interface from outside if you like, but don't only have those, as you need way more to establish whether the implementation is still working. Much like having end to end tests helps but having only end to end tests is not helpful. I like test driven development. It's pretty hard to persuade colleagues to do it so multi-developer stuff is all end to end tested. Everything I write for myself has unit tests that look a lot like the cases I checked in the repl while thinking about the problem. It's an automated recheck-prior-reasoning system, wouldn't want to be without that.

WolfOliver 3y ago
> It leads with considering "unit" to be "the whole system" being bad I do not understand this statement. Could you point out which part of the article you mean.
- 3y ago

nu11ptr 3y ago

I am not going to say that some of these testing religions don't have a place, but mostly they miss the point. By focusing on TDD or "code coverage" the essential questions are missed. Instead of focusing on methodology instead I recommend asking yourself simple questions starting with: 1. How do I meet my quality goals with this project (or module of a project)? This is the root question and it will lead to other questions: 2. What design and testing practice is most likely to lead to this outcome? 3. What is the pay off value for this module for a given type of testing? 4. How can I be confident this project/module will continue to work after refactoring? etc. etc. I have used TDD style unit testing for certain types of modules that were very algorithmic centric. I have also used integration testing only for other modules that were I/O centric without much logic. I personally think choosing a testing strategy as the "one right way" and then trying to come up with different rules to justify it is exactly inverse of how one should be thinking about it (top down vs bottom up design of sorts).

danielovichdk 3y ago

Read https://www.manning.com/books/unit-testing https://www.manning.com/books/unit-testing it's the best book on the subject and is presenting the matter with good evidence. "Tip #4: TDD says the process of writing tests first will/should drive the design of your software. " Yes and if that does not happen during TDD i would argue you are not doing TDD. Sure you always have some sort of boundaries but design up front is a poor choice when you try to iterate towards the best possible solution.

projektfu 3y ago

I think it's both attention-getting and distracting to start with a definition of unit testing that hardly anybody uses. Now I'm not interested in the article because I have to see what your sources are and whether you're gaslighting me. The reason people use the term unit test to mean the size of the system under test is because that's what it's generally meant. Before OO, it would mean module. Now it means class. The original approach would be to have smaller, testable functions that made up the functionality of the module and test them individually. Decoupling was done so that you didn't need to mock the database or the filesystem, just the logic that you're writing. Some people disagree with unit testing and focus on functional testing. For example, the programming style developed by Harlan Mills at IBM was to specify the units very carefully using formal methods and write to the specification. Then, black-box testing was used to gain confidence in the system as a whole. I feel that a refactor shouldn't break unit tests, at least not if the tools are smart enough. If you rename a method or class, its uses should have been renamed in the unit tests. If you push a method down or up in a hierarchy, a failing test tells you that the test is assuming the wrong class. But most cases of failing tests should be places where you made a mistake. However, I agree that functional tests are the hurdle you should have crossed before shipping code. Use unit testing to get 100ms results as you work, functional tests to verify that everything is working correctly. Write them so that you could confidently push to production whenever they're green.

deleted 3y ago

[deleted]

gombosg 3y ago

I think that unit tests are super valuable because when used properly, they serve as micro-specifications for each component involved. These would be super hard to backfill later, because usually only the developer who implements them knows everything about the units (services, methods, classes etc.) in question. With a strongly typed language, a suite of fast unit tests can already be in feature parity with a much slower integration test, because even if mocked out, they essentially test the whole call chain. They can offer even more, because unit tests are supposed to test edge cases, all error cases, wrong/malformed/null inputs etc. By using integration tests only, as the call chain increases on the inside, it would take an exponentially higher amount of integration tests to cover all cases. (E.g. if a call chain contains 3 services, with 3 outcomes each, theoretically it could take up to 27 integration test cases to cover them all.) Also, ballooning unit test sizes or resorting to unit testing private methods give the developer feedback that the service is probably not "single responsibility" enough, providing incentive to split and refactor it. This leads to a more maintainable service architecture, that integration tests don't help with. (Of course, let's not forget that this kind of unit testing is probably only reasonable on the backend. On the frontend, component tests from a functional/user perspective probably bring better results - hence the popularity of frameworks like Storybook and Testing Library. I consider these as integration rather than unit tests.)

skohan 3y ago

I think it also heavily depends on the language you are working with. For instance, unit tests are much more important in a duck-typed language than a strongly typed language, since the compiler is less capable of catching a number of issues.

seanwilson 3y ago

Which companies or large projects use TDD at the moment? There's always such intense discussion about what it is and its benefits, yet I don't see anyone actually doing TDD.

JackMorgan 3y ago
I've been in several multi-million line codebases that all were built with TDD. It's possible. The default way of organizing code with DI makes unit tests extremely expensive to write and maintain. Mocks should be banned if you want to add unit tests or practice TDD. Instead the tested code should be pure. Pure code is easy to test, even if it's calling a dozen helper functions.
hmeh 3y ago
Dozen-ish person team, 3 year project so far, billion dollar revenue company (not a software company), >500k LOC, TDD since the beginning. Have been doing TDD for 18 years or so. Still getting better at it.

andsmedeiros 3y ago

TDD can be valuable but sometimes hindering. I find myself often with an incomplete idea of what I want and, thus, no clear API to start testing. Writing a quick prototype -- sometimes on godbolt or replit -- and then writing tests and production code will actually yield me a better productivity. I usually test all of the public API of something and only it. Exported functions, classes, constants and whatever should be tested and properly documented. If writing tests for the public surface is not enough, most likely the underlying code is poorly written, probably lacking proper abstractions to expose the adequate state associated with a determined behaviour (e.g.: a class that does too much).

tsimionescu 3y ago
I think this is only true upto some point. Ultimately the API of a unit of code is not fully defined by the public VS private language features, it is defined by the conventions for its use. If a field/method is public but it is documented to not be used by anything except, say, some tests, then it shouldn't be considered part of the actual API. Even in languages which have a very powerful type system, there are assumptions that have to be left to documentation (e.g. that the Monad laws are respected by types which match the Monad typeclasses in Haskell). Testing parts which are documented to not be relevant is often actively harmful, since it causes problems with later changes.

dmos62 3y ago

This resonates. I learned the hard way that you want your main tests to integrate all layers of your system: if the system is an HTTP API, the principal tests should be about using that API. All other tests are secondary and optional: can be used if they seem useful during implementation or maintenance, but should never be relied upon to test correctness. Sometimes you have to compromise, because testing the full stack is too expensive, but that's the only reason to compromise. This is largely because if you try to test parts of your system separately, you have to perfectly simulate how they integrate with other parts, otherwise you'll get there worst case scenario: false test passes. That's too hard to do in practice. I suspect that heavy formalization of the parts' interfaces would go a long way here, but I've not yet seen that done.

troupo 3y ago
> if the system is an HTTP API, the principal tests should be about using that API So many times yes! Funnily enough it's also the quickest way to get high code coverage numbers that is still used as a metric everywhere
- shoo 3y ago
  > the quickest way to get high code coverage numbers apochryphal tale of test coverage + Goodhart's law: Team is responsible for a software component that's two-thirds gnarly driver code that's impossible to unit test, and one third trivial stuff that's easy to unit test. Team has 33% unit test coverage. Test coverage becomes an organisational KPI. Teams must hit at least 80% statement coverage. Oh no! Team re-architects their component & adds several abstraction layers that wrap the gnarly driver code, that don't serve any functional purpose. Abstraction layers involve many lines of code, but are elegantly designed so they are easy to test. Now the codebase is 20% gnarly driver code that's impossible to unit test for, 10% trivial stuff that's easy to unit test, and 70% layers of unnecessary nonsense that's easy to unit test. 80% statement coverage target achieved! Raises and promos for everyone!
  - rexpop 3y ago
    I find it hard to imagine anyone stupid and scrupulous enough to go that route when the honorable and beneficial option is to a) push back on arbitrary metrics and b) strive to extract maximal value out of testing practices in earnest.

sebtron 3y ago

When I first leant about unit tests / TDD, I was confused because everyone assumes you are doing OOP. What am I supposed to do with my C code? I can just test a function, right? Or do I have to forcefully turn my program into some OO-syle architecture? But then I realized it does not matter, there is only important thing about unit tests: that they exists. All the rest is implementation detail. Mocking or not, isolated "unit" or full workflow, it does not matter. All I care about is that I can press a button (or type "make test" or whatever) and my tests run and I know if I broke something. Sure, your tests need to be maintainable, you should not need to rewrite them when you make internal changes, and so on. You'll learn as you go. Just write them and make them easy to run.

okl 3y ago
For C code you can use link-time substitution and a mock generator like CMock (http://www.throwtheswitch.org/cmock http://www.throwtheswitch.org/cmock). Link-time substitution means that you swap out certain objects with others when you build your test binaries. For example, let's say your production software binary consists of a main function and objects A, B and C. For a unit test you could use a different main (the test), object B and a mock for object C - leaving out A.

tbrownaw 3y ago

> Tip #1: Write the tests from outside in. > Tip #2: Do not isolate code when you test it. > Tip #3: Never change your code without having a red test. > Tip #4: TDD says the process of writing tests first will/should drive the design of your software. I never understood this. Maybe this works for other people but it does not work for me. It is software architecture 101 — Non-functional requirements (NFR) define your architecture. NFR usually do not play a role when writing unit tests. The one time I ever did "proper" red/green cycle TDD, it worked because I was writing a client library for an existing wire protocol, and knew in adance exactly what it needed to do and how it needed to do it. Item 2 is right, but this also means that #1 is wrong. And knowing what order #2 requires, means knowing how the code is designed (#4).

sheepshear 3y ago
The tips are not contradictory if you follow the advice to start at a higher level. Let's say you had to invent that wire protocol. You would write a test for a client that doesn't care which wire protocol is used.
- HumblyTossed 3y ago
  TDD works great for this. Usually before I am sent a new piece of equipment (has to go through the approval/purchase process) I’m given the docs. I’ll write unit tests using the examples in the docs (or made up examples based on the docs). I’ll write my software controller against that. By the time I get the actual device I’m just confirming my code works.
randomdata 3y ago
TDD was later given the name Behavior Driven Development (before being usurped by the likes of Cucumber, Gerkhin) in an attempt to avoid this confusion. TDD advocates that you test that the client library does what its public interface claims it does – its behavior, not how it is implemented under the hood. The wire protocol is almost irrelevant. The tests should hold true even when the wire protocol is replaced with another protocol.

osigurdson 3y ago

My view on unit testing is if there are no dependencies, there is no real reason not to write tests for all behaviours. While you may have a wonderful integration testing suite, it is still great to know that building blocks work as intended. The problems arise with dependencies as now you need to decide to mock them or use concrete implementations. The concrete implementation might be hard to set up , slow to run in a test - or both. Using a mock, on the other hand, is essentially an alternate implementation. So now your code has the real implementation + one implementation per test (in the limit) which is plainly absurd. My current thinking (after writing a lot of mocks) is to try to shape code so that more of it can be tested without hard to setup dependencies. When this can't be done, think hard about the right approach. Try to put yourself in the shoes of a future maintainer. For example, instead of creating a bespoke mock for just your particular test, consider creating a common test utility that mocks a commonly used dependency in accordance with common testing patterns. This is just one example. Annoyingly, a lot of creativity is required once dependencies of this nature are involved which is why it is great to shape code to avoid it where possible.

patrick451 3y ago
The tests in the codebase I currently work in is a mocking nightmare. It feels like somebody learned about c++ interface classes and gmock for the first time when the codebase was first being put together and went completely bananas. There are almost no classes which don't inherit from a pure interface. Two of the main drawbacks to this are - Classes which have only a single implementation inherit from an interface just so they can be mocked. We often only need polymorphism for testing, but not at runtime. This not only makes the code slower (minor concern, often) but more importantly much more difficult to follow. - The tests rely heavily on implementation details. The typically assertion is NOT on the input/output behavior of some method. Rather, it's on asserting that various mocked dependencies got called at certain times within the method under test. This heavily couples the tests to the implementation and makes refactoring a pain. - We have no tests which tests multiple classes together that aren't at the scale of of system wide, end-to-end tests. So when we DI class Bar into class Foo, we use a mock and don't actually test that Bar and Foo work well together. Personally, I think the code base would be in much better shape if we completely banned gmock.

mannykannot 3y ago

The article highlights this claim: "Now, you change a little thing in your code base, and the only thing the testing suite tells you is that you will be busy the rest of the day rewriting false positive test cases." Whenever this is the case, it would seem at least one of the following is true: 1) There are many ways the 'little change' could break the system. 2) Many of the existing tests are testing for accidental properties which are not relevant to the correct functioning of the system. If only the second proposition describes the situation, then, in my experience, it is usually a consequence of tests written to help get the implementation correct being retained in the test suite. That is not necessarily a bad thing: with slight modification, they might save time in writing tests that are useful in getting the new implementation correct. I should make it clear that I don't think this observation invalidates any of the points the author is making; in fact, I think it supports them.

acidburnNSA 3y ago

Was 'unit' originally intended to be a test you could run in isolation? I don't think so. I'm not an expert in testing history, but this Dec 2000 Software Quality Assurance guide from the Nuclear Regulatory Commission defines Unit Testing as: > Unit Testing - It is defined as testing of a unit of software such as a subroutine that can be compiled or assembled. The unit is relatively small; e.g., on the order of 100 lines. A separate driver is designed and implemented in order to test the unit in the range of its applicability. NUREG-1737 https://www.nrc.gov/docs/ML0101/ML010170081.pdf https://www.nrc.gov/docs/ML0101/ML010170081.pdf Going back, this 1993 nuclear guidance has simililar language: > A unit of software is an element of the software design that can be compiled or assembled and is relatively small (e.g., 100 lines of high-order language code). Require that each software unit be separately tested. NUREG/BR-0167 https://www.nrc.gov/docs/ML0127/ML012750471.pdf https://www.nrc.gov/docs/ML0127/ML012750471.pdf

voiceofunreason 3y ago
See also: ANSI/IEEE 1008-1987

euos 3y ago

I took a break from a big tech and joined a startup. It was infuriating how people were actively opposing my TDD approach. It was redeeming when I was shipping one of my projects (service for integration with 3rd party) - and product managers and others were expecting we will need weeks to test and fix the bugs - but instead the other party just said "it's perfect, no comments". All because I was "wasting my time" on writing "useless tests" that helped me identify and discuss edge cases early in the process. Also, I could continue working on parts even while waiting for a response from product managers or while DevOps were still struggling to provision the resources.

dave333 3y ago

For some classic wisdom about writing tests see the classic "Art of Software Testing" by Glenford Myers. It's $149+++ on Amazon, but only $5 on ebay: https://www.ebay.com/sch/i.html?_from=R40&_trksid=p3671980.m570.l1313&_nkw=art+of+software+testing+myers&_sacat=0 https://www.ebay.com/sch/i.html?_from=R40&_trksid=p3671980.m... This was originally published before TDD was a thing, but is highly applicable.