16 ms·

The Big TDD Misunderstanding

118 points by WolfOliver 5y ago

sorokod 5y ago
The issue with TDD as originally described by Kent Beck (there are as many variations as there are ... ) is the belief that a good enough design emerges from the process.
- qayxc 5y ago
  While I agree that a good architecture cannot be derived from (unit-)tests alone, tests do indeed help a lot with good design. As soon as you need elaborate setup in order to test a single piece of functionality (or function), you will start to rethink your design. In such cases there's too much coupling, too many dependencies, or too many responsibilities within the function under test. Seeing tests as separate from implementation and design is a much bigger issue IMHO. TDD only provides value as opposed to overhead, if you understand it as an integral part of the design and implementation process and use them with that in mind.
  - sorokod 5y ago
    That last sentence seems to do a lot of heavy lifting. Where does it fit in the red-green-refactor mantra?
    - qayxc 5y ago
      Red-green-refactoring doesn't work if you are dealing with existing software, for example (unless there's a well-documented and -understood defect). It's not TDD (emphasis of the "driven"-part) if tests are only used to confirm already implemented behaviour. I found that especially in larger organisations there's all too often hard requirements for unit tests and high test coverage. This doesn't work as intended, though, if the team doesn't "live" by the TDD concept and unit tests are only there to check some box on a "best practices" list. IMO it's perfectly fine to only use TDD for some components of an application and rely on integration- and acceptance tests only for others. Glue code and cross-cutting components don't benefit from TDD. An external API or component that you don't control can also prevent meaningful unit tests and force certain design decisions, rendering TDD an exercise in writing (and maintaining) mocks, etc. What I tried to express is that TDD is one tool of many and not always the best choice; especially so when it is tacked onto an existing process that doesn't use it as intended or cannot benefit from it.
deanCommie 5y ago
The big TDD misunderstanding is that like any tool it is to be wielded when appropriate, not followed rigidly for every single piece of code you write. There are some safety standards and practices in engineering that are matters of life and death. If you're working high above the ground, you need two points of safety clipping, and if you're moving you never disconnect both of them. This is non-negotiable. There is never an appropriate time to violate this. If there is no way to complete the job without constant safety harness restraint, the job doesn't get done. (At least in the developed world.) This is not the category where TDD or rules like "Never change your code without having a red test." fall in. I adore TDD. It made me a better engineer, and I followed the process rigorously in my first 5-10 years of my career. But at some point I realized that what that did is teach me how to write modular, extensible, testable code, and now I can write modules like that upfront, and add tests later. And truthfully, I do go faster. Because I also then focus my tests on JUST the important logic, not the intermediate abstractions in between. I write unit tests for the critical functionality and *integration* tests for the whole system to make sure the whole thing works as expected (because the dirty truth of unit tests is you can have 100% test coverage but a non-functional systems due to frameworks and glue code being incorrectly configured.
- qayxc 5y ago
  High test coverage is the worst metric one can aim for. As soon as frameworks and external APIs are involved, it's basically impossible to achieve without bending over backwards and needlessly creating mocks for half your system. My biggest pet peeve in this context are test coverage requirements for OO languages that feature properties or getters and setters. 100% test coverage would require writing "tests" for those, which basically amounts to testing whether the compiler works - madness!
bedobi 5y ago
Yes! This can't be repeated enough IMO > Terminology > The test that hits the endpoint is commonly referred to as an “integration test”, and the other test is commonly referred to as a “unit test”. > Kent Beck (the originator of test-driven development) defines a unit test as “a test that runs in isolation from other tests”. This is very different from the definition of a unit test as “a test that tests a class/method in isolation from other classes/methods”. The test that hits the endpoint satisfies the original definition. > But it doesn’t really matter if you want to call a given test an “integration” test or a “unit” test. The point is the test fails when something breaks and passes when something is improved. If it does the opposite, it’s not a good test. https://medium.com/expedia-group-tech/test-your-tests-f4e361818720 https://medium.com/expedia-group-tech/test-your-tests-f4e361...
- mrjin 5y ago
  The sadly fact is that lots of people cannot tell the differences between Unit Tests, Functional Tests and Integration Tests even after in the field for over a decade.
  - beebeepka 5y ago
    I welcome you to share your opinion on the matter
    - bckr 5y ago
      I'm going to give it a crack. Caveat that I have not read the TDD scriptures. Unit test: tests a bit of code not much larger than a function, to make sure that the function runs properly and gives the expected outputs. Functional tests: tests some code not much bigger than a module (source file) to make sure it does the right things, from a business logic standpoint. Integration tests: tests some systems not much bigger than one subsystem making an API call to another system, to make sure that the systems can talk to each other properly.
      - Aeolun 5y ago
gleenn 5y ago
I will start that I think TDD is very valuable. But reading this article that quickly goes into a numbered list of "rules" is a little disheartening. Already some other commenters are talking about more nuanced views of when to test. I think the real tragedy is that I see a lot of engineers still not writing tests. So when speaking to the broader audience, you have to be really gung-ho about it to get people to do what I think is a very necessary step. People should err on the side of writing the test first, both to get most people in the right mindset, and also to avoid the broken window syndrome of a lot of missing tests and no one bothering to actually start doing what should be obvious: most code should have at least some level of coverage. Most in fact. If there are "reasons", sure, if you're going to pull the senior engineer move and skip the test, then you better be able to back it up clearly. If not, write the damn test. If not you, someone will appreciate it later.
dwheeler 5y ago
Another huge problem with TDD is a lot of practitioners do not create negative tests. A negative test verifies that something the system shouldn't do is not being done. For example, if you shouldn't allow unauthenticated users unauthenticated users to delete something, there should be a test to verify that authenticated users can't delete something. But if you only add tests that will fail and then make succeed via implementation, you would probably never think to make those kind of test. You certainly can create negative tests with TDD, but most docs on TDD never discuss it.
- jay_kyburz 5y ago
  There should be a test looking for duplicate phrases. (I'm laughing with you not at you :))
- deleted 5y ago
  [deleted]
- dqv 5y ago
  This is how I’ve been making sure multitenancy in my app doesn’t leak data. It shouldn’t leak but how can I be sure? Run tests that try to access data on one tenant from another tenant and fail if the data is accessible.
  - eb0la 5y ago
    Most of the time we make tests when something is in our radar. I guess you have use case in your spec that tells you that's the expected behaviour, or at least some clever developer figured out it should be done that way. But it is not always so "easy": Sometimes you don't have the domain knowledge, or the spec doesn't say a word about negative cases...
  - onion2k 5y ago
    Run tests that try to access data on one tenant from another tenant and fail if the data is accessible. That tells you that the method your tests use to try to access data don't work, not that it's impossible to access data across tenants. If there's a leak you haven't tested for then your app is still insecure. It'd be better to think of a design that actually makes it impossible (assuming that's even possible for your scenario).
ehutch79 5y ago
This is contrary to most testing tutorials and documentation I've read. I get frustrated and forego a lot of testing because the idea of mocking out basically the whole system in order to unit test is insane to me. If function A calls function B, you've got to mock function B out. If you refactor function B, you better update the mock as well. Writing an app twice hardly seems like a good use of time.
- tra3 5y ago
  I only mock functionality that is non deterministic — like sending email.
- rhdunn 5y ago
  That way leads to madness, and I don't follow that style/philosophy of mocking components. Only mock the things you can't easily set up in your unit tests to get them to work, like database connections. For spring-based components, try to use the real components where possible, but where needed use mock components that do enough to get the system working. That does mean that if you are testing function A you are also indirectly testing function B, but that is fine.
- pintxo 5y ago
  If function A does not much more than calling B, maybe testing A is not strictly necessary. If A does something important/complex/critical/etc maybe the important/complex/critical/etc part can be refactored into a dedicated function A1 without external dependency and then tested on its own? That way, the tests focuses on where the money is and gets easier to work with at the same time.
  - adwn 5y ago
    Now you've added an additional, untested point of failure (the glue code), and you've made the interaction between A, A1, and B more difficult to understanding and more error prone.
    - cloogshicer 5y ago
theteapot 5y ago
Article: > "Originally the unit in “unit test” did not refer to the system under test but the test itself. Meaning the test can be executed as one unit and does not depend on other tests to run upfront (see here and here)." I find the links provided to support for this proposition underwhelming. In any case it matters zero. No-one interprets unit test that way any more. Everyone interprets it the other way round, like defined on Wikipedia: > "In computer programming, unit testing is a software testing method by which individual units of source code—sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures—are tested to determine whether they are fit for use. Unit tests are typically automated tests written and run by software developers to ensure that a section of an application (known as the "unit") meets its design and behaves as intended. In procedural programming, a unit could be an entire module, but it is more commonly an individual function or procedure. In object-oriented programming, a unit is often an entire interface, such as a class, or an individual method." -- https://en.wikipedia.org/wiki/Unit_testing https://en.wikipedia.org/wiki/Unit_testing It's generally accepted good practice to not have dependencies between unit tests. Dependencies between unit tests are generally accepted as bad practice.
- deleted 5y ago
  [deleted]
- js8 5y ago
  You could also say that no dependencies between unit tests imply no dependencies between modules under test. Hence, the article's claim is incorrect, as their interpretation implies the interpretation they are refuting.
- thom 5y ago
  The central idea of a ‘unit’ test is that its failure should implicate one, and only one, part of the codebase. So while isolating tests is a necessary part of this (stale state from other tests would cloud the reasons for failure), I’ve never heard this isolation proposed as the primary feature of a unit test.
  - bluGill 5y ago
mlindner 5y ago
Who thinks that "unit test" refers to the unit under test? I've not once in my life heard anyone using that definition.
- paxys 5y ago
  The author is talking about what the "unit" in unit test means. The conventional definition is that it refers to the granular unit of your application being tested. He instead argues it means the test itself is an independent unit within the rest of your test suite.
- tsimionescu 5y ago
  A "unit test" is a test that covers a single unit of code, as opposed to an "integration test" that that covers several integrated units. This is the common definition. This other definition of "a test that doesn't depend on other tests" seems of little use, since no tests, even in something like a UI regression, should depend on other tests.
  - layer8 5y ago
    What is a “unit of code” though. Most units of code call (“integrate”?) other units of code in their implementation. Libraries, language runtime, runtime environment (OS), other functional modules of the application, utility functions, … What is the defining property where it becomes an integration test?
    - tsimionescu 5y ago
      That is more subjective and depends on the team, but for any particular definition, there will typically be a clear distinction (for that particular team/code base) between the two. A typical distinction is that a Unit Test only covers a single function/class method, or tightly coupled group of classes, and almost always no IO (network calls, disk writes). Conversely, integration tests typically cover the public API of an entire publically released library, network service, or CLI executable, usually run in the targeted deployment. For example, for a network service, an integration test may do a PUT and then a GET and check that the resource is returned. It would be running against a full copy of the service which is connected to a real database, but probably one populated with mock data. The same service would typically have unit tests checking something like "the handler for PUT generates an SQL query string that contains the fields defined in the PUT body" (assuming this is a code base without an ORM). Another unit test would check that the same method returns an error if a field is not supported. Another one may check that it doesn't create unsanitized SQL, and so on.
d--b 5y ago
The big tdd misunderstanding is to believe that it provides good test coverage. TDD is about designing how a module is going to be used before actually implementing it. It’s about considering the API as a product, rather than a side effect of the implementation. It’s about putting oneself in the user’s shoes, instead of making it as easy as possible for the implementer. Test coverage depends on the implementation, so you can’t design tests that will ensure 100% test coverage before implementing the stuff.
- shahinghasemi 5y ago
  Have not read the article yet but I guess it can be solved using incremental tests. First we write the tests for the core features of the API then while implementing it to pass the tests, we'll find that there arw some edge cases that we have not consider, so we write tests for them. In addition tests framework like jest provides good API around the coverages.
- davidfactorial 5y ago
  i think you may have suddenly helped me to finally grasp the true essence and nature of tdd. woah this makes so much more sense
- fao_ 5y ago
  > The big tdd misunderstanding is to believe that it provides good test coverage. Have you considered that, because TDD is a design practice (a 'meta-design') for writing code, there is no "one correct" interpretation, or misunderstanding, just different understandings? Yes, the people might be misusing it as per your understanding of it, but you are not the center of the universe here. You are correct that "it's about X", the people who are misusing it are also correct about their interpretation of it. Fundamentally, all TDD is the 5 steps of "writing a test first, and writing code that makes that test pass", along with some mentions of other meta-design approaches like KISS, etc. The way someone understands a meta-design is driven by a lot of factors, the way someone uses a meta-design is subject to the cultural context in the place you use it, the design restrictions and constraints of your tooling, the meta-design constraints of what practices management is willing for you to devote time to, etc. So fundamentally, you cannot create a design practice (or meta-design) that is applicable to all scenarios and circumstances. TDD will be outright inappropriate for some situations. This is exactly what "design patterns" are about, finding similar or identical ways of designing code, categorizing them, and figuring out what circumstances they do and don't work in. Now, I guess we need "design practices patterns", so we as an industry can move on to figuring out what practices are fit for certain scenarios, and what scenarios they are unfit for. Ultimately, anything less than the understanding that "context is king" is likely to lead to people just going around in circles and talking past each other.
furstenheim 5y ago
I would also add, test against a real database (docker) not in memory database. So many bugs that only happen on disk. Also so much functionality that cannot be used to fixed them because it's syntax error on the mock database
dorinlazar 5y ago
Like any TDD proponent (and most cultists, really), the author insists that „you're not doing it right”, despite reading the scriptures. You're always misunderstanding, YOU are the reason TDD doesn't work, no, TDD is itself flawless. There's always a slight misinterpretation of the magic words! But TDD works, you're the sinner for not using it right! And then you have shocked people when they find out that 100% test coverage doesn't mean that you really have a bug-free codebase.
- HelloNurse 5y ago
  There's a hierarchy of smartness: good enough to write perfect code and need no tests, good enough to catch problems with tests, good enough to believe that better techniques would have allowed catching bugs that slipped through tests. Any technique, in retrospect, could have been applied better, in an improved form.
- ribs 5y ago
  The author has described what they thing would make the practice of TDD better. The rationale presented isn’t ridiculous. Reasonable people could disagree on it.
- WolfOliver 5y ago
  No body is saying it does not work per se. But how many test suite have you seen which are less helpful?
  - pech0rin 5y ago
    Quite a few people are saying it doesn’t work. I have worked on TDD codebases where anytime you changed a function or tried to refactor something trivial it took 3x the amount of time to fix the tests even though the code worked correctly as written. This is due to mocking or naming or expecting functions to never change. TDD proponents would say that is the “wrong” way to do it, which Im sure it is but that doesn’t stop it from happening and killing productivity.
    - the_gipsy 5y ago
pictur 5y ago
I didn't quite understand why it was a problem for start with the integration test. okay, some things need to be mocked while writing the integration test, but I think the plus is more than the minus.
- Aeolun 5y ago
  It’s good to start with. Integration tests are great for finding out that ‘something’ is broken. Unit tests are better at finding what exactly is broken.
  - pictur 5y ago
    Thank you for your comment. you're right, it gets harder to find the source of the error, which makes it harder to make changes. I will pay attention to this.
- qayxc 5y ago
  Integration tests are top-down and focus on checking that the system under test behaves as expected. They usually require elaborate setup and configuration and can - depending on the system - take considerable effort to start and run. The idea behind TDD focuses on a bottom-up approach instead - if all the parts behave as expected, it is reasonable to assume that the entire system will. TDD was also intended as a tool to help identify problems with design early (e.g. too many dependencies, tight coupling, too many responsibilities, reduce/eliminate side effects, etc.), allowing for rapid feedback (tests should execute quickly and require little to no setup). Both approaches have different goals in mind and serve a different purpose. Ideally you always have both - run unit tests after each code change (i.e. often), run integration tests after each added or changed behaviour (e.g. daily or as part of an acceptance test process). Starting with integration tests much work up-front and adding unit tests after the fact can introduce considerable overhead for questionable gains.
  - pictur 5y ago
    - Starting with integration tests much work up-front and adding unit tests after the fact can introduce considerable overhead for questionable gains. Thank you for your comment. I will pay attention to this.
rendall 5y ago
"To make Medium work, we log user data. By using Medium, you agree to our Privacy Policy, including cookie policy." No, Medium. No, I do not agree.
cenny 5y ago
> Instead the pyramid says we should write a lot of unit tests. This leads to an inside out approach testing the structure of the system rather its behaviour. No it does not. Unit tests should test behavior. > Only use mocks for truly external systems (e.g. mail service). The database should be part of the tests. Do not stub it. First, mocks and stubs are not the same thing. Secondly, stub fragile dependencies in unit tests is a good rule of thumb if you want easy, fast and non-erratic tests. > Never change your code without having a red test. This is pretty common practice in TDD. No, In TDD we change our code all the time without a red test, we call it refactoring and we only do that when the test is green. There are so many misunderstandings in testing and TDD. I wished more people would read Kent Becks book [Test-Driven Development: by example](https://www.programmingbooks.dev/#test-driven-development https://www.programmingbooks.dev/#test-driven-development) and then the amazing book by Gerard Meszaros [XUnit Test Patterns](https://www.programmingbooks.dev/#xunit-test-patterns https://www.programmingbooks.dev/#xunit-test-patterns). In the end I'm just happy if more people practice TDD and understand it's a skill to learn like everything else and not a silver bullet.
- WolfOliver 5y ago
  > Unit tests should test behavior. Yes, they should but often they do not. The likelihood to get it right is higher using an outside-in approach.
pjmlp 5y ago
> Never change your code without having a red test Now go write a native GUI, a game or an audio engine with that approach.
- bckr 5y ago
  Say a bit more please
  - pjmlp 5y ago
    Write a breaking test for that XAML element to look like Mica in Windows design language. Remember, no code without breaking test.
    - berkes 5y ago
      It sounds like you are testing the wrong things on the wrong level. The test checking the correct color for a 'save button' in specific configuration, would most likely be a unit test for the "SaveButtonComponent" which simply checks that the correct attributes are set or passed on to a library. When you insist on having some coverage for the colors in the layer of e2e tests there are numerous tricks. Most of which you'd need to do only once and certainly not for every permutation of state. Like making a screenshot and diffing that with a fixture then failing on a too-big-delta. Or like running an accessbility-report over the app and failing if the values for things like "readability" drop below a threshold. And so on. You mention XAML, and I have little experience with windows GUI dev, but many setups, frameworks etc employ some adapter or layered architecture. Where the layer that drives the actual GUI library is but an adapter. So that the tests can swap out the actual GUI drawing layer for test-version theref, and do things like "confirmationModal.confirm()" without going through the entire UI. Or where tests can simply call a "confirmatioModal.buttons.map (b)=> b.colors.backGround" to get to the colors that are otherwise passed on to the actual GUI lib. This is why testing-pyramid is so important: to test on the correct level.
      - pjmlp 5y ago
        For application code to be written, regardless in what form and shape, a test must exist, so the TDD religion. You fail to adress that.
eb0la 5y ago
This statement made me think: "The database should be part of the tests. Do not stub it". Databases are both 1) complex piece of software and 2) hard to stub. 5-10 years ago having a database for testing was expensive. Today you don't need to pay licenses for most databases, and you can spin a disposable one in a container. Easier than mocking the DB itself.
- yakshaving_jgt 5y ago
  Why would the database be particularly hard to stub?
  - buzzwordninja 5y ago
    A database is not simply a bag of bits. It has behaviour which matters to the rest of the program.
    - yakshaving_jgt 5y ago
      That's certainly true, but in a typical web application it's quite common that the specific behaviour of a given database isn't interesting enough to warrant testing so rigorously. In those cases, integrating with the database is perhaps a less sensible choice economically.
- hooby 5y ago
  By my personal experience to make TDD work (where you make many small changes, run the tests, make more changes - step by step), tests need to run FAST - as in ideally less than 1 second. Even if it is easy to spin up a container, seed a database inside that container and drop and recreate that database after every single test (so that tests remain a unit and can't influence each other) - it tends to take a few seconds, which instantly makes TDD tedious to do. Thus you start making more changes at once, run the tests less, and drift away from doing TDD... The most typical solution is to run databases in-memory rather than in a container. My personal approach though is to make my code more modular, so that only the "database accessor" class needs to mock the actual database, while the "query builder" needs to mock only that database accessor (much easier) - and the database class needs to only mock the query builder (even easier) - and the actual application code needs to only mock the database class (super easy).
avl999 5y ago
The biggest misunderstanding about TDD is that you need to do TDD to be good engineer and write good code as promoted by some TDD evangelists. Writing tests is important, writing good tests is important... what is not important is the order you write the tests or if you follow an arcane set of "rules" about how you should write tests as if they are some ancient divine wisdom someone discovered on a stone tablet that's the key to writing good software. No... forget about writing tests inside out or outside in, top down or bottom up. Forget about the bs about "TDD is actually there to help you design the code so you SHOULD write tests first to expose flaws in your design". It does not matter as long as you write good tests and feel productive while writing code. Maybe TDD "rules" and rituals are useful when you are a relative beginner and instead of overwhelming them with a ton of information it is nice to have a set of rules and steps you can follow that have a good chance of resulting in a decent final product. But at a certain point you need to what works best for you and ignore the TDD dogma. I am not anti-TDD, I practice TDD for parts of the code but do it in a way that if a TDD purist saw it they would term it blasphemous. I never use TDD as a design tool, on the contrary I always design the general structure of a class or a package based on intuition and experience without writing a single line of test code. Then when I have the general design and skeleton functionality of the module fleshed out I jump into tests and start doing TDD to start filling out the functionality, addressing edge cases and so on. Coupling design with tests slows me down as it results in a lot of thrashing between tests and real code. I think a large number of developers feel this way and have never really bought into the "TDD is for design" hype. If on the otherhand TDD helps you design, go for it. The point is that TDD should not be treated like religious dogma that it often is. The biggest advantage of TDD is the dopamine hit it gives you when a test goes from red to green. That hit is so powerful that I continue to do TDD maybe 50% of the time but don't it dogmatically like TDD purists would want people to do it.
- Aeolun 5y ago
  I think the thing is, you can absolutely write the code and then write the tests afterwards, but you might end up with something that is hard to test (which is probably an indication your code is not properly modularized). You should take that as a sign you need to rewrite the code you’ve just written, but a lot of people don’t want to trash all (or part of) their hard work and instead bend the tests so that they deal with the code. You now have bad code, and you have bad/fragile tests. I think TDD (when enforced) is mostly meant to prevent people from falling for that.
Aeolun 5y ago
> Stuff Congratulations, you have reinvented integration testing!
Ygg2 5y ago
Most of testing wisdom can be summarized in a few words. Tests are worthless, but testing is priceless. TDD fails this. It puts test front and center, when they don't need to be. From my experience doing tests for a limited, dependency light lib is very easy. Trivial even. But going test first never seemed like a clear benefit. Also mocks like Mockito are the plague. Refactors (from tests passing to test passing) in presence of mocks is impossible.
berkes 5y ago
Every proof or point, including the linked articles are at the least underwhelming and at most controversial. They are certainly not "rules". Author calls them "tips" before, which is more apt. But as the "tips/rules" go, they lack all nuance and background. With TDD, as with anything "it depends". All of the rules (tips) proposed, I consider to be harmful and plain wrong in many situations (though not always and everywhere!).
tpoacher 5y ago
> Rule #3: Only use mocks for truly external systems (e.g. mail service). The database should be part of the tests. Do not stub it. Interfacing with external resources is literally the definition of what a unit test is NOT.
morelish 5y ago
Clickbait title I would say. The 'big' issue with TDD is whether it's worth the time needed to do it. If it was obviously always worth the time, then I think we wouldn't even talk about it. (Do we worry about whether breathing is worth it? Nope, we never talk about that because the answer is apparent.) Whether it's worth it depends on the context.
SassyGrapefruit 5y ago
>Do not isolate code when you test it. If you do so, the tests become fragile and do not help you in case you refactor the software. I don't agree with this bit. This is how you make a fragile test suites that are difficult to refactor. Why do we go out of our way to hide information in our code? Answer because code that spaghettis in an out of a lot of other code means small changes cause unexpected regressions, and the changes to fix that regression causes other regressions. Suites that don't isolate code have exactly this problem. They are like a scavenger hunt of assumptions and inevitably lead to people fixing the tests instead of the code. In addition because you have introduced a network effect a small change will likely invalidate a huge number of tests. Meaning you have to update them all. Integration suites have value, but I use them with discretion because I know that they are a challenge to maintain. Tight unit tests that test small pieces of logic thoroughly and make ample use of mocks to ensure deterministic outcomes are a must. These tests are highly durable. The code can change all around it but that function will work as advertised until someone changes it. Then they can update its test suite and you're back in the golden zone. In my experience if you build a complex structure out of a lot of quality components that complex structure will tend to be high quality itself. My pattern usually pushes the complexity into the units and I leave "orchestration" units that tie everything together untested but intentionally simple. I know the code coverage folks will chime for sure, but I don't aim for 100% code coverage. I aim for a balanced test suite that ensures the best possible outcome for the minimum investment with the least amount of hassle when introducing new changes. Trying to test everything never seems to get me there.
- Diggsey 5y ago
  > Answer because code that spaghettis in an out of a lot of other code means small changes cause unexpected regressions That's not a fragile test suite, that's a test suite working well: you want tests to start failing if you introduced a regression to the overall program behaviour. > the changes to fix that regression causes other regressions It sounds like what you're really getting at is something different from test fragility: specifically, if a test fails, how easy is it to identify where in the code a fix needs to be applied. That is also an important consideration when writing tests but it's completely separate from test fragility. > Integration suites have value, but I use them with discretion because I know that they are a challenge to maintain. This is the opposite of how the test pyramid is supposed to work. Integration tests are supposed to be less fragile because they don't rely on specific implementation details or mocks: they're mimicking a real user or a consumer of your API. If you are getting false positives from your integration tests a lot, then either the tests are written poorly, or you're making too many breaking changes to your library/product/whatever. > Tight unit tests that test small pieces of logic thoroughly and make ample use of mocks to ensure deterministic outcomes are a must. The code can change all around it but that function will work as advertised until someone changes it. Then they can update its test suite and you're back in the golden zone. That's completely useless as a test then. Of course if you change code around it the test won't break. The whoe point of a test is that it doesn't break even for (some) changes to the code being tested, and if you always have to update the unit test when you change the code then it is providing no vlaue and giving you a false sense of security that all your tests are green. Unit tests are great when you have a particular irreducable complexity in a piece of code, or more generally they are good when there is some constraint that should be locally upheld that is likely to be more permanent than the specific implementation.
cersa8 5y ago
I'm a bit of a cowboy coder and completely skip all but the very top of the testing pyramid. I strictly run E2E tests and albeit slower than other tests, it tests a much broader spectrum than just a single function. For instance I change the expiration date of a user's subscription, run expire logic, test of this user has received the expected expiration email, this automatically tests the Amazon SES integration, the email template parser and everything in between. Clicking on the expire email call to action button I expect to land on a certain page with elements "missing" and others added, which validates changes in the user's roles. Let's say I swap Amazon SES for Postmark, no new tests have to be written. In my experience this approach works really well for a SaaS like product. It also looks satisfying to see the test suite blast through the site clicking and typing like there is no tomorrow.
lbriner 5y ago
I sitll don't understand whether TDD applies outside of some rather specific domains. In most cases, apart from utility methods, I don't actually know how something will work before I have at least partially implemented it. We have implemented a payments change recently (Paypal to Recurly) and not only are the two very different but we have added additional functionality that we could have done with Paypal but didn't. How do you TDD that? Even writing a simple test like "Go to this page and click <Order>" doesn't really work since when you come to it, you realise you can't just order but you also need to go to the create account page first etc.
- projektfu 5y ago
  A lot of effective functional testing is getting fixtures ready so that your test can reliably hit the important part of the system under test. For external integrations this becomes difficult. Presumably your test uses the api sandbox, so hopefully it gives reproducible results. If not, then you need to test manually and focus on unit testing to assure the software.
- berkes 5y ago
  > How do you TDD that? Counterquestion: how do you know, at each release, that the paypal-stuff still works? If the answer is "I test it by making a purchase at each release", the obvious question is: why not automate this? That is one benefit of TDD. So why not start of with that automation? In your specific case, I would write a simple test first, that, says "on the page of the orange Y-combinator swag, order a shirt and receive a timely confirmation.". Which is subtly, yet importantly different from "go to X and click order". On the level of such acceptence tests, trying to keep away from implementation details like "pages", "buttons" and so on, helps with solving your problem too: you write down the intent&outcome, rather than the implementation. Focus on the business value. Solve that first. And fixate the outcome for all future: regardless of the PSP, pages, exact values (timely vs within 1 sec), actual confirmation-channels and whatnot, the business value is "green" when it works, and "red" when it doesn't. Then I start implementing it. Maybe just add one of these <iframe> things that Paypal offers, even. I don't really care: the quest thing to make it work. as long as it gets me to have that acceptance test green with least work, I'm happy. Happy, but certainly not done. Because the most overlooked part of TDD is the Refactoring. Red-Green-Refactor. Once it is green, start refactoring. Replace the iframe with some library. Go a step further and replace that with my own paypal-API-client. Connect that to my event-stream, or not.
DanielBMarkham 5y ago
I know it's good commenting style on HN to avoid quick one-liners, but TDD is one of those topics that we've kicked around forever. Perhaps a one-liner would help clarify things. TDD is a coding practice to help the programmer understand that their mental model of the code does not reflect the actual code itself. That's it. There are some follow-ons, of course, like if your mental model is exactly in-line with the code then logical errors are unlikely (although business errors can be quite common!). Another follow-on is that TDD is much more about design than testing. Once you realize that it's a practice or habit that's all about sending information from the code back to the mind of the coder, all of that other stuff sorts itself out. Shameless plug: this has been on my mind quite a bit lately since I blogged last week on the unreasonable effectiveness of testing as a way out of both cognitive and coding anti-patterns: https://danielbmarkham.com/cognition-versus-programming-the-big-two/ https://danielbmarkham.com/cognition-versus-programming-the-...
draw_down 5y ago
andrewallbright 5y ago
Reading this article and reading the comments I am reminded that writing automated testing is still an art and less of a science. I highly value testing and it does eat up project time in weird ways. I believe as I get more experience it won't eat as much time. My hope is for productivity gains in the future. I hope in the future most documentation is written with a testing focus. Maybe then I would not have to understand as much as I do now before finishing testing.
deleted 5y ago
[deleted]
throwaway9870 5y ago
I don't take testing advice from anyone who uses generic terms like "test coverage". Test coverage is a complex subject, but without a basic understanding, tests can create a false sense of security. Saying something like "100% coverage" means almost nothing to anyone that understands testing. This is a worthwhile read: https://en.wikipedia.org/wiki/Code_coverage https://en.wikipedia.org/wiki/Code_coverage Without formal methods, software will realistically never be 100% tested. But don't worry, your hardware isn't either!
commandlinefan 5y ago
> The database should be part of the tests. Do not stub it. I've never seen this work well. The tests either rely on a shared/remote database or on a local database instance that has to be set up/torn down between tests. Both approaches cause unending headaches in terms of false positives, onboarding new developers, dependency problems and maintenance.
groby_b 5y ago
Having spent way too much time on the TDD mailing lists in the late 90's/early 2000s, that's... an attempt at retconning, at best. That's not what unit tests meant, and nobody advocated for them that way. C2 still doesn't: https://wiki.c2.com/?UnitTest https://wiki.c2.com/?UnitTest Kent Beck, Robert Martin et.al. didn't. So, no, it's not a misunderstanding.
gdennci 5y ago
There are no "magic bullet" in software development. TDD is my go-to approach if the situation allows it. But it is not always feasible. Sometimes you have to write a bit of code first to find your solution path. Ppl should stop with their dogmatism and start thinking about their approach by themselves.

118 points by WolfOliver 5y ago

The issue with TDD as originally described by Kent Beck (there are as many variations as there are ... ) is the belief that a good enough design emerges from the process.

qayxc 5y ago
While I agree that a good architecture cannot be derived from (unit-)tests alone, tests do indeed help a lot with good design. As soon as you need elaborate setup in order to test a single piece of functionality (or function), you will start to rethink your design. In such cases there's too much coupling, too many dependencies, or too many responsibilities within the function under test. Seeing tests as separate from implementation and design is a much bigger issue IMHO. TDD only provides value as opposed to overhead, if you understand it as an integral part of the design and implementation process and use them with that in mind.
- sorokod 5y ago
  That last sentence seems to do a lot of heavy lifting. Where does it fit in the red-green-refactor mantra?
  - qayxc 5y ago
    Red-green-refactoring doesn't work if you are dealing with existing software, for example (unless there's a well-documented and -understood defect). It's not TDD (emphasis of the "driven"-part) if tests are only used to confirm already implemented behaviour. I found that especially in larger organisations there's all too often hard requirements for unit tests and high test coverage. This doesn't work as intended, though, if the team doesn't "live" by the TDD concept and unit tests are only there to check some box on a "best practices" list. IMO it's perfectly fine to only use TDD for some components of an application and rely on integration- and acceptance tests only for others. Glue code and cross-cutting components don't benefit from TDD. An external API or component that you don't control can also prevent meaningful unit tests and force certain design decisions, rendering TDD an exercise in writing (and maintaining) mocks, etc. What I tried to express is that TDD is one tool of many and not always the best choice; especially so when it is tacked onto an existing process that doesn't use it as intended or cannot benefit from it.

deanCommie 5y ago

The big TDD misunderstanding is that like any tool it is to be wielded when appropriate, not followed rigidly for every single piece of code you write. There are some safety standards and practices in engineering that are matters of life and death. If you're working high above the ground, you need two points of safety clipping, and if you're moving you never disconnect both of them. This is non-negotiable. There is never an appropriate time to violate this. If there is no way to complete the job without constant safety harness restraint, the job doesn't get done. (At least in the developed world.) This is not the category where TDD or rules like "Never change your code without having a red test." fall in. I adore TDD. It made me a better engineer, and I followed the process rigorously in my first 5-10 years of my career. But at some point I realized that what that did is teach me how to write modular, extensible, testable code, and now I can write modules like that upfront, and add tests later. And truthfully, I do go faster. Because I also then focus my tests on JUST the important logic, not the intermediate abstractions in between. I write unit tests for the critical functionality and *integration* tests for the whole system to make sure the whole thing works as expected (because the dirty truth of unit tests is you can have 100% test coverage but a non-functional systems due to frameworks and glue code being incorrectly configured.

qayxc 5y ago
High test coverage is the worst metric one can aim for. As soon as frameworks and external APIs are involved, it's basically impossible to achieve without bending over backwards and needlessly creating mocks for half your system. My biggest pet peeve in this context are test coverage requirements for OO languages that feature properties or getters and setters. 100% test coverage would require writing "tests" for those, which basically amounts to testing whether the compiler works - madness!

bedobi 5y ago

Yes! This can't be repeated enough IMO > Terminology > The test that hits the endpoint is commonly referred to as an “integration test”, and the other test is commonly referred to as a “unit test”. > Kent Beck (the originator of test-driven development) defines a unit test as “a test that runs in isolation from other tests”. This is very different from the definition of a unit test as “a test that tests a class/method in isolation from other classes/methods”. The test that hits the endpoint satisfies the original definition. > But it doesn’t really matter if you want to call a given test an “integration” test or a “unit” test. The point is the test fails when something breaks and passes when something is improved. If it does the opposite, it’s not a good test. https://medium.com/expedia-group-tech/test-your-tests-f4e361818720 https://medium.com/expedia-group-tech/test-your-tests-f4e361...

mrjin 5y ago
The sadly fact is that lots of people cannot tell the differences between Unit Tests, Functional Tests and Integration Tests even after in the field for over a decade.
- beebeepka 5y ago
  I welcome you to share your opinion on the matter
  - bckr 5y ago
    I'm going to give it a crack. Caveat that I have not read the TDD scriptures. Unit test: tests a bit of code not much larger than a function, to make sure that the function runs properly and gives the expected outputs. Functional tests: tests some code not much bigger than a module (source file) to make sure it does the right things, from a business logic standpoint. Integration tests: tests some systems not much bigger than one subsystem making an API call to another system, to make sure that the systems can talk to each other properly.
    - Aeolun 5y ago

gleenn 5y ago

I will start that I think TDD is very valuable. But reading this article that quickly goes into a numbered list of "rules" is a little disheartening. Already some other commenters are talking about more nuanced views of when to test. I think the real tragedy is that I see a lot of engineers still not writing tests. So when speaking to the broader audience, you have to be really gung-ho about it to get people to do what I think is a very necessary step. People should err on the side of writing the test first, both to get most people in the right mindset, and also to avoid the broken window syndrome of a lot of missing tests and no one bothering to actually start doing what should be obvious: most code should have at least some level of coverage. Most in fact. If there are "reasons", sure, if you're going to pull the senior engineer move and skip the test, then you better be able to back it up clearly. If not, write the damn test. If not you, someone will appreciate it later.

dwheeler 5y ago

Another huge problem with TDD is a lot of practitioners do not create negative tests. A negative test verifies that something the system shouldn't do is not being done. For example, if you shouldn't allow unauthenticated users unauthenticated users to delete something, there should be a test to verify that authenticated users can't delete something. But if you only add tests that will fail and then make succeed via implementation, you would probably never think to make those kind of test. You certainly can create negative tests with TDD, but most docs on TDD never discuss it.

jay_kyburz 5y ago
There should be a test looking for duplicate phrases. (I'm laughing with you not at you :))
deleted 5y ago
[deleted]
dqv 5y ago
This is how I’ve been making sure multitenancy in my app doesn’t leak data. It shouldn’t leak but how can I be sure? Run tests that try to access data on one tenant from another tenant and fail if the data is accessible.
- eb0la 5y ago
  Most of the time we make tests when something is in our radar. I guess you have use case in your spec that tells you that's the expected behaviour, or at least some clever developer figured out it should be done that way. But it is not always so "easy": Sometimes you don't have the domain knowledge, or the spec doesn't say a word about negative cases...
- onion2k 5y ago
  Run tests that try to access data on one tenant from another tenant and fail if the data is accessible. That tells you that the method your tests use to try to access data don't work, not that it's impossible to access data across tenants. If there's a leak you haven't tested for then your app is still insecure. It'd be better to think of a design that actually makes it impossible (assuming that's even possible for your scenario).

ehutch79 5y ago

This is contrary to most testing tutorials and documentation I've read. I get frustrated and forego a lot of testing because the idea of mocking out basically the whole system in order to unit test is insane to me. If function A calls function B, you've got to mock function B out. If you refactor function B, you better update the mock as well. Writing an app twice hardly seems like a good use of time.

tra3 5y ago
I only mock functionality that is non deterministic — like sending email.
rhdunn 5y ago
That way leads to madness, and I don't follow that style/philosophy of mocking components. Only mock the things you can't easily set up in your unit tests to get them to work, like database connections. For spring-based components, try to use the real components where possible, but where needed use mock components that do enough to get the system working. That does mean that if you are testing function A you are also indirectly testing function B, but that is fine.
pintxo 5y ago
If function A does not much more than calling B, maybe testing A is not strictly necessary. If A does something important/complex/critical/etc maybe the important/complex/critical/etc part can be refactored into a dedicated function A1 without external dependency and then tested on its own? That way, the tests focuses on where the money is and gets easier to work with at the same time.
- adwn 5y ago
  Now you've added an additional, untested point of failure (the glue code), and you've made the interaction between A, A1, and B more difficult to understanding and more error prone.
  - cloogshicer 5y ago

theteapot 5y ago

Article: > "Originally the unit in “unit test” did not refer to the system under test but the test itself. Meaning the test can be executed as one unit and does not depend on other tests to run upfront (see here and here)." I find the links provided to support for this proposition underwhelming. In any case it matters zero. No-one interprets unit test that way any more. Everyone interprets it the other way round, like defined on Wikipedia: > "In computer programming, unit testing is a software testing method by which individual units of source code—sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures—are tested to determine whether they are fit for use. Unit tests are typically automated tests written and run by software developers to ensure that a section of an application (known as the "unit") meets its design and behaves as intended. In procedural programming, a unit could be an entire module, but it is more commonly an individual function or procedure. In object-oriented programming, a unit is often an entire interface, such as a class, or an individual method." -- https://en.wikipedia.org/wiki/Unit_testing https://en.wikipedia.org/wiki/Unit_testing It's generally accepted good practice to not have dependencies between unit tests. Dependencies between unit tests are generally accepted as bad practice.

deleted 5y ago
[deleted]
js8 5y ago
You could also say that no dependencies between unit tests imply no dependencies between modules under test. Hence, the article's claim is incorrect, as their interpretation implies the interpretation they are refuting.
thom 5y ago
The central idea of a ‘unit’ test is that its failure should implicate one, and only one, part of the codebase. So while isolating tests is a necessary part of this (stale state from other tests would cloud the reasons for failure), I’ve never heard this isolation proposed as the primary feature of a unit test.
- bluGill 5y ago

mlindner 5y ago

Who thinks that "unit test" refers to the unit under test? I've not once in my life heard anyone using that definition.

paxys 5y ago
The author is talking about what the "unit" in unit test means. The conventional definition is that it refers to the granular unit of your application being tested. He instead argues it means the test itself is an independent unit within the rest of your test suite.
tsimionescu 5y ago
A "unit test" is a test that covers a single unit of code, as opposed to an "integration test" that that covers several integrated units. This is the common definition. This other definition of "a test that doesn't depend on other tests" seems of little use, since no tests, even in something like a UI regression, should depend on other tests.
- layer8 5y ago
  What is a “unit of code” though. Most units of code call (“integrate”?) other units of code in their implementation. Libraries, language runtime, runtime environment (OS), other functional modules of the application, utility functions, … What is the defining property where it becomes an integration test?
  - tsimionescu 5y ago
    That is more subjective and depends on the team, but for any particular definition, there will typically be a clear distinction (for that particular team/code base) between the two. A typical distinction is that a Unit Test only covers a single function/class method, or tightly coupled group of classes, and almost always no IO (network calls, disk writes). Conversely, integration tests typically cover the public API of an entire publically released library, network service, or CLI executable, usually run in the targeted deployment. For example, for a network service, an integration test may do a PUT and then a GET and check that the resource is returned. It would be running against a full copy of the service which is connected to a real database, but probably one populated with mock data. The same service would typically have unit tests checking something like "the handler for PUT generates an SQL query string that contains the fields defined in the PUT body" (assuming this is a code base without an ORM). Another unit test would check that the same method returns an error if a field is not supported. Another one may check that it doesn't create unsanitized SQL, and so on.

d--b 5y ago

The big tdd misunderstanding is to believe that it provides good test coverage. TDD is about designing how a module is going to be used before actually implementing it. It’s about considering the API as a product, rather than a side effect of the implementation. It’s about putting oneself in the user’s shoes, instead of making it as easy as possible for the implementer. Test coverage depends on the implementation, so you can’t design tests that will ensure 100% test coverage before implementing the stuff.

shahinghasemi 5y ago
Have not read the article yet but I guess it can be solved using incremental tests. First we write the tests for the core features of the API then while implementing it to pass the tests, we'll find that there arw some edge cases that we have not consider, so we write tests for them. In addition tests framework like jest provides good API around the coverages.
davidfactorial 5y ago
i think you may have suddenly helped me to finally grasp the true essence and nature of tdd. woah this makes so much more sense
fao_ 5y ago
> The big tdd misunderstanding is to believe that it provides good test coverage. Have you considered that, because TDD is a design practice (a 'meta-design') for writing code, there is no "one correct" interpretation, or misunderstanding, just different understandings? Yes, the people might be misusing it as per your understanding of it, but you are not the center of the universe here. You are correct that "it's about X", the people who are misusing it are also correct about their interpretation of it. Fundamentally, all TDD is the 5 steps of "writing a test first, and writing code that makes that test pass", along with some mentions of other meta-design approaches like KISS, etc. The way someone understands a meta-design is driven by a lot of factors, the way someone uses a meta-design is subject to the cultural context in the place you use it, the design restrictions and constraints of your tooling, the meta-design constraints of what practices management is willing for you to devote time to, etc. So fundamentally, you cannot create a design practice (or meta-design) that is applicable to all scenarios and circumstances. TDD will be outright inappropriate for some situations. This is exactly what "design patterns" are about, finding similar or identical ways of designing code, categorizing them, and figuring out what circumstances they do and don't work in. Now, I guess we need "design practices patterns", so we as an industry can move on to figuring out what practices are fit for certain scenarios, and what scenarios they are unfit for. Ultimately, anything less than the understanding that "context is king" is likely to lead to people just going around in circles and talking past each other.

furstenheim 5y ago

I would also add, test against a real database (docker) not in memory database. So many bugs that only happen on disk. Also so much functionality that cannot be used to fixed them because it's syntax error on the mock database

dorinlazar 5y ago

Like any TDD proponent (and most cultists, really), the author insists that „you're not doing it right”, despite reading the scriptures. You're always misunderstanding, YOU are the reason TDD doesn't work, no, TDD is itself flawless. There's always a slight misinterpretation of the magic words! But TDD works, you're the sinner for not using it right! And then you have shocked people when they find out that 100% test coverage doesn't mean that you really have a bug-free codebase.

HelloNurse 5y ago
There's a hierarchy of smartness: good enough to write perfect code and need no tests, good enough to catch problems with tests, good enough to believe that better techniques would have allowed catching bugs that slipped through tests. Any technique, in retrospect, could have been applied better, in an improved form.
ribs 5y ago
The author has described what they thing would make the practice of TDD better. The rationale presented isn’t ridiculous. Reasonable people could disagree on it.
WolfOliver 5y ago
No body is saying it does not work per se. But how many test suite have you seen which are less helpful?
- pech0rin 5y ago
  Quite a few people are saying it doesn’t work. I have worked on TDD codebases where anytime you changed a function or tried to refactor something trivial it took 3x the amount of time to fix the tests even though the code worked correctly as written. This is due to mocking or naming or expecting functions to never change. TDD proponents would say that is the “wrong” way to do it, which Im sure it is but that doesn’t stop it from happening and killing productivity.
  - the_gipsy 5y ago

pictur 5y ago

I didn't quite understand why it was a problem for start with the integration test. okay, some things need to be mocked while writing the integration test, but I think the plus is more than the minus.

Aeolun 5y ago
It’s good to start with. Integration tests are great for finding out that ‘something’ is broken. Unit tests are better at finding what exactly is broken.
- pictur 5y ago
  Thank you for your comment. you're right, it gets harder to find the source of the error, which makes it harder to make changes. I will pay attention to this.
qayxc 5y ago
Integration tests are top-down and focus on checking that the system under test behaves as expected. They usually require elaborate setup and configuration and can - depending on the system - take considerable effort to start and run. The idea behind TDD focuses on a bottom-up approach instead - if all the parts behave as expected, it is reasonable to assume that the entire system will. TDD was also intended as a tool to help identify problems with design early (e.g. too many dependencies, tight coupling, too many responsibilities, reduce/eliminate side effects, etc.), allowing for rapid feedback (tests should execute quickly and require little to no setup). Both approaches have different goals in mind and serve a different purpose. Ideally you always have both - run unit tests after each code change (i.e. often), run integration tests after each added or changed behaviour (e.g. daily or as part of an acceptance test process). Starting with integration tests much work up-front and adding unit tests after the fact can introduce considerable overhead for questionable gains.
- pictur 5y ago
  - Starting with integration tests much work up-front and adding unit tests after the fact can introduce considerable overhead for questionable gains. Thank you for your comment. I will pay attention to this.

rendall 5y ago

"To make Medium work, we log user data. By using Medium, you agree to our Privacy Policy, including cookie policy." No, Medium. No, I do not agree.

cenny 5y ago

> Instead the pyramid says we should write a lot of unit tests. This leads to an inside out approach testing the structure of the system rather its behaviour. No it does not. Unit tests should test behavior. > Only use mocks for truly external systems (e.g. mail service). The database should be part of the tests. Do not stub it. First, mocks and stubs are not the same thing. Secondly, stub fragile dependencies in unit tests is a good rule of thumb if you want easy, fast and non-erratic tests. > Never change your code without having a red test. This is pretty common practice in TDD. No, In TDD we change our code all the time without a red test, we call it refactoring and we only do that when the test is green. There are so many misunderstandings in testing and TDD. I wished more people would read Kent Becks book [Test-Driven Development: by example](https://www.programmingbooks.dev/#test-driven-development https://www.programmingbooks.dev/#test-driven-development) and then the amazing book by Gerard Meszaros [XUnit Test Patterns](https://www.programmingbooks.dev/#xunit-test-patterns https://www.programmingbooks.dev/#xunit-test-patterns). In the end I'm just happy if more people practice TDD and understand it's a skill to learn like everything else and not a silver bullet.

WolfOliver 5y ago
> Unit tests should test behavior. Yes, they should but often they do not. The likelihood to get it right is higher using an outside-in approach.

pjmlp 5y ago

> Never change your code without having a red test Now go write a native GUI, a game or an audio engine with that approach.

bckr 5y ago
Say a bit more please
- pjmlp 5y ago
  Write a breaking test for that XAML element to look like Mica in Windows design language. Remember, no code without breaking test.
  - berkes 5y ago
    It sounds like you are testing the wrong things on the wrong level. The test checking the correct color for a 'save button' in specific configuration, would most likely be a unit test for the "SaveButtonComponent" which simply checks that the correct attributes are set or passed on to a library. When you insist on having some coverage for the colors in the layer of e2e tests there are numerous tricks. Most of which you'd need to do only once and certainly not for every permutation of state. Like making a screenshot and diffing that with a fixture then failing on a too-big-delta. Or like running an accessbility-report over the app and failing if the values for things like "readability" drop below a threshold. And so on. You mention XAML, and I have little experience with windows GUI dev, but many setups, frameworks etc employ some adapter or layered architecture. Where the layer that drives the actual GUI library is but an adapter. So that the tests can swap out the actual GUI drawing layer for test-version theref, and do things like "confirmationModal.confirm()" without going through the entire UI. Or where tests can simply call a "confirmatioModal.buttons.map (b)=> b.colors.backGround" to get to the colors that are otherwise passed on to the actual GUI lib. This is why testing-pyramid is so important: to test on the correct level.
    - pjmlp 5y ago
      For application code to be written, regardless in what form and shape, a test must exist, so the TDD religion. You fail to adress that.

eb0la 5y ago

This statement made me think: "The database should be part of the tests. Do not stub it". Databases are both 1) complex piece of software and 2) hard to stub. 5-10 years ago having a database for testing was expensive. Today you don't need to pay licenses for most databases, and you can spin a disposable one in a container. Easier than mocking the DB itself.

yakshaving_jgt 5y ago
Why would the database be particularly hard to stub?
- buzzwordninja 5y ago
  A database is not simply a bag of bits. It has behaviour which matters to the rest of the program.
  - yakshaving_jgt 5y ago
    That's certainly true, but in a typical web application it's quite common that the specific behaviour of a given database isn't interesting enough to warrant testing so rigorously. In those cases, integrating with the database is perhaps a less sensible choice economically.
hooby 5y ago
By my personal experience to make TDD work (where you make many small changes, run the tests, make more changes - step by step), tests need to run FAST - as in ideally less than 1 second. Even if it is easy to spin up a container, seed a database inside that container and drop and recreate that database after every single test (so that tests remain a unit and can't influence each other) - it tends to take a few seconds, which instantly makes TDD tedious to do. Thus you start making more changes at once, run the tests less, and drift away from doing TDD... The most typical solution is to run databases in-memory rather than in a container. My personal approach though is to make my code more modular, so that only the "database accessor" class needs to mock the actual database, while the "query builder" needs to mock only that database accessor (much easier) - and the database class needs to only mock the query builder (even easier) - and the actual application code needs to only mock the database class (super easy).

avl999 5y ago

The biggest misunderstanding about TDD is that you need to do TDD to be good engineer and write good code as promoted by some TDD evangelists. Writing tests is important, writing good tests is important... what is not important is the order you write the tests or if you follow an arcane set of "rules" about how you should write tests as if they are some ancient divine wisdom someone discovered on a stone tablet that's the key to writing good software. No... forget about writing tests inside out or outside in, top down or bottom up. Forget about the bs about "TDD is actually there to help you design the code so you SHOULD write tests first to expose flaws in your design". It does not matter as long as you write good tests and feel productive while writing code. Maybe TDD "rules" and rituals are useful when you are a relative beginner and instead of overwhelming them with a ton of information it is nice to have a set of rules and steps you can follow that have a good chance of resulting in a decent final product. But at a certain point you need to what works best for you and ignore the TDD dogma. I am not anti-TDD, I practice TDD for parts of the code but do it in a way that if a TDD purist saw it they would term it blasphemous. I never use TDD as a design tool, on the contrary I always design the general structure of a class or a package based on intuition and experience without writing a single line of test code. Then when I have the general design and skeleton functionality of the module fleshed out I jump into tests and start doing TDD to start filling out the functionality, addressing edge cases and so on. Coupling design with tests slows me down as it results in a lot of thrashing between tests and real code. I think a large number of developers feel this way and have never really bought into the "TDD is for design" hype. If on the otherhand TDD helps you design, go for it. The point is that TDD should not be treated like religious dogma that it often is. The biggest advantage of TDD is the dopamine hit it gives you when a test goes from red to green. That hit is so powerful that I continue to do TDD maybe 50% of the time but don't it dogmatically like TDD purists would want people to do it.

Aeolun 5y ago
I think the thing is, you can absolutely write the code and then write the tests afterwards, but you might end up with something that is hard to test (which is probably an indication your code is not properly modularized). You should take that as a sign you need to rewrite the code you’ve just written, but a lot of people don’t want to trash all (or part of) their hard work and instead bend the tests so that they deal with the code. You now have bad code, and you have bad/fragile tests. I think TDD (when enforced) is mostly meant to prevent people from falling for that.

Aeolun 5y ago

> Stuff Congratulations, you have reinvented integration testing!

Ygg2 5y ago

Most of testing wisdom can be summarized in a few words. Tests are worthless, but testing is priceless. TDD fails this. It puts test front and center, when they don't need to be. From my experience doing tests for a limited, dependency light lib is very easy. Trivial even. But going test first never seemed like a clear benefit. Also mocks like Mockito are the plague. Refactors (from tests passing to test passing) in presence of mocks is impossible.

berkes 5y ago

Every proof or point, including the linked articles are at the least underwhelming and at most controversial. They are certainly not "rules". Author calls them "tips" before, which is more apt. But as the "tips/rules" go, they lack all nuance and background. With TDD, as with anything "it depends". All of the rules (tips) proposed, I consider to be harmful and plain wrong in many situations (though not always and everywhere!).

tpoacher 5y ago

> Rule #3: Only use mocks for truly external systems (e.g. mail service). The database should be part of the tests. Do not stub it. Interfacing with external resources is literally the definition of what a unit test is NOT.

morelish 5y ago

Clickbait title I would say. The 'big' issue with TDD is whether it's worth the time needed to do it. If it was obviously always worth the time, then I think we wouldn't even talk about it. (Do we worry about whether breathing is worth it? Nope, we never talk about that because the answer is apparent.) Whether it's worth it depends on the context.

SassyGrapefruit 5y ago

>Do not isolate code when you test it. If you do so, the tests become fragile and do not help you in case you refactor the software. I don't agree with this bit. This is how you make a fragile test suites that are difficult to refactor. Why do we go out of our way to hide information in our code? Answer because code that spaghettis in an out of a lot of other code means small changes cause unexpected regressions, and the changes to fix that regression causes other regressions. Suites that don't isolate code have exactly this problem. They are like a scavenger hunt of assumptions and inevitably lead to people fixing the tests instead of the code. In addition because you have introduced a network effect a small change will likely invalidate a huge number of tests. Meaning you have to update them all. Integration suites have value, but I use them with discretion because I know that they are a challenge to maintain. Tight unit tests that test small pieces of logic thoroughly and make ample use of mocks to ensure deterministic outcomes are a must. These tests are highly durable. The code can change all around it but that function will work as advertised until someone changes it. Then they can update its test suite and you're back in the golden zone. In my experience if you build a complex structure out of a lot of quality components that complex structure will tend to be high quality itself. My pattern usually pushes the complexity into the units and I leave "orchestration" units that tie everything together untested but intentionally simple. I know the code coverage folks will chime for sure, but I don't aim for 100% code coverage. I aim for a balanced test suite that ensures the best possible outcome for the minimum investment with the least amount of hassle when introducing new changes. Trying to test everything never seems to get me there.

Diggsey 5y ago
> Answer because code that spaghettis in an out of a lot of other code means small changes cause unexpected regressions That's not a fragile test suite, that's a test suite working well: you want tests to start failing if you introduced a regression to the overall program behaviour. > the changes to fix that regression causes other regressions It sounds like what you're really getting at is something different from test fragility: specifically, if a test fails, how easy is it to identify where in the code a fix needs to be applied. That is also an important consideration when writing tests but it's completely separate from test fragility. > Integration suites have value, but I use them with discretion because I know that they are a challenge to maintain. This is the opposite of how the test pyramid is supposed to work. Integration tests are supposed to be less fragile because they don't rely on specific implementation details or mocks: they're mimicking a real user or a consumer of your API. If you are getting false positives from your integration tests a lot, then either the tests are written poorly, or you're making too many breaking changes to your library/product/whatever. > Tight unit tests that test small pieces of logic thoroughly and make ample use of mocks to ensure deterministic outcomes are a must. The code can change all around it but that function will work as advertised until someone changes it. Then they can update its test suite and you're back in the golden zone. That's completely useless as a test then. Of course if you change code around it the test won't break. The whoe point of a test is that it doesn't break even for (some) changes to the code being tested, and if you always have to update the unit test when you change the code then it is providing no vlaue and giving you a false sense of security that all your tests are green. Unit tests are great when you have a particular irreducable complexity in a piece of code, or more generally they are good when there is some constraint that should be locally upheld that is likely to be more permanent than the specific implementation.

cersa8 5y ago

I'm a bit of a cowboy coder and completely skip all but the very top of the testing pyramid. I strictly run E2E tests and albeit slower than other tests, it tests a much broader spectrum than just a single function. For instance I change the expiration date of a user's subscription, run expire logic, test of this user has received the expected expiration email, this automatically tests the Amazon SES integration, the email template parser and everything in between. Clicking on the expire email call to action button I expect to land on a certain page with elements "missing" and others added, which validates changes in the user's roles. Let's say I swap Amazon SES for Postmark, no new tests have to be written. In my experience this approach works really well for a SaaS like product. It also looks satisfying to see the test suite blast through the site clicking and typing like there is no tomorrow.

lbriner 5y ago

I sitll don't understand whether TDD applies outside of some rather specific domains. In most cases, apart from utility methods, I don't actually know how something will work before I have at least partially implemented it. We have implemented a payments change recently (Paypal to Recurly) and not only are the two very different but we have added additional functionality that we could have done with Paypal but didn't. How do you TDD that? Even writing a simple test like "Go to this page and click <Order>" doesn't really work since when you come to it, you realise you can't just order but you also need to go to the create account page first etc.

projektfu 5y ago
A lot of effective functional testing is getting fixtures ready so that your test can reliably hit the important part of the system under test. For external integrations this becomes difficult. Presumably your test uses the api sandbox, so hopefully it gives reproducible results. If not, then you need to test manually and focus on unit testing to assure the software.
berkes 5y ago
> How do you TDD that? Counterquestion: how do you know, at each release, that the paypal-stuff still works? If the answer is "I test it by making a purchase at each release", the obvious question is: why not automate this? That is one benefit of TDD. So why not start of with that automation? In your specific case, I would write a simple test first, that, says "on the page of the orange Y-combinator swag, order a shirt and receive a timely confirmation.". Which is subtly, yet importantly different from "go to X and click order". On the level of such acceptence tests, trying to keep away from implementation details like "pages", "buttons" and so on, helps with solving your problem too: you write down the intent&outcome, rather than the implementation. Focus on the business value. Solve that first. And fixate the outcome for all future: regardless of the PSP, pages, exact values (timely vs within 1 sec), actual confirmation-channels and whatnot, the business value is "green" when it works, and "red" when it doesn't. Then I start implementing it. Maybe just add one of these <iframe> things that Paypal offers, even. I don't really care: the quest thing to make it work. as long as it gets me to have that acceptance test green with least work, I'm happy. Happy, but certainly not done. Because the most overlooked part of TDD is the Refactoring. Red-Green-Refactor. Once it is green, start refactoring. Replace the iframe with some library. Go a step further and replace that with my own paypal-API-client. Connect that to my event-stream, or not.

DanielBMarkham 5y ago

I know it's good commenting style on HN to avoid quick one-liners, but TDD is one of those topics that we've kicked around forever. Perhaps a one-liner would help clarify things. TDD is a coding practice to help the programmer understand that their mental model of the code does not reflect the actual code itself. That's it. There are some follow-ons, of course, like if your mental model is exactly in-line with the code then logical errors are unlikely (although business errors can be quite common!). Another follow-on is that TDD is much more about design than testing. Once you realize that it's a practice or habit that's all about sending information from the code back to the mind of the coder, all of that other stuff sorts itself out. Shameless plug: this has been on my mind quite a bit lately since I blogged last week on the unreasonable effectiveness of testing as a way out of both cognitive and coding anti-patterns: https://danielbmarkham.com/cognition-versus-programming-the-big-two/ https://danielbmarkham.com/cognition-versus-programming-the-...

draw_down 5y ago

andrewallbright 5y ago

Reading this article and reading the comments I am reminded that writing automated testing is still an art and less of a science. I highly value testing and it does eat up project time in weird ways. I believe as I get more experience it won't eat as much time. My hope is for productivity gains in the future. I hope in the future most documentation is written with a testing focus. Maybe then I would not have to understand as much as I do now before finishing testing.

deleted 5y ago

[deleted]

throwaway9870 5y ago

I don't take testing advice from anyone who uses generic terms like "test coverage". Test coverage is a complex subject, but without a basic understanding, tests can create a false sense of security. Saying something like "100% coverage" means almost nothing to anyone that understands testing. This is a worthwhile read: https://en.wikipedia.org/wiki/Code_coverage https://en.wikipedia.org/wiki/Code_coverage Without formal methods, software will realistically never be 100% tested. But don't worry, your hardware isn't either!

commandlinefan 5y ago

> The database should be part of the tests. Do not stub it. I've never seen this work well. The tests either rely on a shared/remote database or on a local database instance that has to be set up/torn down between tests. Both approaches cause unending headaches in terms of false positives, onboarding new developers, dependency problems and maintenance.

groby_b 5y ago

Having spent way too much time on the TDD mailing lists in the late 90's/early 2000s, that's... an attempt at retconning, at best. That's not what unit tests meant, and nobody advocated for them that way. C2 still doesn't: https://wiki.c2.com/?UnitTest https://wiki.c2.com/?UnitTest Kent Beck, Robert Martin et.al. didn't. So, no, it's not a misunderstanding.

gdennci 5y ago

There are no "magic bullet" in software development. TDD is my go-to approach if the situation allows it. But it is not always feasible. Sometimes you have to write a bit of code first to find your solution path. Ppl should stop with their dogmatism and start thinking about their approach by themselves.