5 ms·

I rarely write isolated unit tests. Most of the systems I build use a database on which all logic depends, and often a network connection. I've worked on syst

by codr7 2y ago

I rarely write isolated unit tests. Most of the systems I build use a database on which all logic depends, and often a network connection. I've worked on systems where these aspects were mocked, and they eventually grind to a halt because of the effort required to make the tiniest change. First of all you need a way to create a pristine database from code, preferably in memory. Second nested nested transactions are nice, since you can simply rollback the outer transaction per test case; otherwise you need to drop/create the database which is slower. For networked servers, an easy way to start/stop servers in code and send requests to them is all you need. Given these pieces, it's easy to write integration tests that run fast enough and give a lot of bang for the buck. TDD is even more rare for me, I typically only do that when designing API's I'm unsure about, which makes imagining user code difficult. And fixing bugs, because it makes total sense to have a failing test to verify that you fixed it, and that it remains fixed.

hitchstory 2y ago
I still find the skepticism around TDD weird. Except for a few pretty niche scenarios (e.g. it's experimental code or manual testing is cheaper for some obscure reason) i dont really see the point of not doing it. I especially dont see what is gained by writing the test after.
- codr7 2y ago
  Coding is not religion for me, I have no patience for fundamentalism. I will write as many tests as I need to feel confident, which depends on context. And integration tests give me a lot more confidence than mocked unit tests.
  - hitchstory 2y ago
    I hate coding fundamentalism with a passion too. The only thing I get really religious about in coding is the importance of trade offs. The cost/benefit of writing a test before just consistently exceeded doing it after for me. Same for integration, e2e or unit tests (there's never been a rule that says you can only TDD with a unit test). The cost/benefit trade off for tests with mocks vs. database is a different topic - orthogonal to the practise of red/green/refactor, and one where IMO the trade offs are much less obvious.
    - codr7 2y ago
      I honestly can't see how writing integration tests before code would even look in practice, that puzzle usually isn't even close to finished at that point in time. It sometimes makes sense for unit tests; I'll occasionally do that when I'm unsure about the API of the code I'm writing since it allows me to spend some time in the user's shoes. But like I said, I don't do fundamentalism.
- xenomachina 2y ago
  > I especially dont see what is gained by writing the test after. I assume you mean versus writing it first, rather than versus not writing it at all. I've found that TDD works well for bottom-up coding, but not so well for top-down. With bottom up, I can write the test for a piece at the bottom, write the code to pass the test, and move on. With top-down, if I write the test first, it might be a long while before I have that top-level working, because the bottom bits don't exist yet. When I feel it's better to write things top-down, I'll often use TDD for the bottom bits I need to write, but for the bits above that, I'll write the tests "on my way back up".
- simonw 2y ago
  "I especially dont see what is gained by writing the test after." The greatest value in tests is that they help prevent future changes from breaking existing functionality. Writing the test after you write the implementation is equally useful for achieving that as writing the test before you write the implementation.
  - hitchstory 2y ago
    Not the only value though. Red-green-refactor can also provides live feedback about whether your code is behaving correctly as you write it. Requiring the test before writing the code also ensures you dont forget to write a test to match the scenario. So what is gained by test after... is that it is almost as good? I still dont get it.
    - torben-friis 2y ago
      Sometimes you need the unit before you can unit test. So you end up writing the unit's boilerplate, that doesn't yet do anything but needs to compile/run for a suite to test it throwing a adhoc error of the notImplemented sort or whatever, then break flow to set the test suite and check the reds (that aren't telling you anything useful because of course it's red), then actually write the code. I find it flow-breaking and cumbersome for little to no gain. For additions to existing code I find TDD more useful, but even then it's not unusual to decide to move logic around until deciding a final structure, so the units you end up with might be solidified later in time. Writing tests for scraped units is a waste of time then.
      - hitchstory 2y ago
        >Sometimes you need the unit before you can unit test. Right. In those situations I TDD with an e2e or integration test. I dont get why youd restrict yourself to doing TDD with just with low level unit tests.
        torben-friis 2y ago
        I don't, but you agree that in that case the unit test comes after? That was the point I was arguing.
- voiceofunreason 2y ago
  "I still find the skepticism around TDD weird." A small community of programmers, with a disproportionately large audience, foretold that practicing test-driven development would produce great benefits; over twenty five years the audience has found that not to be the case. Compare with "continuous integration" - here, the immediate returns of trying the proposed discipline were so good that pretty much everybody who tried the experiment got positive returns, and leaned into it, and now CI (and later CD) are _everywhere_. As for what is gained, try this spelling: test driven development adds load to your interfaces at a time when you know the least about the problem you are trying to solve, which is to say the period where having your interfaces be flexible is valuable. And thus, the technique gets criticism from both ends -- that design work that should have been done up front is deferred (making the design more difficult to change, therefore introducing costs/delays), and that the investment is being made in testing before you have a clear understanding for which tests are going to be sensitive to the actual errors that you introduce creating the code (thereby both increasing the amount of "waste" in the test suite, in addition to increasing the risk of needing test rewrites). The situation is further not improved by (a) the fact that most TDD demonstrations are problems that are small, stable problems that you can solve in about an hour with any technique at all and (b) the designs produced in support of the TDD practice aren't clearly an improvement on "just doing it", and in some notable cases have been much much worse. So if it is working for you: GREAT, keep it up; no reason for you not to reap the benefits if your local conditions are such that TDD gives you the best positive return on your investment.
  - hitchstory 2y ago
    >As for what is gained, try this spelling: test driven development adds load to your interfaces at a time when you know the least about the problem you are trying to solve If Im writing a single line of production code I should know as much as possible what requirements problem Im actually trying to solve with it first, no? This is actually dovetails into a benefit to writing the test first. If you flesh out a user story scenario in the form of an executable test it can provoke new questions ("hm, actually I'd need the user ID on this new endpoint to satisfy this requirement...") and you can more quickly return to stakeholders ("can you send me a user ID in this API call?") and "fix" your "requirements bugs" before making more expensive lower level changes to the code. This outside-in "flipping between one layer and the layer directly beneath it" is very effective at properly refining requirements, tests and architecture. >And thus, the technique gets criticism from both ends -- that design work that should have been done up front is deferred I dont think "design work" should be done up front if you can help it. I've always felt that the very best architecture emerges as a result of aggressive refactoring done within the confines of a complete set of tests that made as few architectural assumptions as possible. Why? Coz we're all bad at predicting the future and it's better if we dont try. This is a mostly separate issue from TDD though.
motorest 2y ago
> Most of the systems I build use a database on which all logic depends, and often a network connection. Why do you believe this is relevant? Unit tests target units of code, and check if the unit of code verifies specific invariants for specific combinations and ranges of input values. The concept of a network or a database does not exist in unit tests. In fact, you are expected to abstract away these aspects. > I've worked on systems where these aspects were mocked, and they eventually grind to a halt because of the effort required to make the tiniest change. This scenario is totally unrealistic if you knew what you are doing. The fact alone that you're talking about spinning up a database in the context of unit tests already raises red flags. Even if somehow you're talking about integration and end-to-end tests, none of the perceived problems you're mentioning are even a challenge, let alone a problem. > Given these pieces, it's easy to write integration tests that run fast enough and give a lot of bang for the buck. That's perfectly fine. It's besides the point though, and completely ignores the whole point of unit tests, which is to have a bunch of fast tests that check if your code continues to do the right thing between changes. Unit tests also compel developers to think through how they create/update code to prevent them from introducing handled scenarios. There are many reasons why the test pyramid is a thing, and why the unit test layer includes far more tests than any other layer.
- simonw 2y ago
  Tests that mock databases are bad tests. This is a hill I am happy to die on. Spinning up a fresh database instance for each testing run is trivially easy in every testing framework worth its salt.
- rcxdude 2y ago
  This works fine if the difficult part of your software is in nicely abstracted boxes, but in my experience the content of the boxes is usually fairly trivial: you can write a huge number of tests for them, but they're rarely where problems occur. The integration and interaction are where you get by far the most value out of testing (it's where most errors will occur: units are usually small enough to fit in your head while working on them, but the system as a whole it not), and for that you should abstract out the bare minimum needed to control your test environment and get acceptable performance, not mock absolutely everything by default. Unit tests are the last tests that should be added to any system, and I've yet to see any system where the effort isn't better spent on integration testing (maybe once you're as thorough as sqlite?).

by codr7 2y ago

hitchstory 2y ago

I still find the skepticism around TDD weird. Except for a few pretty niche scenarios (e.g. it's experimental code or manual testing is cheaper for some obscure reason) i dont really see the point of not doing it. I especially dont see what is gained by writing the test after.

codr7 2y ago
Coding is not religion for me, I have no patience for fundamentalism. I will write as many tests as I need to feel confident, which depends on context. And integration tests give me a lot more confidence than mocked unit tests.
- hitchstory 2y ago
  I hate coding fundamentalism with a passion too. The only thing I get really religious about in coding is the importance of trade offs. The cost/benefit of writing a test before just consistently exceeded doing it after for me. Same for integration, e2e or unit tests (there's never been a rule that says you can only TDD with a unit test). The cost/benefit trade off for tests with mocks vs. database is a different topic - orthogonal to the practise of red/green/refactor, and one where IMO the trade offs are much less obvious.
  - codr7 2y ago
    I honestly can't see how writing integration tests before code would even look in practice, that puzzle usually isn't even close to finished at that point in time. It sometimes makes sense for unit tests; I'll occasionally do that when I'm unsure about the API of the code I'm writing since it allows me to spend some time in the user's shoes. But like I said, I don't do fundamentalism.
xenomachina 2y ago
> I especially dont see what is gained by writing the test after. I assume you mean versus writing it first, rather than versus not writing it at all. I've found that TDD works well for bottom-up coding, but not so well for top-down. With bottom up, I can write the test for a piece at the bottom, write the code to pass the test, and move on. With top-down, if I write the test first, it might be a long while before I have that top-level working, because the bottom bits don't exist yet. When I feel it's better to write things top-down, I'll often use TDD for the bottom bits I need to write, but for the bits above that, I'll write the tests "on my way back up".
simonw 2y ago
"I especially dont see what is gained by writing the test after." The greatest value in tests is that they help prevent future changes from breaking existing functionality. Writing the test after you write the implementation is equally useful for achieving that as writing the test before you write the implementation.
- hitchstory 2y ago
  Not the only value though. Red-green-refactor can also provides live feedback about whether your code is behaving correctly as you write it. Requiring the test before writing the code also ensures you dont forget to write a test to match the scenario. So what is gained by test after... is that it is almost as good? I still dont get it.
  - torben-friis 2y ago
    Sometimes you need the unit before you can unit test. So you end up writing the unit's boilerplate, that doesn't yet do anything but needs to compile/run for a suite to test it throwing a adhoc error of the notImplemented sort or whatever, then break flow to set the test suite and check the reds (that aren't telling you anything useful because of course it's red), then actually write the code. I find it flow-breaking and cumbersome for little to no gain. For additions to existing code I find TDD more useful, but even then it's not unusual to decide to move logic around until deciding a final structure, so the units you end up with might be solidified later in time. Writing tests for scraped units is a waste of time then.
    - hitchstory 2y ago
      >Sometimes you need the unit before you can unit test. Right. In those situations I TDD with an e2e or integration test. I dont get why youd restrict yourself to doing TDD with just with low level unit tests.
      - torben-friis 2y ago
        I don't, but you agree that in that case the unit test comes after? That was the point I was arguing.
voiceofunreason 2y ago
"I still find the skepticism around TDD weird." A small community of programmers, with a disproportionately large audience, foretold that practicing test-driven development would produce great benefits; over twenty five years the audience has found that not to be the case. Compare with "continuous integration" - here, the immediate returns of trying the proposed discipline were so good that pretty much everybody who tried the experiment got positive returns, and leaned into it, and now CI (and later CD) are _everywhere_. As for what is gained, try this spelling: test driven development adds load to your interfaces at a time when you know the least about the problem you are trying to solve, which is to say the period where having your interfaces be flexible is valuable. And thus, the technique gets criticism from both ends -- that design work that should have been done up front is deferred (making the design more difficult to change, therefore introducing costs/delays), and that the investment is being made in testing before you have a clear understanding for which tests are going to be sensitive to the actual errors that you introduce creating the code (thereby both increasing the amount of "waste" in the test suite, in addition to increasing the risk of needing test rewrites). The situation is further not improved by (a) the fact that most TDD demonstrations are problems that are small, stable problems that you can solve in about an hour with any technique at all and (b) the designs produced in support of the TDD practice aren't clearly an improvement on "just doing it", and in some notable cases have been much much worse. So if it is working for you: GREAT, keep it up; no reason for you not to reap the benefits if your local conditions are such that TDD gives you the best positive return on your investment.
- hitchstory 2y ago
  >As for what is gained, try this spelling: test driven development adds load to your interfaces at a time when you know the least about the problem you are trying to solve If Im writing a single line of production code I should know as much as possible what requirements problem Im actually trying to solve with it first, no? This is actually dovetails into a benefit to writing the test first. If you flesh out a user story scenario in the form of an executable test it can provoke new questions ("hm, actually I'd need the user ID on this new endpoint to satisfy this requirement...") and you can more quickly return to stakeholders ("can you send me a user ID in this API call?") and "fix" your "requirements bugs" before making more expensive lower level changes to the code. This outside-in "flipping between one layer and the layer directly beneath it" is very effective at properly refining requirements, tests and architecture. >And thus, the technique gets criticism from both ends -- that design work that should have been done up front is deferred I dont think "design work" should be done up front if you can help it. I've always felt that the very best architecture emerges as a result of aggressive refactoring done within the confines of a complete set of tests that made as few architectural assumptions as possible. Why? Coz we're all bad at predicting the future and it's better if we dont try. This is a mostly separate issue from TDD though.

motorest 2y ago

> Most of the systems I build use a database on which all logic depends, and often a network connection. Why do you believe this is relevant? Unit tests target units of code, and check if the unit of code verifies specific invariants for specific combinations and ranges of input values. The concept of a network or a database does not exist in unit tests. In fact, you are expected to abstract away these aspects. > I've worked on systems where these aspects were mocked, and they eventually grind to a halt because of the effort required to make the tiniest change. This scenario is totally unrealistic if you knew what you are doing. The fact alone that you're talking about spinning up a database in the context of unit tests already raises red flags. Even if somehow you're talking about integration and end-to-end tests, none of the perceived problems you're mentioning are even a challenge, let alone a problem. > Given these pieces, it's easy to write integration tests that run fast enough and give a lot of bang for the buck. That's perfectly fine. It's besides the point though, and completely ignores the whole point of unit tests, which is to have a bunch of fast tests that check if your code continues to do the right thing between changes. Unit tests also compel developers to think through how they create/update code to prevent them from introducing handled scenarios. There are many reasons why the test pyramid is a thing, and why the unit test layer includes far more tests than any other layer.

simonw 2y ago
Tests that mock databases are bad tests. This is a hill I am happy to die on. Spinning up a fresh database instance for each testing run is trivially easy in every testing framework worth its salt.
rcxdude 2y ago
This works fine if the difficult part of your software is in nicely abstracted boxes, but in my experience the content of the boxes is usually fairly trivial: you can write a huge number of tests for them, but they're rarely where problems occur. The integration and interaction are where you get by far the most value out of testing (it's where most errors will occur: units are usually small enough to fit in your head while working on them, but the system as a whole it not), and for that you should abstract out the bare minimum needed to control your test environment and get acceptable performance, not mock absolutely everything by default. Unit tests are the last tests that should be added to any system, and I've yet to see any system where the effort isn't better spent on integration testing (maybe once you're as thorough as sqlite?).