3 ms·

The tests get in the way. Because my design does not have low coupling, I end up with tests that also do not have low coupling. Not to be smug, but I feel like

by chubot 9y ago

The tests get in the way. Because my design does not have low coupling, I end up with tests that also do not have low coupling. Not to be smug, but I feel like this is a rookie mistake I learned 10 years ago immediately after starting TDD. The slogan I use in my head is that testing calcifies interfaces. Once you have a test against an interface, it's hard to change it. If you find yourself changing tests and code AT THE SAME TIME, e.g. while refactoring, then your tests become less useful, and are just slowing you down. Instead, you want to test against stable interfaces -- ones you did NOT create. That could be HTTP/WSGI/Rack for web services, or stdin/stdout/argv for command line tools. Unit test frameworks and in particular mocking frameworks can lead you into this trap. I've never used a mocking library -- they are the worst. There are pretty straightforward solutions to this problem. If I want to be fancy then I will say I write "bespoke test frameworks", but all this means is: write some simple Python or shell scripts to test your code from a coarse-grained level. Your tests can often be in a different langauge than your code. The last two posts on my blog are about this: "How I Use Tests": http://www.oilshell.org/blog/2017/06/22.html http://www.oilshell.org/blog/2017/06/22.html "How I Plan to Use Tests: Transforming OSH": http://www.oilshell.org/blog/2017/06/24.html http://www.oilshell.org/blog/2017/06/24.html -- I want to change the LANGUAGE my code is written in, but preserve the tests, and use them as a guide. And definitely these kinds of tests work better for data manipulation rather than heavily stateful code. But the point is that testing teaches you good design, and good design is to separate your data manipulation from your program state as much as possible. State is hard, and logic is easy (if you have isolated it and tested it.) Summary: I use TDD, it absolutely works. But I use more coarse-grained tests against STABLE INTERFACES I didn't create.

daxfohl 9y ago
This is terrific. You know, I've spent the last ten years perfecting coding strategies around unit test frameworks and mocks. I'm really really good at that. Really. I won't be humble; if somebody wants this, then I'm one of the best in the business. Yet, I'm slowly coming to the realization that that's a perfectly useless skill. Perhaps, even a negative skill. It depends on the project, but for most projects I've worked on, the most difficult parts are the integrations with external systems. Figuring out what headers you need to pass in a call to Facebook, or what certificates you need to access some third-party API, or what data to send over USB to activate some device. Unit tests / mocks let you blithely ignore all those things. You mock them out, hide behind an interface, write your "application code" that uses these interfaces to do whatever your application does, make unit tests with mocks that behave the way you'd like, and viola your app is done! With almost 100% code coverage even! And it's even fairly well-designed with fairly low coupling. Except, it doesn't work at all, the hard work is still ahead of you, and your interfaces are all probably leaky abstractions that you're going to have to change substantially to make it work for real. Anyway that's the hole I've dug for my current project. I'm pretty quickly coming to the conclusion that I need to unlearn quite a bit, retrain my instincts. I like your posts. Not even much for the content, but mainly for the concept. Rather than cargo culting onto "unit-testing-in-framework-X-to-achieve-100%-code-coverage-because-that's-how-you-make-sure-your-app-works,-right?", it's more of an intentional approach. Step 1: determine what we really need to make sure of, step 2: determine the best way to achieve that. Ultimately, what I think is wrong with TDD as most people know it, in a word, is that it's a shortcut. It's easy. You populate your mocks with fake data and it's infinitely repeatable so yay. Populating an actual database with fake data and making sure it's deleted/refreshed/whatever between tests is difficult. But that doesn't mean it's not the right thing to do.
- bluGill 9y ago
  I can argue both sides of this (in fact I have). I would response here that you need to consider your facade design pattern. You should have a facade that you test your code with by careful mocks and fakes. Then the facade translates between your code (which now works) and the API. Particularity if the third party is infamous for breaking changes all the time you want to ensure that when things break you only have to figure out what they changed and fix the glue code. Of course if all you have is glue code (this is likely: a lot of real world problems are just getting data from one system to another) there is nothing to test.
nvarsj 9y ago
Thanks for this. TDD against coarse interfaces as you put it is the correct approach. Sometimes called end to end tests. The use of TDD at the class level, with mocking hell, is the cause of most of TFA's issues.
- atom-morgan 9y ago
  > The use of TDD at the class level, with mocking hell, is the cause of most of TFA's issues. What's your approach with testing? Do you typically do E2E tests only?

by chubot 9y ago

daxfohl 9y ago

This is terrific. You know, I've spent the last ten years perfecting coding strategies around unit test frameworks and mocks. I'm really really good at that. Really. I won't be humble; if somebody wants this, then I'm one of the best in the business. Yet, I'm slowly coming to the realization that that's a perfectly useless skill. Perhaps, even a negative skill. It depends on the project, but for most projects I've worked on, the most difficult parts are the integrations with external systems. Figuring out what headers you need to pass in a call to Facebook, or what certificates you need to access some third-party API, or what data to send over USB to activate some device. Unit tests / mocks let you blithely ignore all those things. You mock them out, hide behind an interface, write your "application code" that uses these interfaces to do whatever your application does, make unit tests with mocks that behave the way you'd like, and viola your app is done! With almost 100% code coverage even! And it's even fairly well-designed with fairly low coupling. Except, it doesn't work at all, the hard work is still ahead of you, and your interfaces are all probably leaky abstractions that you're going to have to change substantially to make it work for real. Anyway that's the hole I've dug for my current project. I'm pretty quickly coming to the conclusion that I need to unlearn quite a bit, retrain my instincts. I like your posts. Not even much for the content, but mainly for the concept. Rather than cargo culting onto "unit-testing-in-framework-X-to-achieve-100%-code-coverage-because-that's-how-you-make-sure-your-app-works,-right?", it's more of an intentional approach. Step 1: determine what we really need to make sure of, step 2: determine the best way to achieve that. Ultimately, what I think is wrong with TDD as most people know it, in a word, is that it's a shortcut. It's easy. You populate your mocks with fake data and it's infinitely repeatable so yay. Populating an actual database with fake data and making sure it's deleted/refreshed/whatever between tests is difficult. But that doesn't mean it's not the right thing to do.

bluGill 9y ago
I can argue both sides of this (in fact I have). I would response here that you need to consider your facade design pattern. You should have a facade that you test your code with by careful mocks and fakes. Then the facade translates between your code (which now works) and the API. Particularity if the third party is infamous for breaking changes all the time you want to ensure that when things break you only have to figure out what they changed and fix the glue code. Of course if all you have is glue code (this is likely: a lot of real world problems are just getting data from one system to another) there is nothing to test.

nvarsj 9y ago

Thanks for this. TDD against coarse interfaces as you put it is the correct approach. Sometimes called end to end tests. The use of TDD at the class level, with mocking hell, is the cause of most of TFA's issues.

atom-morgan 9y ago
> The use of TDD at the class level, with mocking hell, is the cause of most of TFA's issues. What's your approach with testing? Do you typically do E2E tests only?