4 ms·

One thing I have never understood is how to apply TDD when the code that you are writing is nearly 100% dependent on external services and "black-box" library c

by douche 12y ago

One thing I have never understood is how to apply TDD when the code that you are writing is nearly 100% dependent on external services and "black-box" library code. For instance, I rely on several more or less undocumented Microsoft APIs in my day job. How do I even know what correct outputs to code interacting with those components would be? It could depend on server policies, registry settings, firewall settings, SQL server settings, Active directory settings, and a hundred other things. My application code is bog-simple next to all of the code handling possible FUBAR states of all of this environment.

BurningFrog 12y ago
What you're saying is that it's not clear to you what your code needs to do. That's a fundamental problem that neither TDD or any other methodology can really solve.
- douche 12y ago
  It's clear to me what my code needs to do, however there are a myriad of environmental factors that can fuck up what my simple code needs to do. For instance, a common thing that I need to do is to look up one Active Directory property based on another. Depending on the environment, I may have access to that property using the logged-in user's credentials, or I may need to use a special service account set of credentials in my application, defined in a database or an app/web.config. At some point, an over-zealous administrator can disable the rights for the account that my application is using inside a client organization. Then I end up on the phone for an hour or more, figuring out why an account that used to work no longer does, and getting my clients to raise a change request with the admins that broke our application in the first place to put things back.
  - coldtea 12y ago
    >It's clear to me what my code needs to do, however there are a myriad of environmental factors that can fuck up what my simple code needs to do. For instance, a common thing that I need to do is to look up one Active Directory property based on another. Depending on the environment, I may have access to that property using the logged-in user's credentials, or I may need to use a special service account set of credentials in my application, defined in a database or an app/web.config. Some stuff you test in unit tests, other you do functional testing for. What you describe is probably higher level than unit tests. >At some point, an over-zealous administrator can disable the rights for the account that my application is using inside a client organization. Then I end up on the phone for an hour or more, figuring out why an account that used to work no longer does, and getting my clients to raise a change request with the admins that broke our application in the first place to put things back. So? That's orthogonal to your app. You test that your app works if the rights are working OK, and you test what it shows to the user if the rights are not OK. Other than that, obviously you cannot ensure it works without things it needs to have, nor do you need to test each and every externally dependent failure mode (e.g. user wiped out his hard drive).
  - BurningFrog 12y ago
    If you actually know all these myriad environmental factors, writing tests for them is no harder than for anything else. It could be a lot of work, of course, but that just reflects that you're building a complex system. Test 1: Look up AD prop with user credentials Test 2: Look up AD prop when user credentials don't work You'd almost certainly want to mock out the environment. Don't use the actual services.
  - timv 12y ago
    These two comments (below) seem to be in opposition to one another. It's clear to me what my code needs to do (Above) How do I even know what correct outputs to code interacting with those components would be? (From your original comment) If you know all the possible environmental factors that you need to deal with, then you can write up front tests for them (whether that is economical or not, is a different question). However, if the complete set of environmental factors is unknown (though potentially discoverable) then you have a requirements problem, not a code problem. TDD can't solve that problem (although it might help make it obvious that the problem exists). TDD is based on the assumption that it is possible to know when your code "works", and requires you to define that criteria up front. But any development process you follow is must have some way of answering the "am I done yet?" question, or you'd never ship anything. Perhaps your concern is that TDD doesn't help solve the "I don't have a complete set of requirements" problem. If so, then that's true, but it never claims to. If you don't have the requirements yet, then TDD says you're not ready to write code, and some other process has to be followed in order to produce requirements. It is possible to do TDD if you only have some (but not all) requirements - but you can only write code for the requirements you do have.
- millstone 12y ago
  That reply reads disparaging, perhaps implying that the author needs to spend more time designing. But that's not necessarily true! Have you ever changed your code to work around a bug in some version of a web browser, or compiler? If so, you discovered a limitation of TDD as it relates to components outside your control. Saying that "no methodology can solve this" is defeatist: there are lots of things you can do. One technique is to write your tests to use the actual component, instead of mocks. For example, I never mock the filesystem: I always design tests to run using the real thing. This reduces the chances that I've encoded some bogus assumption into my tests, and makes it more likely that I'll encounter the individual quirks of the filesystem.
  - BurningFrog 12y ago
    I just meant it sounded like the requirements aren't fully known. That's hardly OPs fault, and neither more design or more TDD can solve it.
    - collyw 12y ago
      Agreed. Much of the time the users can't accurately describe what they want. Its easier to build what you think they want quickly and show them it, then get them to say what they want different. (Though a problem with this approach is feature creep).
TheDong 12y ago
Your issue is that you're not mocking correctly. You need to write mocks for services you depend on, ideally by recording and replaying real api interactions, such that you can verify that given the libraries you depend on your tests will be correct. Then write TDD tests, then write code, then write integ tests which will verify your original mocks work as assumed.
- millstone 12y ago
  When your software is tightly coupled to some complicated external component, the most likely cause of bugs is bad assumptions about how that component behaves. The integration tests are therefore the most important, since those are what will expose the bad assumptions. I wrote a program that used the computer's MAC address. The API that reports the MAC strips leading 0s, but my MAC had no leading 0s and so I I did not notice this behavior during development. If I had just captured and replayed this result, my tests would have passed on all machines, even affected ones. But because my tests exercised the real API, I encountered the bug and was able to work around it.
- mikekchar 12y ago
  Another approach (which is not really so different) is to build adapters around external services and then write your code to use the adapters. The upside is that you can avoid writing integration tests which might be very difficult to set up (depending on the situation). The downside is that you might implement more in the adapter than you need. In practice, I tend to do a mixture. I will mock/fake the external services and then build adapters and then finally remove the mocks/fakes. In the final part you may have to stub the adapters. The unit tests in the adapters will alert me to changes in the external services, so this is relatively safe to do. Your main sources of error will be subtle bugs where you have insufficient coverage in the adapters. You must also use restraint in working on the adapters independent of the code that uses it, because you have no end to end tests. Building fakes instead of mocks in this situation is also a good way to really test your understanding of the external services. It takes more time, but it can pay dividends. Many times I will write a fake and then think, "Does it really work that way?" only to realize that I can't actually use the external service in the way I envisioned. With a mock, you completely divorce yourself from implementation so it is relatively easy to assume that the external service can do impossible things. You can go a long way before you realize that your code is unusable. One thing I will caution -- mocks, fakes, stubs and integration tests tend to attract people with almost religious views in how they should be used. I tend to believe that this is stuff is hard and that current industry practice is very naive. I think there is a lot of room for experimentation with a very big upside. But you are likely to attract a lot of criticism if you wander outside someone's boundaries, so it is best to spend significant amounts of time pairing to make sure that everybody is comfortable with the approaches you use. Nothing is worse that "test wars" where people are going in 100 different directions in the tests.
maxthegeek1 12y ago
You can either have your tests call the external service or you can mock out the service. You could also replicate the service for the purposes of testing, for example its pretty standard to have both a test and a production database.
greggman 12y ago
Here's an answer I posted on SO that might be helpful http://stackoverflow.com/questions/2611280/how-can-i-effectively-test-against-the-windows-api/2620506#2620506 http://stackoverflow.com/questions/2611280/how-can-i-effecti...
henrik_w 12y ago
In my opinion [1], TDD is best suited for algorithmic logic, but doesn't really give any benefits for coordinating code. This is explained well in "Selective Unit Testing – Costs and Benefits" http://blog.stevensanderson.com/2009/11/04/selective-unit-testing-costs-and-benefits/ http://blog.stevensanderson.com/2009/11/04/selective-unit-te... [1] http://henrikwarne.com/2014/02/19/5-unit-testing-mistakes/ http://henrikwarne.com/2014/02/19/5-unit-testing-mistakes/
- cowardlydragon 12y ago
  So.... it's basically useless for 90% of enterprise development?
- the_af 12y ago
  I've read the opposite argued by proponents of TDD (either here or on StackOverflow, I forget): that TDD is definitely not for deriving algorithmic logic, but for run-of-the-mill CRUDs. For me the definite example that TDD for algorithmic design/exploration is not very useful is the infamous Sudoku Debacle, but I've also never seen anything more than toy examples (e.g. fibonacci) of algorithmic exploration using TDD. Is there any real-world example of complex algorithmic design using TDD where the author doesn't cheat by using hidden domain knowledge to make "magical" leaps of inference in some of the TDD steps? PS: just read your blog post, but you seem to be talking about unit testing with JUnit & mocking frameworks. Just to be sure we're on the same page: we do agree that TDD and Unit Testing are not the same and do not even share the same goal, right?

by douche 12y ago

BurningFrog 12y ago

What you're saying is that it's not clear to you what your code needs to do. That's a fundamental problem that neither TDD or any other methodology can really solve.

douche 12y ago
It's clear to me what my code needs to do, however there are a myriad of environmental factors that can fuck up what my simple code needs to do. For instance, a common thing that I need to do is to look up one Active Directory property based on another. Depending on the environment, I may have access to that property using the logged-in user's credentials, or I may need to use a special service account set of credentials in my application, defined in a database or an app/web.config. At some point, an over-zealous administrator can disable the rights for the account that my application is using inside a client organization. Then I end up on the phone for an hour or more, figuring out why an account that used to work no longer does, and getting my clients to raise a change request with the admins that broke our application in the first place to put things back.
- coldtea 12y ago
  >It's clear to me what my code needs to do, however there are a myriad of environmental factors that can fuck up what my simple code needs to do. For instance, a common thing that I need to do is to look up one Active Directory property based on another. Depending on the environment, I may have access to that property using the logged-in user's credentials, or I may need to use a special service account set of credentials in my application, defined in a database or an app/web.config. Some stuff you test in unit tests, other you do functional testing for. What you describe is probably higher level than unit tests. >At some point, an over-zealous administrator can disable the rights for the account that my application is using inside a client organization. Then I end up on the phone for an hour or more, figuring out why an account that used to work no longer does, and getting my clients to raise a change request with the admins that broke our application in the first place to put things back. So? That's orthogonal to your app. You test that your app works if the rights are working OK, and you test what it shows to the user if the rights are not OK. Other than that, obviously you cannot ensure it works without things it needs to have, nor do you need to test each and every externally dependent failure mode (e.g. user wiped out his hard drive).
- BurningFrog 12y ago
  If you actually know all these myriad environmental factors, writing tests for them is no harder than for anything else. It could be a lot of work, of course, but that just reflects that you're building a complex system. Test 1: Look up AD prop with user credentials Test 2: Look up AD prop when user credentials don't work You'd almost certainly want to mock out the environment. Don't use the actual services.
- timv 12y ago
  These two comments (below) seem to be in opposition to one another. It's clear to me what my code needs to do (Above) How do I even know what correct outputs to code interacting with those components would be? (From your original comment) If you know all the possible environmental factors that you need to deal with, then you can write up front tests for them (whether that is economical or not, is a different question). However, if the complete set of environmental factors is unknown (though potentially discoverable) then you have a requirements problem, not a code problem. TDD can't solve that problem (although it might help make it obvious that the problem exists). TDD is based on the assumption that it is possible to know when your code "works", and requires you to define that criteria up front. But any development process you follow is must have some way of answering the "am I done yet?" question, or you'd never ship anything. Perhaps your concern is that TDD doesn't help solve the "I don't have a complete set of requirements" problem. If so, then that's true, but it never claims to. If you don't have the requirements yet, then TDD says you're not ready to write code, and some other process has to be followed in order to produce requirements. It is possible to do TDD if you only have some (but not all) requirements - but you can only write code for the requirements you do have.
millstone 12y ago
That reply reads disparaging, perhaps implying that the author needs to spend more time designing. But that's not necessarily true! Have you ever changed your code to work around a bug in some version of a web browser, or compiler? If so, you discovered a limitation of TDD as it relates to components outside your control. Saying that "no methodology can solve this" is defeatist: there are lots of things you can do. One technique is to write your tests to use the actual component, instead of mocks. For example, I never mock the filesystem: I always design tests to run using the real thing. This reduces the chances that I've encoded some bogus assumption into my tests, and makes it more likely that I'll encounter the individual quirks of the filesystem.
- BurningFrog 12y ago
  I just meant it sounded like the requirements aren't fully known. That's hardly OPs fault, and neither more design or more TDD can solve it.
  - collyw 12y ago
    Agreed. Much of the time the users can't accurately describe what they want. Its easier to build what you think they want quickly and show them it, then get them to say what they want different. (Though a problem with this approach is feature creep).

TheDong 12y ago

Your issue is that you're not mocking correctly. You need to write mocks for services you depend on, ideally by recording and replaying real api interactions, such that you can verify that given the libraries you depend on your tests will be correct. Then write TDD tests, then write code, then write integ tests which will verify your original mocks work as assumed.

millstone 12y ago
When your software is tightly coupled to some complicated external component, the most likely cause of bugs is bad assumptions about how that component behaves. The integration tests are therefore the most important, since those are what will expose the bad assumptions. I wrote a program that used the computer's MAC address. The API that reports the MAC strips leading 0s, but my MAC had no leading 0s and so I I did not notice this behavior during development. If I had just captured and replayed this result, my tests would have passed on all machines, even affected ones. But because my tests exercised the real API, I encountered the bug and was able to work around it.
mikekchar 12y ago
Another approach (which is not really so different) is to build adapters around external services and then write your code to use the adapters. The upside is that you can avoid writing integration tests which might be very difficult to set up (depending on the situation). The downside is that you might implement more in the adapter than you need. In practice, I tend to do a mixture. I will mock/fake the external services and then build adapters and then finally remove the mocks/fakes. In the final part you may have to stub the adapters. The unit tests in the adapters will alert me to changes in the external services, so this is relatively safe to do. Your main sources of error will be subtle bugs where you have insufficient coverage in the adapters. You must also use restraint in working on the adapters independent of the code that uses it, because you have no end to end tests. Building fakes instead of mocks in this situation is also a good way to really test your understanding of the external services. It takes more time, but it can pay dividends. Many times I will write a fake and then think, "Does it really work that way?" only to realize that I can't actually use the external service in the way I envisioned. With a mock, you completely divorce yourself from implementation so it is relatively easy to assume that the external service can do impossible things. You can go a long way before you realize that your code is unusable. One thing I will caution -- mocks, fakes, stubs and integration tests tend to attract people with almost religious views in how they should be used. I tend to believe that this is stuff is hard and that current industry practice is very naive. I think there is a lot of room for experimentation with a very big upside. But you are likely to attract a lot of criticism if you wander outside someone's boundaries, so it is best to spend significant amounts of time pairing to make sure that everybody is comfortable with the approaches you use. Nothing is worse that "test wars" where people are going in 100 different directions in the tests.

maxthegeek1 12y ago

You can either have your tests call the external service or you can mock out the service. You could also replicate the service for the purposes of testing, for example its pretty standard to have both a test and a production database.

greggman 12y ago

Here's an answer I posted on SO that might be helpful http://stackoverflow.com/questions/2611280/how-can-i-effectively-test-against-the-windows-api/2620506#2620506 http://stackoverflow.com/questions/2611280/how-can-i-effecti...

henrik_w 12y ago

In my opinion [1], TDD is best suited for algorithmic logic, but doesn't really give any benefits for coordinating code. This is explained well in "Selective Unit Testing – Costs and Benefits" http://blog.stevensanderson.com/2009/11/04/selective-unit-testing-costs-and-benefits/ http://blog.stevensanderson.com/2009/11/04/selective-unit-te... [1] http://henrikwarne.com/2014/02/19/5-unit-testing-mistakes/ http://henrikwarne.com/2014/02/19/5-unit-testing-mistakes/

cowardlydragon 12y ago
So.... it's basically useless for 90% of enterprise development?
the_af 12y ago
I've read the opposite argued by proponents of TDD (either here or on StackOverflow, I forget): that TDD is definitely not for deriving algorithmic logic, but for run-of-the-mill CRUDs. For me the definite example that TDD for algorithmic design/exploration is not very useful is the infamous Sudoku Debacle, but I've also never seen anything more than toy examples (e.g. fibonacci) of algorithmic exploration using TDD. Is there any real-world example of complex algorithmic design using TDD where the author doesn't cheat by using hidden domain knowledge to make "magical" leaps of inference in some of the TDD steps? PS: just read your blog post, but you seem to be talking about unit testing with JUnit & mocking frameworks. Just to be sure we're on the same page: we do agree that TDD and Unit Testing are not the same and do not even share the same goal, right?