4 ms·

I've always been curious how people do integration tests on things with multiple servers (like SOA stuff), it seems like it'd be dreadful, unless you use Vagran

by PLejeck 13y ago

I've always been curious how people do integration tests on things with multiple servers (like SOA stuff), it seems like it'd be dreadful, unless you use Vagrant to (quite slowly) spawn a new VM every time you want to test. Anyone with experience care to enlighten me?

saryant 13y ago
We heavily utilize Akka clustering for our backend and we do this using Akka's multi-node testing feature. http://doc.akka.io/docs/akka/2.2.1/dev/multi-node-testing.html http://doc.akka.io/docs/akka/2.2.1/dev/multi-node-testing.ht... For the most part we only spin up multiple JVMs on the same machine but it's not a huge leap to use multiple machines. We don't do so regularly because, as you said, Vagrant takes forever to spin up a new VM. Our Jenkins server does spin up multiple JVMs for our integration tests though.
AznHisoka 13y ago
I don't have such a problem, but in my current project, I am integrating with various APIs such as Twitter + Facebook, and unit testing my code can be a nightmare at times since I'm dependent on them being stable.
tunesmith 13y ago
I've used mocking for this purpose. Not in the same way you would for unit testing, of course. But if you want to integration test the (entire) workings of a consumer of a service, then your test starts with its entry point (whether a form submission, an API request, or whatever). Then if part of your consumer's business logic relies on a call to an external SOA service, you mock that service. In other words you say, "Given this particular form of request that would be sent to the SOA service, respond with this particular service response". Then your consumer will continue on with its business logic, all the way up until it returns its response to your original request, and you can apply your assertions then. It's not a true end-to-end test, which would require all your services to be up, but it is an integration test since it tests the entire workings of the consumer. The assumption is that other test suites would test each of the services, and maybe a ping test to test the availability of the services themselves.
- PLejeck 13y ago
  I've never been able to fully grasp mocking and stubbing.
  - nl 13y ago
    Mock = Responding to requests with no (or minimal) handling of logic. In SOA it can be as simple as putting a set of files containing the expected responses (in XML or JSON form) on a webserver and using mod_rewrite to make sure it responds with the correct file to a given request.
  - tunesmith 13y ago
    The most common objection I've heard to mocking is along the lines of, "If you're mocking, what is the point of the test?" The answer lies in having a clear understanding of what you are testing. For instance, method A might call another class's method B. If you're writing a unit test for method A, you don't also want to test method B. (Other unit tests would test B in isolation, instead.) So in A's test, you make A call a mock B rather than the real B, and you ensure B returns a particular kind of response. In other words, you want to make sure method A behaves correctly when it gets certain responses from B. So in english, a test is basically saying, "When B gives this type of response (which I'm forcing it to respond with), make sure A behaves as I expect", and a second test says, "When B gives this other type of response (which I'm forcing it to respond with), make sure A behaves as I expect. So when you write a unit test for A, you mock B by making it send back a canned response and make A use it. Then you call A with some parameters, and then you make sure that A's return type is as you expect. One advantage of this is that if a future code enhancement breaks one of the methods deep in the code, then the failing unit test will tell you exactly where the problem is. Without mocking, a failed unit test will make you have to examine several layers of stack trace to find the bug.
    - justincormack 13y ago
      Also getting the right kind of response out of a back end server may be hard, eg getting certain kinds of error or unusual responses, so mocking is the only way to get them reliably. They are not a substitute for end to end tests though, as you cannot tell if some other assumption is broken.
nl 13y ago
Docker might work pretty well. (But generally I prefer the Mock method anyway)
- nl 13y ago
  Wow.. downvotes? I guess I need to add an explanation or something? Docker allows very quick startup of containers (and for testing purposes the distinctions between containers and VM aren't important. Additionally, the easy command line and app ecosystem Docker gives you over just using Linux containers is a big advantage here.). I prefer the Mock method for two reasons: 1) It's extremely fast - much faster than even docker would be. 2) The data you use for mocks can form part of your documentation (if done properly).
  - coconutrandom 13y ago
    +1 docker and jenkins and is a happy medium between speed and isolation.
twic 13y ago
I work on a system split into 4-6 applications, depending on what you consider to be part of the system. We have a tier of integration tests that run across several of them. I can confirm that it is indeed often dreadful. At the moment, we don't use VMs, we just fire up multiple servers on the same box. They use different ports, so it's no big deal. The tests are framed as being for one particular application in the system (eg a test that asserts that the front-end server displays the right profit and loss is a test for the front-end server, even though it uses the price server and the calculation server). Tests for a given application will use the current checkout of that application's code, and obtain packaged versions of the other applications from our internal artifact store. Tests always uses the latest versions of those available; the assumption is that the latest version will have been deployed by the time the code under test is deployed. There are plenty of things that the multiple-apps-on-one-box approach doesn't test, mostly around networking, so we are trying to get our VM tooling up to speed to run multiple VMs instead. We already use the tooling to deploy VMs for apps in production, but it needs some tweaking to work well on desktops and in CI. It's not clear that this approach will completely replace the multiple-apps-on-one-box approach, because of the overhead in starting the VMs. We'll play it by ear. Some of the dreadfulness of this approach comes from mismatches between test and production around versioning and networking. Most of it comes from the fact that in practice, it is much harder to track down the source of an error than when testing a single monolithic application. Now, it could be the case that this pain, and the other pain resulting from splitting a system into multiple small applications, is worth it, because that splitting brings other advantages. I don't have a parallel version of our system implemented as a monolith, so i can't tell you if that's true. I have serious doubts about it, though.

by PLejeck 13y ago

saryant 13y ago

We heavily utilize Akka clustering for our backend and we do this using Akka's multi-node testing feature. http://doc.akka.io/docs/akka/2.2.1/dev/multi-node-testing.html http://doc.akka.io/docs/akka/2.2.1/dev/multi-node-testing.ht... For the most part we only spin up multiple JVMs on the same machine but it's not a huge leap to use multiple machines. We don't do so regularly because, as you said, Vagrant takes forever to spin up a new VM. Our Jenkins server does spin up multiple JVMs for our integration tests though.

AznHisoka 13y ago

I don't have such a problem, but in my current project, I am integrating with various APIs such as Twitter + Facebook, and unit testing my code can be a nightmare at times since I'm dependent on them being stable.

tunesmith 13y ago

I've used mocking for this purpose. Not in the same way you would for unit testing, of course. But if you want to integration test the (entire) workings of a consumer of a service, then your test starts with its entry point (whether a form submission, an API request, or whatever). Then if part of your consumer's business logic relies on a call to an external SOA service, you mock that service. In other words you say, "Given this particular form of request that would be sent to the SOA service, respond with this particular service response". Then your consumer will continue on with its business logic, all the way up until it returns its response to your original request, and you can apply your assertions then. It's not a true end-to-end test, which would require all your services to be up, but it is an integration test since it tests the entire workings of the consumer. The assumption is that other test suites would test each of the services, and maybe a ping test to test the availability of the services themselves.

PLejeck 13y ago
I've never been able to fully grasp mocking and stubbing.
- nl 13y ago
  Mock = Responding to requests with no (or minimal) handling of logic. In SOA it can be as simple as putting a set of files containing the expected responses (in XML or JSON form) on a webserver and using mod_rewrite to make sure it responds with the correct file to a given request.
- tunesmith 13y ago
  The most common objection I've heard to mocking is along the lines of, "If you're mocking, what is the point of the test?" The answer lies in having a clear understanding of what you are testing. For instance, method A might call another class's method B. If you're writing a unit test for method A, you don't also want to test method B. (Other unit tests would test B in isolation, instead.) So in A's test, you make A call a mock B rather than the real B, and you ensure B returns a particular kind of response. In other words, you want to make sure method A behaves correctly when it gets certain responses from B. So in english, a test is basically saying, "When B gives this type of response (which I'm forcing it to respond with), make sure A behaves as I expect", and a second test says, "When B gives this other type of response (which I'm forcing it to respond with), make sure A behaves as I expect. So when you write a unit test for A, you mock B by making it send back a canned response and make A use it. Then you call A with some parameters, and then you make sure that A's return type is as you expect. One advantage of this is that if a future code enhancement breaks one of the methods deep in the code, then the failing unit test will tell you exactly where the problem is. Without mocking, a failed unit test will make you have to examine several layers of stack trace to find the bug.
  - justincormack 13y ago
    Also getting the right kind of response out of a back end server may be hard, eg getting certain kinds of error or unusual responses, so mocking is the only way to get them reliably. They are not a substitute for end to end tests though, as you cannot tell if some other assumption is broken.

nl 13y ago

Docker might work pretty well. (But generally I prefer the Mock method anyway)

nl 13y ago
Wow.. downvotes? I guess I need to add an explanation or something? Docker allows very quick startup of containers (and for testing purposes the distinctions between containers and VM aren't important. Additionally, the easy command line and app ecosystem Docker gives you over just using Linux containers is a big advantage here.). I prefer the Mock method for two reasons: 1) It's extremely fast - much faster than even docker would be. 2) The data you use for mocks can form part of your documentation (if done properly).
- coconutrandom 13y ago
  +1 docker and jenkins and is a happy medium between speed and isolation.

twic 13y ago

I work on a system split into 4-6 applications, depending on what you consider to be part of the system. We have a tier of integration tests that run across several of them. I can confirm that it is indeed often dreadful. At the moment, we don't use VMs, we just fire up multiple servers on the same box. They use different ports, so it's no big deal. The tests are framed as being for one particular application in the system (eg a test that asserts that the front-end server displays the right profit and loss is a test for the front-end server, even though it uses the price server and the calculation server). Tests for a given application will use the current checkout of that application's code, and obtain packaged versions of the other applications from our internal artifact store. Tests always uses the latest versions of those available; the assumption is that the latest version will have been deployed by the time the code under test is deployed. There are plenty of things that the multiple-apps-on-one-box approach doesn't test, mostly around networking, so we are trying to get our VM tooling up to speed to run multiple VMs instead. We already use the tooling to deploy VMs for apps in production, but it needs some tweaking to work well on desktops and in CI. It's not clear that this approach will completely replace the multiple-apps-on-one-box approach, because of the overhead in starting the VMs. We'll play it by ear. Some of the dreadfulness of this approach comes from mismatches between test and production around versioning and networking. Most of it comes from the fact that in practice, it is much harder to track down the source of an error than when testing a single monolithic application. Now, it could be the case that this pain, and the other pain resulting from splitting a system into multiple small applications, is worth it, because that splitting brings other advantages. I don't have a parallel version of our system implemented as a monolith, so i can't tell you if that's true. I have serious doubts about it, though.