6 ms·

Summary of the article: At google-scale, integration tests take too long because it triggers so many service calls. So the solution is to use static hard-coded

Summary of the article: At google-scale, integration tests take too long because it triggers so many service calls. So the solution is to use static hard-coded mock data. But also create tests to verify the mock data with real service calls. So previously, if you had N tests calling a rest service A, it would result in N calls to service A. But now, those N tests go to a hardcoded mock data, and there is a single test verifying the mock data by calling A. So N tests can run fast and there will be only a single call to A to verify the mock. However, that would be the best case, which is the mock data is sufficient for all N tests. That is not likely to be true, but probably a M number of mocks can still cover all N tests. And M < N in most cases so there's still savings in number of service calls. In the worst case, N tests will need N mocks and it will be the same as before.

sly010 10y ago
In a large graph you would re-test the same part of the code over and over, because many top level inputs is going to do trigger the same code-path deep down in the graph, which is wasteful. In a sense they memoize service call answers, so any tested code-path only needs to run once. If they actually do this across their code they will never have to run "real service calls", because those services are also tested the same way. It's mocking all the way down.
- bit_logic 10y ago
  Paraphrasing the famous quote: There are only two hard problems in CS: naming things and cache expiration. An issue I see with this is that there's a potential window when tests pass with bad data. For example, tests using a mock and the mock is periodically verified with a service call. The mock could become bad data, but won't be marked as bad until the next service call verification. Until that happens, the tests using the mock will all pass. It's not clear from the article how they address this.
  - dmoy 10y ago
    ... and off-by-one errors.
  - paulddraper 10y ago
    Perhaps Google tracks the changes to the dependencies, and reruns tests against a a real service when it changes. That's how I would do it, at least. If you have task-oriented build system with dependencies (like Google does with Blaze), it'd fit right in. Though depending on how you deploy, you may have old and new services at different versions. Assuming a level backwards/forwards compatibility may be reasonable.
  - jaawn 10y ago
    I imagine that if the service call to verify mock data fails, it should retroactively invalidate all tests based on that mock data, or at least this should be understood to be the case by the people reviewing test reports.
lifeisstillgood 10y ago
I have always thought this is how mocks really should work - basically a two way stub. Mocks are usually (inevitably?) a per-test structure which makes them outrageously fragile and tempting to over mock to get a passing test I prefer building something that acts like the mocked thing (ie a database) but does So both ends (it looks like a database to the web service and a web service to the database. Then the stub is my contract It's awkward
rifung 10y ago
> In the worst case, N tests will need N mocks and it will be the same as before. Well it's still better because you only have to run the services you are actually testing right?
- paulddraper 10y ago
  Yes, though presumably some extra services isn't much overhead. The transitive closure of services is.
zerd 10y ago
The important part is that you are able to temporally decouple running the tests against the service mock, and the tests running against the real service. This sounds very similar to Consumer Driven Contracts, such as https://docs.pact.io/ https://docs.pact.io/ You record your expectations towards the service mock, and then later verify that these expectations hold against the real service. This decoupling allows you to run your "end-to-end" test suite very often, say on every check in, and then only verify the contract say every hour or day, depending on how stable your test environments are. The theory is that this gives quicker feedback, reduce flakyness and avoid combinatorial explosion of number of tests needed to run.
lucideer 10y ago
> In the worst case, N tests will need N mocks and it will be the same as before. This very much neglects the probably significant overhead of writing, maintaining, understanding the M mocks on top of the base of N tests, but since, as you said, this is "at Google scale" the trade-off seemingly becomes worthwhile
Fifer82 10y ago
This should be at the end of every article. "basically...."

sly010 10y ago

In a large graph you would re-test the same part of the code over and over, because many top level inputs is going to do trigger the same code-path deep down in the graph, which is wasteful. In a sense they memoize service call answers, so any tested code-path only needs to run once. If they actually do this across their code they will never have to run "real service calls", because those services are also tested the same way. It's mocking all the way down.

bit_logic 10y ago
Paraphrasing the famous quote: There are only two hard problems in CS: naming things and cache expiration. An issue I see with this is that there's a potential window when tests pass with bad data. For example, tests using a mock and the mock is periodically verified with a service call. The mock could become bad data, but won't be marked as bad until the next service call verification. Until that happens, the tests using the mock will all pass. It's not clear from the article how they address this.
- dmoy 10y ago
  ... and off-by-one errors.
- paulddraper 10y ago
  Perhaps Google tracks the changes to the dependencies, and reruns tests against a a real service when it changes. That's how I would do it, at least. If you have task-oriented build system with dependencies (like Google does with Blaze), it'd fit right in. Though depending on how you deploy, you may have old and new services at different versions. Assuming a level backwards/forwards compatibility may be reasonable.
- jaawn 10y ago
  I imagine that if the service call to verify mock data fails, it should retroactively invalidate all tests based on that mock data, or at least this should be understood to be the case by the people reviewing test reports.

lifeisstillgood 10y ago

I have always thought this is how mocks really should work - basically a two way stub. Mocks are usually (inevitably?) a per-test structure which makes them outrageously fragile and tempting to over mock to get a passing test I prefer building something that acts like the mocked thing (ie a database) but does So both ends (it looks like a database to the web service and a web service to the database. Then the stub is my contract It's awkward

rifung 10y ago

> In the worst case, N tests will need N mocks and it will be the same as before. Well it's still better because you only have to run the services you are actually testing right?

paulddraper 10y ago
Yes, though presumably some extra services isn't much overhead. The transitive closure of services is.

zerd 10y ago

The important part is that you are able to temporally decouple running the tests against the service mock, and the tests running against the real service. This sounds very similar to Consumer Driven Contracts, such as https://docs.pact.io/ https://docs.pact.io/ You record your expectations towards the service mock, and then later verify that these expectations hold against the real service. This decoupling allows you to run your "end-to-end" test suite very often, say on every check in, and then only verify the contract say every hour or day, depending on how stable your test environments are. The theory is that this gives quicker feedback, reduce flakyness and avoid combinatorial explosion of number of tests needed to run.

lucideer 10y ago

> In the worst case, N tests will need N mocks and it will be the same as before. This very much neglects the probably significant overhead of writing, maintaining, understanding the M mocks on top of the base of N tests, but since, as you said, this is "at Google scale" the trade-off seemingly becomes worthwhile

Fifer82 10y ago

This should be at the end of every article. "basically...."