3 ms·

I assume the parent commenter meant unit tests must take < 100ms. Unit tests should be mocking databases in-memory. So latency to databases in unit tests should

I assume the parent commenter meant unit tests must take < 100ms. Unit tests should be mocking databases in-memory. So latency to databases in unit tests should be less than a microsecond. It's okay (and expected) if integration and end-to-end tests take more than 100ms.

patrec 6y ago
My point isn't that it's OK to take more than 100ms for and end to end test. My point is there is no reason why you shouldn't be able to reliably run tests that hit databases, multiple times, within a 100ms deadline and without adding flakiness. It's 2020, computers are fast. You can do tens of thousands of database transactions in a second. Starting up postgres takes a fraction of a second and consumes negligible resources. If you treat your DB files as a build artifact that includes schema migration.
- tylerhou 6y ago
  Spinning up a local instance of postgres defeats the whole purpose of an end-to-end test. The purpose of an end-to-end test is to test the whole system end-to-end; this includes network, other servers, etc. Each of these can break, which is why e2e tests are inherently more flaky than unit tests. And network latency and transfer speeds mean that taking more than 100ms may happen often. Another example of an e2e test that takes a long time are screenshot diffs. Spinning up Selenium and rendering a webpage might take more than 100ms in rare instances. iOS/Android tests are in the same vein where reloading an app takes > 100ms much of the time.
  - patrec 6y ago
    > Spinning up a local instance of postgres defeats the whole purpose of an end-to-end test. So run some end to end test in a fully featured staging-style environment as well? I'm not disagreeing at all that having some environment that closely resembles production for very late phases as well is nice to have. I'm arguing against nonsense like mocking out database calls because of weird superstitions. > And network latency and transfer speeds mean that taking more than 100ms may happen often. The operative word being "may". If your staging server in Timbutku needs to talk to your other staging server on the South Pole you will blow your 100ms window all the time. If it all runs within a single good DC, network latency ought to be ~0.1ms roundtrip and you can shovel around Gigabytes per second.

patrec 6y ago

My point isn't that it's OK to take more than 100ms for and end to end test. My point is there is no reason why you shouldn't be able to reliably run tests that hit databases, multiple times, within a 100ms deadline and without adding flakiness. It's 2020, computers are fast. You can do tens of thousands of database transactions in a second. Starting up postgres takes a fraction of a second and consumes negligible resources. If you treat your DB files as a build artifact that includes schema migration.

tylerhou 6y ago
Spinning up a local instance of postgres defeats the whole purpose of an end-to-end test. The purpose of an end-to-end test is to test the whole system end-to-end; this includes network, other servers, etc. Each of these can break, which is why e2e tests are inherently more flaky than unit tests. And network latency and transfer speeds mean that taking more than 100ms may happen often. Another example of an e2e test that takes a long time are screenshot diffs. Spinning up Selenium and rendering a webpage might take more than 100ms in rare instances. iOS/Android tests are in the same vein where reloading an app takes > 100ms much of the time.
- patrec 6y ago
  > Spinning up a local instance of postgres defeats the whole purpose of an end-to-end test. So run some end to end test in a fully featured staging-style environment as well? I'm not disagreeing at all that having some environment that closely resembles production for very late phases as well is nice to have. I'm arguing against nonsense like mocking out database calls because of weird superstitions. > And network latency and transfer speeds mean that taking more than 100ms may happen often. The operative word being "may". If your staging server in Timbutku needs to talk to your other staging server on the South Pole you will blow your 100ms window all the time. If it all runs within a single good DC, network latency ought to be ~0.1ms roundtrip and you can shovel around Gigabytes per second.