4 ms·

If you can't hit your database in CI within 100ms or talking to it makes your tests flaky, you got problems.

• nogabebop23 6y ago
  Tests that hit dynamic data stores like DBs don't fail very often due to network speed constraints compared to configuration, system updates, security, schema changes, and a myriad of other time-consuming hard to diagnose problems. It's easier to just prevent these situations. In the abstract I'm not sure what a test that talks to the database is actually testing.
  • patrec 6y ago
    How do you effectively develop if you can't reliably talk to the DB in dev due to configuration, system updates, security, schema changes, and a myriad of other time-consuming hard to diagnose problems? > In the abstract I'm not sure what a test that talks to the database is actually testing. Well, presumably the reason that you have a DB in the first place is that your application requires a DB for some of its functionality. So how do you make sure this functionality works if you don't talk to the DB? Don't get me wrong -- I love that you refuse to accept ridiculously slow tests and enforce a 100ms deadline; I wish everything I was working on had the same approach. But unless you have some specific requirements (like talking to a proprietary cloud DB), you should maybe investigate if you might not have to accept slow and flaky DBs in dev or CI either. My experience is you don't.
• tylerhou 6y ago
  I assume the parent commenter meant unit tests must take < 100ms. Unit tests should be mocking databases in-memory. So latency to databases in unit tests should be less than a microsecond. It's okay (and expected) if integration and end-to-end tests take more than 100ms.
  • patrec 6y ago
    My point isn't that it's OK to take more than 100ms for and end to end test. My point is there is no reason why you shouldn't be able to reliably run tests that hit databases, multiple times, within a 100ms deadline and without adding flakiness. It's 2020, computers are fast. You can do tens of thousands of database transactions in a second. Starting up postgres takes a fraction of a second and consumes negligible resources. If you treat your DB files as a build artifact that includes schema migration.
    • tylerhou 6y ago
      Spinning up a local instance of postgres defeats the whole purpose of an end-to-end test. The purpose of an end-to-end test is to test the whole system end-to-end; this includes network, other servers, etc. Each of these can break, which is why e2e tests are inherently more flaky than unit tests. And network latency and transfer speeds mean that taking more than 100ms may happen often. Another example of an e2e test that takes a long time are screenshot diffs. Spinning up Selenium and rendering a webpage might take more than 100ms in rare instances. iOS/Android tests are in the same vein where reloading an app takes > 100ms much of the time.
      • patrec 6y ago
        > Spinning up a local instance of postgres defeats the whole purpose of an end-to-end test. So run some end to end test in a fully featured staging-style environment as well? I'm not disagreeing at all that having some environment that closely resembles production for very late phases as well is nice to have. I'm arguing against nonsense like mocking out database calls because of weird superstitions. > And network latency and transfer speeds mean that taking more than 100ms may happen often. The operative word being "may". If your staging server in Timbutku needs to talk to your other staging server on the South Pole you will blow your 100ms window all the time. If it all runs within a single good DC, network latency ought to be ~0.1ms roundtrip and you can shovel around Gigabytes per second.