6 ms·

We just implemented a rule where tests taking > 100ms to execute are considered a failure. You can experiment with the parameters or limits but this sort of rul

We just implemented a rule where tests taking > 100ms to execute are considered a failure. You can experiment with the parameters or limits but this sort of rule has paid huge dividends. The majority of the tests it has disqualified sucked and didn't really "test" anything. They also tended to be the most brittle. You don't write tests that hit APIs and databases when you have strict time limits and this eliminates a lot of "flaky" tests.

patrec 6y ago
If you can't hit your database in CI within 100ms or talking to it makes your tests flaky, you got problems.
- nogabebop23 6y ago
  Tests that hit dynamic data stores like DBs don't fail very often due to network speed constraints compared to configuration, system updates, security, schema changes, and a myriad of other time-consuming hard to diagnose problems. It's easier to just prevent these situations. In the abstract I'm not sure what a test that talks to the database is actually testing.
  - patrec 6y ago
    How do you effectively develop if you can't reliably talk to the DB in dev due to configuration, system updates, security, schema changes, and a myriad of other time-consuming hard to diagnose problems? > In the abstract I'm not sure what a test that talks to the database is actually testing. Well, presumably the reason that you have a DB in the first place is that your application requires a DB for some of its functionality. So how do you make sure this functionality works if you don't talk to the DB? Don't get me wrong -- I love that you refuse to accept ridiculously slow tests and enforce a 100ms deadline; I wish everything I was working on had the same approach. But unless you have some specific requirements (like talking to a proprietary cloud DB), you should maybe investigate if you might not have to accept slow and flaky DBs in dev or CI either. My experience is you don't.
- tylerhou 6y ago
  I assume the parent commenter meant unit tests must take < 100ms. Unit tests should be mocking databases in-memory. So latency to databases in unit tests should be less than a microsecond. It's okay (and expected) if integration and end-to-end tests take more than 100ms.
  - patrec 6y ago
    My point isn't that it's OK to take more than 100ms for and end to end test. My point is there is no reason why you shouldn't be able to reliably run tests that hit databases, multiple times, within a 100ms deadline and without adding flakiness. It's 2020, computers are fast. You can do tens of thousands of database transactions in a second. Starting up postgres takes a fraction of a second and consumes negligible resources. If you treat your DB files as a build artifact that includes schema migration.
    - tylerhou 6y ago
      Spinning up a local instance of postgres defeats the whole purpose of an end-to-end test. The purpose of an end-to-end test is to test the whole system end-to-end; this includes network, other servers, etc. Each of these can break, which is why e2e tests are inherently more flaky than unit tests. And network latency and transfer speeds mean that taking more than 100ms may happen often. Another example of an e2e test that takes a long time are screenshot diffs. Spinning up Selenium and rendering a webpage might take more than 100ms in rare instances. iOS/Android tests are in the same vein where reloading an app takes > 100ms much of the time.
      - patrec 6y ago
        > Spinning up a local instance of postgres defeats the whole purpose of an end-to-end test. So run some end to end test in a fully featured staging-style environment as well? I'm not disagreeing at all that having some environment that closely resembles production for very late phases as well is nice to have. I'm arguing against nonsense like mocking out database calls because of weird superstitions. > And network latency and transfer speeds mean that taking more than 100ms may happen often. The operative word being "may". If your staging server in Timbutku needs to talk to your other staging server on the South Pole you will blow your 100ms window all the time. If it all runs within a single good DC, network latency ought to be ~0.1ms roundtrip and you can shovel around Gigabytes per second.
tylerhou 6y ago
Maybe an area you want to carve out an exception for are benchmarking tests; i.e. tests that ensure that a function runs faster/slower than a certain speed. In this case, you probably need more than 100ms of testing for accurate results. For example, password checking functions should be slow, and extremely hot/performance sensitive functions should be fast. Of course, you should have fairly wide margins to avoid flakes.
nhoughto 6y ago
obviously the context matters, but putting time limits on some tests can really cause headaches. We've had problems where builds failed when many jobs ran together because build runner contention meant that jobs took just a bit longer than normal.. and of course the re-run would work because contention is lower.. the function still behaves as expected just a bit slower, is that worth failing a test and hurting productivity? The obvious exception is benchmark performance critical code, but run of the mill functional tests should maybe have an unreasonably high time limit bc it hurts less?
dharmab 6y ago
Are there exceptions? Our codebase has some functions related to generating secure cryptographic keys and certificate chains, and that test suit takes many seconds to run since it actually stresses the CPU.
davewritescode 6y ago
Testing budgets are important by disqualifying a test because it's "slow" is really silly. 1. Tests that hit APIs and databases are important because when you mock out the database and the API you're writing a test that asserts your code integrates correctly with what you think the database or API should be doing. 2. Some systems are highly asynchronous and thus very difficult to test individual cases in under 100 ms. If I have a suite of 100 tests that completes in 10 seconds because everything runs concurrently, that's a lot more important than limiting and individual test to some arbitrary timespan. Again, make a budget for tests and meet it instead of making hard rules about how fast or slow a given test should be.

patrec 6y ago

If you can't hit your database in CI within 100ms or talking to it makes your tests flaky, you got problems.

nogabebop23 6y ago
Tests that hit dynamic data stores like DBs don't fail very often due to network speed constraints compared to configuration, system updates, security, schema changes, and a myriad of other time-consuming hard to diagnose problems. It's easier to just prevent these situations. In the abstract I'm not sure what a test that talks to the database is actually testing.
- patrec 6y ago
  How do you effectively develop if you can't reliably talk to the DB in dev due to configuration, system updates, security, schema changes, and a myriad of other time-consuming hard to diagnose problems? > In the abstract I'm not sure what a test that talks to the database is actually testing. Well, presumably the reason that you have a DB in the first place is that your application requires a DB for some of its functionality. So how do you make sure this functionality works if you don't talk to the DB? Don't get me wrong -- I love that you refuse to accept ridiculously slow tests and enforce a 100ms deadline; I wish everything I was working on had the same approach. But unless you have some specific requirements (like talking to a proprietary cloud DB), you should maybe investigate if you might not have to accept slow and flaky DBs in dev or CI either. My experience is you don't.
tylerhou 6y ago
I assume the parent commenter meant unit tests must take < 100ms. Unit tests should be mocking databases in-memory. So latency to databases in unit tests should be less than a microsecond. It's okay (and expected) if integration and end-to-end tests take more than 100ms.
- patrec 6y ago
  My point isn't that it's OK to take more than 100ms for and end to end test. My point is there is no reason why you shouldn't be able to reliably run tests that hit databases, multiple times, within a 100ms deadline and without adding flakiness. It's 2020, computers are fast. You can do tens of thousands of database transactions in a second. Starting up postgres takes a fraction of a second and consumes negligible resources. If you treat your DB files as a build artifact that includes schema migration.
  - tylerhou 6y ago
    Spinning up a local instance of postgres defeats the whole purpose of an end-to-end test. The purpose of an end-to-end test is to test the whole system end-to-end; this includes network, other servers, etc. Each of these can break, which is why e2e tests are inherently more flaky than unit tests. And network latency and transfer speeds mean that taking more than 100ms may happen often. Another example of an e2e test that takes a long time are screenshot diffs. Spinning up Selenium and rendering a webpage might take more than 100ms in rare instances. iOS/Android tests are in the same vein where reloading an app takes > 100ms much of the time.
    - patrec 6y ago
      > Spinning up a local instance of postgres defeats the whole purpose of an end-to-end test. So run some end to end test in a fully featured staging-style environment as well? I'm not disagreeing at all that having some environment that closely resembles production for very late phases as well is nice to have. I'm arguing against nonsense like mocking out database calls because of weird superstitions. > And network latency and transfer speeds mean that taking more than 100ms may happen often. The operative word being "may". If your staging server in Timbutku needs to talk to your other staging server on the South Pole you will blow your 100ms window all the time. If it all runs within a single good DC, network latency ought to be ~0.1ms roundtrip and you can shovel around Gigabytes per second.

tylerhou 6y ago

Maybe an area you want to carve out an exception for are benchmarking tests; i.e. tests that ensure that a function runs faster/slower than a certain speed. In this case, you probably need more than 100ms of testing for accurate results. For example, password checking functions should be slow, and extremely hot/performance sensitive functions should be fast. Of course, you should have fairly wide margins to avoid flakes.

nhoughto 6y ago

obviously the context matters, but putting time limits on some tests can really cause headaches. We've had problems where builds failed when many jobs ran together because build runner contention meant that jobs took just a bit longer than normal.. and of course the re-run would work because contention is lower.. the function still behaves as expected just a bit slower, is that worth failing a test and hurting productivity? The obvious exception is benchmark performance critical code, but run of the mill functional tests should maybe have an unreasonably high time limit bc it hurts less?

dharmab 6y ago

Are there exceptions? Our codebase has some functions related to generating secure cryptographic keys and certificate chains, and that test suit takes many seconds to run since it actually stresses the CPU.

davewritescode 6y ago

Testing budgets are important by disqualifying a test because it's "slow" is really silly. 1. Tests that hit APIs and databases are important because when you mock out the database and the API you're writing a test that asserts your code integrates correctly with what you think the database or API should be doing. 2. Some systems are highly asynchronous and thus very difficult to test individual cases in under 100 ms. If I have a suite of 100 tests that completes in 10 seconds because everything runs concurrently, that's a lot more important than limiting and individual test to some arbitrary timespan. Again, make a budget for tests and meet it instead of making hard rules about how fast or slow a given test should be.