3 ms·

Not at Google, but making reliable tests was one thing I really struggled with professionally. There are many, many possible causes of non-determinism. Even som

Not at Google, but making reliable tests was one thing I really struggled with professionally. There are many, many possible causes of non-determinism. Even something as simple as using multiple threads means that you suddenly realize that you must decide how long you're willing to wait for the other thread to complete. (You always kind of did, but when all the logic was on one thread, the waiting was not as obvious and errors were less likely to cause hangups.) Unfortunately, under high load, I found that threads can be starved of CPU for seconds at a time. The CI machine was flooring it trying to build the software and run the tests. Setting the timeout high enough that there were no random failures caused by CPU starvation meant that build failures would take a very long time, delaying feedback for errors. That was nearly as bad as the flakiness. I mostly fixed the flakiness by making the number of threads configurable and only running one thread in the testing environment, but I think a better way would be to reduce the number of parallel jobs to keep the test machines' CPU utilization rate at something more reasonable. Then, you can set timeouts that reflect real performance expectations for your software.

sebcat 8y ago
When doing anything concurrently and/or in parallel, it is always good to have an input queue and an upper bound on the number of tasks (/threads/processes/...). Also an asynchronous flow. You need to handle other problems (back-pressure, priorities, timeouts, ...) but if you need it you need it.
- zenexer 8y ago
  But how do you go about creating tests for all of that?
  - cyphar 8y ago
    You don't need to write your own and test it -- use something like GNU parallel (if your tests can be run individually as commands or shell scripts) or a test framework that supports parallelism.
sorokod 8y ago
In such cases it is better to await on some outcome of the asynchronous calculation. In Java ecosystem we have awaitility (https://github.com/awaitility/awaitility https://github.com/awaitility/awaitility) which provides a bunch of utility methods in this spirit.
- slavik81 8y ago
  That's how our tests worked, but you still must choose a timeout or you will wait forever when you do not receive the response you're listening for. Setting up the CI system to report failure on the first error would have helped a lot too. Waiting on a single timeout was not too bad. The problem was waiting on hundreds of them.

sebcat 8y ago

When doing anything concurrently and/or in parallel, it is always good to have an input queue and an upper bound on the number of tasks (/threads/processes/...). Also an asynchronous flow. You need to handle other problems (back-pressure, priorities, timeouts, ...) but if you need it you need it.

zenexer 8y ago
But how do you go about creating tests for all of that?
- cyphar 8y ago
  You don't need to write your own and test it -- use something like GNU parallel (if your tests can be run individually as commands or shell scripts) or a test framework that supports parallelism.

sorokod 8y ago

In such cases it is better to await on some outcome of the asynchronous calculation. In Java ecosystem we have awaitility (https://github.com/awaitility/awaitility https://github.com/awaitility/awaitility) which provides a bunch of utility methods in this spirit.

slavik81 8y ago
That's how our tests worked, but you still must choose a timeout or you will wait forever when you do not receive the response you're listening for. Setting up the CI system to report failure on the first error would have helped a lot too. Waiting on a single timeout was not too bad. The problem was waiting on hundreds of them.