3 ms·
> maybe not in the first run, but after a week or two in continuous integration You'd use a different seed for each CI run?? That sounds like a nightmare of n
by paulddraper 3y ago
> maybe not in the first run, but after a week or two in continuous integration
You'd use a different seed for each CI run??
That sounds like a nightmare of non-determinism, and lost of trust in CI system in general.
- forty 3y agoNot if you log the seed of the failing runs
- hedora 3y agoYep; I definitely log the seed and re-seed every few seconds. Most software I work on these days is non-deterministic anyway (it involves the network, etc.), so CI is fundamentally going to fail some runs and not others. Even stuff like deterministic simulation has this property: Those test suites rely on having a large number of randomly generated schedules, so there's always a chance that running the test one more time (with a new seed) will find a new bug.
- jhardy54 3y agoIf you’re using Git, I’d strongly recommend the hash of the current tree (not the commit). That way your tests are deterministic based on the contents of your tree. For example, if you add a commit and then revert, you’ll end up with the same test seed as if you hadn’t committed.
- hedora 3y agoThe important thing to log is the seed passed to the rng. (It’s usually wallclock time measured at nanosecond granularity.) In a typical night for a production quality system, 100-1000+ hours of such tests would run, all with different seeds, and in diverse configurations, etc, so the seed isn’t derived from the git sha, or the source code.
- paulddraper 3y agoOr someone just punches retry
- hedora 3y agoIgnoring test failures is a different issue. It can range from known bugs and rational tradeoffs to a hiring/hr issue. Establishing a healthy dev/engineering culture is hard.
- paulddraper 3y agoYou don't have auto retries anywhere in your tests?