5 ms·

Agreed with that. In a world where you can't statically enforce that tests are deterministic, you absolutely must be diligent. However, it still happens. An

Agreed with that. In a world where you can't statically enforce that tests are deterministic, you absolutely must be diligent. However, it still happens. An example of "innocent intermittence" recently came up at my current employer: there was a test that invoked a web service twice, verifying idempotence. It then asserted the responses were identical. This worked almost all of the time, but when the two service invocations straddled a minute boundary, then some output text that included a rendered timestamp (rounded to the minute), which would cause the test to fail. It's extremely hard to get an engineering organization to just inherently know not to call, say, datetime.now() in their code, especially when the tests pass most of the time.

crdoconnor 11y ago
It sounds like your problem was with the test, not necessarily the developers calling datetime.now().
- implicit 11y ago
  The problem is that developers do not know that they mustn't call datetime.now() and there will always be a developer who doesn't know this, either because people have a tough time knowing everything, or just because you're hiring new people. Wouldn't it be nice if you could just obstruct such patterns within the first millisecond of their first interaction with the compiler?
  - crdoconnor 11y ago
    I don't have a problem with it. I send a mocked time to the application under test using libfaketime using this: http://hitchtest.com http://hitchtest.com Et voila, no more time issues.
  - peteretep 11y ago
    > developers do not know that they mustn't call > datetime.now() Plenty of languages where it's easy to mock the equivalent to that call.
peteretep 11y ago
> An example of "innocent intermittence" recently came up > at my current employer: there was a test that invoked a > web service twice, verifying idempotence. It then > asserted the responses were identical. This is not innocent intermittence, this is a bad test. No-one should get in trouble for it, but I would certainly expect senior developers to get involved to find out: Why is idempotence being tested for on the response side rather than on the execution end? If the call is truly idempotent, then a changing date-stamp means that the test has captured the information envelope as well as the data payload, and is testing that, which is sloppy - you'd probably be upset to see the HTTP headers being tested in the same string, so why other pieces of the envelope too? If a test is calling a "web service", then it's either an integration test (so why's it testing for idempotence, rather than testing the envelope?) or it's a unit test gone awry. Are there other tests that do this too? Could this all be abstracted out in to a sensible library to make it easier for more junior developers to not make these mistakes? The moment you start accepting less than 100% test pass rates as being OK, the moment madness descends and you lose much of the value of your test suite.

crdoconnor 11y ago

It sounds like your problem was with the test, not necessarily the developers calling datetime.now().

implicit 11y ago
The problem is that developers do not know that they mustn't call datetime.now() and there will always be a developer who doesn't know this, either because people have a tough time knowing everything, or just because you're hiring new people. Wouldn't it be nice if you could just obstruct such patterns within the first millisecond of their first interaction with the compiler?
- crdoconnor 11y ago
  I don't have a problem with it. I send a mocked time to the application under test using libfaketime using this: http://hitchtest.com http://hitchtest.com Et voila, no more time issues.
- peteretep 11y ago
  > developers do not know that they mustn't call > datetime.now() Plenty of languages where it's easy to mock the equivalent to that call.

peteretep 11y ago

> An example of "innocent intermittence" recently came up > at my current employer: there was a test that invoked a > web service twice, verifying idempotence. It then > asserted the responses were identical. This is not innocent intermittence, this is a bad test. No-one should get in trouble for it, but I would certainly expect senior developers to get involved to find out: Why is idempotence being tested for on the response side rather than on the execution end? If the call is truly idempotent, then a changing date-stamp means that the test has captured the information envelope as well as the data payload, and is testing that, which is sloppy - you'd probably be upset to see the HTTP headers being tested in the same string, so why other pieces of the envelope too? If a test is calling a "web service", then it's either an integration test (so why's it testing for idempotence, rather than testing the envelope?) or it's a unit test gone awry. Are there other tests that do this too? Could this all be abstracted out in to a sensible library to make it easier for more junior developers to not make these mistakes? The moment you start accepting less than 100% test pass rates as being OK, the moment madness descends and you lose much of the value of your test suite.