3 ms·

We once had a bug that would cause a failure in our test suite, but very rarely. It was so rare that it would just "go away" and not show up for weeks. So we th

by atonse 9y ago

We once had a bug that would cause a failure in our test suite, but very rarely. It was so rare that it would just "go away" and not show up for weeks. So we thought it was odd (maybe bad test data, etc) but moved on. When we decided to look into it when we had some downtime, we found out that it was a date bug that only reared its head on the 31st day of the month, hence only happening every two months. I'm not sure bisect would've helped but still was really funny when we did find the issue.

michaelcampbell 9y ago
Was this the kind of test that only ran once a day?
- atonse 9y ago
  It wasn't, but it was the sort of thing where we never modified that test or even a related area, so it just wouldn't make sense that the test would fail. But the bigger issue was that since it would "fix itself" we never prioritized it and just left it alone. But looking back, the system (automated tests) was functioning as designed. :) We just changed our "DateTime.now" to concrete dates.
adrianratnapala 9y ago
Thanks for the anecdote, because it gives me a new idea. We should be monitoring the rate of test failures -- per test -- in a timeseries and looking for periodicities. Just graphing them and seeing regular spikes will help, but having real numerics looking for approximate periodicities will catch this sort of thing (after a few months!).
- bArray 9y ago
  Unless you are writing date specific code or have seen this bug arise, I probably wouldn't bother. Time bugs will likely creep in regardless, for example how would you find a bug that happens one month a year? How would you find a bug once every four years? Sometimes human knowledge/experience can't be tested out of a system. All test failures probably warrant investigation of some type. But, as a general rule, I would try to brute force or fuzz your "inputs", a.k.a data out of your control. Random numbers, dates, user input, network packets, etc. Users and environments often act in unexpected ways and you want to make sure your bug finding isn't reduced to accidentally be working on the right day! :)
scorcher 9y ago
Thats where property based tests come in very useful
brainfire 9y ago
Related bug- OpenOffice can't print on Tuesdays: https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161/comments/28 https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161... Previous HN thread: https://news.ycombinator.com/item?id=8171956 https://news.ycombinator.com/item?id=8171956

by atonse 9y ago

michaelcampbell 9y ago

Was this the kind of test that only ran once a day?

atonse 9y ago
It wasn't, but it was the sort of thing where we never modified that test or even a related area, so it just wouldn't make sense that the test would fail. But the bigger issue was that since it would "fix itself" we never prioritized it and just left it alone. But looking back, the system (automated tests) was functioning as designed. :) We just changed our "DateTime.now" to concrete dates.

adrianratnapala 9y ago

Thanks for the anecdote, because it gives me a new idea. We should be monitoring the rate of test failures -- per test -- in a timeseries and looking for periodicities. Just graphing them and seeing regular spikes will help, but having real numerics looking for approximate periodicities will catch this sort of thing (after a few months!).

bArray 9y ago
Unless you are writing date specific code or have seen this bug arise, I probably wouldn't bother. Time bugs will likely creep in regardless, for example how would you find a bug that happens one month a year? How would you find a bug once every four years? Sometimes human knowledge/experience can't be tested out of a system. All test failures probably warrant investigation of some type. But, as a general rule, I would try to brute force or fuzz your "inputs", a.k.a data out of your control. Random numbers, dates, user input, network packets, etc. Users and environments often act in unexpected ways and you want to make sure your bug finding isn't reduced to accidentally be working on the right day! :)

scorcher 9y ago

Thats where property based tests come in very useful

brainfire 9y ago

Related bug- OpenOffice can't print on Tuesdays: https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161/comments/28 https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161... Previous HN thread: https://news.ycombinator.com/item?id=8171956 https://news.ycombinator.com/item?id=8171956