3 ms·

Maybe worth mentioning: this was a bug that existed for ~2 years before I started on the project. It's easy for this kinda subtlety to slip past review and debu

Maybe worth mentioning: this was a bug that existed for ~2 years before I started on the project. It's easy for this kinda subtlety to slip past review and debugging.

jerf 11y ago
I once worked for a failed startup, where the CEO insisted that we needed to write code to deal with billing people on arbitrary days of the month. I really wanted to do it on just the first of the month unconditionally, because I didn't think this was a good use of our time [1], but I was overridden. I had a Perl library that was much like the Ruby one, where it could add a month. Having by then caught the unit testing bug, I wrote my billing code, with just a couple lines as the payload, then wrote myself a unit test that simulated a customer joining us on any of the possible 731 days they could do so (365 days in a non-leap year, 366 in a leap year), then ran through two years of billing, validating that we billed them on a day as close as possible to the correct day (i.e., Feb 28th if you were on a 30 cycle), never missing a bill or double-billing. Sounds simple, right? In my nearly 20 years of coding, this is the single most frustrating bit of code I have ever tried to write and debug. Just absolutely unbelievable. Billing code that worked perfectly, except on months with 30 days joined in a leap year. Billing code that worked perfectly, except if you joined on a 31-day month that was followed by another 31-day month (which is July, but the bug was essentially caused by that problem). And no matter how I beat the code, no matter how I fixed one bug, some other blasted corner case popped out. Even now as I write this, I say to myself "surely this isn't that hard of a problem"... but it was. It really should have worked, but it didn't. Calendars are evil. And, what prompted me to reply, it would have been really easy for any version of this code to ship and not be noticed for a long, long time, if I hadn't written the exhaustive test cases I did. There was code that literally worked in 99.5% of the cases, and would only have broken a couple of times a year, and probably would have been written off as flukes or mysteries. Should you ever find yourself in a similar situation, I strongly recommend the exhaustive testing approach. You will be stunned what crap you find in your seemingly-simple code for handling dates. This also really solidified my view on the utility of automated/unit testing. You simply can not manually test a 731-element matrix, and I was getting an incredibly rich array of varying failures across that matrix, in billing code, which is very important code. [1]: This is many years ago, before there were so many great billing services off the shelf available, and we still had a significant challenge to be able to take payments at all. And lest I unhumbly make myself sound wiser than I was/am, I learned a lot about what code is and isn't worth writing at that startup... the hard way. Mercifully I don't feel my gross errors really contributed to the failure in the end... the startup died of one-more-feature-before-we-can-release-itis: https://news.ycombinator.com/item?id=9640335 https://news.ycombinator.com/item?id=9640335
- meesterdude 11y ago
  Thank you for this comment! that certainly sounds like a nightmare of edgecases. The first of the month certainly seems like the better way to go! This was sorta like that - specs would fail some days, but not others. The code worked, but the tests failed seemingly randomly.
jobigoud 11y ago
That's interesting because if I understand the bug correctly, it should have failed as soon as you performed the test less than 5 days before the end of the current month. (Except in December and July for which the next month has the same number of days in it.).
- meesterdude 11y ago
  if i recall, it was a bug that would come and go from our tests depending on when you ran them. Sometimes the math worked out, other times it was off. We knew the code worked, but couldn't figure out why the specs kept failing for it randomly.
  - bazzargh 11y ago
    eek. Time tests should be using something like https://github.com/travisjeffery/timecop https://github.com/travisjeffery/timecop for repeatability. If you do mean to allow random starting values like that, you should look into property-based testing http://www.troikatech.com/blog/2014/04/02/property-based-testing-in-ruby http://www.troikatech.com/blog/2014/04/02/property-based-tes... - test hundreds of possible values at a time, instead of just the one time you ran the tests at.

Maybe worth mentioning: this was a bug that existed for ~2 years before I started on the project. It's easy for this kinda subtlety to slip past review and debugging.

jerf 11y ago

I once worked for a failed startup, where the CEO insisted that we needed to write code to deal with billing people on arbitrary days of the month. I really wanted to do it on just the first of the month unconditionally, because I didn't think this was a good use of our time [1], but I was overridden. I had a Perl library that was much like the Ruby one, where it could add a month. Having by then caught the unit testing bug, I wrote my billing code, with just a couple lines as the payload, then wrote myself a unit test that simulated a customer joining us on any of the possible 731 days they could do so (365 days in a non-leap year, 366 in a leap year), then ran through two years of billing, validating that we billed them on a day as close as possible to the correct day (i.e., Feb 28th if you were on a 30 cycle), never missing a bill or double-billing. Sounds simple, right? In my nearly 20 years of coding, this is the single most frustrating bit of code I have ever tried to write and debug. Just absolutely unbelievable. Billing code that worked perfectly, except on months with 30 days joined in a leap year. Billing code that worked perfectly, except if you joined on a 31-day month that was followed by another 31-day month (which is July, but the bug was essentially caused by that problem). And no matter how I beat the code, no matter how I fixed one bug, some other blasted corner case popped out. Even now as I write this, I say to myself "surely this isn't that hard of a problem"... but it was. It really should have worked, but it didn't. Calendars are evil. And, what prompted me to reply, it would have been really easy for any version of this code to ship and not be noticed for a long, long time, if I hadn't written the exhaustive test cases I did. There was code that literally worked in 99.5% of the cases, and would only have broken a couple of times a year, and probably would have been written off as flukes or mysteries. Should you ever find yourself in a similar situation, I strongly recommend the exhaustive testing approach. You will be stunned what crap you find in your seemingly-simple code for handling dates. This also really solidified my view on the utility of automated/unit testing. You simply can not manually test a 731-element matrix, and I was getting an incredibly rich array of varying failures across that matrix, in billing code, which is very important code. [1]: This is many years ago, before there were so many great billing services off the shelf available, and we still had a significant challenge to be able to take payments at all. And lest I unhumbly make myself sound wiser than I was/am, I learned a lot about what code is and isn't worth writing at that startup... the hard way. Mercifully I don't feel my gross errors really contributed to the failure in the end... the startup died of one-more-feature-before-we-can-release-itis: https://news.ycombinator.com/item?id=9640335 https://news.ycombinator.com/item?id=9640335

meesterdude 11y ago
Thank you for this comment! that certainly sounds like a nightmare of edgecases. The first of the month certainly seems like the better way to go! This was sorta like that - specs would fail some days, but not others. The code worked, but the tests failed seemingly randomly.

jobigoud 11y ago

That's interesting because if I understand the bug correctly, it should have failed as soon as you performed the test less than 5 days before the end of the current month. (Except in December and July for which the next month has the same number of days in it.).

meesterdude 11y ago
if i recall, it was a bug that would come and go from our tests depending on when you ran them. Sometimes the math worked out, other times it was off. We knew the code worked, but couldn't figure out why the specs kept failing for it randomly.
- bazzargh 11y ago
  eek. Time tests should be using something like https://github.com/travisjeffery/timecop https://github.com/travisjeffery/timecop for repeatability. If you do mean to allow random starting values like that, you should look into property-based testing http://www.troikatech.com/blog/2014/04/02/property-based-testing-in-ruby http://www.troikatech.com/blog/2014/04/02/property-based-tes... - test hundreds of possible values at a time, instead of just the one time you ran the tests at.