4 ms·

There's no proof it was a leap year bug. If they were truly sending an out of range value, the response code would probably be a 500, not a 503.

by avree 7y ago

deleted 7y ago
[deleted]
btown 7y ago
But it could also be the case that a service they use times out if you ask for invalid data e.g. from the future. In which case that could cause a cascade of backlogged tasks that would eventually lead to 503 errors under load.
falcolas 7y ago
One thing to consider is that this exact bug came up 4 years ago on March 2nd. Correlation is not causation, but it is a big old spotlight that says "Hey, have a look over here!"
ksaj 7y ago
The numbers can be totally wrong without being out of range. How many days was it calculating until March 1st, and how was it handling the next day which should have been +1, which surprisingly (to the code) wasn't March 1st? How many business hours passed from sun up Feb 28 until sun down March 1? Which day was the following Monday? No over- or underflows needed here, but clearly coding without taking these into account will provide something other than the correct answer. As a guy who works in infosec since the early 90's, I see these errors showing up a lot in different, almost (anti)ingenious ways. This Robinhood event is just one for the collection, but not one that surprises me or anyone else in the field given enough time or enough education. Fun fact: Y2K was a leap year. And we survived both events back-to-back better than Robinhood survived the obvious singular event in 2020. That is the problem here. Not Leap Year. But the fact that some programmers refuse to learn from history. At least during the Y2K debacle, everyone understood what they needed to do to code for leap years. I used to teach the Easter Problem in a certain big blue coding classroom: From scratch, calculate the Easter dates for any given AD year. It gets messy quickly and it is not trivial. You have to think about Leap Years and Y2K's, which is why I was teaching it to rooms full of technical folk that hopefully remember it. Of course given enough classroom discussion, everyone got to some sort of code that predicts Easter for any given year, but not after quite a few failures. Then we talked about leap seconds, although those were a lot less painful. By that point, they totally understood how barely stable our calendar cycles are. It proved a good lesson that I thought was rather important once they already understood the counting-fenceposts issue, which is yet another failure to understand cycles in exactly the same way. Needless to say, nobody at Robinhood was in my classes as far as I can tell. It's not like we don't have these "changes everything" roll overs fairly often. Anyway, my job was to react if the bank I was working for ended up with crashing systems due to viruses, worms and whatever other malware they might have that weren't Y2K compatible. So this subject is near and dear to me. I'm literally blown away that Robin Hood and other companies could still have this problem today. The last leap year was, gee, how many years ago? They didn't see that coming?

There's no proof it was a leap year bug. If they were truly sending an out of range value, the response code would probably be a 500, not a 503.

by avree 7y ago

There's no proof it was a leap year bug. If they were truly sending an out of range value, the response code would probably be a 500, not a 503.

deleted 7y ago

[deleted]

btown 7y ago

But it could also be the case that a service they use times out if you ask for invalid data e.g. from the future. In which case that could cause a cascade of backlogged tasks that would eventually lead to 503 errors under load.

falcolas 7y ago

One thing to consider is that this exact bug came up 4 years ago on March 2nd. Correlation is not causation, but it is a big old spotlight that says "Hey, have a look over here!"

ksaj 7y ago

The numbers can be totally wrong without being out of range. How many days was it calculating until March 1st, and how was it handling the next day which should have been +1, which surprisingly (to the code) wasn't March 1st? How many business hours passed from sun up Feb 28 until sun down March 1? Which day was the following Monday? No over- or underflows needed here, but clearly coding without taking these into account will provide something other than the correct answer. As a guy who works in infosec since the early 90's, I see these errors showing up a lot in different, almost (anti)ingenious ways. This Robinhood event is just one for the collection, but not one that surprises me or anyone else in the field given enough time or enough education. Fun fact: Y2K was a leap year. And we survived both events back-to-back better than Robinhood survived the obvious singular event in 2020. That is the problem here. Not Leap Year. But the fact that some programmers refuse to learn from history. At least during the Y2K debacle, everyone understood what they needed to do to code for leap years. I used to teach the Easter Problem in a certain big blue coding classroom: From scratch, calculate the Easter dates for any given AD year. It gets messy quickly and it is not trivial. You have to think about Leap Years and Y2K's, which is why I was teaching it to rooms full of technical folk that hopefully remember it. Of course given enough classroom discussion, everyone got to some sort of code that predicts Easter for any given year, but not after quite a few failures. Then we talked about leap seconds, although those were a lot less painful. By that point, they totally understood how barely stable our calendar cycles are. It proved a good lesson that I thought was rather important once they already understood the counting-fenceposts issue, which is yet another failure to understand cycles in exactly the same way. Needless to say, nobody at Robinhood was in my classes as far as I can tell. It's not like we don't have these "changes everything" roll overs fairly often. Anyway, my job was to react if the bank I was working for ended up with crashing systems due to viruses, worms and whatever other malware they might have that weren't Y2K compatible. So this subject is near and dear to me. I'm literally blown away that Robin Hood and other companies could still have this problem today. The last leap year was, gee, how many years ago? They didn't see that coming?