3 ms·

There was a discussion last night about this. Some of the more technically competent posters dismissed it because it's unbelievable that a financial platform wo

There was a discussion last night about this. Some of the more technically competent posters dismissed it because it's unbelievable that a financial platform would roll their own date-time implementation.

rriepe 7y ago
While I agree in spirit (Occam tells me it was just unprecedented load), that explanation doesn't make any sense to me in terms of disproving the Leap Year theory. The mistake wouldn't even have to be in their code. As for believing/unbelieving, I mean, I've seen dumber mistakes in financial products I've worked on. Not as high stakes or damaging, but definitely dumber.
notyourday 7y ago
That means people should reevaluate the weight of the opinions of those technically competent posters - we have a screen shot that robinhood did roll their own implementation because yesterday was March 2nd and its app was requesting March 3rd. That at least means that some portion of their stack used roll your own datetime library. That would not actually be a problem as long as the entire stack got the same library or the same rules for datetime. The problem is of course that it probably did not so the libraries were not bug compatible which of course caused errors and those errors need to be handled, preferably every fast. Error load and error handling is the least tested part of every system because it can only be properly tested in production and, unless your company embraces the Chaos Monkey approach to testing, the C-level would have a heart attack when anyone proposes doing it.
- vsareto 7y ago
  That particular API endpoint is just returning market open times, in which a request for tomorrow might be perfectly reasonable. https://api.robinhood.com/markets/XASE/hours/2020-03-03/ https://api.robinhood.com/markets/XASE/hours/2020-03-03/ vs https://api.robinhood.com/markets/XASE/hours/2020-03-07/ https://api.robinhood.com/markets/XASE/hours/2020-03-07/ This whole discussion lacks context to the nature of the requests, aka a front end code review.
  - notyourday 7y ago
    The issue is going to end up being related to some sort of date being injected by the front-end and propagating that incorrect date into the infrastructure. Somewhere within the infrastructure there's going to be an assertion such as if (max_drift_delta > delta(order_live_date,system_live_date) { # oh crap, something is completely broken in our system where did this come from? blow_up("terrible things are happening! how did we get this order") } which is an excellent and correct catcher for "terrible things are happening" since those things should never happen. That blow_up() code path is likely to be very expensive which kills performance of the system, which in turn means that it no longer can handle the load. And since RH has lots of people who use apps, it is not that they can just push an immediate bugfix.
  - notyourday 7y ago
    This definitely does not come from a front-end: https://twitter.com/jhyu/status/1234617361467990018 https://twitter.com/jhyu/status/1234617361467990018
hermitdev 7y ago
I work in finance, and at a previous employer (not Robinhood), I partially rolled a datetime implementation. Mostly it was a wrapper around Boost Date Time [0]. It was a facade that smoothed out the interface and implemented some missing functionally, like a cross-platform strptime and loading of the Olson timezone database. I spent a better part of a year working on it (along with other things). Modeled the interface after Python's datetime module (which I think is one of the simpler and easy to use date-time libraries across various languages I've used). More than $20B USD trades using that library. The motivation was the firm used to use RogueWave's date-time facilities, but we moved away after they jacked up the licensing. Think we used to have a site-wide license, but they were moving to a per-core licensing, wanting something like $2K per core annually. Needless to say, testing was extensive. Overflows were found in Boost in far distant dates, had to work around those. Tested against a ton of historical dates. Tested against Northern and Southern Hemisphere daylight saving time (a lot of people don't realize that Southern is inverted from Northern). Learned a lot about timezones and the history of timezones along the way. [0] https://www.boost.org/doc/libs/1_72_0/doc/html/date_time.html https://www.boost.org/doc/libs/1_72_0/doc/html/date_time.htm...

rriepe 7y ago

While I agree in spirit (Occam tells me it was just unprecedented load), that explanation doesn't make any sense to me in terms of disproving the Leap Year theory. The mistake wouldn't even have to be in their code. As for believing/unbelieving, I mean, I've seen dumber mistakes in financial products I've worked on. Not as high stakes or damaging, but definitely dumber.

notyourday 7y ago

That means people should reevaluate the weight of the opinions of those technically competent posters - we have a screen shot that robinhood did roll their own implementation because yesterday was March 2nd and its app was requesting March 3rd. That at least means that some portion of their stack used roll your own datetime library. That would not actually be a problem as long as the entire stack got the same library or the same rules for datetime. The problem is of course that it probably did not so the libraries were not bug compatible which of course caused errors and those errors need to be handled, preferably every fast. Error load and error handling is the least tested part of every system because it can only be properly tested in production and, unless your company embraces the Chaos Monkey approach to testing, the C-level would have a heart attack when anyone proposes doing it.

vsareto 7y ago
That particular API endpoint is just returning market open times, in which a request for tomorrow might be perfectly reasonable. https://api.robinhood.com/markets/XASE/hours/2020-03-03/ https://api.robinhood.com/markets/XASE/hours/2020-03-03/ vs https://api.robinhood.com/markets/XASE/hours/2020-03-07/ https://api.robinhood.com/markets/XASE/hours/2020-03-07/ This whole discussion lacks context to the nature of the requests, aka a front end code review.
- notyourday 7y ago
  The issue is going to end up being related to some sort of date being injected by the front-end and propagating that incorrect date into the infrastructure. Somewhere within the infrastructure there's going to be an assertion such as if (max_drift_delta > delta(order_live_date,system_live_date) { # oh crap, something is completely broken in our system where did this come from? blow_up("terrible things are happening! how did we get this order") } which is an excellent and correct catcher for "terrible things are happening" since those things should never happen. That blow_up() code path is likely to be very expensive which kills performance of the system, which in turn means that it no longer can handle the load. And since RH has lots of people who use apps, it is not that they can just push an immediate bugfix.
- notyourday 7y ago
  This definitely does not come from a front-end: https://twitter.com/jhyu/status/1234617361467990018 https://twitter.com/jhyu/status/1234617361467990018

hermitdev 7y ago

I work in finance, and at a previous employer (not Robinhood), I partially rolled a datetime implementation. Mostly it was a wrapper around Boost Date Time [0]. It was a facade that smoothed out the interface and implemented some missing functionally, like a cross-platform strptime and loading of the Olson timezone database. I spent a better part of a year working on it (along with other things). Modeled the interface after Python's datetime module (which I think is one of the simpler and easy to use date-time libraries across various languages I've used). More than $20B USD trades using that library. The motivation was the firm used to use RogueWave's date-time facilities, but we moved away after they jacked up the licensing. Think we used to have a site-wide license, but they were moving to a per-core licensing, wanting something like $2K per core annually. Needless to say, testing was extensive. Overflows were found in Boost in far distant dates, had to work around those. Tested against a ton of historical dates. Tested against Northern and Southern Hemisphere daylight saving time (a lot of people don't realize that Southern is inverted from Northern). Learned a lot about timezones and the history of timezones along the way. [0] https://www.boost.org/doc/libs/1_72_0/doc/html/date_time.html https://www.boost.org/doc/libs/1_72_0/doc/html/date_time.htm...