4 ms·

I haven't looked into the technical credibility of this at all [EDIT: I should have more strongly indicated my doubt here], so this isn't any sort of condemnati

by rriepe 7y ago

I haven't looked into the technical credibility of this at all [EDIT: I should have more strongly indicated my doubt here], so this isn't any sort of condemnation on my end, but I thought the Leap Day theory was interesting: https://twitter.com/jtech63/status/1234600045787394048 https://twitter.com/jtech63/status/1234600045787394048

eshyong 7y ago
That was probably a joke, see Robinhood's reply: https://twitter.com/AskRobinhood/status/1234861941413351434 https://twitter.com/AskRobinhood/status/1234861941413351434 Seems like infra problems.
- bob1029 7y ago
  It's starting to seem like they use some sort of message bus to tie everything together. If the servers actually processing messages can't handle the volume, the entire show comes grinding to a halt. I've seen similar things in other industries where the entire system looks like its perfectly healthy, and then before you know it 100% of your systems are down because you simply can't push peak message volume or because one participant is infinite-looping messages.
  - ewheeler 7y ago
    Correct, here is their Kafka-based stream processing package that presumably ties everything together: https://faust.readthedocs.io/en/latest/ https://faust.readthedocs.io/en/latest/
  - eshyong 7y ago
    Their director of infra tried to recruit me and basically said that they wanted to replace every relational data store with message queues instead... that struck me as a bit weird and overzealous.
- zarmin 7y ago
  It's worth noting that they were down 4 years ago on March 2 as well.
Humdeee 7y ago
There was a discussion last night about this. Some of the more technically competent posters dismissed it because it's unbelievable that a financial platform would roll their own date-time implementation.
- rriepe 7y ago
  While I agree in spirit (Occam tells me it was just unprecedented load), that explanation doesn't make any sense to me in terms of disproving the Leap Year theory. The mistake wouldn't even have to be in their code. As for believing/unbelieving, I mean, I've seen dumber mistakes in financial products I've worked on. Not as high stakes or damaging, but definitely dumber.
- notyourday 7y ago
  That means people should reevaluate the weight of the opinions of those technically competent posters - we have a screen shot that robinhood did roll their own implementation because yesterday was March 2nd and its app was requesting March 3rd. That at least means that some portion of their stack used roll your own datetime library. That would not actually be a problem as long as the entire stack got the same library or the same rules for datetime. The problem is of course that it probably did not so the libraries were not bug compatible which of course caused errors and those errors need to be handled, preferably every fast. Error load and error handling is the least tested part of every system because it can only be properly tested in production and, unless your company embraces the Chaos Monkey approach to testing, the C-level would have a heart attack when anyone proposes doing it.
  - vsareto 7y ago
    That particular API endpoint is just returning market open times, in which a request for tomorrow might be perfectly reasonable. https://api.robinhood.com/markets/XASE/hours/2020-03-03/ https://api.robinhood.com/markets/XASE/hours/2020-03-03/ vs https://api.robinhood.com/markets/XASE/hours/2020-03-07/ https://api.robinhood.com/markets/XASE/hours/2020-03-07/ This whole discussion lacks context to the nature of the requests, aka a front end code review.
thefreeman 7y ago
I am seeing this parroted everywhere this comes up and it is very unlikely to be the case. For one we have no idea what that API call is doing. The date parameter could be an upper bound, ie. it's just asking for all of the most recent data. In addition, Robinhood was up over the weekend, after the leap year occurred, and was even available early on Monday before the market opened.
- notyourday 7y ago
  Robinhood only trades in the US and the US markets are closed on Saturday and Sunday which means there were no live orders on those days which in turn means that certain code paths were not executed and errors were not thrown on Saturday and Sunday the two days when it would have been detected.

by rriepe 7y ago

eshyong 7y ago

That was probably a joke, see Robinhood's reply: https://twitter.com/AskRobinhood/status/1234861941413351434 https://twitter.com/AskRobinhood/status/1234861941413351434 Seems like infra problems.

bob1029 7y ago
It's starting to seem like they use some sort of message bus to tie everything together. If the servers actually processing messages can't handle the volume, the entire show comes grinding to a halt. I've seen similar things in other industries where the entire system looks like its perfectly healthy, and then before you know it 100% of your systems are down because you simply can't push peak message volume or because one participant is infinite-looping messages.
- ewheeler 7y ago
  Correct, here is their Kafka-based stream processing package that presumably ties everything together: https://faust.readthedocs.io/en/latest/ https://faust.readthedocs.io/en/latest/
- eshyong 7y ago
  Their director of infra tried to recruit me and basically said that they wanted to replace every relational data store with message queues instead... that struck me as a bit weird and overzealous.
zarmin 7y ago
It's worth noting that they were down 4 years ago on March 2 as well.

Humdeee 7y ago

There was a discussion last night about this. Some of the more technically competent posters dismissed it because it's unbelievable that a financial platform would roll their own date-time implementation.

rriepe 7y ago
While I agree in spirit (Occam tells me it was just unprecedented load), that explanation doesn't make any sense to me in terms of disproving the Leap Year theory. The mistake wouldn't even have to be in their code. As for believing/unbelieving, I mean, I've seen dumber mistakes in financial products I've worked on. Not as high stakes or damaging, but definitely dumber.
notyourday 7y ago
That means people should reevaluate the weight of the opinions of those technically competent posters - we have a screen shot that robinhood did roll their own implementation because yesterday was March 2nd and its app was requesting March 3rd. That at least means that some portion of their stack used roll your own datetime library. That would not actually be a problem as long as the entire stack got the same library or the same rules for datetime. The problem is of course that it probably did not so the libraries were not bug compatible which of course caused errors and those errors need to be handled, preferably every fast. Error load and error handling is the least tested part of every system because it can only be properly tested in production and, unless your company embraces the Chaos Monkey approach to testing, the C-level would have a heart attack when anyone proposes doing it.
- vsareto 7y ago
  That particular API endpoint is just returning market open times, in which a request for tomorrow might be perfectly reasonable. https://api.robinhood.com/markets/XASE/hours/2020-03-03/ https://api.robinhood.com/markets/XASE/hours/2020-03-03/ vs https://api.robinhood.com/markets/XASE/hours/2020-03-07/ https://api.robinhood.com/markets/XASE/hours/2020-03-07/ This whole discussion lacks context to the nature of the requests, aka a front end code review.

thefreeman 7y ago

I am seeing this parroted everywhere this comes up and it is very unlikely to be the case. For one we have no idea what that API call is doing. The date parameter could be an upper bound, ie. it's just asking for all of the most recent data. In addition, Robinhood was up over the weekend, after the leap year occurred, and was even available early on Monday before the market opened.

notyourday 7y ago
Robinhood only trades in the US and the US markets are closed on Saturday and Sunday which means there were no live orders on those days which in turn means that certain code paths were not executed and errors were not thrown on Saturday and Sunday the two days when it would have been detected.