5 ms·

Apparently leap day is the cause: https://mobile.twitter.com/jtech63/status/1234600045787394048 https://mobile.twitter.com/jtech63/status/123460004578739404...

Apparently leap day is the cause: https://mobile.twitter.com/jtech63/status/1234600045787394048 https://mobile.twitter.com/jtech63/status/123460004578739404... Also rumor has it they “lost user data and the backups are not working as expected” Edit: This is the tweet. Not sure if it is legit: https://mobile.twitter.com/classrobinhood/status/1234577426488811520 https://mobile.twitter.com/classrobinhood/status/12345774264...

pfarnsworth 7y ago
This looks like a terrible source. I would wait until they issue a statement.
- halotrope 7y ago
  Very true! Unfortunately RH went completely dark. I sincerely hope they fix the issue and provide a post mortem.
vsareto 7y ago
If this is true and nothing happens to this company, everyone should feel free to use bugs to cover up any and all IT negligence.
- Craighead 7y ago
  Are you blaming the lack of programming logic around leap years on IT?
- halotrope 7y ago
  I doubt it will be possible to just shake it off as there is likely millions of dollars lost for customers. Not in some abstract way like when github goes down but literally because people cannot close out positions and options expire. If it is really just a leap year bug It should be gross negligence. Not sure if their TOS will hold then.
  - deleted 7y ago
    [deleted]
nodesocket 7y ago
As somebody who does DevOps consulting I was baffled how they could be down all day if it was a pure operations issue. That tweet is just speculation though. Hopefully more facts and details come out. My bet is still on it being a database issue. If they indeed are using AWS RDS, I wouldn’t be surprised if they have half of AWS support engineers working on it. Once again, if I had to bet, I say they rolled their own "half-baked" database solution instead of using a AWS native offering.
- yushuf 7y ago
  I recently interviewed there and they said they're still using Postgres master-slave setup most likely on some cloud provider. Could be catastrophic failure with multiple shards going down at the same time. This might also explain why they mentioned their "backups" not working
  - nodesocket 7y ago
    I believe they are on AWS, so maybe RDS PostgreSQL. If so they should be getting unlimited support from top tier AWS engineers.
    - opsunit 7y ago
      If that is true and they're not on the ball they have another hard deadline fast approaching: https://aws.amazon.com/blogs/database/amazon-rds-customers-update-your-ssl-tls-certificates-by-february-5-2020/ https://aws.amazon.com/blogs/database/amazon-rds-customers-u... (yes: the URI says February 5th whilst the article says March 5th!)
    - aeyes 7y ago
      Why would they get unlimited support from "top tier" AWS engineers? A company like Robinhood doesn't strike me as a strong candidate for contracting AWS Enterprise support. For the price you get almost nothing out of it.
      - vasco 7y ago
enlyth 7y ago
It looks like the guy on Twitter doesn't understand that the API is down so you get 503 and a CORS error cause the pre-flight request failed and doesn't have the right headers. It will fail for any date. The question is really does the front-end normally try to query the next day and what's going on in their order execution system on the backend, which we don't know because this source is just speculation based on console logs in the browser.
manigandham 7y ago
This person is probing an API which is already down so of course there are errors. Robinhood would be using a proper date library anyway to handle timezones. The system was fine on the 29th, 1st and even this morning pre-market. Seems more like a capacity issue that cascaded into failure.
- chance_state 7y ago
  The chances that it was also down exactly 4 years ago today (it was) are astronomical though.
  - UncleMeat 7y ago
    The 2nd was a Wednesday four years ago. That wouldn’t make any sense for a leap year error. Why wouldn’t it have gone down on the Monday two days prior?
ecnahc515 7y ago
My favorite part is this comment: https://twitter.com/holman/status/1234620062398464002 https://twitter.com/holman/status/1234620062398464002 https://www.reddit.com/r/RobinHood/comments/48mep4/robinhood_not_working/ https://www.reddit.com/r/RobinHood/comments/48mep4/robinhood... Additionally, if you’re a conspiracy theorist — and who isn't, in this wild 2020 ride — Robinhood was down exactly four years ago today, too, lol
- nodesocket 7y ago
  > Same here CALL ROBINHOOD ASAP. I spoke with them myself and they are aware. IF YOU NEED TO MAKE TRADES THEY CAN DO IT FOR YOU! I didn't even know they had a phone number you could call. This should have been emailed out after the first hour of the outage. Or, how about you have a better system down page then just a blank website? If API health checks fail, serve a static site from S3 (route53 can do this). The ops failure of a company that can afford to pay quality DevOps engineers $200-300k a year is beyond belief. Simple as: "We are experiencing a technical outage... <corporate blah blah blah here>. If you need to make trades, please do so via phone at 1-800-SHORT-RH"
  - EpicEng 7y ago
    >"We are experiencing a technical outage... <corporate blah blah blah here>. If you need to make trades, please do so via phone at 1-800-SHORT-RH" "You're call will be taken in the order it was rexeived. Current wait time: 1,732 minutes."

pfarnsworth 7y ago

This looks like a terrible source. I would wait until they issue a statement.

halotrope 7y ago
Very true! Unfortunately RH went completely dark. I sincerely hope they fix the issue and provide a post mortem.

vsareto 7y ago

If this is true and nothing happens to this company, everyone should feel free to use bugs to cover up any and all IT negligence.

Craighead 7y ago
Are you blaming the lack of programming logic around leap years on IT?
halotrope 7y ago
I doubt it will be possible to just shake it off as there is likely millions of dollars lost for customers. Not in some abstract way like when github goes down but literally because people cannot close out positions and options expire. If it is really just a leap year bug It should be gross negligence. Not sure if their TOS will hold then.
- deleted 7y ago
  [deleted]

nodesocket 7y ago

As somebody who does DevOps consulting I was baffled how they could be down all day if it was a pure operations issue. That tweet is just speculation though. Hopefully more facts and details come out. My bet is still on it being a database issue. If they indeed are using AWS RDS, I wouldn’t be surprised if they have half of AWS support engineers working on it. Once again, if I had to bet, I say they rolled their own "half-baked" database solution instead of using a AWS native offering.

yushuf 7y ago
I recently interviewed there and they said they're still using Postgres master-slave setup most likely on some cloud provider. Could be catastrophic failure with multiple shards going down at the same time. This might also explain why they mentioned their "backups" not working
- nodesocket 7y ago
  I believe they are on AWS, so maybe RDS PostgreSQL. If so they should be getting unlimited support from top tier AWS engineers.
  - opsunit 7y ago
    If that is true and they're not on the ball they have another hard deadline fast approaching: https://aws.amazon.com/blogs/database/amazon-rds-customers-update-your-ssl-tls-certificates-by-february-5-2020/ https://aws.amazon.com/blogs/database/amazon-rds-customers-u... (yes: the URI says February 5th whilst the article says March 5th!)
  - aeyes 7y ago
    Why would they get unlimited support from "top tier" AWS engineers? A company like Robinhood doesn't strike me as a strong candidate for contracting AWS Enterprise support. For the price you get almost nothing out of it.
    - vasco 7y ago

enlyth 7y ago

It looks like the guy on Twitter doesn't understand that the API is down so you get 503 and a CORS error cause the pre-flight request failed and doesn't have the right headers. It will fail for any date. The question is really does the front-end normally try to query the next day and what's going on in their order execution system on the backend, which we don't know because this source is just speculation based on console logs in the browser.

manigandham 7y ago

This person is probing an API which is already down so of course there are errors. Robinhood would be using a proper date library anyway to handle timezones. The system was fine on the 29th, 1st and even this morning pre-market. Seems more like a capacity issue that cascaded into failure.

chance_state 7y ago
The chances that it was also down exactly 4 years ago today (it was) are astronomical though.
- UncleMeat 7y ago
  The 2nd was a Wednesday four years ago. That wouldn’t make any sense for a leap year error. Why wouldn’t it have gone down on the Monday two days prior?

ecnahc515 7y ago

My favorite part is this comment: https://twitter.com/holman/status/1234620062398464002 https://twitter.com/holman/status/1234620062398464002 https://www.reddit.com/r/RobinHood/comments/48mep4/robinhood_not_working/ https://www.reddit.com/r/RobinHood/comments/48mep4/robinhood... Additionally, if you’re a conspiracy theorist — and who isn't, in this wild 2020 ride — Robinhood was down exactly four years ago today, too, lol

nodesocket 7y ago
> Same here CALL ROBINHOOD ASAP. I spoke with them myself and they are aware. IF YOU NEED TO MAKE TRADES THEY CAN DO IT FOR YOU! I didn't even know they had a phone number you could call. This should have been emailed out after the first hour of the outage. Or, how about you have a better system down page then just a blank website? If API health checks fail, serve a static site from S3 (route53 can do this). The ops failure of a company that can afford to pay quality DevOps engineers $200-300k a year is beyond belief. Simple as: "We are experiencing a technical outage... <corporate blah blah blah here>. If you need to make trades, please do so via phone at 1-800-SHORT-RH"
- EpicEng 7y ago
  >"We are experiencing a technical outage... <corporate blah blah blah here>. If you need to make trades, please do so via phone at 1-800-SHORT-RH" "You're call will be taken in the order it was rexeived. Current wait time: 1,732 minutes."