5 ms·

I know quite a few people that were personally affected by this and lost money due to the two outages and they are all pulling their money from Robinhood. The f

I know quite a few people that were personally affected by this and lost money due to the two outages and they are all pulling their money from Robinhood. The fact that they can't offer any compensation might be a big problem for them, since they already have zero trading fees, which is what most brokerages offer as compensation. Personally it doesn't pass the smell test for me. The load was much higher the previous week and load problems go away once the load disappears. They probably had a lot less load the rest of the day, so the fact they were down the entire day suggests it was something else. I would need a fully transparent post mortem before I believed anything they said.

solidasparagus 7y ago
Failures due to high load can take a while to resolve - you often need to fix the broken infrastructure, process the backlog, and catch up to live.
- hcknwscommenter 7y ago
  You can't process the backlog on a trading platform. If i put in a trade at 2:20 pm and the system goes down, I don't want my trade to execute next morning at market open. That's insane. Especially the RH flavor of YOLO infinite leverage call option nonsense.
  - radicaldreamer 7y ago
    Exactly, you have to default to fill or kill within the trading day. You just can’t treat certain products like a standard queue... sometimes time is the most important component
    - stiglitz 7y ago
      FYI, FillOrKill/ImmediateOrCancel are not the same as a day order. FoK/IoC means “do not queue this order”. It’s immediately filled (or not) (or, for IoC, partially) based on whatever orders are already in the book, and then you’re done. Whereas a day order is queued until the end of the day or until it’s filled, whichever comes first.
  - solidasparagus 7y ago
    True for RH trades but I'm sure there is a lot of other data being handled - such as market data.
rpdillon 7y ago
I'm not sure it's fair to assume that service gets automatically restored when load dissipates after failures due to high load.
balls187 7y ago
How did they lose money?
- jiqiren 7y ago
  Quick example: They bought puts on Friday and couldn't unload them for a full day + following morning. Monday morning puts were down - it was obvious the market was recovering in a big way. Instead of cutting losses at ~20% in the morning they lost ~99% of their position. Some lost 100% since the options expired EOD.
  - UncleMeat 7y ago
    > it was obvious the market was recovering in a big way Was it? Markets started up today but ended way lower.
    - akchin 7y ago
      He is talking about Monday not today, Tuesday. RH didn’t work on Monday either.
      - kortilla 7y ago
        Yes, there was nothing “obvious” about the market on Monday morning. Saying otherwise implies the ability to predict the market.
        harikb 7y ago
        I think they meant to say the “loss makes sense”, not that Friday/Monday’s move was predictable
      - UncleMeat 7y ago
        Yes and the point is that today it looked the same as yesterday in the morning but it didn't turn out to be a bounce. It wasn't "obvious" that everything would rise on Monday. Only in retrospect.
deleted 7y ago
[deleted]
driverdan 7y ago
This isn't something new, downtime is the norm for Robinhood. Anyone trusting them with more than play money is foolish.
- neuronic 7y ago
  This is the correct sentiment. People who put anything more than play money into Robinhood should not be surprised when their financial life is ruined.
deleted 7y ago
[deleted]
rolltiide 7y ago
> The fact that they can't offer any compensation might be a big problem for them, since they already have zero trading fees Robinhood makes the most money than any known firm on Wall Street by getting paid specifically to leak user's trades to other traders. SEC requires a periodic report on that which shows compensation. Can't believe people are still buying Robinhood's pitch of misdirection.
- LatteLazy 7y ago
  Is there a source you can cite for that? Why would anyone want retail investor order data? Especially since most of their orders execute immediately, so you can just get the trade data from the venue...
  - JumpCrisscross 7y ago
    > Why would anyone want retail investor order data? Former market maker here. Retail flow is low risk. If I buy $100mm of institutional flow, I could get a bunch of corporate hedging orders. Or I could make a single bet against George Soros. With retail, one tends to find lots of small orders. Even if there are some with high information, i.e. they're smart money and I'm going to lose money trading against them, they're small enough to be manageable. Retail is also low information. At an old job, we bought a prominent retail broker's options flow. The number of in-the-money unexercised options that would come through that pipe was mind-blowing. (Today, whoever was buying Robinhood's flow likely got the same.)
  - a2h 7y ago
    Rule 606 disclosure for the source. And it's not about the trade data it's the order itself. Second link is a very thorough explanation. https://cdn.robinhood.com/assets/robinhood/legal/RHS%20SEC%20Rule%20606%20Report%20Disclosure%20-%20Q4%202019.pdf https://cdn.robinhood.com/assets/robinhood/legal/RHS%20SEC%2... https://www.google.com/url?sa=t&source=web&rct=j&url=http://blog.themistrading.com/wp-content/uploads/2015/03/What-Every-Retail-Investor-Needs-to-Know.pdf&ved=2ahUKEwi475aysYDoAhWuY98KHR0HCPEQFjAAegQIBxAC&usg=AOvVaw0M_rFmz-YkQZu5pOnkiYVK https://www.google.com/url?sa=t&source=web&rct=j&url=http://...
- 7y ago
afc 7y ago
Load problems don't go away when the load disappears. If the system isn't engineered very carefully (this takes a lot of work!), you may have cascading failures that may take hours to resolve, especially if you have bad retry policies (their mention of thundering herd problem seems to indicate that they might). We wrote a bit about this here: https://landing.google.com/sre/sre-book/chapters/addressing-cascading-failures/ https://landing.google.com/sre/sre-book/chapters/addressing-... I would strongly caution anyone who thinks this subject is trivial, just add a bit of load shedding and you're done. I wrote a bit about my team's work (including a simplified view of some of the considerations that go into how we do retries) here: https://landing.google.com/sre/sre-book/chapters/handling-overload/ https://landing.google.com/sre/sre-book/chapters/handling-ov...
- jennyyang 7y ago
  They specifically said it lead to a DNS failure. They didn't mention anything else, like corrupt data, etc. Sure there are plenty of ways that outages, not just load problems, can cause significant outages, but what Robinhood specifically said was that they had load issues that lead to a DNS failure. They should be more forthcoming with exactly happened if they want people to trust them.

solidasparagus 7y ago

Failures due to high load can take a while to resolve - you often need to fix the broken infrastructure, process the backlog, and catch up to live.

hcknwscommenter 7y ago
You can't process the backlog on a trading platform. If i put in a trade at 2:20 pm and the system goes down, I don't want my trade to execute next morning at market open. That's insane. Especially the RH flavor of YOLO infinite leverage call option nonsense.
- radicaldreamer 7y ago
  Exactly, you have to default to fill or kill within the trading day. You just can’t treat certain products like a standard queue... sometimes time is the most important component
  - stiglitz 7y ago
    FYI, FillOrKill/ImmediateOrCancel are not the same as a day order. FoK/IoC means “do not queue this order”. It’s immediately filled (or not) (or, for IoC, partially) based on whatever orders are already in the book, and then you’re done. Whereas a day order is queued until the end of the day or until it’s filled, whichever comes first.
- solidasparagus 7y ago
  True for RH trades but I'm sure there is a lot of other data being handled - such as market data.

rpdillon 7y ago

I'm not sure it's fair to assume that service gets automatically restored when load dissipates after failures due to high load.

balls187 7y ago

How did they lose money?

jiqiren 7y ago
Quick example: They bought puts on Friday and couldn't unload them for a full day + following morning. Monday morning puts were down - it was obvious the market was recovering in a big way. Instead of cutting losses at ~20% in the morning they lost ~99% of their position. Some lost 100% since the options expired EOD.
- UncleMeat 7y ago
  > it was obvious the market was recovering in a big way Was it? Markets started up today but ended way lower.
  - akchin 7y ago
    He is talking about Monday not today, Tuesday. RH didn’t work on Monday either.
    - kortilla 7y ago
      Yes, there was nothing “obvious” about the market on Monday morning. Saying otherwise implies the ability to predict the market.
      - harikb 7y ago
        I think they meant to say the “loss makes sense”, not that Friday/Monday’s move was predictable
    - UncleMeat 7y ago
      Yes and the point is that today it looked the same as yesterday in the morning but it didn't turn out to be a bounce. It wasn't "obvious" that everything would rise on Monday. Only in retrospect.

deleted 7y ago

[deleted]

driverdan 7y ago

This isn't something new, downtime is the norm for Robinhood. Anyone trusting them with more than play money is foolish.

neuronic 7y ago
This is the correct sentiment. People who put anything more than play money into Robinhood should not be surprised when their financial life is ruined.

deleted 7y ago

[deleted]

rolltiide 7y ago

> The fact that they can't offer any compensation might be a big problem for them, since they already have zero trading fees Robinhood makes the most money than any known firm on Wall Street by getting paid specifically to leak user's trades to other traders. SEC requires a periodic report on that which shows compensation. Can't believe people are still buying Robinhood's pitch of misdirection.

LatteLazy 7y ago
Is there a source you can cite for that? Why would anyone want retail investor order data? Especially since most of their orders execute immediately, so you can just get the trade data from the venue...
- JumpCrisscross 7y ago
  > Why would anyone want retail investor order data? Former market maker here. Retail flow is low risk. If I buy $100mm of institutional flow, I could get a bunch of corporate hedging orders. Or I could make a single bet against George Soros. With retail, one tends to find lots of small orders. Even if there are some with high information, i.e. they're smart money and I'm going to lose money trading against them, they're small enough to be manageable. Retail is also low information. At an old job, we bought a prominent retail broker's options flow. The number of in-the-money unexercised options that would come through that pipe was mind-blowing. (Today, whoever was buying Robinhood's flow likely got the same.)
- a2h 7y ago
  Rule 606 disclosure for the source. And it's not about the trade data it's the order itself. Second link is a very thorough explanation. https://cdn.robinhood.com/assets/robinhood/legal/RHS%20SEC%20Rule%20606%20Report%20Disclosure%20-%20Q4%202019.pdf https://cdn.robinhood.com/assets/robinhood/legal/RHS%20SEC%2... https://www.google.com/url?sa=t&source=web&rct=j&url=http://blog.themistrading.com/wp-content/uploads/2015/03/What-Every-Retail-Investor-Needs-to-Know.pdf&ved=2ahUKEwi475aysYDoAhWuY98KHR0HCPEQFjAAegQIBxAC&usg=AOvVaw0M_rFmz-YkQZu5pOnkiYVK https://www.google.com/url?sa=t&source=web&rct=j&url=http://...
7y ago

afc 7y ago

Load problems don't go away when the load disappears. If the system isn't engineered very carefully (this takes a lot of work!), you may have cascading failures that may take hours to resolve, especially if you have bad retry policies (their mention of thundering herd problem seems to indicate that they might). We wrote a bit about this here: https://landing.google.com/sre/sre-book/chapters/addressing-cascading-failures/ https://landing.google.com/sre/sre-book/chapters/addressing-... I would strongly caution anyone who thinks this subject is trivial, just add a bit of load shedding and you're done. I wrote a bit about my team's work (including a simplified view of some of the considerations that go into how we do retries) here: https://landing.google.com/sre/sre-book/chapters/handling-overload/ https://landing.google.com/sre/sre-book/chapters/handling-ov...

jennyyang 7y ago
They specifically said it lead to a DNS failure. They didn't mention anything else, like corrupt data, etc. Sure there are plenty of ways that outages, not just load problems, can cause significant outages, but what Robinhood specifically said was that they had load issues that lead to a DNS failure. They should be more forthcoming with exactly happened if they want people to trust them.