5 ms·

Kudos, these are moderate sized systems you've built over your career. There are lot bigger and more mission critical systems in the world and you might build t

Kudos, these are moderate sized systems you've built over your career. There are lot bigger and more mission critical systems in the world and you might build them one day. I understand GP's tone wasn't exactly nice here. But here's the rub with RH's outage. RH is unfortunately in an industry (Finance, Healthcare, Aviation, Food, etc.) where people _need_ to trust them to be successful. The consequences of failure in these industries is very catastrophic not only for them but their clients. Sure failures happen but the scale at which RH has failed and the lukewarm response they've put out has pissed off people. I don't recall any brokerage, old or new, that has failed so catastrophically and has responded to it so poorly. If you think you have a worse example, I am all ears.

tinus_hn 7y ago
Catastrophe would be if they actually lost money. This is all indirect damage that is probably disclaimed in their terms of service. No service guarantees 100% availability, it doesn’t exist.
deleted 7y ago
[deleted]
deleted 7y ago
[deleted]
kasey_junk 7y ago
Hard to judge _worse_ in this context but while I was trading all of CME globex was down for 4 hours canceling active orders. https://www.profit-loss.com/cme-hit-by-globex-outage/ https://www.profit-loss.com/cme-hit-by-globex-outage/ I don’t remember them offering any apology or explanation at all. That’s an exchange mind you where things like the global price of oil and s&p futures trade. Not a small boutique brokerage. Further they have planned downtime every week & at that point still had planned daily downtime I think. I think Robinhood screwed up. I think they should learn a hard lesson. But people thinking that trading is some high reliability industry haven’t spent any time in it. The scary thing to me is are healthcare, aviation & food the same?
- vsareto 7y ago
  Part of AWS's sell with elasticity is only spending what you need, but those industries have redundancies or unused capacity. Someone in one of these threads said there's a hidden DNS within VPCs that can fail and isn't scaled, so if that's true, they might just have to architect around that unless they can get AWS to change it. It's on RH for not knowing that but it's also kind of on AWS too. But as far as what you can do, you can really only split your cash across brokerages if you want to engineer the same redundancy yourself. Otherwise, RH would need to route everything to another exchange to keep satisfying orders, and even that is just another system that could fail. Keeping all of your money in one brokerage doesn't seem ideal if you want to completely avoid downtime. Doing the same redundancy yourself with those industries isn't really practical.
C1sc0cat 7y ago
The Problems with the TSB in the Uk come to mind and NatWest /RBS had a similar SNAFU a few years back.
wonderwonder 7y ago
Boeing's failures have killed hundreds of people. Governments still pay them and people still fly on their planes. Stores sell salmonella contaminated products all the time and people still shop there. RH's failure pales in comparison. Crypto exchanges fail all the time, people still use them. RH may lose a few customers in the short term but I see no reason they wont bounce back, they provide a product people like and the majority of people dont like change and will stay with them provided stability returns soon. Non technical people dont want a technical apology, they just want an 'our bad, working on it' which is what was provided. The company will be fine. Should they be is another question all together.
- raiyu 7y ago
  Technically people still fly the old Boeing planes that don't crash. The 737MAX is still not in service, and there is a likelihood that it may never go back into service. All future orders are cancelled, and there isn't a clear pathway to the plane re-certified and more importantly for people to trust them again. High trust systems require just that, high trust. And once broken it's hard to re-establish. Crypto exchanges certainly have their fair issues of downtime, but don't forget that crypto exchanges for a long time operated purely for early adopters as crypto wasn't something that everyone traded. There was also less availability of competition, because again the industry was newer and there were fewer choices. And certainly Coinbase helped to popularize crypto trading and they had their fair issues, but I don't believe they had an outage of this exact magnitude, and again they were in an early adopter area where mistakes are seen as part of the process. If not expressly, then at least subconsciously.
  - wonderwonder 7y ago
    I think that we have entered a new 'trust' phase, where we pretty much don't care about it and just want familiarity. Look at Facebook, privacy has been violated a thousand times, and we still keep logging in. Experian is still chugging along. People used and paid for AOL for years when they did not have to. Online consumption is different than in person. You go to a restaurant and the food is bad you probably don't go back. Online the bulk of consumers just keep going back because that's what they are used to. We love our favorites. I remember all of AWS going down a couple years ago. Boeing itself is fine even though one product killed hundreds. Robin Hood is going to be fine. This will be forgotten in a week.

tinus_hn 7y ago

Catastrophe would be if they actually lost money. This is all indirect damage that is probably disclaimed in their terms of service. No service guarantees 100% availability, it doesn’t exist.

deleted 7y ago

[deleted]

deleted 7y ago

[deleted]

kasey_junk 7y ago

Hard to judge _worse_ in this context but while I was trading all of CME globex was down for 4 hours canceling active orders. https://www.profit-loss.com/cme-hit-by-globex-outage/ https://www.profit-loss.com/cme-hit-by-globex-outage/ I don’t remember them offering any apology or explanation at all. That’s an exchange mind you where things like the global price of oil and s&p futures trade. Not a small boutique brokerage. Further they have planned downtime every week & at that point still had planned daily downtime I think. I think Robinhood screwed up. I think they should learn a hard lesson. But people thinking that trading is some high reliability industry haven’t spent any time in it. The scary thing to me is are healthcare, aviation & food the same?

vsareto 7y ago
Part of AWS's sell with elasticity is only spending what you need, but those industries have redundancies or unused capacity. Someone in one of these threads said there's a hidden DNS within VPCs that can fail and isn't scaled, so if that's true, they might just have to architect around that unless they can get AWS to change it. It's on RH for not knowing that but it's also kind of on AWS too. But as far as what you can do, you can really only split your cash across brokerages if you want to engineer the same redundancy yourself. Otherwise, RH would need to route everything to another exchange to keep satisfying orders, and even that is just another system that could fail. Keeping all of your money in one brokerage doesn't seem ideal if you want to completely avoid downtime. Doing the same redundancy yourself with those industries isn't really practical.

C1sc0cat 7y ago

The Problems with the TSB in the Uk come to mind and NatWest /RBS had a similar SNAFU a few years back.

wonderwonder 7y ago

Boeing's failures have killed hundreds of people. Governments still pay them and people still fly on their planes. Stores sell salmonella contaminated products all the time and people still shop there. RH's failure pales in comparison. Crypto exchanges fail all the time, people still use them. RH may lose a few customers in the short term but I see no reason they wont bounce back, they provide a product people like and the majority of people dont like change and will stay with them provided stability returns soon. Non technical people dont want a technical apology, they just want an 'our bad, working on it' which is what was provided. The company will be fine. Should they be is another question all together.

raiyu 7y ago
Technically people still fly the old Boeing planes that don't crash. The 737MAX is still not in service, and there is a likelihood that it may never go back into service. All future orders are cancelled, and there isn't a clear pathway to the plane re-certified and more importantly for people to trust them again. High trust systems require just that, high trust. And once broken it's hard to re-establish. Crypto exchanges certainly have their fair issues of downtime, but don't forget that crypto exchanges for a long time operated purely for early adopters as crypto wasn't something that everyone traded. There was also less availability of competition, because again the industry was newer and there were fewer choices. And certainly Coinbase helped to popularize crypto trading and they had their fair issues, but I don't believe they had an outage of this exact magnitude, and again they were in an early adopter area where mistakes are seen as part of the process. If not expressly, then at least subconsciously.
- wonderwonder 7y ago
  I think that we have entered a new 'trust' phase, where we pretty much don't care about it and just want familiarity. Look at Facebook, privacy has been violated a thousand times, and we still keep logging in. Experian is still chugging along. People used and paid for AOL for years when they did not have to. Online consumption is different than in person. You go to a restaurant and the food is bad you probably don't go back. Online the bulk of consumers just keep going back because that's what they are used to. We love our favorites. I remember all of AWS going down a couple years ago. Boeing itself is fine even though one product killed hundreds. Robin Hood is going to be fine. This will be forgotten in a week.