7 ms·

Sometimes you just have to cut them some slack. Have you engineered a highly available cluster before? I'm not talking about the hot-standby postgres master th

Sometimes you just have to cut them some slack. Have you engineered a highly available cluster before? I'm not talking about the hot-standby postgres master that gets called on once every 2 years, but I'm talking about a 180 node Cassandra cluster thats doing 15,000 writes a second 24/7 and peaking at 60,000 writes a second every day, and you have to do node replacements every week or two because of the high load. Or I'm talking about a 200 node hadoop cluster thats doing the electrical metering and billing for 8 million people, and is NOT allowed to stop. Or the trading platform thats running sub millisecond trades and downtime means 300,000 $ USD per minute. These are systems I have engineered over the last 10 years, and I can say: These things are complex and have failures in 1000 different ways, and while you're monitoring 999 of them that one thing you're not looking at is festering under the surface (your monitoring system is tracking IRQ hardware interrupt response times, right???) Part of being in a team is everyone pulling together, and yes it's stressful at the time, but even very good management cant see all ends, just like very good engineering cant predict everything. I don't think it's useful to start pointing the finger at management and "asking some pointed questions at leadership" because sometimes everyone is doing their best. Yes we should analyse our failures so we can do better, but your tone is very accusatory, and I believe that a better approach is an all inclusive chat about how we can do better, and management saying "great job engineering" for fixing it, and giving them a break after the stressful event.

raiyu 7y ago
No doubt there are many complex systems and they inevitably go down. Every provider has suffered meaningful outages. I think the issue here isn’t so much that the system went down but the blog post. It’s very light on details and doesn’t go far enough in terms of re-establishing trust with the customers that were affected. Which by the looks of it is everyone attempting any trade most of the day on Monday.
LaserToy 7y ago
It is not about scale, it is about the fact that people lost real money. If you can’t make it work you should not be in that business, and I don’t really care how hard they work. I’m taking my account off their platform.
- malux85 7y ago
  Is this your first day of trading or something? People lose money in trading all the time, for hundreds of reasons and some of those reasons are infrastructure downtime. If your risk profile doesn't reflect that, maybe you should take your money out of trading altogether.
  - LaserToy 7y ago
    I will take it off this platform for sure.
    - LaserToy 7y ago
      And why -2 points??? Already moving money out. What is annoying - only 50k per day is allowed.
      - blackearl 7y ago
        Because complaining about RH being a low quality trading platform is like being angry at the burger quality at McDonald's. You get what you pay for.
        EpicEng 7y ago
        Nearly every brokerage now has zero fee trades, just like RH. They weren't down Monday.
  - anon102010 7y ago
    I carry reasonable investment balances - I’m not an active trader but in this space I expect availability. I’d never put my money on RH - and this has nothing to do w risk profile
    - deleted 7y ago
      [deleted]
  - 7y ago
luckylion 7y ago
On the other hand, they've had plenty of time and resources to do just that in a reliable fashion, it's not like it's one guy in his bedroom (I hope!). It's not like they are volunteers doing this open source for the community, they are getting paid (very well, I assume) to run the system. And Management is getting paid (even better, I assume) to make sure the priorities are right and correct decisions are taken. "Who could've known there might be a lot more traffic" sounds like somebody failed in Management, and engineering might have failed by not foreseeing the issue and/or informing Management. Sure, don't burn people at the stake, but "hey, it's hard, don't blame them, they are doing their best" doesn't cut it for me. I'm sure they're expecting to be paid and not for someone to "do their best" to pay them.
- malux85 7y ago
  Can you give me a concrete example of a massive distributed system that has zero downtime? Because the largest distributed system I have seen and worked on was at Apple (or maybe DFP at Google) - and even though they had some of the smartest people in the world and literally billions of dollars behind them, there were still an endless list of problems and downtime events. Spoiler alert: It doesn't exist.
  - luckylion 7y ago
    The point isn't that "a system cannot fail", the point is "if the system fails, it's no big deal, shit happens, cut them some slack" is a weird way to look at it for corporate systems, especially in sensitive areas. If you're running a HA system and you only need one nine to express your availability percentage, sure, sure, you have the smartest people etc and you're doing such a great job, and yeah, yeah, show me one system that has 100% uptime etc.
    - malux85 7y ago
      It didn't say it's no big deal, you're extrapolating and exaggerating my words because your argument is weak. My point was that failure is inevitable in any complex system, and I was responding to the parents point that he immediately pointed the finger at management in an accusatory way, and I was saying that's not constructive. Also your point "They expect to be paid" is actually implicitly "I expect management do do their best to pay me" - there could be a failure in the payroll system, there could be a failure in the banks, there could be many reasons outside managements control that means I'm not getting paid. I can say "why don't you have redundant payroll systems" (which is a stupid waste of resources given the cost/benefit/low failure rate) But my point is again - complex systems have failures - and SOMETIMES, JUST SOMETIMES, YOU CAN CUT THEM SOME SLACK.
techie128 7y ago
Kudos, these are moderate sized systems you've built over your career. There are lot bigger and more mission critical systems in the world and you might build them one day. I understand GP's tone wasn't exactly nice here. But here's the rub with RH's outage. RH is unfortunately in an industry (Finance, Healthcare, Aviation, Food, etc.) where people _need_ to trust them to be successful. The consequences of failure in these industries is very catastrophic not only for them but their clients. Sure failures happen but the scale at which RH has failed and the lukewarm response they've put out has pissed off people. I don't recall any brokerage, old or new, that has failed so catastrophically and has responded to it so poorly. If you think you have a worse example, I am all ears.
- tinus_hn 7y ago
  Catastrophe would be if they actually lost money. This is all indirect damage that is probably disclaimed in their terms of service. No service guarantees 100% availability, it doesn’t exist.
- deleted 7y ago
  [deleted]
- deleted 7y ago
  [deleted]
- kasey_junk 7y ago
  Hard to judge _worse_ in this context but while I was trading all of CME globex was down for 4 hours canceling active orders. https://www.profit-loss.com/cme-hit-by-globex-outage/ https://www.profit-loss.com/cme-hit-by-globex-outage/ I don’t remember them offering any apology or explanation at all. That’s an exchange mind you where things like the global price of oil and s&p futures trade. Not a small boutique brokerage. Further they have planned downtime every week & at that point still had planned daily downtime I think. I think Robinhood screwed up. I think they should learn a hard lesson. But people thinking that trading is some high reliability industry haven’t spent any time in it. The scary thing to me is are healthcare, aviation & food the same?
  - 7y ago
dirtydroog 7y ago
If you used Scylla you'd have only needed 90 nodes. (Don't believe the instability rumours)
- gshulegaard 7y ago
  I've was a primary contributor on a migration of time series data to Scylla. As an anecdote, I once emailed our business contact about tracking down why we appeared to have data inconsistencies between our new (Scylla backed) and old system. I thought the e-mail got lost since we never heard back...until 8 months later (long after we had de-prioritized the migration since our old system was "good enough") asking if we had tried the newly released version which fixed a data loss issue. Blew. My. Mind. Not only because of the radio silence and then dropping back in out of the blue as if no time had passed, but also because they had a data loss issue. So rechecked out my previous branch, upgraded Scylla versions and sure enough the data differences we were noticing before appeared to be resolved. I couldn't believe the amount of time I had spent combing through my code to see if I had a hard to detect bug somewhere...but nope, it was ScyllaDB (although I am sure there were plenty of other bugs...just they weren't the cause of this specific symptom). I am actually a fan of ScyllaDB and what is trying to do. Performance was great (as advertised) and management was simple enough; but they are going to need to work pretty hard to convince me "instability" is just rumor after that experience not too many years ago.
  - ethbro 7y ago
    And this is why folks still run DB2.
  - dirtydroog 7y ago
    Well, we moved to it from Cassandra. It's yet to fall over and we're querying it maybe 200k times per second. No changes were needed from the client driver side of things too. YMMV.
TheCondor 7y ago
Does the duration of their downtime suggest a “1/1000” unmonitored oversight? Or is it more like a threshold that was meet and probably could/should have been observed? And FWIW, they have down time every day and weekend, at least in a virtual sense; the load does drop off in a very real sense too. You are spiritually correct, they should pull together and sort it out, and they owe nobody money here (don’t use a discount broker if you want some sort of guarantee about trades) but as a general rule you should ever feel too sorry for banker under just about any circumstances. The harshest lesson here, for everybody, was the only thing they would do for you was give you some commission free trades but that won’t work with this one, so a non-apology is what you get.
- mumblemumble 7y ago
  I think you may be focusing on the finger instead of the thing that it's pointing at. The post reads to me like all those examples were meant to be concrete examples to drive home a more general argument that complex systems are, well, complex, and that there's an element of hubris in taking potshots from the peanut gallery.
  - tmpz22 7y ago
    I think the original point in this sub-thread boils down to: basic micro-level human error like typos + bad configuration deploys is completely understandable (to a certain extent), but macro level failures that happen by ignoring obvious trends and best practices is malfeasance. Personally I don't think Robinhood will ever release a full honest post-mortem and so we'll never know (and never be able to judge fairly). If the system failed by virtue of being too complex, that is also malfeasance because any devops/SRE worth their salt (as might be expected at a 7 BILLION DOLLAR company) should smell unnecessary complexity from a mile away and slowly refactor it away over the course of several years - which looking at Robinhoods downtime history they never did. The closest example to Robinhoods engineering woes is Reddit, which throughout its early history made fairly poor infrastructure and data modeling decisions but have since repaired and improved on. We should hold Robinhood to higher expectations then Reddit for obvious reasons. Them having similar engineering capability to circa ~2012 start-up reddit is INEXCUSABLE.
Ntrails 7y ago
> Or the trading platform thats running sub millisecond trades and downtime means 300,000 USD per minute. I mean, I'll bite. Assuming you only traded 6 hours a day (ie US) that'd be a 27bn dollar a year strategy, and the only way for returns to be linear and trading to be sub milli is market making/arbitrage. That is a lot of half spreads...
- deleted 7y ago
  [deleted]
- deleted 7y ago
  [deleted]

raiyu 7y ago

No doubt there are many complex systems and they inevitably go down. Every provider has suffered meaningful outages. I think the issue here isn’t so much that the system went down but the blog post. It’s very light on details and doesn’t go far enough in terms of re-establishing trust with the customers that were affected. Which by the looks of it is everyone attempting any trade most of the day on Monday.

LaserToy 7y ago

It is not about scale, it is about the fact that people lost real money. If you can’t make it work you should not be in that business, and I don’t really care how hard they work. I’m taking my account off their platform.

malux85 7y ago
Is this your first day of trading or something? People lose money in trading all the time, for hundreds of reasons and some of those reasons are infrastructure downtime. If your risk profile doesn't reflect that, maybe you should take your money out of trading altogether.
- LaserToy 7y ago
  I will take it off this platform for sure.
  - LaserToy 7y ago
    And why -2 points??? Already moving money out. What is annoying - only 50k per day is allowed.
    - blackearl 7y ago
      Because complaining about RH being a low quality trading platform is like being angry at the burger quality at McDonald's. You get what you pay for.
      - EpicEng 7y ago
        Nearly every brokerage now has zero fee trades, just like RH. They weren't down Monday.
- anon102010 7y ago
  I carry reasonable investment balances - I’m not an active trader but in this space I expect availability. I’d never put my money on RH - and this has nothing to do w risk profile
  - deleted 7y ago
    [deleted]
- 7y ago

luckylion 7y ago

On the other hand, they've had plenty of time and resources to do just that in a reliable fashion, it's not like it's one guy in his bedroom (I hope!). It's not like they are volunteers doing this open source for the community, they are getting paid (very well, I assume) to run the system. And Management is getting paid (even better, I assume) to make sure the priorities are right and correct decisions are taken. "Who could've known there might be a lot more traffic" sounds like somebody failed in Management, and engineering might have failed by not foreseeing the issue and/or informing Management. Sure, don't burn people at the stake, but "hey, it's hard, don't blame them, they are doing their best" doesn't cut it for me. I'm sure they're expecting to be paid and not for someone to "do their best" to pay them.

malux85 7y ago
Can you give me a concrete example of a massive distributed system that has zero downtime? Because the largest distributed system I have seen and worked on was at Apple (or maybe DFP at Google) - and even though they had some of the smartest people in the world and literally billions of dollars behind them, there were still an endless list of problems and downtime events. Spoiler alert: It doesn't exist.
- luckylion 7y ago
  The point isn't that "a system cannot fail", the point is "if the system fails, it's no big deal, shit happens, cut them some slack" is a weird way to look at it for corporate systems, especially in sensitive areas. If you're running a HA system and you only need one nine to express your availability percentage, sure, sure, you have the smartest people etc and you're doing such a great job, and yeah, yeah, show me one system that has 100% uptime etc.
  - malux85 7y ago
    It didn't say it's no big deal, you're extrapolating and exaggerating my words because your argument is weak. My point was that failure is inevitable in any complex system, and I was responding to the parents point that he immediately pointed the finger at management in an accusatory way, and I was saying that's not constructive. Also your point "They expect to be paid" is actually implicitly "I expect management do do their best to pay me" - there could be a failure in the payroll system, there could be a failure in the banks, there could be many reasons outside managements control that means I'm not getting paid. I can say "why don't you have redundant payroll systems" (which is a stupid waste of resources given the cost/benefit/low failure rate) But my point is again - complex systems have failures - and SOMETIMES, JUST SOMETIMES, YOU CAN CUT THEM SOME SLACK.

techie128 7y ago

Kudos, these are moderate sized systems you've built over your career. There are lot bigger and more mission critical systems in the world and you might build them one day. I understand GP's tone wasn't exactly nice here. But here's the rub with RH's outage. RH is unfortunately in an industry (Finance, Healthcare, Aviation, Food, etc.) where people _need_ to trust them to be successful. The consequences of failure in these industries is very catastrophic not only for them but their clients. Sure failures happen but the scale at which RH has failed and the lukewarm response they've put out has pissed off people. I don't recall any brokerage, old or new, that has failed so catastrophically and has responded to it so poorly. If you think you have a worse example, I am all ears.

tinus_hn 7y ago
Catastrophe would be if they actually lost money. This is all indirect damage that is probably disclaimed in their terms of service. No service guarantees 100% availability, it doesn’t exist.
deleted 7y ago
[deleted]
deleted 7y ago
[deleted]
kasey_junk 7y ago
Hard to judge _worse_ in this context but while I was trading all of CME globex was down for 4 hours canceling active orders. https://www.profit-loss.com/cme-hit-by-globex-outage/ https://www.profit-loss.com/cme-hit-by-globex-outage/ I don’t remember them offering any apology or explanation at all. That’s an exchange mind you where things like the global price of oil and s&p futures trade. Not a small boutique brokerage. Further they have planned downtime every week & at that point still had planned daily downtime I think. I think Robinhood screwed up. I think they should learn a hard lesson. But people thinking that trading is some high reliability industry haven’t spent any time in it. The scary thing to me is are healthcare, aviation & food the same?
- 7y ago

dirtydroog 7y ago

If you used Scylla you'd have only needed 90 nodes. (Don't believe the instability rumours)

gshulegaard 7y ago
I've was a primary contributor on a migration of time series data to Scylla. As an anecdote, I once emailed our business contact about tracking down why we appeared to have data inconsistencies between our new (Scylla backed) and old system. I thought the e-mail got lost since we never heard back...until 8 months later (long after we had de-prioritized the migration since our old system was "good enough") asking if we had tried the newly released version which fixed a data loss issue. Blew. My. Mind. Not only because of the radio silence and then dropping back in out of the blue as if no time had passed, but also because they had a data loss issue. So rechecked out my previous branch, upgraded Scylla versions and sure enough the data differences we were noticing before appeared to be resolved. I couldn't believe the amount of time I had spent combing through my code to see if I had a hard to detect bug somewhere...but nope, it was ScyllaDB (although I am sure there were plenty of other bugs...just they weren't the cause of this specific symptom). I am actually a fan of ScyllaDB and what is trying to do. Performance was great (as advertised) and management was simple enough; but they are going to need to work pretty hard to convince me "instability" is just rumor after that experience not too many years ago.
- ethbro 7y ago
  And this is why folks still run DB2.
- dirtydroog 7y ago
  Well, we moved to it from Cassandra. It's yet to fall over and we're querying it maybe 200k times per second. No changes were needed from the client driver side of things too. YMMV.

TheCondor 7y ago

Does the duration of their downtime suggest a “1/1000” unmonitored oversight? Or is it more like a threshold that was meet and probably could/should have been observed? And FWIW, they have down time every day and weekend, at least in a virtual sense; the load does drop off in a very real sense too. You are spiritually correct, they should pull together and sort it out, and they owe nobody money here (don’t use a discount broker if you want some sort of guarantee about trades) but as a general rule you should ever feel too sorry for banker under just about any circumstances. The harshest lesson here, for everybody, was the only thing they would do for you was give you some commission free trades but that won’t work with this one, so a non-apology is what you get.

mumblemumble 7y ago
I think you may be focusing on the finger instead of the thing that it's pointing at. The post reads to me like all those examples were meant to be concrete examples to drive home a more general argument that complex systems are, well, complex, and that there's an element of hubris in taking potshots from the peanut gallery.
- tmpz22 7y ago
  I think the original point in this sub-thread boils down to: basic micro-level human error like typos + bad configuration deploys is completely understandable (to a certain extent), but macro level failures that happen by ignoring obvious trends and best practices is malfeasance. Personally I don't think Robinhood will ever release a full honest post-mortem and so we'll never know (and never be able to judge fairly). If the system failed by virtue of being too complex, that is also malfeasance because any devops/SRE worth their salt (as might be expected at a 7 BILLION DOLLAR company) should smell unnecessary complexity from a mile away and slowly refactor it away over the course of several years - which looking at Robinhoods downtime history they never did. The closest example to Robinhoods engineering woes is Reddit, which throughout its early history made fairly poor infrastructure and data modeling decisions but have since repaired and improved on. We should hold Robinhood to higher expectations then Reddit for obvious reasons. Them having similar engineering capability to circa ~2012 start-up reddit is INEXCUSABLE.

Ntrails 7y ago

> Or the trading platform thats running sub millisecond trades and downtime means 300,000 USD per minute. I mean, I'll bite. Assuming you only traded 6 hours a day (ie US) that'd be a 27bn dollar a year strategy, and the only way for returns to be linear and trading to be sub milli is market making/arbitrage. That is a lot of half spreads...

deleted 7y ago
[deleted]
deleted 7y ago
[deleted]