8 ms·

If this were an outage directly caused by a natural disaster, I could understand. This outage was an availability problem. This clearly points to some prioritiz

If this were an outage directly caused by a natural disaster, I could understand. This outage was an availability problem. This clearly points to some prioritization problems within the leadership layers if robust and resilient infrastructure was not emphasized. The prioritization problems may not be due to ignorance or malice though, and may be justifiable if there are other fires that are burning brighter. It's still pointing to problems though, and I think it's completely legitimate for engineers to question the stability of the company when this sort of thing happens. At the very least as an engineer I would be asking some pointed questions of my leadership. Maybe not dusting off the resume yet, but still I'd want to get reassurance from internally that the leadership problems that caused this are being addressed.

donavanm 7y ago
Ive seen bigger, scarier, potentially costlier time based bugs personally. I dont think this would make me reevaluate my employment if I was at robinhood. As the parent says you either learn these lessons the hard way or you havent learned them yet. Thats doesnt translate to being a “leadership failure.” Your smaller point about prioritization is spot on though. I dont believe Ive seen any similar incidents lead to business ending outcomes. I personally point to sony or, more recently, equifax as examples of the disparity between actual business impact and technical abhorrence. In light of that why is it worth trying to preemptively solve technical challenges instead of business needs? Every calorie spent on “what if” subtracts from “whats needed.”
malux85 7y ago
Sometimes you just have to cut them some slack. Have you engineered a highly available cluster before? I'm not talking about the hot-standby postgres master that gets called on once every 2 years, but I'm talking about a 180 node Cassandra cluster thats doing 15,000 writes a second 24/7 and peaking at 60,000 writes a second every day, and you have to do node replacements every week or two because of the high load. Or I'm talking about a 200 node hadoop cluster thats doing the electrical metering and billing for 8 million people, and is NOT allowed to stop. Or the trading platform thats running sub millisecond trades and downtime means 300,000 $ USD per minute. These are systems I have engineered over the last 10 years, and I can say: These things are complex and have failures in 1000 different ways, and while you're monitoring 999 of them that one thing you're not looking at is festering under the surface (your monitoring system is tracking IRQ hardware interrupt response times, right???) Part of being in a team is everyone pulling together, and yes it's stressful at the time, but even very good management cant see all ends, just like very good engineering cant predict everything. I don't think it's useful to start pointing the finger at management and "asking some pointed questions at leadership" because sometimes everyone is doing their best. Yes we should analyse our failures so we can do better, but your tone is very accusatory, and I believe that a better approach is an all inclusive chat about how we can do better, and management saying "great job engineering" for fixing it, and giving them a break after the stressful event.
- raiyu 7y ago
  No doubt there are many complex systems and they inevitably go down. Every provider has suffered meaningful outages. I think the issue here isn’t so much that the system went down but the blog post. It’s very light on details and doesn’t go far enough in terms of re-establishing trust with the customers that were affected. Which by the looks of it is everyone attempting any trade most of the day on Monday.
- LaserToy 7y ago
  It is not about scale, it is about the fact that people lost real money. If you can’t make it work you should not be in that business, and I don’t really care how hard they work. I’m taking my account off their platform.
  - malux85 7y ago
    Is this your first day of trading or something? People lose money in trading all the time, for hundreds of reasons and some of those reasons are infrastructure downtime. If your risk profile doesn't reflect that, maybe you should take your money out of trading altogether.
    - LaserToy 7y ago
      I will take it off this platform for sure.
      - LaserToy 7y ago
        And why -2 points??? Already moving money out. What is annoying - only 50k per day is allowed.
        blackearl 7y ago
        Because complaining about RH being a low quality trading platform is like being angry at the burger quality at McDonald's. You get what you pay for.

donavanm 7y ago

Ive seen bigger, scarier, potentially costlier time based bugs personally. I dont think this would make me reevaluate my employment if I was at robinhood. As the parent says you either learn these lessons the hard way or you havent learned them yet. Thats doesnt translate to being a “leadership failure.” Your smaller point about prioritization is spot on though. I dont believe Ive seen any similar incidents lead to business ending outcomes. I personally point to sony or, more recently, equifax as examples of the disparity between actual business impact and technical abhorrence. In light of that why is it worth trying to preemptively solve technical challenges instead of business needs? Every calorie spent on “what if” subtracts from “whats needed.”

malux85 7y ago

Sometimes you just have to cut them some slack. Have you engineered a highly available cluster before? I'm not talking about the hot-standby postgres master that gets called on once every 2 years, but I'm talking about a 180 node Cassandra cluster thats doing 15,000 writes a second 24/7 and peaking at 60,000 writes a second every day, and you have to do node replacements every week or two because of the high load. Or I'm talking about a 200 node hadoop cluster thats doing the electrical metering and billing for 8 million people, and is NOT allowed to stop. Or the trading platform thats running sub millisecond trades and downtime means 300,000 $ USD per minute. These are systems I have engineered over the last 10 years, and I can say: These things are complex and have failures in 1000 different ways, and while you're monitoring 999 of them that one thing you're not looking at is festering under the surface (your monitoring system is tracking IRQ hardware interrupt response times, right???) Part of being in a team is everyone pulling together, and yes it's stressful at the time, but even very good management cant see all ends, just like very good engineering cant predict everything. I don't think it's useful to start pointing the finger at management and "asking some pointed questions at leadership" because sometimes everyone is doing their best. Yes we should analyse our failures so we can do better, but your tone is very accusatory, and I believe that a better approach is an all inclusive chat about how we can do better, and management saying "great job engineering" for fixing it, and giving them a break after the stressful event.

raiyu 7y ago
No doubt there are many complex systems and they inevitably go down. Every provider has suffered meaningful outages. I think the issue here isn’t so much that the system went down but the blog post. It’s very light on details and doesn’t go far enough in terms of re-establishing trust with the customers that were affected. Which by the looks of it is everyone attempting any trade most of the day on Monday.
LaserToy 7y ago
It is not about scale, it is about the fact that people lost real money. If you can’t make it work you should not be in that business, and I don’t really care how hard they work. I’m taking my account off their platform.
- malux85 7y ago
  Is this your first day of trading or something? People lose money in trading all the time, for hundreds of reasons and some of those reasons are infrastructure downtime. If your risk profile doesn't reflect that, maybe you should take your money out of trading altogether.
  - LaserToy 7y ago
    I will take it off this platform for sure.
    - LaserToy 7y ago
      And why -2 points??? Already moving money out. What is annoying - only 50k per day is allowed.
      - blackearl 7y ago
        Because complaining about RH being a low quality trading platform is like being angry at the burger quality at McDonald's. You get what you pay for.