14 ms·

On the profession side of this, if you're an engineer at RH in the thick of this - many have been there. It seems dire now, but in a few years the fog, panic, a

by czbond 7y ago

On the profession side of this, if you're an engineer at RH in the thick of this - many have been there. It seems dire now, but in a few years the fog, panic, and haze of no sleep will become a story you tell your peers at happy hour. Many will cast stones - but they have been there too. If they haven't, well maybe their day will also come. You may feel bad at the moment - but the best way professionally forward is "We try our best tomorrow"

cheschire 7y ago
If this were an outage directly caused by a natural disaster, I could understand. This outage was an availability problem. This clearly points to some prioritization problems within the leadership layers if robust and resilient infrastructure was not emphasized. The prioritization problems may not be due to ignorance or malice though, and may be justifiable if there are other fires that are burning brighter. It's still pointing to problems though, and I think it's completely legitimate for engineers to question the stability of the company when this sort of thing happens. At the very least as an engineer I would be asking some pointed questions of my leadership. Maybe not dusting off the resume yet, but still I'd want to get reassurance from internally that the leadership problems that caused this are being addressed.
- donavanm 7y ago
  Ive seen bigger, scarier, potentially costlier time based bugs personally. I dont think this would make me reevaluate my employment if I was at robinhood. As the parent says you either learn these lessons the hard way or you havent learned them yet. Thats doesnt translate to being a “leadership failure.” Your smaller point about prioritization is spot on though. I dont believe Ive seen any similar incidents lead to business ending outcomes. I personally point to sony or, more recently, equifax as examples of the disparity between actual business impact and technical abhorrence. In light of that why is it worth trying to preemptively solve technical challenges instead of business needs? Every calorie spent on “what if” subtracts from “whats needed.”
- malux85 7y ago
  Sometimes you just have to cut them some slack. Have you engineered a highly available cluster before? I'm not talking about the hot-standby postgres master that gets called on once every 2 years, but I'm talking about a 180 node Cassandra cluster thats doing 15,000 writes a second 24/7 and peaking at 60,000 writes a second every day, and you have to do node replacements every week or two because of the high load. Or I'm talking about a 200 node hadoop cluster thats doing the electrical metering and billing for 8 million people, and is NOT allowed to stop. Or the trading platform thats running sub millisecond trades and downtime means 300,000 $ USD per minute. These are systems I have engineered over the last 10 years, and I can say: These things are complex and have failures in 1000 different ways, and while you're monitoring 999 of them that one thing you're not looking at is festering under the surface (your monitoring system is tracking IRQ hardware interrupt response times, right???) Part of being in a team is everyone pulling together, and yes it's stressful at the time, but even very good management cant see all ends, just like very good engineering cant predict everything. I don't think it's useful to start pointing the finger at management and "asking some pointed questions at leadership" because sometimes everyone is doing their best. Yes we should analyse our failures so we can do better, but your tone is very accusatory, and I believe that a better approach is an all inclusive chat about how we can do better, and management saying "great job engineering" for fixing it, and giving them a break after the stressful event.
  - raiyu 7y ago
    No doubt there are many complex systems and they inevitably go down. Every provider has suffered meaningful outages. I think the issue here isn’t so much that the system went down but the blog post. It’s very light on details and doesn’t go far enough in terms of re-establishing trust with the customers that were affected. Which by the looks of it is everyone attempting any trade most of the day on Monday.
  - LaserToy 7y ago
    It is not about scale, it is about the fact that people lost real money. If you can’t make it work you should not be in that business, and I don’t really care how hard they work. I’m taking my account off their platform.
kerng 7y ago
Reminds me of the book Showstopper and the personal stories in - its about the creation of Windows NT. Pretty interesting how things where not so differnet some 30 years ago In case anyone is interested: https://www.amazon.com/Show-Stopper-Breakneck-Generation-Microsoft/dp/0029356717 https://www.amazon.com/Show-Stopper-Breakneck-Generation-Mic...
- dmix 7y ago
  Interesting, so it took 5yrs, $150-million, and 250-employees to get NT shipped. Adding this one to my reading list!
vinaypai 7y ago
This is all true for a company that is actually pushing any boundaries as opposed to failing pathetically at a well solved problem.
- unicornmama 7y ago
  Pushing the boundaries? They wrote an app that gamifies stock and options trading...
- indecisive_user 7y ago
  Robinhood opened up stock trading to a large portion of the population that would otherwise not have been interested in traditional trading platforms with high commissions. Their success helped to pressure companies such as TD and Schwab to mostly get rid of commissions as well, which is great for the average trader I think Robinhood has a lot of problems, but to say they're not pushing any boundaries ignores the huge changes they've brought to the industry.
  - C1sc0cat 7y ago
    This is the 21st century low cost trading has been around for several decades now
    - kube-system 7y ago
      Commission-free stock trading hasn’t, though. If you’re only trading a couple of shares at a time, $5-10 for a trade is a pretty steep fee.
      - C1sc0cat 7y ago
        Fix your stock exchanges to have stock splits then like the LSE does.
        kube-system 7y ago
        The fees I am referencing were imposed by brokers, not exchanges. Our exchanges have stock splits, but that still doesn't make a $10 fee on a single $50 share very palatable to the small-time investor.
bertil 7y ago
Important step though: have a retro, many maybe and write a report explaining what was messed up and how you might mitigate in the future. It looks like it’s going to be a good one. If you can share a sanitised version publicly, that would hopefully make it all a little bit more worth it. I think I speak for everyone here if I say that, if that report is public and interesting, everyone on this thread will be happy to get you a drink.

by czbond 7y ago

cheschire 7y ago

If this were an outage directly caused by a natural disaster, I could understand. This outage was an availability problem. This clearly points to some prioritization problems within the leadership layers if robust and resilient infrastructure was not emphasized. The prioritization problems may not be due to ignorance or malice though, and may be justifiable if there are other fires that are burning brighter. It's still pointing to problems though, and I think it's completely legitimate for engineers to question the stability of the company when this sort of thing happens. At the very least as an engineer I would be asking some pointed questions of my leadership. Maybe not dusting off the resume yet, but still I'd want to get reassurance from internally that the leadership problems that caused this are being addressed.

donavanm 7y ago
Ive seen bigger, scarier, potentially costlier time based bugs personally. I dont think this would make me reevaluate my employment if I was at robinhood. As the parent says you either learn these lessons the hard way or you havent learned them yet. Thats doesnt translate to being a “leadership failure.” Your smaller point about prioritization is spot on though. I dont believe Ive seen any similar incidents lead to business ending outcomes. I personally point to sony or, more recently, equifax as examples of the disparity between actual business impact and technical abhorrence. In light of that why is it worth trying to preemptively solve technical challenges instead of business needs? Every calorie spent on “what if” subtracts from “whats needed.”
malux85 7y ago
Sometimes you just have to cut them some slack. Have you engineered a highly available cluster before? I'm not talking about the hot-standby postgres master that gets called on once every 2 years, but I'm talking about a 180 node Cassandra cluster thats doing 15,000 writes a second 24/7 and peaking at 60,000 writes a second every day, and you have to do node replacements every week or two because of the high load. Or I'm talking about a 200 node hadoop cluster thats doing the electrical metering and billing for 8 million people, and is NOT allowed to stop. Or the trading platform thats running sub millisecond trades and downtime means 300,000 $ USD per minute. These are systems I have engineered over the last 10 years, and I can say: These things are complex and have failures in 1000 different ways, and while you're monitoring 999 of them that one thing you're not looking at is festering under the surface (your monitoring system is tracking IRQ hardware interrupt response times, right???) Part of being in a team is everyone pulling together, and yes it's stressful at the time, but even very good management cant see all ends, just like very good engineering cant predict everything. I don't think it's useful to start pointing the finger at management and "asking some pointed questions at leadership" because sometimes everyone is doing their best. Yes we should analyse our failures so we can do better, but your tone is very accusatory, and I believe that a better approach is an all inclusive chat about how we can do better, and management saying "great job engineering" for fixing it, and giving them a break after the stressful event.
- raiyu 7y ago
  No doubt there are many complex systems and they inevitably go down. Every provider has suffered meaningful outages. I think the issue here isn’t so much that the system went down but the blog post. It’s very light on details and doesn’t go far enough in terms of re-establishing trust with the customers that were affected. Which by the looks of it is everyone attempting any trade most of the day on Monday.
- LaserToy 7y ago
  It is not about scale, it is about the fact that people lost real money. If you can’t make it work you should not be in that business, and I don’t really care how hard they work. I’m taking my account off their platform.

kerng 7y ago

Reminds me of the book Showstopper and the personal stories in - its about the creation of Windows NT. Pretty interesting how things where not so differnet some 30 years ago In case anyone is interested: https://www.amazon.com/Show-Stopper-Breakneck-Generation-Microsoft/dp/0029356717 https://www.amazon.com/Show-Stopper-Breakneck-Generation-Mic...

dmix 7y ago
Interesting, so it took 5yrs, $150-million, and 250-employees to get NT shipped. Adding this one to my reading list!

vinaypai 7y ago

This is all true for a company that is actually pushing any boundaries as opposed to failing pathetically at a well solved problem.

unicornmama 7y ago
Pushing the boundaries? They wrote an app that gamifies stock and options trading...
indecisive_user 7y ago
Robinhood opened up stock trading to a large portion of the population that would otherwise not have been interested in traditional trading platforms with high commissions. Their success helped to pressure companies such as TD and Schwab to mostly get rid of commissions as well, which is great for the average trader I think Robinhood has a lot of problems, but to say they're not pushing any boundaries ignores the huge changes they've brought to the industry.
- C1sc0cat 7y ago
  This is the 21st century low cost trading has been around for several decades now
  - kube-system 7y ago
    Commission-free stock trading hasn’t, though. If you’re only trading a couple of shares at a time, $5-10 for a trade is a pretty steep fee.
    - C1sc0cat 7y ago
      Fix your stock exchanges to have stock splits then like the LSE does.
      - kube-system 7y ago
        The fees I am referencing were imposed by brokers, not exchanges. Our exchanges have stock splits, but that still doesn't make a $10 fee on a single $50 share very palatable to the small-time investor.

bertil 7y ago

Important step though: have a retro, many maybe and write a report explaining what was messed up and how you might mitigate in the future. It looks like it’s going to be a good one. If you can share a sanitised version publicly, that would hopefully make it all a little bit more worth it. I think I speak for everyone here if I say that, if that report is public and interesting, everyone on this thread will be happy to get you a drink.