5 ms·

Merrill Lynch / Bank of America's trading platform is also down... so this isn't limited to Robin Hood. I was completely unable to place any trades or even che

Merrill Lynch / Bank of America's trading platform is also down... so this isn't limited to Robin Hood. I was completely unable to place any trades or even check balances during the market opening this morning. However, engineering HEADS SHOULD ROLL. In the era of on demand computing, near limitless cloud resources, etc, these engineering teams should have already had capacity planning ready to go for these trading platforms. I'm extremely disappointed - these engineers LIVE for this stuff, and they dropped the ball. Again, heads should roll. This is not some data center that caught fire, this is capacity planning at its lowest common denominator.

avvt4avaw 7y ago
The market hit a circuit breaker this morning when it was 7% down. No one could trade at the open, not even professionals, because the market was stopped.
- blantonl 7y ago
  No, I couldn't event place any orders whatsoever, and couldn't check balances etc. Order PLACEMENT and order execution are two different things. The former was impossible as well.
- Frost1x 7y ago
  That doesn't seem to be the reason Robinhood is having issues. I don't know if it was the case for the two other platforms OP mentioned but Robinhood seems to have some issues handling the request volumes.
throwaway3157 7y ago
> In the era of on demand computing, near limitless cloud resources Should be pointed out that BoA decided to build their own "cloud"
deleted 7y ago
[deleted]
sf_rob 7y ago
Agree that heads should role, but it may very well be management decisions that brought them to this point. We don't know if engineering was resource/strategically limited (seems unlikely at a well funded startup but it may have been). Having worked places where terminations were the result of failures, it's not always the party most responsible that is let go.
papito 7y ago
Sometimes engineers are not at fault. They are well aware of technical debt and possible performance mine fields. It's the management that does not properly prioritize performance and scale work because there is no apparent "value" to the customer. Until it's too late.
jetru 7y ago
Bad idea. You would be just rolling heads of people who just learnt a valuable lesson in high throughput services. Keep them around, they'll get better than hiring someone else who will have to spend more time learning the same lesson.
city41 7y ago
Designing a real time system to scale to this magnitude is not easy at all. I wonder how many engineers would be capable of designing such a system, I sure couldn't. This isn't like most scaling where delaying/queuing can be an acceptable trade off.
- wasdfff 7y ago
  Maybe they shouldn’t be in business if they can’t hire the engineering to do the job. Every other discount broker seems to have no problem hiring this level of talent.
yibg 7y ago
A couple of things wrong with this: 1) you make it sound like scaling a system, especially one doing lots of financial transactions, is trivial. Just because you can get access to nearly unlimited computing resources doesn’t mean everything just magically scales infinitely. 2) why should heads in engineering roll? Do you know it was incompetence on their part? Do you even know what the issue is? How did you diagnose the fundamental issue here without any further info? I’d hate to do a post mortem with you. Sounds like you’ll be basically calling people involved idiots and for them to be fired.
- LaserToy 7y ago
  he is the customer, not a employee. As a customer I don't actually care about postmortems, i just want things to work. If it doesn't work, I use another product. I'm leaving robinhood due to their issues, and it sucks it takes so long due to their transfer limits
  - yibg 7y ago
    That's fine and for every customer to evaluate for themselves. But there is a difference between saying "as a customer the product isn't acceptable so I will use something else" and "engineering is incompetent and someone should be fired". Especially if you don't actually know what happened.
    - LaserToy 7y ago
      People are emotional, especially when they lose something of value. And they should not be fired, they should just cover the loses...
raz32dust 7y ago
There should be accountability for sure. I am not sure if rolling heads is enough or even the right thing by itself - but there definitely needs to be a 3rd party audit, and a public report.

avvt4avaw 7y ago

The market hit a circuit breaker this morning when it was 7% down. No one could trade at the open, not even professionals, because the market was stopped.

blantonl 7y ago
No, I couldn't event place any orders whatsoever, and couldn't check balances etc. Order PLACEMENT and order execution are two different things. The former was impossible as well.
Frost1x 7y ago
That doesn't seem to be the reason Robinhood is having issues. I don't know if it was the case for the two other platforms OP mentioned but Robinhood seems to have some issues handling the request volumes.

throwaway3157 7y ago

> In the era of on demand computing, near limitless cloud resources Should be pointed out that BoA decided to build their own "cloud"

deleted 7y ago

[deleted]

sf_rob 7y ago

Agree that heads should role, but it may very well be management decisions that brought them to this point. We don't know if engineering was resource/strategically limited (seems unlikely at a well funded startup but it may have been). Having worked places where terminations were the result of failures, it's not always the party most responsible that is let go.

papito 7y ago

Sometimes engineers are not at fault. They are well aware of technical debt and possible performance mine fields. It's the management that does not properly prioritize performance and scale work because there is no apparent "value" to the customer. Until it's too late.

jetru 7y ago

Bad idea. You would be just rolling heads of people who just learnt a valuable lesson in high throughput services. Keep them around, they'll get better than hiring someone else who will have to spend more time learning the same lesson.

city41 7y ago

Designing a real time system to scale to this magnitude is not easy at all. I wonder how many engineers would be capable of designing such a system, I sure couldn't. This isn't like most scaling where delaying/queuing can be an acceptable trade off.

wasdfff 7y ago
Maybe they shouldn’t be in business if they can’t hire the engineering to do the job. Every other discount broker seems to have no problem hiring this level of talent.

yibg 7y ago

A couple of things wrong with this: 1) you make it sound like scaling a system, especially one doing lots of financial transactions, is trivial. Just because you can get access to nearly unlimited computing resources doesn’t mean everything just magically scales infinitely. 2) why should heads in engineering roll? Do you know it was incompetence on their part? Do you even know what the issue is? How did you diagnose the fundamental issue here without any further info? I’d hate to do a post mortem with you. Sounds like you’ll be basically calling people involved idiots and for them to be fired.

LaserToy 7y ago
he is the customer, not a employee. As a customer I don't actually care about postmortems, i just want things to work. If it doesn't work, I use another product. I'm leaving robinhood due to their issues, and it sucks it takes so long due to their transfer limits
- yibg 7y ago
  That's fine and for every customer to evaluate for themselves. But there is a difference between saying "as a customer the product isn't acceptable so I will use something else" and "engineering is incompetent and someone should be fired". Especially if you don't actually know what happened.
  - LaserToy 7y ago
    People are emotional, especially when they lose something of value. And they should not be fired, they should just cover the loses...

raz32dust 7y ago

There should be accountability for sure. I am not sure if rolling heads is enough or even the right thing by itself - but there definitely needs to be a 3rd party audit, and a public report.