7 ms·

Wow... Being down once - OK, it was record-high volume, servers were strained. Twice - well, it was only a day after the first time and it takes time to fix the

Wow... Being down once - OK, it was record-high volume, servers were strained. Twice - well, it was only a day after the first time and it takes time to fix these things. But a third time, nearly a week after the first incident (and once again on a volatile day where users stand to gain/lose a lot of money)? Really hard to think of any acceptable justification for that.

Traster 7y ago
The simple answer is that the reason the system failed the first time wasn't easy to fix in the short term. Either they need to acquire more hardware resources or redesign some system so that it scales better. Not everything can be fixed overnight. Take a look at reddit - that's a website that collapsed repeatedly for months and months and months. Sometimes it's not something you can just 'fix'.
- nightski 7y ago
  If they KNEW it was broken and were working on a fix, then they should of shut down the service. I'm sorry but this is simply unacceptable. I don't use Robinhood, but when you are handling very time sensitive contracts that can make or lose people a lot of money in a very short period of time if the service is not stable it should not of been in operation until it was fixed. I can't believe HN is making excuses for this.
  - usefulcat 7y ago
    If they do that, then they're definitely harming any customers who have a position they need to get out of, as opposed to only potentially harming them.
    - nightski 7y ago
      That's exactly what is happening right now anyways. Except in the past week many new positions were opened on a broken platform. If this was Wells Fargo we'd see the pitchforks. But it's a SV company so all is good.
      - usefulcat 7y ago
        If you believe that the platform is so broken that it shouldn't be used (and I'm not arguing against that, btw) then a more reasonable solution would be to only allow position-reducing trades. That would also reduce overall trading volume on their platform. Not that I expect them to do any of this, just pointing out that 'shut everything down' is not necessarily the best approach.
      - to11mtm 7y ago
        I mean, the investment community knows, and even on other sites there was a lot of encouragement to get -out- before the next disaster struck. IMO anyone who opened a position on RH in the last week should not be shocked that the same thing happened again in this high volume crapstorm.
- blantonl 7y ago
  Cloud computing resources are nearly limitless. Provision the resources and take the financial hit in costs. If your architecture wasn't able to scale horizontally because it was poorly designed, heads should roll. This isn't some social network - these are financial platforms where literally the individual customer is financially dependent. Heads. Should. Roll. Tell us the post-mortems, and then tell us who got the boot. Completely unacceptable.
  - MrStonedOne 7y ago
    The only company in the world who can claim to have made a perfectly horizontal system at all levels of macro and micro is amazon, and even they would be lying to do so. A system like robinhood can have hundreds of moving parts, if 99 of them are horizontally scalable but 1 is not, eventually that 1 piece will become the bottle neck, and the fact it hadn't been made horizontally scalable yet is more likely to be a testament to how much work would have to go into doing so. > Heads. Should. Roll. Tell us the post-mortems, and then tell us who got the boot. Completely unacceptable. This is an unfortunate viewpoint. How quickly did we forget that robinhood is literally providing a service that no other company was able to do before. You want 100% uptime, you won't find it in an online service, let alone an online fee free service, you'll find it on the floor of the exchange.
    - manigandham 7y ago
      All brokers were able to do it, they just didn't because commissions were easy profit and there was no competition. $0 commissions aren't a big deal either. If you just buy and hold, it makes no difference. If you trade actively then a real broker with better tech and order management is worth way more than the fees.
    - dehrmann 7y ago
      > a service that no other company was ~~able~~ willing to do before ftfy
    - dodobirdlord 7y ago
      > The only company in the world who can claim to have made a perfectly horizontal system at all levels of macro and micro is amazon, and even they would be lying to do so. Haha, true. Call up you AWS account representative and you may find that certain service limits can’t be increased for love or money.
- CaptainZapp 7y ago
  I'd argue that the consequences of reddit not being available for a certain amount of time are absolutely not comparable with a financial services' firms infrastructure. Mistakes, any mistakes, are very hard to excuse when people, or other financial entities happen to lose money. Comparing it with a site like reddit is, frankly, a bit disengineous.
  - dpcx 7y ago
    Consequences, no. Infrastructure failures, absolutely. What's being hosted isn't as important when it's broken and you're struggling to fix it. Sure, when it's a financial service's infrastructure, you're worried about other peoples money, but it's still something that's broken and may take time to fix.
    - manquer 7y ago
      The load testing you need to do for trading is considerably higher than any chat forum to go live . Your customers are paying for speed and performance . In a industry where users spend millions to be closer to the exchange if you are not provisioning for some multiple of your peak load you are going to get slaughtered in the market
      - rednixion 7y ago
        I don't think that argument really holds up when Robinhood is the subject. I mean they still don't even have phone support or same day guarantees covering them responding to your email.
        grvdrm 7y ago
        Their support is horrific. Will never use them again. M1 is delightful in comparison.
- manigandham 7y ago
  Robinhood is nothing like Reddit. It has a much bigger valuation, more resources, strict regulations to follow, and the responsibility of safely handling money for millions of people. This should be fixed within days even if it means redesigning the entire system. The last blog post blamed overloaded DNS so it doesn't seem like a major architectural problem, which is why it's surprising that they're down again.
  - SkyPuncher 7y ago
    Also, Reddit can afford to queue and buffer like crazy. Nearly every write-action on Reddit appears to hit a queue. If your vote or comment doesn't go up for 30 seconds, that's not a problem. If your trade takes 30 seconds, that could be a huge deal.
  - swsieber 7y ago
    > strict regulations to follow, and the responsibility of safely handling money for millions of people. This should be fixed within days even if it means redesigning the entire system Somehow, in my mind, the responsibility of safely handling money of millions and redesigning the entire system don't really mix. Ditto for strict regulations and redesigning the entire system.
    - thfuran 7y ago
      I know it's not the same regulations but I'm pretty sure the next FDA auditor would have a heart attack if we slipped "rewrote the whole product" into one of our patch notes at work.
  - yibg 7y ago
    Being in a regulated environment with lots of peoples money should make it harder and take longer to redesign and implement anything. I don’t see a complete redesign of a non-trivial part of a trading platform happening in a week.
    - manigandham 7y ago
      That’s before outages, priority now is to restore service ASAP, whatever it takes.
- deleted 7y ago
  [deleted]
AgloeDreams 7y ago
I would imagine the volume differential scale is massive here, they probably are scaled at 1X peak, so they are capable of handling ~3X peak. They are probably at ~10X peak traffic at times. Just are not prepped for a hardcore crash.
chisleu 7y ago
Looks like they had a bug w/ leap years and load.
- shawnz 7y ago
  The issue a few days ago was not related to the leap year. That was only a theory at the time. They have since claimed it was a DNS issue.
neuronic 7y ago
I would not be surprised if many trading places have been told to shut down temporarily. [1] https://www.ccn.com/dow-crashes-so-hard-might-close-stock-market/ https://www.ccn.com/dow-crashes-so-hard-might-close-stock-ma... [2] https://www.businessinsider.de/international/stock-market-news-today-indices-plunge-oil-market-chaos-coronavirus-2020-3 https://www.businessinsider.de/international/stock-market-ne... [3] https://www.bloomberg.com/news/articles/2020-03-08/rout-in-u-s-stock-futures-would-trigger-trading-curbs-at-5 https://www.bloomberg.com/news/articles/2020-03-08/rout-in-u...
- IAmEveryone 7y ago
  That’s about a set of specific, known “circuit-breakers”. They shut down trading at the actual exchange, not “ask trading places“ to do so.
- manigandham 7y ago
  The market makers and exchanges stop executing orders because a series of circuit-breakers fired (set by the regulating organizations). Robinhood and other brokers don't execute any trades, they just forward orders from users to their exchange partners and show up to date quotes. They have a much easier load to bear than actual exchanges.
zachware 7y ago
What sets financial services apart when they fail is the extreme legibility of the losses. One can argue that if say Reddit is down, they have losses, but measuring the impact in terms of dollars lost is difficult. It's a lot easier with something like Robinhood. So, ethics aside, the level of measurable risk they have now is existential. Lawsuits will flow.
throwsprtsdy 7y ago
I get the impression that the people running Robinhood actually don't understand how difficult it is to implement what they want to offer, both on the reliability side (outages) and on the logical side (infinite margin).
dehrmann 7y ago
This is pretty interesting for capacity planning. I worked at a minor ecommerce company when Black Friday was still a big deal. In the month leading up to it, we went into code freeze and made sure all our systems can handle added load. What Robinhood is facing is like that, but with no notice. It means they ~always need to be ready for a 4x(?) increase in load.
- notwedtm 7y ago
  This is one of the problems that "the cloud" is supposed to solve. No need to keep infra up for that unicorn event, just to ensure that you can scale properly when it does happen.
  - darkwater 7y ago
    Until you hit some kind of aggregation server that it's not trivial to scale out properly unless you take a completely different approach or that you may scale vertically up (but eventually you will hit the ceiling).

Traster 7y ago

The simple answer is that the reason the system failed the first time wasn't easy to fix in the short term. Either they need to acquire more hardware resources or redesign some system so that it scales better. Not everything can be fixed overnight. Take a look at reddit - that's a website that collapsed repeatedly for months and months and months. Sometimes it's not something you can just 'fix'.

nightski 7y ago
If they KNEW it was broken and were working on a fix, then they should of shut down the service. I'm sorry but this is simply unacceptable. I don't use Robinhood, but when you are handling very time sensitive contracts that can make or lose people a lot of money in a very short period of time if the service is not stable it should not of been in operation until it was fixed. I can't believe HN is making excuses for this.
- usefulcat 7y ago
  If they do that, then they're definitely harming any customers who have a position they need to get out of, as opposed to only potentially harming them.
  - nightski 7y ago
    That's exactly what is happening right now anyways. Except in the past week many new positions were opened on a broken platform. If this was Wells Fargo we'd see the pitchforks. But it's a SV company so all is good.
    - usefulcat 7y ago
      If you believe that the platform is so broken that it shouldn't be used (and I'm not arguing against that, btw) then a more reasonable solution would be to only allow position-reducing trades. That would also reduce overall trading volume on their platform. Not that I expect them to do any of this, just pointing out that 'shut everything down' is not necessarily the best approach.
    - to11mtm 7y ago
      I mean, the investment community knows, and even on other sites there was a lot of encouragement to get -out- before the next disaster struck. IMO anyone who opened a position on RH in the last week should not be shocked that the same thing happened again in this high volume crapstorm.
blantonl 7y ago
Cloud computing resources are nearly limitless. Provision the resources and take the financial hit in costs. If your architecture wasn't able to scale horizontally because it was poorly designed, heads should roll. This isn't some social network - these are financial platforms where literally the individual customer is financially dependent. Heads. Should. Roll. Tell us the post-mortems, and then tell us who got the boot. Completely unacceptable.
- MrStonedOne 7y ago
  The only company in the world who can claim to have made a perfectly horizontal system at all levels of macro and micro is amazon, and even they would be lying to do so. A system like robinhood can have hundreds of moving parts, if 99 of them are horizontally scalable but 1 is not, eventually that 1 piece will become the bottle neck, and the fact it hadn't been made horizontally scalable yet is more likely to be a testament to how much work would have to go into doing so. > Heads. Should. Roll. Tell us the post-mortems, and then tell us who got the boot. Completely unacceptable. This is an unfortunate viewpoint. How quickly did we forget that robinhood is literally providing a service that no other company was able to do before. You want 100% uptime, you won't find it in an online service, let alone an online fee free service, you'll find it on the floor of the exchange.
  - manigandham 7y ago
    All brokers were able to do it, they just didn't because commissions were easy profit and there was no competition. $0 commissions aren't a big deal either. If you just buy and hold, it makes no difference. If you trade actively then a real broker with better tech and order management is worth way more than the fees.
  - dehrmann 7y ago
    > a service that no other company was ~~able~~ willing to do before ftfy
  - dodobirdlord 7y ago
    > The only company in the world who can claim to have made a perfectly horizontal system at all levels of macro and micro is amazon, and even they would be lying to do so. Haha, true. Call up you AWS account representative and you may find that certain service limits can’t be increased for love or money.
CaptainZapp 7y ago
I'd argue that the consequences of reddit not being available for a certain amount of time are absolutely not comparable with a financial services' firms infrastructure. Mistakes, any mistakes, are very hard to excuse when people, or other financial entities happen to lose money. Comparing it with a site like reddit is, frankly, a bit disengineous.
- dpcx 7y ago
  Consequences, no. Infrastructure failures, absolutely. What's being hosted isn't as important when it's broken and you're struggling to fix it. Sure, when it's a financial service's infrastructure, you're worried about other peoples money, but it's still something that's broken and may take time to fix.
  - manquer 7y ago
    The load testing you need to do for trading is considerably higher than any chat forum to go live . Your customers are paying for speed and performance . In a industry where users spend millions to be closer to the exchange if you are not provisioning for some multiple of your peak load you are going to get slaughtered in the market
    - rednixion 7y ago
      I don't think that argument really holds up when Robinhood is the subject. I mean they still don't even have phone support or same day guarantees covering them responding to your email.
      - grvdrm 7y ago
        Their support is horrific. Will never use them again. M1 is delightful in comparison.
manigandham 7y ago
Robinhood is nothing like Reddit. It has a much bigger valuation, more resources, strict regulations to follow, and the responsibility of safely handling money for millions of people. This should be fixed within days even if it means redesigning the entire system. The last blog post blamed overloaded DNS so it doesn't seem like a major architectural problem, which is why it's surprising that they're down again.
- SkyPuncher 7y ago
  Also, Reddit can afford to queue and buffer like crazy. Nearly every write-action on Reddit appears to hit a queue. If your vote or comment doesn't go up for 30 seconds, that's not a problem. If your trade takes 30 seconds, that could be a huge deal.
- swsieber 7y ago
  > strict regulations to follow, and the responsibility of safely handling money for millions of people. This should be fixed within days even if it means redesigning the entire system Somehow, in my mind, the responsibility of safely handling money of millions and redesigning the entire system don't really mix. Ditto for strict regulations and redesigning the entire system.
  - thfuran 7y ago
    I know it's not the same regulations but I'm pretty sure the next FDA auditor would have a heart attack if we slipped "rewrote the whole product" into one of our patch notes at work.
- yibg 7y ago
  Being in a regulated environment with lots of peoples money should make it harder and take longer to redesign and implement anything. I don’t see a complete redesign of a non-trivial part of a trading platform happening in a week.
  - manigandham 7y ago
    That’s before outages, priority now is to restore service ASAP, whatever it takes.
deleted 7y ago
[deleted]

AgloeDreams 7y ago

I would imagine the volume differential scale is massive here, they probably are scaled at 1X peak, so they are capable of handling ~3X peak. They are probably at ~10X peak traffic at times. Just are not prepped for a hardcore crash.

chisleu 7y ago

Looks like they had a bug w/ leap years and load.

shawnz 7y ago
The issue a few days ago was not related to the leap year. That was only a theory at the time. They have since claimed it was a DNS issue.

neuronic 7y ago

I would not be surprised if many trading places have been told to shut down temporarily. [1] https://www.ccn.com/dow-crashes-so-hard-might-close-stock-market/ https://www.ccn.com/dow-crashes-so-hard-might-close-stock-ma... [2] https://www.businessinsider.de/international/stock-market-news-today-indices-plunge-oil-market-chaos-coronavirus-2020-3 https://www.businessinsider.de/international/stock-market-ne... [3] https://www.bloomberg.com/news/articles/2020-03-08/rout-in-u-s-stock-futures-would-trigger-trading-curbs-at-5 https://www.bloomberg.com/news/articles/2020-03-08/rout-in-u...

IAmEveryone 7y ago
That’s about a set of specific, known “circuit-breakers”. They shut down trading at the actual exchange, not “ask trading places“ to do so.
manigandham 7y ago
The market makers and exchanges stop executing orders because a series of circuit-breakers fired (set by the regulating organizations). Robinhood and other brokers don't execute any trades, they just forward orders from users to their exchange partners and show up to date quotes. They have a much easier load to bear than actual exchanges.

zachware 7y ago

What sets financial services apart when they fail is the extreme legibility of the losses. One can argue that if say Reddit is down, they have losses, but measuring the impact in terms of dollars lost is difficult. It's a lot easier with something like Robinhood. So, ethics aside, the level of measurable risk they have now is existential. Lawsuits will flow.

throwsprtsdy 7y ago

I get the impression that the people running Robinhood actually don't understand how difficult it is to implement what they want to offer, both on the reliability side (outages) and on the logical side (infinite margin).

dehrmann 7y ago

This is pretty interesting for capacity planning. I worked at a minor ecommerce company when Black Friday was still a big deal. In the month leading up to it, we went into code freeze and made sure all our systems can handle added load. What Robinhood is facing is like that, but with no notice. It means they ~always need to be ready for a 4x(?) increase in load.

notwedtm 7y ago
This is one of the problems that "the cloud" is supposed to solve. No need to keep infra up for that unicorn event, just to ensure that you can scale properly when it does happen.
- darkwater 7y ago
  Until you hit some kind of aggregation server that it's not trivial to scale out properly unless you take a completely different approach or that you may scale vertically up (but eventually you will hit the ceiling).